menu

Feed available - Subscribe to our feed to stay up to date on upcoming maintenance and incidents.

Identity services - Oregon performance degradation

Incident window: August 2, 2019 17:23 UTC to August 3, 2019 21:02 UTC

Impacted Cloud services:

Impact Level : high

Identity services performance in Oregon is significantly degraded starting at 17:23 UTC. The performance of Identity services in other regions is not affected. We are investigating.

[2019-08-02 18:22 UTC] Performance has more significantly degraded since the initial detection and Identity services may be unavailable. We are continuing to investigate and look to mitigate the issue as soon as possible.

[2019-08-02 19:20 UTC] As we continue to investigate, we are scaling up our infrastructure in an attempt to help mitigate the symptoms and restore service availability and performance.

[2019-08-02 20:23 UTC] Scaling up the infrastructure has not yielded significant improvements. We are considering other options as we continue to investigate.

[2019-08-02 21:22 UTC] Resolved. We have replaced the running infrastructure and can see that the service is available and performing with normal latency as of 21:16 UTC. We are continuing to investigate the underlying cause and will be continuing to monitor the service closely.

[2019-08-03 14:40 UTC] We are seeing degraded performance and are working to restore full services.

[2019-08-03 19:00 UTC] We continuing to see recurring issues and are working on additional mitigations.

[2019-08-03 21:02 UTC] Resolved. Service is restored and we are closely monitoring. We continue the work to identify the root cause.

[2019-08-04 02:00 UTC] The services are currently performing within normal parameters.

[2019-08-06 15:29 UTC] The services have been continuing to perform within normal parameters. We are continuing our investigation to identify the root cause.

[2019-08-06 22:33 UTC] On August 7, between 06:00 UTC and 08:00 UTC, we will be performing maintenance for further stabilization of the Identity services in the Oregon region. We have already contacted specific customers who may be affected though no downtime or performance disruption is anticipated.

[2019-08-07 07:30 UTC] The stabilization maintenance to improve load balancing was completed during the stated window without any issues. Services are performing within normal parameters.

[2019-08-07 17:21 UTC] The services have been continuing to perform within normal parameters. We are continuing our investigation to identify the root cause.

[2019-08-08 14:10 UTC] The services have been continuing to perform within normal parameters. We are continuing our investigation to identify the root cause.

[2019-08-09 13:14 UTC] The services have been continuing to perform within normal parameters. We are continuing our investigation to identify the root cause.

[2019-08-12 18:09 UTC] The services have been continuing to perform within normal parameters. We are continuing our investigation to identify the root cause.

[2019-08-14 11:33 UTC] The services have been continuing to perform within normal parameters. We are continuing our investigation to identify the root cause.

[2019-08-15 11:49 UTC] The services have been continuing to perform within normal parameters. We are continuing our investigation to identify the root cause.

[2019-08-19 12:19 UTC] The services have been continuing to perform within normal parameters. We are continuing our investigation to identify the root cause.