menu

Feed available - Subscribe to our feed to stay up to date on upcoming maintenance and incidents.

Workspace services - Oregon is unavailable

Incident window: November 4, 2019 17:21 UTC to 18:26 UTC

Impacted Cloud services:

Impact Level : high

Workspace services in Oregon have become unavailable starting at 17:21 UTC. The availability and performance of Workspace services in other regions is not affected. We are investigating.

infrastructure failure of a DB. The database has been recovered and the systems did not auto-restart as expected. We have opened internal development tickets on this scenario and will have the team correct this as we expect the workspace systems to survive this type of issue without intervention.

[2019-11-04 18:41 UTC] Resolved. The underlying Oregon Workspace database master node had failed and automatically failed over to the replica node, which had been promoted as the new master (and a new replica created after). However, while the database was able to recover with our automation, the Oregon Workspace application did not automatically recover as would be expected and was still yielding errors until we intervened by restarting. Once restarted at 18:26 UTC, the Oregon Workspace services have been available and performing within expected levels. We will be continuing to monitor. We will also be coordinating with our development team to identify why the Workspace application was unable to automatically recover and to work toward a solution to mitigate these types of incidents requiring our attention beyond our automated incident response controls in the future.