menu

Feed available - Subscribe to our feed to stay up to date on upcoming maintenance and incidents.

Cloud Management Console - intermittent timeouts

Incident window: February 4, 2019 07:00 UTC to February 5, 2019 03:10 UTC

Impacted Cloud services:

Impact Level : high

During our V8 SP4 release, upgrades were performed on infrastructure load balancers used by the Cloud management console. The maintenance inadvertently reset any custom timeout values that had been previously applied to these load balancers back to a default timeout period, which has led to some customers experiencing issues (e.g., retrieving apps from workspace services, saving a service, publishing an app). Only those customers who have a larger number of apps/services that would necessitate a longer timeout would have been affected. Note that this incident is not affecting service availability in customer runtime environments, including published applications / services. We are applying updates to the affected infrastructure and will share an update once all custom timeout values have been propagated and affected service operations have been restored.

[2019-02-04 17:50 UTC] Custom timeout values for the affected infrastructure load balancers have been restored for the following regions: Oregon, Sydney, Tokyo. Remaining regions that are in progress: Virginia (most have been completed), São Paulo, Ireland, Frankfurt, Singapore.

[2019-02-04 19:14 UTC] Custom timeout values for infrastructure load balancers have been restored for the Virginia region. São Paulo, Ireland, Frankfurt, & Singapore are still in progress.

[2019-02-04 19:31 UTC] Custom timeout values for infrastructure load balancers have been restored for the Singapore region. São Paulo, Ireland, & Frankfurt are still in progress.

[2019-02-04 19:40 UTC] Custom timeout values for infrastructure load balancers have been restored for the São Paulo region. Ireland & Frankfurt are still in progress.

[2019-02-04 20:49 UTC] Custom timeout values for infrastructure load balancers have been restored for the Frankfurt region. Ireland is still in progress.

[2019-02-04 21:49 UTC] Custom timeout values for infrastructure load balancers have been restored for the remaining region, Ireland. However, we are detecting app/service save issues still occurring in the Virginia region and are investigating.

[2019-02-05 03:10 UTC] Resolved. During the deployment across the main infrastructure and the other seven regions that were involved in the Fabric V8 SP4 deployment, there was a firewall rule that did not show in initial health and deployment checks, but did interfere with certain system communications when saving some projects in the Cloud management console. The firewall issue was resolved and systems are operating normally.