menu

Feed available - Subscribe to our feed to stay up to date on upcoming maintenance and incidents.

Reporting Services - Significant performance degradation and unavailability

Incident window: October 25, 2020 00:01 UTC to November 11, 2020 00:01 UTC

Impacted Cloud services:

Impact Level : high

Reporting services performance is significantly degraded as analytics ingestion is encountering issues.

We are working to recover the services and will provide an ETA as soon as possible.

Data from the 19th onward may not be available yet. No analytics data was lost during the incident window and should be reflected in newly generated reports.

[2020-10-29 00:20 UTC] We are currently recovering the reporting system and have approximately 750M messages to process. We are able to ingest about 150,000 messages/minute, but this includes new data in addition to the backlog. At present we estimate about six more days before we might catch up to normal ingestion latency.

[2020-11-09 16:00 UTC] We are approximately 1 day behind in processing queued messages, with the backlog down to under 70M messages.

[2020-11-11 19:45 UTC] Resolved. All queued messages have been processed.