Improved Cloud Management Console Support for Latest 9.2 Release

• Cloud Management Console

  • Implemented calling stored procedures by ordinal position of parameters.

  • Added support for Oracle CLOB and NCLOB data types.

  • Fixed a timestamp parsing issue in stored procedures and functions.

No downtime is expected for the impacted Cloud services while this maintenance is being performed. The scheduled maintenance is designed to mitigate disruptions to service availability and performance for the impacted Cloud services. However, it is possible for the impacted Cloud services to be unavailable and/or performance degraded for a short period of time during the maintenance window. Note that no changes are being applied for other Cloud services outside of the list of impacted services above and no service availability or performance disruption is expected for other Cloud services.

Reporting Services - Significant performance degradation and unavailability

• Reporting Services

Reporting services performance is significantly degraded as analytics ingestion is encountering issues.

We are working to recover the services and will provide an ETA as soon as possible.

There were slight delays in processing analytics beginning on September 14th, which have grown large enough to cause a noticeable issue on the 18th. No analytics data is lost during the window, and will be reflected in newly generated reports as our ingestion systems catch up with the influx of messages to be processed.

[2021-09-23 21:00 UTC] Resolved. The queue is now cleared and metrics/reports are now performing at normal levels.

Cloud Management Console and Workspace Maintenance

• Cloud Management Console

  • Composite app publish status refresh issues fixed

  • Publish time out increased from default 20 seconds to 60 seconds to allow bigger customer apps to be publishable

• Workspace Maintenance

  • Composite app publish status refresh issues fixed

  • Publish time out increased from default 20 seconds to 60 seconds to allow bigger customer apps to be publishable

No downtime is expected for the impacted Cloud services while this maintenance is being performed. The scheduled maintenance is designed to mitigate disruptions to service availability and performance for the impacted Cloud services. However, it is possible for the impacted Cloud services to be unavailable and/or performance degraded for a short period of time during the maintenance window. Note that no changes are being applied for other Cloud services outside of the list of impacted services above and no service availability or performance disruption is expected for other Cloud services.

Quantum V9 SP3 Release

• Fabric Integration Server

  • Added support for WebSockets to the Quantum Fabric Cloud

  • Added support to reconfigure the scheme for Swagger 2.0 services

  • Added the Fabric Workflow Adapter in Integration Services

  • Enhancements to Workflow Audit history

  • Middleware Integration with Temenos Metering Service Client Library

  • Updates to Server Events APIs

• Cloud Management Console

  • Supporting Modularity of Fabric Apps

  • Added OpenAPI Specification support for Object Verb Mapping

  • Added support for Custom Front-End URLs in Object Services

  • Added the Integration Service trigger type to Workflow services

  • Added support to view linked Java Pre and Post Processors for a Fabric App

  • Added a Version Upgrade mechanism to MFCLI

  • Updates to Engagement Adapter

• Fabric Identity

  • Added support to use PKCE in JavaScript SDK

  • Added support to replace Identity Service references

• Engagement

  • Improved support for Events notifications

  • Enhancements to Email API to send emails without audience member creation.

  • Support the engagement application creation without the certificate configuration.

  • Enhancements to the engagement adapter to support API key.

  • Deprecated support for Blackberry

• App Factory

  • Added support for Microservices

  • Added support for Cross Application Testing

  • Added OAuth support for testing deployed web apps

• Installer

  • Added support for WebLogic 14c

  • Cloud Installer on MS Azure

  • Added Docker Support for OpenShift

Brief downtime is expected for the impacted Cloud services while this maintenance is being performed. The scheduled maintenance is designed to mitigate disruptions to service availability and performance for the impacted Cloud services. However, it is possible for the impacted Cloud services to be unavailable and/or performance degraded for a short period of time during the maintenance window. Note that no changes are being applied for other Cloud services outside of the list of impacted services above and no service availability or performance disruption is expected for other Cloud services.

[2021-07-04 21:30 UTC] Database upgrades are running longer than expected, therefore updating the impact to high. We are taking what steps we can to minimize the outage durations.

Maintenance for Identity

• Identity

  • Fixed issue where Custom identity provider’s Claim token obtained in refresh flow was not working for Fabric integration services.

No downtime is expected for the impacted Cloud services while this maintenance is being performed. The scheduled maintenance is designed to mitigate disruptions to service availability and performance for the impacted Cloud services. However, it is possible for the impacted Cloud services to be unavailable and/or performance degraded for a short period of time during the maintenance window. Note that no changes are being applied for other Cloud services outside of the list of impacted services above and no service availability or performance disruption is expected for other Cloud services.

Quantum V9 SP2 Release Fixpack 2

• Cloud Management Console

  • A new feature is added to lock services in a Fabric app. Locking enables partners to deliver pre-build apps, which can be extended by their customers, while still providing the partners the ability to upgrade the pre-build solutions to latest versions of pre-build app

• Workspace

  • A new feature is added to lock services in a Fabric app. Locking enables partners to deliver pre-build apps, which can be extended by their customers, while still providing the partners the ability to upgrade the pre-build solutions to latest versions of pre-build app

The maintenance window start and end times are local to the region in which your Clouds are hosted. If you are unsure where your Clouds are hosted, you can hover over a Cloud Name in the Manage Clouds page of the Cloud Management Console and the region will be displayed.

Identity Service Availability Improvements

• Identity

  • Under some circumstances, specifically constant monitoring of applications using monitors that login frequently or applications with extremely long session or idle durations, stale data can be kept ‘in-cache’ when no longer needed. Over time this can lead to cache congestion. In some rare circumstances, slow cache response times have resulted. The slow cache responses have caused serious impacts on Identity response times. The cache support libraries leveraged by Identity have been updated to resolve this issue.

  • As part of the updates, the Identity configuration has been adjusted to provide additional bandwidth to the cache

  • As part of the deployment, infrastructure will be upgraded to a more recent version of the cache engine along with hardware tuning

  • Dedicated Identity services use a different cache engine and are not affected by this issue

No downtime is expected for the impacted Cloud services while this maintenance is being performed. The scheduled maintenance is designed to mitigate disruptions to service availability and performance for the impacted Cloud services. However, it is possible for the impacted Cloud services to be unavailable and/or performance degraded for a short period of time during the maintenance window. Note that no changes are being applied for other Cloud services outside of the list of impacted services above and no service availability or performance disruption is expected for other Cloud services.

Maintenance for Cloud Management Console, Identity, and Workspace

• WorkSpace

  • Better tolerate publish subsequent issues where a previous deployment may have failed

• MobileFabric Console

  • Enhancements to the Custom Identity Provider that allow custom providers to be informed in more detail on malicious traffic and set throttles for flagging same

• Identity

  • Enhancements to the Custom Identity Provider that allow custom providers to be informed in more detail on malicious traffic and set throttles for flagging same

No downtime is expected for the impacted Cloud services while this maintenance is being performed. The scheduled maintenance is designed to mitigate disruptions to service availability and performance for the impacted Cloud services. However, it is possible for the impacted Cloud services to be unavailable and/or performance degraded for a short period of time during the maintenance window. Note that no changes are being applied for other Cloud services outside of the list of impacted services above and no service availability or performance disruption is expected for other Cloud services.

Tokyo performance degradation

• Cloud services in Tokyo

  • Issue affects all customers using multi-tenant Identity services in Tokyo

The performance and availability of Cloud services in other regions are not impacted.

[2021-03-19 01:11 UTC] Excessive latency for Identity services in the Tokyo region.

[2021-03-19 01:15 UTC] Latency averages have worsened to 50 seconds effecting most APIs.

[2021-03-19 01:25 UTC] Latency averages have improved to 20 seconds, but still impacting most APIs.

[2021-03-19 02:30 UTC] We are continuing to work to restore services.

[2021-03-19 04:20 UTC] Latency averages have improved to 15 seconds, but still impacting most APIs.

[2021-03-19 04:36 UTC] Latency averages have retuned to normal.

[2021-03-19 06:00 UTC] We are continuing to investigate the root cause but there is a risk of reoccurrence.

[2021-03-20 02:00 UTC] Systems are performing normally for the last 24 hours. Escalated issue to Engineering to assist with RCA.

[2021-03-21 02:00 UTC] Systems are performing normally for the last 24 hours. Investigations continue.

[2021-03-22 02:00 UTC] Systems are performing normally for the last 24 hours. Investigations continue.

[2021-03-23 02:00 UTC] Systems are performing normally for the last 24 hours. Investigations continue.

[2021-03-24 00:56 UTC] Excessive latency issue has reoccurred for Identity services in the Tokyo region.

[2021-03-24 01:05 UTC] We are working to mitigate the issue.

[2021-03-24 01:25 UTC] Latency averages have degraded to 25 seconds and are impacting most APIs.

[2021-03-24 01:10 UTC] Latency averages have improved to 10 seconds and are still impacting most APIs.

[2021-03-24 02:00 UTC] Identified high latency on cache cluster, working to mitigate the impact.

[2021-03-24 02:17 UTC] Latency averages have improved to 15 seconds and are impacting most APIs.

[2021-03-24 02:48 UTC] New cache cluster created, reprovisioning systems to use the new cluster.

[2021-03-24 03:11 UTC] New set of Identity nodes start taking requests.

[2021-03-24 03:22 UTC] Removed all old nodes from load balancers.

[2021-03-24 03:24 UTC] [Mitigated] Removed all old nodes from load balancers.

[2021-03-24 06:00 UTC] We are continuing to investigate the root cause but there is a risk of reoccurrence.

[2021-03-25 02:00 UTC] Systems are performing normally for the last 24 hours. We suspect a client cache library issue.

[2021-03-26 02:00 UTC] Systems are performing normally for the last 24 hours. RCA identified as stale data accumulating in an unmanaged cache key. Internally testing a potential fix.

[2021-03-27 02:00 UTC] Systems are performing normally for the last 24 hours. Internal testing validated, functional test in progress. No risk of reoccurrence as mitigation plan in place.

[2021-03-28 02:00 UTC] Systems are performing normally for the last 24 hours. Manually clearing cache daily. Test candidate released internally. Fix Performance testing in progress.

[2021-03-29 02:00 UTC] Systems are performing normally for the last 24 hours. Manually clearing cache daily. Fix Performance testing in progress.

[2021-03-30 02:00 UTC] Systems are performing normally for the last 24 hours. Manually clearing cache daily. QA candidate released for full QA testing.

[2021-04-01 02:00 UTC] Systems are performing normally for the last 24 hours. Manually clearing cache daily. GA candidate released. Scheduling for staging (pre-production) deployment.

[2021-04-01 02:00 UTC] GA candidate approved. Requesting to schedule for production deployment.

[2021-04-01 02:00 UTC] [Resolved] See fix maintenance notice