Kony Cloud status
Current status and incident report
Improved Cloud Management Console Support for Latest 9.2 Release
Maintenance window: September 27, 2021 00:01 to 04:00 UTCCloud Management Console
Implemented calling stored procedures by ordinal position of parameters.
Added support for Oracle CLOB and NCLOB data types.
Fixed a timestamp parsing issue in stored procedures and functions.
Impact Level : low
No downtime is expected for the impacted Cloud services while this maintenance is being performed. The scheduled maintenance is designed to mitigate disruptions to service availability and performance for the impacted Cloud services. However, it is possible for the impacted Cloud services to be unavailable and/or performance degraded for a short period of time during the maintenance window. Note that no changes are being applied for other Cloud services outside of the list of impacted services above and no service availability or performance disruption is expected for other Cloud services.
Reporting Services - Significant performance degradation and unavailability
Incident window: September 14, 2021 00:01 UTC to September 23, 2021 21:00 UTC Impacted Cloud services:Reporting Services
Impact Level : high
Reporting services performance is significantly degraded as analytics ingestion is encountering issues.
We are working to recover the services and will provide an ETA as soon as possible.
There were slight delays in processing analytics beginning on September 14th, which have grown large enough to cause a noticeable issue on the 18th. No analytics data is lost during the window, and will be reflected in newly generated reports as our ingestion systems catch up with the influx of messages to be processed.
[2021-09-23 21:00 UTC] Resolved. The queue is now cleared and metrics/reports are now performing at normal levels.
Cloud Management Console and Workspace Maintenance
Maintenance window: August 30, 2021 00:01 to 04:00Cloud Management Console
Composite app publish status refresh issues fixed
Publish time out increased from default 20 seconds to 60 seconds to allow bigger customer apps to be publishable
Workspace Maintenance
Composite app publish status refresh issues fixed
Publish time out increased from default 20 seconds to 60 seconds to allow bigger customer apps to be publishable
Impact Level : low
No downtime is expected for the impacted Cloud services while this maintenance is being performed. The scheduled maintenance is designed to mitigate disruptions to service availability and performance for the impacted Cloud services. However, it is possible for the impacted Cloud services to be unavailable and/or performance degraded for a short period of time during the maintenance window. Note that no changes are being applied for other Cloud services outside of the list of impacted services above and no service availability or performance disruption is expected for other Cloud services.
Quantum V9 SP3 Release
Maintenance window: August 2, 2021 00:01 to 04:00Fabric Integration Server
Added support for WebSockets to the Quantum Fabric Cloud
Added support to reconfigure the scheme for Swagger 2.0 services
Added the Fabric Workflow Adapter in Integration Services
Enhancements to Workflow Audit history
Middleware Integration with Temenos Metering Service Client Library
Updates to Server Events APIs
Cloud Management Console
Supporting Modularity of Fabric Apps
Added OpenAPI Specification support for Object Verb Mapping
Added support for Custom Front-End URLs in Object Services
Added the Integration Service trigger type to Workflow services
Added support to view linked Java Pre and Post Processors for a Fabric App
Added a Version Upgrade mechanism to MFCLI
Updates to Engagement Adapter
Fabric Identity
Added support to use PKCE in JavaScript SDK
Added support to replace Identity Service references
Engagement
Improved support for Events notifications
Enhancements to Email API to send emails without audience member creation.
Support the engagement application creation without the certificate configuration.
Enhancements to the engagement adapter to support API key.
Deprecated support for Blackberry
App Factory
Added support for Microservices
Added support for Cross Application Testing
Added OAuth support for testing deployed web apps
Installer
Added support for WebLogic 14c
Cloud Installer on MS Azure
Added Docker Support for OpenShift
Impact Level : high
Brief downtime is expected for the impacted Cloud services while this maintenance is being performed. The scheduled maintenance is designed to mitigate disruptions to service availability and performance for the impacted Cloud services. However, it is possible for the impacted Cloud services to be unavailable and/or performance degraded for a short period of time during the maintenance window. Note that no changes are being applied for other Cloud services outside of the list of impacted services above and no service availability or performance disruption is expected for other Cloud services.
[2021-07-04 21:30 UTC] Database upgrades are running longer than expected, therefore updating the impact to high. We are taking what steps we can to minimize the outage durations.
Maintenance for Identity
Maintenance window: June 14, 2021 00:01 to 04:00Identity
Fixed issue where Custom identity provider’s Claim token obtained in refresh flow was not working for Fabric integration services.
Impact Level : minor
No downtime is expected for the impacted Cloud services while this maintenance is being performed. The scheduled maintenance is designed to mitigate disruptions to service availability and performance for the impacted Cloud services. However, it is possible for the impacted Cloud services to be unavailable and/or performance degraded for a short period of time during the maintenance window. Note that no changes are being applied for other Cloud services outside of the list of impacted services above and no service availability or performance disruption is expected for other Cloud services.
Quantum V9 SP2 Release Fixpack 2
Maintenance window: May 31, 2021 00:01 to 04:00 Impacted Cloud services:Cloud Management Console
A new feature is added to lock services in a Fabric app. Locking enables partners to deliver pre-build apps, which can be extended by their customers, while still providing the partners the ability to upgrade the pre-build solutions to latest versions of pre-build app
Workspace
A new feature is added to lock services in a Fabric app. Locking enables partners to deliver pre-build apps, which can be extended by their customers, while still providing the partners the ability to upgrade the pre-build solutions to latest versions of pre-build app
Impact Level : minor
The maintenance window start and end times are local to the region in which your Clouds are hosted. If you are unsure where your Clouds are hosted, you can hover over a Cloud Name in the Manage Clouds page of the Cloud Management Console and the region will be displayed.
Identity Service Availability Improvements
Maintenance window: April 8, 2021 00:01 to 04:00Identity
Under some circumstances, specifically constant monitoring of applications using monitors that login frequently or applications with extremely long session or idle durations, stale data can be kept ‘in-cache’ when no longer needed. Over time this can lead to cache congestion. In some rare circumstances, slow cache response times have resulted. The slow cache responses have caused serious impacts on Identity response times. The cache support libraries leveraged by Identity have been updated to resolve this issue.
As part of the updates, the Identity configuration has been adjusted to provide additional bandwidth to the cache
As part of the deployment, infrastructure will be upgraded to a more recent version of the cache engine along with hardware tuning
Dedicated Identity services use a different cache engine and are not affected by this issue
Impact Level : minor
No downtime is expected for the impacted Cloud services while this maintenance is being performed. The scheduled maintenance is designed to mitigate disruptions to service availability and performance for the impacted Cloud services. However, it is possible for the impacted Cloud services to be unavailable and/or performance degraded for a short period of time during the maintenance window. Note that no changes are being applied for other Cloud services outside of the list of impacted services above and no service availability or performance disruption is expected for other Cloud services.
Maintenance for Cloud Management Console, Identity, and Workspace
Maintenance window: March 29, 2021 00:01 to 04:00WorkSpace
Better tolerate publish subsequent issues where a previous deployment may have failed
MobileFabric Console
Enhancements to the Custom Identity Provider that allow custom providers to be informed in more detail on malicious traffic and set throttles for flagging same
Identity
Enhancements to the Custom Identity Provider that allow custom providers to be informed in more detail on malicious traffic and set throttles for flagging same
Impact Level : minor
No downtime is expected for the impacted Cloud services while this maintenance is being performed. The scheduled maintenance is designed to mitigate disruptions to service availability and performance for the impacted Cloud services. However, it is possible for the impacted Cloud services to be unavailable and/or performance degraded for a short period of time during the maintenance window. Note that no changes are being applied for other Cloud services outside of the list of impacted services above and no service availability or performance disruption is expected for other Cloud services.
Tokyo performance degradation
Incident window: March 19, 2021 01:11 to April 1 02:00 Impacted Cloud services:Cloud services in Tokyo
Issue affects all customers using multi-tenant Identity services in Tokyo
Impact Level : high
The performance and availability of Cloud services in other regions are not impacted.
[2021-03-19 01:11 UTC] Excessive latency for Identity services in the Tokyo region.
[2021-03-19 01:15 UTC] Latency averages have worsened to 50 seconds effecting most APIs.
[2021-03-19 01:25 UTC] Latency averages have improved to 20 seconds, but still impacting most APIs.
[2021-03-19 02:30 UTC] We are continuing to work to restore services.
[2021-03-19 04:20 UTC] Latency averages have improved to 15 seconds, but still impacting most APIs.
[2021-03-19 04:36 UTC] Latency averages have retuned to normal.
[2021-03-19 06:00 UTC] We are continuing to investigate the root cause but there is a risk of reoccurrence.
[2021-03-20 02:00 UTC] Systems are performing normally for the last 24 hours. Escalated issue to Engineering to assist with RCA.
[2021-03-21 02:00 UTC] Systems are performing normally for the last 24 hours. Investigations continue.
[2021-03-22 02:00 UTC] Systems are performing normally for the last 24 hours. Investigations continue.
[2021-03-23 02:00 UTC] Systems are performing normally for the last 24 hours. Investigations continue.
[2021-03-24 00:56 UTC] Excessive latency issue has reoccurred for Identity services in the Tokyo region.
[2021-03-24 01:05 UTC] We are working to mitigate the issue.
[2021-03-24 01:25 UTC] Latency averages have degraded to 25 seconds and are impacting most APIs.
[2021-03-24 01:10 UTC] Latency averages have improved to 10 seconds and are still impacting most APIs.
[2021-03-24 02:00 UTC] Identified high latency on cache cluster, working to mitigate the impact.
[2021-03-24 02:17 UTC] Latency averages have improved to 15 seconds and are impacting most APIs.
[2021-03-24 02:48 UTC] New cache cluster created, reprovisioning systems to use the new cluster.
[2021-03-24 03:11 UTC] New set of Identity nodes start taking requests.
[2021-03-24 03:22 UTC] Removed all old nodes from load balancers.
[2021-03-24 03:24 UTC] [Mitigated] Removed all old nodes from load balancers.
[2021-03-24 06:00 UTC] We are continuing to investigate the root cause but there is a risk of reoccurrence.
[2021-03-25 02:00 UTC] Systems are performing normally for the last 24 hours. We suspect a client cache library issue.
[2021-03-26 02:00 UTC] Systems are performing normally for the last 24 hours. RCA identified as stale data accumulating in an unmanaged cache key. Internally testing a potential fix.
[2021-03-27 02:00 UTC] Systems are performing normally for the last 24 hours. Internal testing validated, functional test in progress. No risk of reoccurrence as mitigation plan in place.
[2021-03-28 02:00 UTC] Systems are performing normally for the last 24 hours. Manually clearing cache daily. Test candidate released internally. Fix Performance testing in progress.
[2021-03-29 02:00 UTC] Systems are performing normally for the last 24 hours. Manually clearing cache daily. Fix Performance testing in progress.
[2021-03-30 02:00 UTC] Systems are performing normally for the last 24 hours. Manually clearing cache daily. QA candidate released for full QA testing.
[2021-04-01 02:00 UTC] Systems are performing normally for the last 24 hours. Manually clearing cache daily. GA candidate released. Scheduling for staging (pre-production) deployment.
[2021-04-01 02:00 UTC] GA candidate approved. Requesting to schedule for production deployment.
[2021-04-01 02:00 UTC] [Resolved] See fix maintenance notice