menu

Feed available - Subscribe to our feed to stay up to date on upcoming maintenance and incidents.

Kony Cloud status
Current status and incident report

Identity Service Availability Improvements

Maintenance window: April 8, 2021 00:01 to 04:00
The maintenance window start and end times are local to the region in which your Clouds are hosted. If you are unsure where your Clouds are hosted, you can hover over a Cloud Name in the Manage Clouds page of the Cloud Management Console and the region will be displayed.
Impacted Cloud services:
  • Identity

    • Under some circumstances, specifically constant monitoring of applications using monitors that login frequently or applications with extremely long session or idle durations, stale data can be kept ‘in-cache’ when no longer needed. Over time this can lead to cache congestion. In some rare circumstances, slow cache response times have resulted. The slow cache responses have caused serious impacts on Identity response times. The cache support libraries leveraged by Identity have been updated to resolve this issue.

    • As part of the updates, the Identity configuration has been adjusted to provide additional bandwidth to the cache

    • As part of the deployment, infrastructure will be upgraded to a more recent version of the cache engine along with hardware tuning

    • Dedicated Identity services use a different cache engine and are not affected by this issue


Impact Level : minor

No downtime is expected for the impacted Cloud services while this maintenance is being performed. The scheduled maintenance is designed to mitigate disruptions to service availability and performance for the impacted Cloud services. However, it is possible for the impacted Cloud services to be unavailable and/or performance degraded for a short period of time during the maintenance window. Note that no changes are being applied for other Cloud services outside of the list of impacted services above and no service availability or performance disruption is expected for other Cloud services.

Maintenance for Cloud Management Console, Identity, and Workspace

Maintenance window: March 29, 2021 00:01 to 04:00
The maintenance window start and end times are local to the region in which your Clouds are hosted. If you are unsure where your Clouds are hosted, you can hover over a Cloud Name in the Manage Clouds page of the Cloud Management Console and the region will be displayed.
Impacted Cloud services:
  • WorkSpace

    • Better tolerate publish subsequent issues where a previous deployment may have failed

  • MobileFabric Console

    • Enhancements to the Custom Identity Provider that allow custom providers to be informed in more detail on malicious traffic and set throttles for flagging same

  • Identity

    • Enhancements to the Custom Identity Provider that allow custom providers to be informed in more detail on malicious traffic and set throttles for flagging same


Impact Level : minor

No downtime is expected for the impacted Cloud services while this maintenance is being performed. The scheduled maintenance is designed to mitigate disruptions to service availability and performance for the impacted Cloud services. However, it is possible for the impacted Cloud services to be unavailable and/or performance degraded for a short period of time during the maintenance window. Note that no changes are being applied for other Cloud services outside of the list of impacted services above and no service availability or performance disruption is expected for other Cloud services.

Tokyo performance degradation

Incident window: March 19, 2021 01:11 to April 1 02:00
Impacted Cloud services:
  • Cloud services in Tokyo

    • Issue affects all customers using multi-tenant Identity services in Tokyo


Impact Level : high

The performance and availability of Cloud services in other regions are not impacted.

[2021-03-19 01:11 UTC] Excessive latency for Identity services in the Tokyo region.

[2021-03-19 01:15 UTC] Latency averages have worsened to 50 seconds effecting most APIs.

[2021-03-19 01:25 UTC] Latency averages have improved to 20 seconds, but still impacting most APIs.

[2021-03-19 02:30 UTC] We are continuing to work to restore services.

[2021-03-19 04:20 UTC] Latency averages have improved to 15 seconds, but still impacting most APIs.

[2021-03-19 04:36 UTC] Latency averages have retuned to normal.

[2021-03-19 06:00 UTC] We are continuing to investigate the root cause but there is a risk of reoccurrence.

[2021-03-20 02:00 UTC] Systems are performing normally for the last 24 hours. Escalated issue to Engineering to assist with RCA.

[2021-03-21 02:00 UTC] Systems are performing normally for the last 24 hours. Investigations continue.

[2021-03-22 02:00 UTC] Systems are performing normally for the last 24 hours. Investigations continue.

[2021-03-23 02:00 UTC] Systems are performing normally for the last 24 hours. Investigations continue.

[2021-03-24 00:56 UTC] Excessive latency issue has reoccurred for Identity services in the Tokyo region.

[2021-03-24 01:05 UTC] We are working to mitigate the issue.

[2021-03-24 01:25 UTC] Latency averages have degraded to 25 seconds and are impacting most APIs.

[2021-03-24 01:10 UTC] Latency averages have improved to 10 seconds and are still impacting most APIs.

[2021-03-24 02:00 UTC] Identified high latency on cache cluster, working to mitigate the impact.

[2021-03-24 02:17 UTC] Latency averages have improved to 15 seconds and are impacting most APIs.

[2021-03-24 02:48 UTC] New cache cluster created, reprovisioning systems to use the new cluster.

[2021-03-24 03:11 UTC] New set of Identity nodes start taking requests.

[2021-03-24 03:22 UTC] Removed all old nodes from load balancers.

[2021-03-24 03:24 UTC] [Mitigated] Removed all old nodes from load balancers.

[2021-03-24 06:00 UTC] We are continuing to investigate the root cause but there is a risk of reoccurrence.

[2021-03-25 02:00 UTC] Systems are performing normally for the last 24 hours. We suspect a client cache library issue.

[2021-03-26 02:00 UTC] Systems are performing normally for the last 24 hours. RCA identified as stale data accumulating in an unmanaged cache key. Internally testing a potential fix.

[2021-03-27 02:00 UTC] Systems are performing normally for the last 24 hours. Internal testing validated, functional test in progress. No risk of reoccurrence as mitigation plan in place.

[2021-03-28 02:00 UTC] Systems are performing normally for the last 24 hours. Manually clearing cache daily. Test candidate released internally. Fix Performance testing in progress.

[2021-03-29 02:00 UTC] Systems are performing normally for the last 24 hours. Manually clearing cache daily. Fix Performance testing in progress.

[2021-03-30 02:00 UTC] Systems are performing normally for the last 24 hours. Manually clearing cache daily. QA candidate released for full QA testing.

[2021-04-01 02:00 UTC] Systems are performing normally for the last 24 hours. Manually clearing cache daily. GA candidate released. Scheduling for staging (pre-production) deployment.

[2021-04-01 02:00 UTC] GA candidate approved. Requesting to schedule for production deployment.

[2021-04-01 02:00 UTC] [Resolved] See fix maintenance notice

Cloud Management Console and Workspace Maintenance

Maintenance window: March 15, 2021 00:01 to 04:00
The maintenance window start and end times are local to the region in which your Clouds are hosted. If you are unsure where your Clouds are hosted, you can hover over a Cloud Name in the Manage Clouds page of the Cloud Management Console and the region will be displayed.
Impacted Cloud services:
  • Cloud Management Console

    • Support for Identity Provider authentication in RedHat Adapter Services

    • Enhancements to Custom Identity Provider

  • Workspace

    • Enhancements to Custom Identity Provider


Impact Level : minor

Minor downtime is possible for the impacted Cloud services while this maintenance is being performed. The scheduled maintenance is designed to mitigate disruptions to service availability and performance for the impacted Cloud services. However, it is possible for the impacted Cloud services to be unavailable and/or performance degraded for a short period of time during the maintenance window. Note that no changes are being applied for other Cloud services outside of the list of impacted services above and no service availability or performance disruption is expected for other Cloud services.

Quantum V9 SP2 Release Fixpack 1

Maintenance window: February 10, 2021 00:01 to 04:00
The maintenance window start and end times are local to the region in which your Clouds are hosted. If you are unsure where your Clouds are hosted, you can hover over a Cloud Name in the Manage Clouds page of the Cloud Management Console and the region will be displayed.
Impacted Cloud services:
  • Fabric Integration Server and Cloud Management Console

    • Added new APIs to Red Hat PAM adapter in integration services


Impact Level : medium

Quantum V9 SP2 Release

Maintenance window: January 18, 2021 00:01 to 04:00
The maintenance window start and end times are local to the region in which your Clouds are hosted. If you are unsure where your Clouds are hosted, you can hover over a Cloud Name in the Manage Clouds page of the Cloud Management Console and the region will be displayed.
Impacted Cloud services:
  • Fabric Integration Server

    • A new adapter added for integration with Red Hat Process Automation Manager. Using the new adapter users can invoke and explore Rules, Process and Cases defined in their Red Hat Process Automation Manager Instance

    • Open API adapter is enhanced to support OpenAPI 3.0 files. Users can now import their OpenAPI 3.0 files to create services in Fabric

    • Support for triggering Workflows by raising Fabric events is added

    • Support for Signal Events (Signal Start, Catch Signal, Throw Signal) is added in Fabric Workflow which enables users to pause and restore the workflows

    • Support for retrying a failed service tasks from admin console is added

    • Workflow Audit History view in Admin Console is enhanced for better traceability of tasks

  • Cloud Management Console

    • Application Publish Enhancements

      • Incremental Application publish support is added to significantly reduce the publish time after the first publish of an application
      • New reconfiguration option added to skip swagger generation steps during publish
      • Improved error messaging in case of publish failures
      • Quick feedback to the user in case the server is busy with other application publish
    • New deployment package is introduced which can be downloaded from snapshot package of an environment and promoted to another environment without being impacted by the development changes in Console

    • A new simplified format of reconfiguration file is supported to better editing support

    • Support for RedHat PAM is added

    • Support for importing OpenAPI 3.0 files in existing OpenAPI adapter

    • Support for multiple enterprise logins

  • Fabric Identity

    • OAuth 2.0 identity providers enhancements

      • Support added for Proof Key for Code Exchange (PKCE) in Authorization code flow
      • Refresh Token Grant Type support is added for oAuth Providers
    • Console Access control is added for identity services

  • Engagement

    • Support for the provisioning of the single tenant engagement server


Impact Level : minor

Minor downtime is possible for the impacted Cloud services while this maintenance is being performed. The scheduled maintenance is designed to mitigate disruptions to service availability and performance for the impacted Cloud services. However, it is possible for the impacted Cloud services to be unavailable and/or performance degraded for a short period of time during the maintenance window. Note that no changes are being applied for other Cloud services outside of the list of impacted services above and no service availability or performance disruption is expected for other Cloud services.

Intermittent Service Degradation for the Management Console

Incident window: January 8, 2020 18:00 to 20:37 UTC
The maintenance window start and end times are local to the region in which your Clouds are hosted. If you are unsure where your Clouds are hosted, you can hover over a Cloud Name in the Manage Clouds page of the Cloud Management Console and the region will be displayed.
Impacted Cloud services:
  • Cloud Management Console


    Impact Level : medium

    We’re experiencing performance degradation for manage.kony.com. This should not affect any runtime environments. Engineering is looking into the incident.

    Cloud Management Console - Performance Degradation

    Incident window: December 9, 2020 17:00 UTC to December 10, 2020, 02:20 UTC
    The maintenance window start and end times are local to the region in which your Clouds are hosted. If you are unsure where your Clouds are hosted, you can hover over a Cloud Name in the Manage Clouds page of the Cloud Management Console and the region will be displayed.
    Impacted Cloud services:
    • Cloud Management Console

      • Access to management console impacted


    Impact Level : medium

    We’re experiencing performance degradation for manage.kony.com. This should not affect any runtime environments. Engineering is looking into the incident.

    [2020-12-09 17:52 UTC] Resolved Issue identified and performance restored.

    [2020-12-10 00:45 UTC] Performance degradation observed. Engineering team has restored performance.

    [2020-12-10 02:20 UTC] The root cause has been identified and engineering has implemented the fix.

    Cloud Management Console - Performance Degradation

    Incident window: November 23, 2020 20:00 to 23:40 UTC
    Impacted Cloud services:
    • Cloud Management Console

      • Access to management console impacted


    Impact Level : medium

    We’re experiencing performance degradation for manage.kony.com. This should not affect any runtime environments. Engineering is looking into the incident.

    [2020-11-23 23:40 UTC] Resolved Issue identified and performance restored.