menu

Feed available - Subscribe to our feed to stay up to date on upcoming maintenance and incidents.

Kony Cloud status
Current status and incident report

Tokyo performance degradation

Incident window: March 19, 2021 01:11 to April 1 02:00
Impacted Cloud services:
  • Cloud services in Tokyo

    • Issue affects all customers using multi-tenant Identity services in Tokyo


Impact Level : high

The performance and availability of Cloud services in other regions are not impacted.

[2021-03-19 01:11 UTC] Excessive latency for Identity services in the Tokyo region.

[2021-03-19 01:15 UTC] Latency averages have worsened to 50 seconds effecting most APIs.

[2021-03-19 01:25 UTC] Latency averages have improved to 20 seconds, but still impacting most APIs.

[2021-03-19 02:30 UTC] We are continuing to work to restore services.

[2021-03-19 04:20 UTC] Latency averages have improved to 15 seconds, but still impacting most APIs.

[2021-03-19 04:36 UTC] Latency averages have retuned to normal.

[2021-03-19 06:00 UTC] We are continuing to investigate the root cause but there is a risk of reoccurrence.

[2021-03-20 02:00 UTC] Systems are performing normally for the last 24 hours. Escalated issue to Engineering to assist with RCA.

[2021-03-21 02:00 UTC] Systems are performing normally for the last 24 hours. Investigations continue.

[2021-03-22 02:00 UTC] Systems are performing normally for the last 24 hours. Investigations continue.

[2021-03-23 02:00 UTC] Systems are performing normally for the last 24 hours. Investigations continue.

[2021-03-24 00:56 UTC] Excessive latency issue has reoccurred for Identity services in the Tokyo region.

[2021-03-24 01:05 UTC] We are working to mitigate the issue.

[2021-03-24 01:25 UTC] Latency averages have degraded to 25 seconds and are impacting most APIs.

[2021-03-24 01:10 UTC] Latency averages have improved to 10 seconds and are still impacting most APIs.

[2021-03-24 02:00 UTC] Identified high latency on cache cluster, working to mitigate the impact.

[2021-03-24 02:17 UTC] Latency averages have improved to 15 seconds and are impacting most APIs.

[2021-03-24 02:48 UTC] New cache cluster created, reprovisioning systems to use the new cluster.

[2021-03-24 03:11 UTC] New set of Identity nodes start taking requests.

[2021-03-24 03:22 UTC] Removed all old nodes from load balancers.

[2021-03-24 03:24 UTC] [Mitigated] Removed all old nodes from load balancers.

[2021-03-24 06:00 UTC] We are continuing to investigate the root cause but there is a risk of reoccurrence.

[2021-03-25 02:00 UTC] Systems are performing normally for the last 24 hours. We suspect a client cache library issue.

[2021-03-26 02:00 UTC] Systems are performing normally for the last 24 hours. RCA identified as stale data accumulating in an unmanaged cache key. Internally testing a potential fix.

[2021-03-27 02:00 UTC] Systems are performing normally for the last 24 hours. Internal testing validated, functional test in progress. No risk of reoccurrence as mitigation plan in place.

[2021-03-28 02:00 UTC] Systems are performing normally for the last 24 hours. Manually clearing cache daily. Test candidate released internally. Fix Performance testing in progress.

[2021-03-29 02:00 UTC] Systems are performing normally for the last 24 hours. Manually clearing cache daily. Fix Performance testing in progress.

[2021-03-30 02:00 UTC] Systems are performing normally for the last 24 hours. Manually clearing cache daily. QA candidate released for full QA testing.

[2021-04-01 02:00 UTC] Systems are performing normally for the last 24 hours. Manually clearing cache daily. GA candidate released. Scheduling for staging (pre-production) deployment.

[2021-04-01 02:00 UTC] GA candidate approved. Requesting to schedule for production deployment.

[2021-04-01 02:00 UTC] [Resolved] See fix maintenance notice

Cloud Management Console and Workspace Maintenance

Maintenance window: March 15, 2021 00:01 to 04:00
The maintenance window start and end times are local to the region in which your Clouds are hosted. If you are unsure where your Clouds are hosted, you can hover over a Cloud Name in the Manage Clouds page of the Cloud Management Console and the region will be displayed.
Impacted Cloud services:
  • Cloud Management Console

    • Support for Identity Provider authentication in RedHat Adapter Services

    • Enhancements to Custom Identity Provider

  • Workspace

    • Enhancements to Custom Identity Provider


Impact Level : minor

Minor downtime is possible for the impacted Cloud services while this maintenance is being performed. The scheduled maintenance is designed to mitigate disruptions to service availability and performance for the impacted Cloud services. However, it is possible for the impacted Cloud services to be unavailable and/or performance degraded for a short period of time during the maintenance window. Note that no changes are being applied for other Cloud services outside of the list of impacted services above and no service availability or performance disruption is expected for other Cloud services.

Quantum V9 SP2 Release Fixpack 1

Maintenance window: February 10, 2021 00:01 to 04:00
The maintenance window start and end times are local to the region in which your Clouds are hosted. If you are unsure where your Clouds are hosted, you can hover over a Cloud Name in the Manage Clouds page of the Cloud Management Console and the region will be displayed.
Impacted Cloud services:
  • Fabric Integration Server and Cloud Management Console

    • Added new APIs to Red Hat PAM adapter in integration services


Impact Level : medium

Quantum V9 SP2 Release

Maintenance window: January 18, 2021 00:01 to 04:00
The maintenance window start and end times are local to the region in which your Clouds are hosted. If you are unsure where your Clouds are hosted, you can hover over a Cloud Name in the Manage Clouds page of the Cloud Management Console and the region will be displayed.
Impacted Cloud services:
  • Fabric Integration Server

    • A new adapter added for integration with Red Hat Process Automation Manager. Using the new adapter users can invoke and explore Rules, Process and Cases defined in their Red Hat Process Automation Manager Instance

    • Open API adapter is enhanced to support OpenAPI 3.0 files. Users can now import their OpenAPI 3.0 files to create services in Fabric

    • Support for triggering Workflows by raising Fabric events is added

    • Support for Signal Events (Signal Start, Catch Signal, Throw Signal) is added in Fabric Workflow which enables users to pause and restore the workflows

    • Support for retrying a failed service tasks from admin console is added

    • Workflow Audit History view in Admin Console is enhanced for better traceability of tasks

  • Cloud Management Console

    • Application Publish Enhancements

      • Incremental Application publish support is added to significantly reduce the publish time after the first publish of an application
      • New reconfiguration option added to skip swagger generation steps during publish
      • Improved error messaging in case of publish failures
      • Quick feedback to the user in case the server is busy with other application publish
    • New deployment package is introduced which can be downloaded from snapshot package of an environment and promoted to another environment without being impacted by the development changes in Console

    • A new simplified format of reconfiguration file is supported to better editing support

    • Support for RedHat PAM is added

    • Support for importing OpenAPI 3.0 files in existing OpenAPI adapter

    • Support for multiple enterprise logins

  • Fabric Identity

    • OAuth 2.0 identity providers enhancements

      • Support added for Proof Key for Code Exchange (PKCE) in Authorization code flow
      • Refresh Token Grant Type support is added for oAuth Providers
    • Console Access control is added for identity services

  • Engagement

    • Support for the provisioning of the single tenant engagement server


Impact Level : minor

Minor downtime is possible for the impacted Cloud services while this maintenance is being performed. The scheduled maintenance is designed to mitigate disruptions to service availability and performance for the impacted Cloud services. However, it is possible for the impacted Cloud services to be unavailable and/or performance degraded for a short period of time during the maintenance window. Note that no changes are being applied for other Cloud services outside of the list of impacted services above and no service availability or performance disruption is expected for other Cloud services.

Intermittent Service Degradation for the Management Console

Incident window: January 8, 2020 18:00 to 20:37 UTC
The maintenance window start and end times are local to the region in which your Clouds are hosted. If you are unsure where your Clouds are hosted, you can hover over a Cloud Name in the Manage Clouds page of the Cloud Management Console and the region will be displayed.
Impacted Cloud services:
  • Cloud Management Console


    Impact Level : medium

    We’re experiencing performance degradation for manage.kony.com. This should not affect any runtime environments. Engineering is looking into the incident.

    Cloud Management Console - Performance Degradation

    Incident window: December 9, 2020 17:00 UTC to December 10, 2020, 02:20 UTC
    The maintenance window start and end times are local to the region in which your Clouds are hosted. If you are unsure where your Clouds are hosted, you can hover over a Cloud Name in the Manage Clouds page of the Cloud Management Console and the region will be displayed.
    Impacted Cloud services:
    • Cloud Management Console

      • Access to management console impacted


    Impact Level : medium

    We’re experiencing performance degradation for manage.kony.com. This should not affect any runtime environments. Engineering is looking into the incident.

    [2020-12-09 17:52 UTC] Resolved Issue identified and performance restored.

    [2020-12-10 00:45 UTC] Performance degradation observed. Engineering team has restored performance.

    [2020-12-10 02:20 UTC] The root cause has been identified and engineering has implemented the fix.

    Cloud Management Console - Performance Degradation

    Incident window: November 23, 2020 20:00 to 23:40 UTC
    Impacted Cloud services:
    • Cloud Management Console

      • Access to management console impacted


    Impact Level : medium

    We’re experiencing performance degradation for manage.kony.com. This should not affect any runtime environments. Engineering is looking into the incident.

    [2020-11-23 23:40 UTC] Resolved Issue identified and performance restored.

    Cloud SSL Certificate Updates

    Maintenance window: November 14, 2020 08:01 UTC to November 15, 2020 00:01 UTC
    Impacted Cloud services:
    • Cloud SSL certificates for Identity services (.auth.konycloud.com), Engagement services (.messaging.konycloud.com), App services (.konycloud.com), and Sync services (.sync.konycloud.com)

      • ⚠️ Note: If you have not pinned Kony certificates in your application, no application updates will be necessary. Customers that have pinned SSL certificates will need to download the new certificates, rebuild the applications, including both the old and new certificates, and publish the updated binaries to the various app stores. Your applications should be published before Kony updates the certificates on the cloud servers or customers will no longer be able to connect. New certificates can be found on the Kony Cloud Certificate Preview page. The new certificates can also be downloaded by executing the following commands:

        • Identity: openssl s_client -showcerts -connect konycertificatepreview.auth.konycloud.com:443

        • Engagement: openssl s_client -showcerts -connect konycertificatepreview.messaging.konycloud.com:7443

        • App: openssl s_client -showcerts -connect konycertificatepreview.konycloud.com:8443

        • Sync: openssl s_client -showcerts -connect konycertificatepreview.sync.konycloud.com:9443

      • Customers who have pinned the public key instead of the full certificate (which we strongly recommend and was made available in V8 SP4) may not be required to update their applications. The updated certificates will have the same public keys as the existing certificates.

        • If necessary, you can submit your applications for expedited approval (e.g., Apple has an expedited approval process for critical bugs, or in this case, pinned certificates).


          Impact Level : high

          Customer applications that have pinned SSL certificates will need to be updated as described above prior to this maintenance window. Customer applications that have not pinned SSL certificates will not be affected and will experience no service disruptions during this maintenance window.

          Please refer to our documentation for how to pin the public key of a certificate (which we strongly recommend and was made available in V8 SP4) or how to pin an SSL certificate in your apps (deprecated).

          [2020-11-14] 17:45 UTC] Resolved Infrastructure certificates rotation has completed. We have verified the SSL certs on all affected multi-tenant services in all regions.

          Identity maintenance

          Maintenance window: November 14, 2020 00:01 to 04:00
          The maintenance window start and end times are local to the region in which your Clouds are hosted. If you are unsure where your Clouds are hosted, you can hover over a Cloud Name in the Manage Clouds page of the Cloud Management Console and the region will be displayed.
          Impacted Cloud services:
          • Identity

            • Fixed issue where revoke URL is not getting auto-generated for OKTA identity service

            • Added grant type ‘refresh_token’ support to OAuth identity provider


          Impact Level : minor

          No downtime is expected for the impacted Cloud services while this maintenance is being performed. The scheduled maintenance is designed to mitigate disruptions to service availability and performance for the impacted Cloud services. However, it is possible for the impacted Cloud services to be unavailable and/or performance degraded for a short period of time during the maintenance window. Note that no changes are being applied for other Cloud services outside of the list of impacted services above and no service availability or performance disruption is expected for other Cloud services.