menu

Feed available - Subscribe to our feed to stay up to date on upcoming maintenance and incidents.

Kony Cloud status
Current status and incident report

Cloud SSL Certificate Updates

Maintenance window: November 16, 2019 00:01 UTC to November 17, 2019 00:01 UTC
Impacted Cloud services:
  • Cloud SSL certificates for Identity services (.auth.konycloud.com), Engagement services (.messaging.konycloud.com), App services (.konycloud.com), and Sync services (.sync.konycloud.com)

    • ⚠️ Note: If you have not pinned Kony certificates in your application, no application updates will be necessary. Customers that have pinned SSL certificates will need to download the new certificates, rebuild the applications, including both the old and new certificates, and publish the updated binaries to the various app stores. Your applications should be published before Kony updates the certificates on the cloud servers or customers will no longer be able to connect. New certificates can be found on the Kony Cloud Certificate Preview page. The new certificates can also be downloaded by executing the following commands:

      • Identity: openssl s_client -showcerts -connect konycertificatepreview.auth.konycloud.com:443

      • Engagement: openssl s_client -showcerts -connect konycertificatepreview.messaging.konycloud.com:7443

      • App: openssl s_client -showcerts -connect konycertificatepreview.konycloud.com:8443

      • Sync: openssl s_client -showcerts -connect konycertificatepreview.sync.konycloud.com:9443

    • Customers who have pinned the public key instead of the full certificate (which we strongly recommend and was made available in V8 SP4) may not be required to update their applications. The updated certificates will have the same public keys as the existing certificates.

      • If necessary, you can submit your applications for expedited approval (e.g., Apple has an expedited approval process for critical bugs, or in this case, pinned certificates).


        Impact Level : high

        Customer applications that have pinned SSL certificates will need to be updated as described above prior to this maintenance window. Customer applications that have not pinned SSL certificates will not be affected and will experience no service disruptions during this maintenance window.

        Please refer to our documentation for how to pin the public key of a certificate (which we strongly recommend and was made available in V8 SP4) or how to pin an SSL certificate in your apps (deprecated).

        Integration Server V8 SP4 FP3 HF8 Release

        Maintenance window: November 11, 2019 00:01 to 04:00
        The maintenance window start and end times are local to the region in which your Clouds are hosted. If you are unsure where your Clouds are hosted, you can hover over a Cloud Name in the Manage Clouds page of the Cloud Management Console and the region will be displayed.
        Impacted Cloud services:
        • Integration Server

          • Fix inconsistent formatting of x-kony-reportingparams in request headers when application invokes integration service. In this case, it is url encoded, which is not correct. Instead, it should be returned as a json string, similar to how it is returned via Object service.

          • ⚠️ Any existing Clouds containing dedicated (i.e., single-tenant) Kony Integration Server runtime environments will NOT be affected during the maintenance window (i.e., will NOT be automatically upgraded). If you wish to upgrade, please open a support case, specifying your Cloud(s), desired version (V8 SP4 FP3 HF8), and desired maintenance window (day, time, and timezone) when we can apply the upgrade. Existing Clouds containing shared (i.e., multi-tenant) Kony Integration Server runtime environments (e.g., AppPlatform Gold & Platinum tiers (excluding Platinum Plus), free Fabric services) WILL be automatically upgraded during the maintenance window. For additional details regarding this hotfix, please refer to our release notes page, which will be updated in the next few days with notes for this latest release.


        Impact Level : minor

        No downtime is expected for the impacted Cloud services while this maintenance is being performed. The scheduled maintenance is designed to mitigate disruptions to service availability and performance for the impacted Cloud services. However, it is possible for the impacted Cloud services to be unavailable and/or performance degraded for a short period of time during the maintenance window. Note that no changes are being applied for other Cloud services outside of the list of impacted services above and no service availability or performance disruption is expected for other Cloud services.

        Workspace services - Oregon is unavailable

        Incident window: November 8, 2019 22:16 UTC to 22:48 UTC
        Impacted Cloud services:
        • Workspace (Oregon region only)

          • Other regions are not affected


        Impact Level : high

        Workspace services in Oregon have become unavailable starting at 22:16 UTC. The availability and performance of Workspace services in other regions is not affected. We are investigating.

        [2019-11-08 22:48 UTC] Resolved. The underlying Oregon Workspace database master node had failed and automatically failed over to the replica node, which had been promoted as the new master (and a new replica created after). However, while the database was able to recover with our automation, the Oregon Workspace application did not automatically recover as would be expected and was still yielding errors until we intervened by restarting. Once restarted at 22:48 UTC, the Oregon Workspace services have been available and performing within expected levels. We will be continuing to monitor. We will also be continuing to coordinate with our development team to identify why the Workspace application was unable to automatically recover and to work toward a solution to mitigate these types of incidents requiring our attention beyond our automated incident response controls in the future.

        Workspace services - Oregon is unavailable

        Incident window: November 7, 2019 16:30 UTC to 16:45 UTC
        Impacted Cloud services:
        • Workspace (Oregon region only)

          • Other regions are not affected


        Impact Level : high

        Workspace services in Oregon have become unavailable starting at 16:30 UTC. The availability and performance of Workspace services in other regions is not affected. We are investigating.

        [2019-11-07 16:59 UTC] Resolved. The underlying Oregon Workspace database master node had failed and automatically failed over to the replica node, which had been promoted as the new master (and a new replica created after). However, while the database was able to recover with our automation, the Oregon Workspace application did not automatically recover as would be expected and was still yielding errors until we intervened by restarting. Once restarted at 16:45 UTC, the Oregon Workspace services have been available and performing within expected levels. We will be continuing to monitor. We will also be continuing to coordinate with our development team to identify why the Workspace application was unable to automatically recover and to work toward a solution to mitigate these types of incidents requiring our attention beyond our automated incident response controls in the future.

        Workspace services - Oregon is unavailable

        Incident window: November 4, 2019 17:21 UTC to 18:26 UTC
        Impacted Cloud services:
        • Workspace (Oregon region only)

          • Other regions are not affected


        Impact Level : high

        Workspace services in Oregon have become unavailable starting at 17:21 UTC. The availability and performance of Workspace services in other regions is not affected. We are investigating.

        infrastructure failure of a DB. The database has been recovered and the systems did not auto-restart as expected. We have opened internal development tickets on this scenario and will have the team correct this as we expect the workspace systems to survive this type of issue without intervention.

        [2019-11-04 18:41 UTC] Resolved. The underlying Oregon Workspace database master node had failed and automatically failed over to the replica node, which had been promoted as the new master (and a new replica created after). However, while the database was able to recover with our automation, the Oregon Workspace application did not automatically recover as would be expected and was still yielding errors until we intervened by restarting. Once restarted at 18:26 UTC, the Oregon Workspace services have been available and performing within expected levels. We will be continuing to monitor. We will also be coordinating with our development team to identify why the Workspace application was unable to automatically recover and to work toward a solution to mitigate these types of incidents requiring our attention beyond our automated incident response controls in the future.

        Cloud Management console and Identity services - severe performance degradation

        Incident window: October 28, 2019 17:01 to 17:56 UTC
        Impacted Cloud services:
        • Cloud management console

          • Identity


            Impact Level : high

            We have detected significant performance degradation for the impacted Cloud services, resulting in increased latency and error rates for the responses. We are working to restore service performance and availability to normal operating levels.

            [2019-10-28 18:15 UTC] Resolved. We have restored performance and availability for the Cloud management console (https://manage.kony.com) and Identity services as of 17:56 UTC.

            [2019-10-28 21:24 UTC] We would like to share some additional information concerning the root cause of the incident today and the additional measures we are taking to reduce the likelihood of recurrence as well as to improve our ability to more quickly respond to and address similar events.

            In preparation for database certificate rotations, a new certificate bundle was loaded into our staging area. The automation properly replicated that bundle, but it was not expected that the systems would activate the bundle until sometime later.

            Unfortunately, the Identity systems reloaded the certificate bundle prematurely. This caused connection errors to the databases, which very quickly locked all of the existing Identity servers out because the errors exceeded the connection threshold. This threshold is in place to prevent attacks on the database from the same IP trying to connect and guess passwords, etc.

            We corrected the issue with the assets and the automation was able to propagate the updates to the systems. However, having been locked out, the updates had no effect as the existing servers were unable to reach the database because of the now blacklisted IP. Similarly, we were unable to connect and manually reset the database from any of the authorized instances due to the instances all now having a blacklisted IP.

            Once it was determined that automation had pulled the new bundle to all of the instances (and there would be no quick way to connect to the database from any existing servers), the fastest solution was to replaced the entire fleet of Identity servers in such a way that all of the new servers were ensured not to get one of the now blacklisted IPs.

            The new servers were built with the proper assets and services were restored as these systems came on-online.

            We have identified the following improvement actions that we will be pursuing in an effort to mitigate the recurrence of this type of issue altogether, but also to improve our incident response procedures to more rapidly resolve similar future issues:

            1. Review the ‘attack’ threshold to ensure it is not overly aggressive.
            2. Investigate why the Identity services activated the new asset without the expected coordinated restart.
            3. Investigate the creation of additional highly restricted management instances that may provide access in cases like this to cut the time it takes to gain access for any manual intervention.

            V8 SP4 Releases

            Maintenance window: October 21, 2019 00:01 to 04:00
            The maintenance window start and end times are local to the region in which your Clouds are hosted. If you are unsure where your Clouds are hosted, you can hover over a Cloud Name in the Manage Clouds page of the Cloud Management Console and the region will be displayed.
            Impacted Cloud services:
            • Cloud Management Console

              • Add MFA support for custom identity provider

            • Engagement

              • Fix bulk push issue resulting in BatchUpdateException: Failed to save messageentry_status duplicate entry ‘' for key 'PRIMARY'

              • The frequency of the procedure specifically used for sending ‘scheduled’ pushes for windows, androidjpush, blackberry osTypes has been increased from 5 min to 60 min to reduce database load (as there are no customers actively using these osTypes). Note that ‘immediate’ pushes can still be performed outside of the ‘scheduled’ 60 min frequency.

            • Identity

              • Fix browser logout issues in case of Azure AD SAML

              • Fix login and UI issues in case of OAuth provider

              • Fix for cors-config does not set console-url in the config to prevent console from getting blocked

              • Add MFA support for custom identity provider

            • Integration Server

              • Fix intermittent socket exceptions occurring on login to DBX when idle for some time

              • Fix issue introduced in in V8 SP4 FP3 HF6 in which service metadata API calls were failing

              • ⚠️ Any existing Clouds containing dedicated (i.e., single-tenant) Kony Integration Server runtime environments will not be affected during the maintenance window (i.e., will not be automatically upgraded). If you wish to upgrade, please open a support case, specifying your Cloud(s), desired version (V8 SP4 FP3 HF7), and desired maintenance window (day, time, and timezone) when we can apply the upgrade. Existing Clouds containing shared (i.e., multi-tenant) Kony Integration Server runtime environments (e.g., AppPlatform Gold & Platinum tiers (excluding Platinum Plus), free Fabric services) will be automatically upgraded during the maintenance window.


            Impact Level : minor

            No downtime is expected for the impacted Cloud services while this maintenance is being performed. The scheduled maintenance is designed to mitigate disruptions to service availability and performance for the impacted Cloud services. However, it is possible for the impacted Cloud services to be unavailable and/or performance degraded for a short period of time during the maintenance window. Note that no changes are being applied for other Cloud services outside of the list of impacted services above and no service availability or performance disruption is expected for other Cloud services.

            Engagement Services Hotfix

            Maintenance window: October 14, 2019 00:01 to 04:00
            The maintenance window start and end times are local to the region in which your Clouds are hosted. If you are unsure where your Clouds are hosted, you can hover over a Cloud Name in the Manage Clouds page of the Cloud Management Console and the region will be displayed.
            Impacted Cloud services:
            • Engagement

              • Address out of memory issues that could lead to service becoming significantly degraded or unavailable

              • Fix bulk push issue in which entry creation stops and yields an error message indicating that a different object with the same identifier value was already associated with the session

              • Introduce performance improvement for Fetch Push Sent Payload API

              • Fix issue related to subscription when content type is ‘application/x-www-form-urlencoded’


            Impact Level : minor

            No downtime is expected for the impacted Cloud services while this maintenance is being performed. The scheduled maintenance is designed to mitigate disruptions to service availability and performance for the impacted Cloud services. However, it is possible for the impacted Cloud services to be unavailable and/or performance degraded for a short period of time during the maintenance window. Note that no changes are being applied for other Cloud services outside of the list of impacted services above and no service availability or performance disruption is expected for other Cloud services.

            [2019-10-03 18:20 UTC] Please note that while we do not anticipate any impacts while the maintenance is being performed and have assigned an impact level of ‘minor’, this fix is ‘critical’ to the stability and overall health of Engagement services.

            [2019-10-06 18:07 UTC] The scheduled maintenance has been postponed to a new date (TBD). We will provide an update once we can announce the new date.

            [2019-10-09 12:21 UTC] The maintenance has been rescheduled for October 14, 2019 between 00:01 and 04:00 local to the regional timezone in which your Clouds are hosted.

            Identity Services Hotfix

            Maintenance window: October 7, 2019 00:01 to 04:00
            The maintenance window start and end times are local to the region in which your Clouds are hosted. If you are unsure where your Clouds are hosted, you can hover over a Cloud Name in the Manage Clouds page of the Cloud Management Console and the region will be displayed.
            Impacted Cloud services:
            • Identity

              • Fix login failure when using a custom identity provider and MFA is enabled and integrity is on

              • ⚠️ For customers who have dedicated Identity services, we will start to apply the Identity services upgrade during this maintenance window, but expect to complete for all customers’ dedicated Identity services over the next few weeks. In general, we do not expect impacts to your Identity services during the upgrade process. If we believe there may be any impacts during your Identity services upgrade, we will notify owners and admins in your Cloud account(s) via support case to communicate any impacts and coordinate on a suitable upgrade window


            Impact Level : minor

            No downtime is expected for the impacted Cloud services while this maintenance is being performed. The scheduled maintenance is designed to mitigate disruptions to service availability and performance for the impacted Cloud services. However, it is possible for the impacted Cloud services to be unavailable and/or performance degraded for a short period of time during the maintenance window. Note that no changes are being applied for other Cloud services outside of the list of impacted services above and no service availability or performance disruption is expected for other Cloud services.

            [2019-10-04 19:01 UTC] Please note that while we do not anticipate any impacts while the maintenance is being performed and have assigned an impact level of ‘minor’, this fix is ‘critical’ for supporting our upcoming DBX release (4.2.5).

            [2019-10-07 17:02 UTC] We finished applying updates for customers with dedicated Identity services earlier today.