menu

Feed available - Subscribe to our feed to stay up to date on upcoming maintenance and incidents.

Kony Cloud status
Current status and incident report

Cloud SSL Certificate Updates

Maintenance window: September 24, 2022 00:01 to 04:00 UTC
Impacted Cloud services:
  • Cloud SSL certificates for Identity services (.auth.konycloud.com), Engagement services (.messaging.konycloud.com), App services (.konycloud.com), and Sync services (.sync.konycloud.com)

    • ⚠️ Note: If your application uses public key pinning, or your application has not pinned Kony certificates, no application updates will be necessary. Customers that have pinned the full SSL certificates will need to download the new certificates and rebuild the applications, including both the old and new certificates, and publish the updated binaries to the various app stores. Your applications should be published before Kony updates the certificates on the cloud servers or the applications will no longer be able to connect to Kony servers. New certificates can be found on the Kony Cloud Certificate Preview page. The new certificates can also be downloaded by executing the following commands:

      • Identity: openssl s_client -showcerts -connect konycertificatepreview.auth.konycloud.com:443

      • Engagement: openssl s_client -showcerts -connect konycertificatepreview.messaging.konycloud.com:7443

      • App: openssl s_client -showcerts -connect konycertificatepreview.konycloud.com:8443

      • Sync: openssl s_client -showcerts -connect konycertificatepreview.sync.konycloud.com:9443

    • Customers who have pinned the public key instead of the full certificate (which we strongly recommend and was made available in V8 SP4) will not be required to update their applications. The updated certificates will use the same public keys as the existing certificates.

      • If necessary, you can submit your applications for expedited approval (e.g., Apple has an expedited approval process for critical bugs, or in this case, pinned certificates).


        Impact Level : high

        Customer applications that have pinned SSL certificates will need to be updated as described above prior to this maintenance window. Customer applications that have not pinned SSL certificates will not be affected and will experience no service disruptions during this maintenance window.

        Please refer to our documentation for how to pin the public key of a certificate (which we strongly recommend and was made available in V8 SP4) or how to pin an SSL certificate in your apps (deprecated).

        Quantum V9 SP6 Release

        Maintenance window: August 01, 2022 00:01 to 04:00
        The maintenance window start and end times are local to the region in which your Clouds are hosted. If you are unsure where your Clouds are hosted, you can hover over a Cloud Name in the Manage Clouds page of the Cloud Management Console and the region will be displayed.
        Impacted Cloud services:
        • Fabric Integration Server

          • Enhanced Temenos Microservices connector for simplified Developer experience

          • Support for multiple mappings for Data Models in Object services

          • Improved performance for Reports

        • Appfactory

          • Ability to include / exclude deploying a web app while performing Appfactory Fabric builds

          • Ability to capture screenshots while recording the test cases and compare them with screenshots taken while the automation running

          • Appfactory is now running on latest Jenkins and third-party plugins.


        Impact Level : minor

        Minor downtime is possible for the impacted Cloud services while this maintenance is being performed. The scheduled maintenance is designed to mitigate disruptions to service availability and performance for the impacted Cloud services. However, it is possible for the impacted Cloud services to be unavailable and/or performance degraded for a short period of time during the maintenance window. Note that no changes are being applied for other Cloud services outside of the list of impacted services above and no service availability or performance disruption is expected for other Cloud services.

        Reporting Services delayed processing of analytics

        Incident window: April 05, 2022 19:00 UTC to May 29, 2022 23:15
        Customers that require an estimate of dates where data may have been lost should open a support ticket and request that the Cloud Operations team provide an estimate. The teams will be able to provide only an estimate for the data range and potential numnber of messages lost.
        Impacted Cloud services:
        • Cloud Analytics


          Impact Level : medium

          The processing of analytics data was impacted by a customer application error in combination with impacting traffic from several DDoS attacks on specific customers in April. [There were no breaches from any DDoS activity]. We did not notice the growing analytics backlog until alerts raised our awareness. However, the alerts were configured in 2016 as a percentage of maximum assumed capacity, and that capacity is several fold over normal requirememts. Once alerted, we determined the quantity of analytics produced by the DDoS requests, growth on Kony Cloud, and increases in customer device subscriptions since 2016 would present an issue where we could not process all of the backlog fast enough to prevent data loss. We have identified a bottleneck in our analytics processing and have tasked engineering to provide an architectural solution. The data loss will vary by customer and reports may show reduced/missing data between April 5th and May 5th, with the most likly loss of data occurring before April 24th. Analytics from the backlog are still being processed and we expect recovery efforts to last until May 23rd. We would note that this is the first loss of analytics data of any kind since the Kony Cloud opened and internally we are devistated at this occurance.

          [2022-04-17 13:20 UTC] We have stopped the data loss and are working on processing the backlog of analytics data. We estimate 10B data records are queued for processing.

          [2022-04-24 18:14 UTC] We have increased capacity, but there is an archectural limit in the analytics design that gates the maximum processing throughput. We are currently processing 15M records per hour. We are working to identify and provision additional processing capacity in strategic areas.

          [2022-05-03 16:20 UTC] We have added additional capacity and are now processing 18M records per hour. We are continuing to process the backlog of analytics data (oldest data that has not yet been processed being approximately 10 days old). At the current file count and processing rate, we estimate that the backlog (along with processing of current inbound analytics data) will not complete until sometime after May 22nd. We are continuing to monitor and will continue to provide updates until the backlog has been fully processed.

          [2022-05-12 13:40 UTC] We use an at-least-once delivery queue for analytics records, but the order of delivery is not sequential. With multiple readers processing the data we can, and do, get duplicate records. Because of the extreme number of readers in place to deal with the backlog of records the numnber of duplicates being processed will also increase. Due to the necessary pause for scrubbing the data warehouse of duplicates customers may see increased counts in analytics reports for data after Mar 25th. We will not be able to remove duplicates from the database until after we have cleared the backlog, now estimated at May 25th.

          [2022-05-17 13:35 UTC] We have collected all outstanding messages from all accounts into our main processing queue. Collection of data from customer accounts is now running with the normal backlog, which is about 10min latency in the customer accounts. We have ingested approximalty 25% of all outstanding collected data. At the current ingestion rate, baring any new issues, we are still projecting to have all data ingested on or about May 25th.

          [2022-05-21 14:15 UTC] We have now processed more than half of the analytics backlog. Older messages are not guaranteed to be processed first due to the way that the underlying message queuing system works. This is not normally an issue because we can read and process all of the messages in one pass. However, with such a large backlog, we are seeing the oldest message measurement grow. In order to prevent the expiration of collected messages we are pausing message collection as of May 21 00:00 UTC. We will complete processing the currently collected messages. There will be a slight impact to the completion estimate as a tradeoff for processing all of the current message backlog. We believe we can process the collected analytics in 72hrs. We will update the full completion estimate once the backlog is processed.

          [2022-05-23 12:56 UTC] We have injested all data into the warehouse up to Saturday May 21 12:00 UTC. We are currently processing the remaining data and expect to have all data collected and injested today. Once this current activity is complete we will pause collection and processing in order to clean up duplicates from the database. We expect that the duplicate processing to take approximately 12 hours. Currently we expect to have all of the data processed and systems back on line by EOD May 25th.

          [2022-05-24 15:41 UTC] We have enabled automatic collection of all customer analytics data and have a current backlog of 44M messages in our queue. We expect these will clear in a few hours. Once the systems are fully caught up we will decide on the timing for maintenance window that is required to clean up duplicate records. During the maintenance window we will pause collection again, but customers will be able to access the reports services in the consoles.

          [2022-05-26 17:45 UTC] We have ingested all analytics from the backlog and the system is running normally for the last 36 hours. Due to the quantity of duplicate records in the dataset we project the cleanup to take approximately 12 to 18 hours. We are scheduling the cleanup for Saturday, May 28th. Reporing services will be available with momentary outages as we resize the data warehouse systems to handle the scale of this cleanup. Any outage is expected to be less than 5 minutes while the system failover.

          [2022-05-28 13:00 UTC] We are pausing processing of analytics while the data warehouse contents are processed for duplicate records. Reports against the data will be available during the process, but there will be a database restarts during the procedure. Any outage would be only for a few moments. For any error, please wait about file minutes and rerun your report.

          [2022-05-29 05:00 UTC] We have completed the maintenance to remove duplicate records. We will begin processing data from May 28 13:00 UTC and expect to be caught up by early Sunday morning US EST.

          [2022-05-29 23:15 UTC] Resolved. The maintenance has completed and the analytics systems are running normally.

          Quantum V9 SP5 Release

          Maintenance window: April 18, 2022 00:01 to 04:00
          The maintenance window start and end times are local to the region in which your Clouds are hosted. If you are unsure where your Clouds are hosted, you can hover over a Cloud Name in the Manage Clouds page of the Cloud Management Console and the region will be displayed.
          Impacted Cloud services:
          • Fabric Integration Server

            • New Temenos Microservices connector for simplified Developer experience

            • Enhanced functionality for Microapps

            • Support for authentication into Fabric Console using AD SAML

            • Improved Service Monitoring

          • AppFactory

            • Cucumber Support for Fabric Testing in AppFactory

            • Fabric Build Process from AppFactory Command Line

            • IOS & Android Cross App Testing support

            • Target Test Data from environments


          Impact Level : minor

          Minor downtime is possible for the impacted Cloud services while this maintenance is being performed. The scheduled maintenance is designed to mitigate disruptions to service availability and performance for the impacted Cloud services. However, it is possible for the impacted Cloud services to be unavailable and/or performance degraded for a short period of time during the maintenance window. Note that no changes are being applied for other Cloud services outside of the list of impacted services above and no service availability or performance disruption is expected for other Cloud services.

          Maintenance for Cloud Management Console, Identity, and Workspace

          Maintenance window: March 7, 2022 00:01 to 04:00
          The maintenance window start and end times are local to the region in which your Clouds are hosted. If you are unsure where your Clouds are hosted, you can hover over a Cloud Name in the Manage Clouds page of the Cloud Management Console and the region will be displayed.
          Impacted Cloud services:
          • Cloud Management Console

            • Developer Portal Members will now be able to see the list of other registered users

            • Enable additional Server Side Validations for Email Hyperlink Injection attacks

          • Workspace

            • For some customers, snapshot creation of extended services failed during Publish. This has been fixed.


          Impact Level : minor

          Minor downtime is possible for the impacted Cloud services while this maintenance is being performed. The scheduled maintenance is designed to mitigate disruptions to service availability and performance for the impacted Cloud services. However, it is possible for the impacted Cloud services to be unavailable and/or performance degraded for a short period of time during the maintenance window. Note that no changes are being applied for other Cloud services outside of the list of impacted services above and no service availability or performance disruption is expected for other Cloud services.

          Pause to outbound Mail Services

          Incident window: March 4, 2022 22:15 UTC to March 9, 2022 10:54 UTC
          Impacted Cloud services:
          • Cloud Management Console

            • Excessive registrations (forming a DDoS attack) from China has resulted in our mail services being over the sending limits. Combined with increase bounce rates, we have had to pause all outbound mail services while we implement appropriate filters. We are working with AWS to resolve the issue, and expect to have mail services resumed by end of day in US EST.

            • Emails generated by Fabric monitoring will also be affected. Teams should manually monitor any alerts configured in the Quantum Fabric consoles until we resume email deliveries.


          Impact Level : medium

          Mail services have been paused as of March 4th; expected resolution is end of day (March 9th).

          [2022-03-09 18:54 UTC] Resolved. Outbound mails services have been restored.

          Maintenance for Cloud Management Console, Identity, and Workspace

          Maintenance window: February 21, 2022 00:01 to 04:00
          The maintenance window start and end times are local to the region in which your Clouds are hosted. If you are unsure where your Clouds are hosted, you can hover over a Cloud Name in the Manage Clouds page of the Cloud Management Console and the region will be displayed.
          Impacted Cloud services:
          • Cloud Management Console

            • Added Content Security Policy in response header to protect against clickjacking attacks

            • Fixed Unable to load WSDL file error for SOAP Services

          • Fabric Identity

            • Improved security against clickjacking attacks

            • jQuery upgrade from 3.5.0 to 3.6.0

          • Workspace

            • Fixed SAML provider configuration validation issue

            • Fixed issue where headers could not be added or saved for locked services

            • Fixed issue where custom operations could not be deleted for locked operations


          Impact Level : minor

          Minor downtime is possible for the impacted Cloud services while this maintenance is being performed. The scheduled maintenance is designed to mitigate disruptions to service availability and performance for the impacted Cloud services. However, it is possible for the impacted Cloud services to be unavailable and/or performance degraded for a short period of time during the maintenance window. Note that no changes are being applied for other Cloud services outside of the list of impacted services above and no service availability or performance disruption is expected for other Cloud services.

          Quantum V9 SP4 Release

          Maintenance window: January 17, 2022 00:01 to 04:00
          The maintenance window start and end times are local to the region in which your Clouds are hosted. If you are unsure where your Clouds are hosted, you can hover over a Cloud Name in the Manage Clouds page of the Cloud Management Console and the region will be displayed.
          Impacted Cloud services:
          • Cloud Management Console

            • Enhanced extensibility for Integration and Object services

          • Fabric Integration Server

            • Advanced Workflow capabilities with Parallel paths

            • Added Looping support for workflows

            • Improved security for caching using Redis

          • Fabric Identity

            • Improved support for OAuth connections using private key JWT

          • AppFactory

            • Support for Istanbul testing code coverage for Quantum built apps using the Quantum Testing Framework. Ensure every line of your code is tested, every time you test.

            • Group failed tests together and re-run those test as part of AppFactory’s testing process with the Quantum Testing Framework.

            • Web based cross app testing backported to 9.2 to allow testers to instrument test scenarios that take the test case outside the Quantum Testing Framework, like OAuth.

            • Micro app test case support for combining base micro app test cases to a larger test case within a composite app.

            • Command line support for create new AppFactory projects and building Visualizer based apps within AppFactory from the command line.

            • Apple API key support for signing iOS based apps.

            • Regenerate Fabric object models for visualizer based apps every time you build with AppFactory.


          Impact Level : minor

          Minor downtime is possible for the impacted Cloud services while this maintenance is being performed. The scheduled maintenance is designed to mitigate disruptions to service availability and performance for the impacted Cloud services. However, it is possible for the impacted Cloud services to be unavailable and/or performance degraded for a short period of time during the maintenance window. Note that no changes are being applied for other Cloud services outside of the list of impacted services above and no service availability or performance disruption is expected for other Cloud services.

          AWS Outage affecting US Virgina customers in one datacenter

          Incident window: December 22, 2021 12:33 UTC to December 23, 2021 00:22 UTC
          The AWS Service Status page will have the lastest information about the known issues and actions that AWS is taking.
          Impacted Cloud services:
          • AWS (Virgina region only) instances and network issues

            • AWS power outage for compute, network and disk services in a single datacenter. As physical datacenters are allocated randomly, only a subset of Kony customers would see any impact.

            • Kony Cloud runtime services are generally not affected, and survived the initial outage by design. However, power restoration has not been orderly leading to recovery issues.

            • Other regions are not affected


          Impact Level : medium

          AWS Regional outage in Virgina cause by power failure in one of the AWS datacenters.

          [2021-12-22 15:45 UTC] The underlying Virginia region is returning API errors for many services. We are working with AWS to understand the issues, impacts, and recovery timeline.

          [2021-12-22 13:56 UTC] The underlying Virginia region is returning to normal. There were failovers for the Kony Cloud runtime services, and the system continued to run on a single datacenter. However, power-on activities, while mostly successful, have see sporadic incidents. We will launch a full recovery inventory on the underlying AWS issues are resolved.

          [2021-12-22 23:15 UTC] AWS continues to recover compute and disk resources. Kony Cloud systems are on line and running normally, but customers should report any issues that are noticed.

          [2021-12-23 00:22 UTC] Resolved. AWS has restored power and recovered compute and disks. Databases have automatically failed over, and any local disk or compute failure has automatically been replaced. Please report any issues with Fabric services.