Workspace services - Oregon is unavailable

• Workspace (Oregon region only)

  • Other regions are not affected

Workspace services in Oregon have become unavailable starting at 17:21 UTC. The availability and performance of Workspace services in other regions is not affected. We are investigating.

infrastructure failure of a DB. The database has been recovered and the systems did not auto-restart as expected. We have opened internal development tickets on this scenario and will have the team correct this as we expect the workspace systems to survive this type of issue without intervention.

[2019-11-04 18:41 UTC] Resolved. The underlying Oregon Workspace database master node had failed and automatically failed over to the replica node, which had been promoted as the new master (and a new replica created after). However, while the database was able to recover with our automation, the Oregon Workspace application did not automatically recover as would be expected and was still yielding errors until we intervened by restarting. Once restarted at 18:26 UTC, the Oregon Workspace services have been available and performing within expected levels. We will be continuing to monitor. We will also be coordinating with our development team to identify why the Workspace application was unable to automatically recover and to work toward a solution to mitigate these types of incidents requiring our attention beyond our automated incident response controls in the future.

Cloud Management console and Identity services - severe performance degradation

• Cloud management console

• Identity

We have detected significant performance degradation for the impacted Cloud services, resulting in increased latency and error rates for the responses. We are working to restore service performance and availability to normal operating levels.

[2019-10-28 18:15 UTC] Resolved. We have restored performance and availability for the Cloud management console (https://manage.kony.com) and Identity services as of 17:56 UTC.

[2019-10-28 21:24 UTC] We would like to share some additional information concerning the root cause of the incident today and the additional measures we are taking to reduce the likelihood of recurrence as well as to improve our ability to more quickly respond to and address similar events.

In preparation for database certificate rotations, a new certificate bundle was loaded into our staging area. The automation properly replicated that bundle, but it was not expected that the systems would activate the bundle until sometime later.

Unfortunately, the Identity systems reloaded the certificate bundle prematurely. This caused connection errors to the databases, which very quickly locked all of the existing Identity servers out because the errors exceeded the connection threshold. This threshold is in place to prevent attacks on the database from the same IP trying to connect and guess passwords, etc.

We corrected the issue with the assets and the automation was able to propagate the updates to the systems. However, having been locked out, the updates had no effect as the existing servers were unable to reach the database because of the now blacklisted IP. Similarly, we were unable to connect and manually reset the database from any of the authorized instances due to the instances all now having a blacklisted IP.

Once it was determined that automation had pulled the new bundle to all of the instances (and there would be no quick way to connect to the database from any existing servers), the fastest solution was to replaced the entire fleet of Identity servers in such a way that all of the new servers were ensured not to get one of the now blacklisted IPs.

The new servers were built with the proper assets and services were restored as these systems came on-online.

We have identified the following improvement actions that we will be pursuing in an effort to mitigate the recurrence of this type of issue altogether, but also to improve our incident response procedures to more rapidly resolve similar future issues:

1. Review the ‘attack’ threshold to ensure it is not overly aggressive.
2. Investigate why the Identity services activated the new asset without the expected coordinated restart.
3. Investigate the creation of additional highly restricted management instances that may provide access in cases like this to cut the time it takes to gain access for any manual intervention.

V8 SP4 Releases

• Cloud Management Console

  • Add MFA support for custom identity provider

• Engagement

  • Fix bulk push issue resulting in BatchUpdateException: Failed to save messageentry_status duplicate entry ‘' for key 'PRIMARY'

  • The frequency of the procedure specifically used for sending ‘scheduled’ pushes for windows, androidjpush, blackberry osTypes has been increased from 5 min to 60 min to reduce database load (as there are no customers actively using these osTypes). Note that ‘immediate’ pushes can still be performed outside of the ‘scheduled’ 60 min frequency.

• Identity

  • Fix browser logout issues in case of Azure AD SAML

  • Fix login and UI issues in case of OAuth provider

  • Fix for cors-config does not set console-url in the config to prevent console from getting blocked

  • Add MFA support for custom identity provider

• Integration Server

  • Fix intermittent socket exceptions occurring on login to DBX when idle for some time

  • Fix issue introduced in in V8 SP4 FP3 HF6 in which service metadata API calls were failing

  • ⚠️ Any existing Clouds containing dedicated (i.e., single-tenant) Kony Integration Server runtime environments will not be affected during the maintenance window (i.e., will not be automatically upgraded). If you wish to upgrade, please open a support case, specifying your Cloud(s), desired version (V8 SP4 FP3 HF7), and desired maintenance window (day, time, and timezone) when we can apply the upgrade. Existing Clouds containing shared (i.e., multi-tenant) Kony Integration Server runtime environments (e.g., AppPlatform Gold & Platinum tiers (excluding Platinum Plus), free Fabric services) will be automatically upgraded during the maintenance window.

No downtime is expected for the impacted Cloud services while this maintenance is being performed. The scheduled maintenance is designed to mitigate disruptions to service availability and performance for the impacted Cloud services. However, it is possible for the impacted Cloud services to be unavailable and/or performance degraded for a short period of time during the maintenance window. Note that no changes are being applied for other Cloud services outside of the list of impacted services above and no service availability or performance disruption is expected for other Cloud services.

Engagement Services Hotfix

• Engagement

  • Address out of memory issues that could lead to service becoming significantly degraded or unavailable

  • Fix bulk push issue in which entry creation stops and yields an error message indicating that a different object with the same identifier value was already associated with the session

  • Introduce performance improvement for Fetch Push Sent Payload API

  • Fix issue related to subscription when content type is ‘application/x-www-form-urlencoded’

No downtime is expected for the impacted Cloud services while this maintenance is being performed. The scheduled maintenance is designed to mitigate disruptions to service availability and performance for the impacted Cloud services. However, it is possible for the impacted Cloud services to be unavailable and/or performance degraded for a short period of time during the maintenance window. Note that no changes are being applied for other Cloud services outside of the list of impacted services above and no service availability or performance disruption is expected for other Cloud services.

[2019-10-03 18:20 UTC] Please note that while we do not anticipate any impacts while the maintenance is being performed and have assigned an impact level of ‘minor’, this fix is ‘critical’ to the stability and overall health of Engagement services.

[2019-10-06 18:07 UTC] The scheduled maintenance has been postponed to a new date (TBD). We will provide an update once we can announce the new date.

[2019-10-09 12:21 UTC] The maintenance has been rescheduled for October 14, 2019 between 00:01 and 04:00 local to the regional timezone in which your Clouds are hosted.

Identity Services Hotfix

• Identity

  • Fix login failure when using a custom identity provider and MFA is enabled and integrity is on

  • ⚠️ For customers who have dedicated Identity services, we will start to apply the Identity services upgrade during this maintenance window, but expect to complete for all customers’ dedicated Identity services over the next few weeks. In general, we do not expect impacts to your Identity services during the upgrade process. If we believe there may be any impacts during your Identity services upgrade, we will notify owners and admins in your Cloud account(s) via support case to communicate any impacts and coordinate on a suitable upgrade window

No downtime is expected for the impacted Cloud services while this maintenance is being performed. The scheduled maintenance is designed to mitigate disruptions to service availability and performance for the impacted Cloud services. However, it is possible for the impacted Cloud services to be unavailable and/or performance degraded for a short period of time during the maintenance window. Note that no changes are being applied for other Cloud services outside of the list of impacted services above and no service availability or performance disruption is expected for other Cloud services.

[2019-10-04 19:01 UTC] Please note that while we do not anticipate any impacts while the maintenance is being performed and have assigned an impact level of ‘minor’, this fix is ‘critical’ for supporting our upcoming DBX release (4.2.5).

[2019-10-07 17:02 UTC] We finished applying updates for customers with dedicated Identity services earlier today.

Cloud Management Console and Workspace Services Releases

• Cloud Management Console

  • Fix issue in which the template parameters in a SOAP service are not read while generating the data model in the Object services created on top of SOAP Integration services. Visual mapper diagram was not loaded as a result of this issue.

• Workspace

  • Fix issue in which the template parameters in a SOAP service are not read while generating the data model in the Object services created on top of SOAP Integration services. Visual mapper diagram was not loaded as a result of this issue.

  • Support for host name alias for reporting services

No downtime is expected for the impacted Cloud services while this maintenance is being performed. The scheduled maintenance is designed to mitigate disruptions to service availability and performance for the impacted Cloud services. However, it is possible for the impacted Cloud services to be unavailable and/or performance degraded for a short period of time during the maintenance window. Note that no changes are being applied for other Cloud services outside of the list of impacted services above and no service availability or performance disruption is expected for other Cloud services.

Visualization services - ability to access and publish prototypes is unavailable

• Visualization

  • The ability to access and publish prototypes in Visualizer clients and prototype viewer is unavailable (appears to be limited to users who are using a proxy or behind a firewall when connecting to Kony Cloud Visualization services)

[2019-09-26 15:23 UTC] We are coordinating with one of our vendors to resolve.

[2019-09-26 20:08 UTC] Our initial investigation suggests that the issue appears to be limited to users who are using a proxy or behind a firewall when connecting to Kony Cloud Visualization services. We are continuing to investigate.

[2019-09-26 21:04 UTC] Resolved. As we continue our investigation, we believe that this issue may be restricted to a single customer and due to how they have configured Visualizer clients to connect to the backend Visualization services in Kony Cloud. We do not suspect that the impacts extend to any other customer, but are continuing to review to rule that out. To note, the backend Visualization services in Kony Cloud have been 100% available since the onset of this incident, but the ability for the Visualizer client to connect to the backend services in Kony Cloud to access or publish prototypes was what was failing.

Engagement Services - Performance improvements for Oregon

• Engagement - Oregon region

  • Add support for HTTP/2 protocol when sending iOS push messages, resulting in much higher push throughput, immediate push feedback, and ability to immediately deactivate bad subscription tokens.

  • Add support for token based authentication for Apple Push Notification Service, with backward compatibility for Cert-based authentication.

Only Engagement services in the Oregon region will be updated. Engagement services in other regions will not be affected.

No downtime is expected for the impacted Cloud services while this maintenance is being performed. The scheduled maintenance is designed to mitigate disruptions to service availability and performance for the impacted Cloud services. However, it is possible for the impacted Cloud services to be unavailable and/or performance degraded for a short period of time during the maintenance window. Note that no changes are being applied for other Cloud services outside of the list of impacted services above and no service availability or performance disruption is expected for other Cloud services.

[2019-08-04 22:00 UTC] The scheduled maintenance has been postponed to a new date (TBD). We will provide an update once we can announce the new date.

[2019-08-08 15:29 UTC] The scheduled maintenance has been rescheduled for August 12 between 00:01 and 04:00 (local to the region in which your Clouds are hosted).

[2019-08-12 00:01 UTC] The rescheduled maintenance has been postponed again until further notice. We will provide another update once we can announce the new date.

[2019-09-19 18:19 UTC] The scheduled maintenance has been rescheduled for September 23 between 00:01 and 04:00 (local to the region in which your Clouds are hosted).

Cloud Management Console and Identity Services Releases

• Cloud Management Console

  • Fix issue in which Open API operation response output parameters are not generated in a proper format

  • Enhancements to OAuth provider identity type for Engage Lite implementation

• Identity

  • Fix Custom Identity test login not working when a custom header is sent for backend

  • Fix custom Identity login failing in webapp case when passing integration header X-Kony-API-Version and the app is hosted in a domain other than Kony Identity

  • ⚠️ For customers who have dedicated Identity services, we will start to apply the Identity services upgrade during this maintenance window, but expect to complete for all customers’ dedicated Identity services over the next few weeks. In general, we do not expect impacts to your Identity services during the upgrade process. If we believe there may be any impacts during your Identity services upgrade, we will notify owners and admins in your Cloud account(s) via support case to communicate any impacts and coordinate on a suitable upgrade window

No downtime is expected for the impacted Cloud services while this maintenance is being performed. The scheduled maintenance is designed to mitigate disruptions to service availability and performance for the impacted Cloud services. However, it is possible for the impacted Cloud services to be unavailable and/or performance degraded for a short period of time during the maintenance window. Note that no changes are being applied for other Cloud services outside of the list of impacted services above and no service availability or performance disruption is expected for other Cloud services.