Identity Services Rollback to v8 - Expired token gateway exceptions

• Identity services

  • A handful of customers using older SDKs were experiencing a high frequency of invalid / expired token gateway exceptions have been encountered in the latest version that was deployed earlier today. We are rolling back to prior version of Identity services to mitigate these issues until our development teams can investigate further.

  • After further review, the invalid / expired token gateway exceptions were due to a recent clock drift issue that affected specific hardware. Kony mitigated the issue and has since replaced all of the systems running on the affected hardware.

  • The rollback of Identity services was proactive and done out of an abundance of caution. The clock drift issue led to expired claims tokens being returned from Identity service API calls. The V8 SP1 version of Identity was not a contributor to any of the issues observed. We will schedule the redeployment of V8 SP1 in the next maintenance window.

  • Until we replaced affected hardware, some customers also experienced issues when attempting to access the Cloud Management Console.

No significant downtime is expected. The scheduled maintenance is designed for minimal disruption of service availability; however, it is possible for your service(s) to be unavailable for a short period of time during the maintenance window.

[2017-12-18 23:20 UTC] These exceptions appear to be occurring for customers who are on older SDKs.

[2017-12-19 15:27 UTC] After further review, the invalid / expired token gateway exceptions were due to a recent clock drift issue for specific hardware, which was acknowledged by our infrastructure provider. Our provider anticipates being able to deploy a fix in 48 hours. Once we understood that this was only affecting specific hardware, earlier today, we were able to reprovision onto new hardware that was not subject to the clock drift issue. Since the hardware was replaced at 04:00 UTC, the issue with system clocks affecting the validity of claims tokens has been resolved. Customers who had opened support tickets also confirmed that their applications were behaving properly after the hardware change.

[2017-12-19 19:25 UTC] Our review of existing systems has shown that the time drift patches have been applied to the affected hardware categories by the infrastructure provider.

[2017-12-19 19:35 UTC] Until we replaced the affected hardware, some customers also experienced issues when attempting to access the Cloud Management Console.