Management Services - inability to release iOS and Android apps using EMM

• Management services

We are investigating.

[2019-04-09 21:57 UTC] Resolved. We have identified and addressed the underlying issue (rolling out a fix to all regions by April 9 21:24 UTC) and will be continuing to monitor the service closely. If you were experiencing issues releasing iOS apps using EMM since the onset of this incident, you can retry at this time. If you encounter any further issues, please open a support case.

[2019-04-10 13:23 UTC] After addressing the iOS issue, the same issue was discovered for Android releases. We addressed the issue for Android apps in all regions by April 10 01:56 UTC and are continuing to monitor the service closely. Again, if you encounter any further issues, please open a support case.

[2019-04-10 23:41 UTC] We have reviewed this incident and determined the underlying cause to be due to mistakes in the execution of standard operating procedures, which allowed a packaging issue to be introduced in the most recent Management services hotfix deployment on April 8th. The build server had its operating system recently upgraded, which accidentally removed a critical dependency from the packaged build that would be used to convert the files to the correct format at the time of deployment. Initial testing was performed using the on-premises installer that would be analogous to the build intended to be deployed to Kony Cloud; however, the on-premises installer would differ slightly in that it performed the necessary file conversion as part of the installation process and thereby had circumvented and masked the issue. Initial testing yielded successful results for the new build. When the new build was later deployed to Kony Cloud testing environments for next steps of verification, the team detected the files issue and traced it back to the missing dependency, but attributed it to incorrect machine permissions in the test environments and determined that the the missing dependency was low risk and would be able to be manually introduced and executed instead of rejecting the build entirely and requiring it to be rebuilt and redeployed. The workaround that was executed in the testing environment to install the missing dependency and convert the files was not repeated at the time of the production deployment on April 8th as it was believed to be a permissions-related issue that was isolated to the testing environments. Following the April 8th deployment, when we noticed an increase in iOS build errors and were able to trace them back to the same issue encountered in the test environments, we were able to address the issue in production on April 9th. We subsequently noticed that the same issue also extended to Android builds, which we were able to address in production on April 10th. As a result of this incident and following our analysis, we have identified several corrective actions that we will be working toward to mitigate the occurrence of similar incidents in the future. We apologize for the impact this incident has caused to our customers and partners and we are committed to doing everything we can to learn from this event and improve.

⚠️ It is worth noting that Management services MDM, MAM, and MCM will be reaching end of life on September 30, 2019. Moving forward, Kony will be focusing on Enterprise App Store (EAS). If you are currently using MDM, MAM, or MCM features, please review our end of life announcement to understand how this will impact your services and support.