[Europe] Signature management apps partially offline for a subset of users
Incident Report for CodeTwo
Postmortem

Here’s a root analysis provided by Microsoft:

The Microsoft Azure Team has investigated the issue reported and we have identified it was due to a dependency that the Azure Web App service takes on Azure Storage.
 
In this specific scenario, a storage cluster experienced a larger than normal network traffic volume. Engineers initially triaged this to be due to a possible increased traffic load. Thus, the first mitigation performed was to modify a configuration setting to allow for an additional volume increase. However, this change alleviated some of the pressure but did not solve the underlying problem. Additional troubleshooting determined that the majority of the remaining impact was due to a lock contention in the storage volume, primarily due to the cluster being overloaded from the initial traffic increase.
 
After mitigating the issue, engineers were able to solve the issue of the increased traffic volume by putting in place mechanisms that would distribute the volume for the specific scenario that resulted in this incident. Additionally, future improvements are planned to address the interaction between storage cluster dependencies and network volume fluctuations.
 
We apologize for any inconvenience.
Regards,
The Microsoft Azure Team

Posted Jan 14, 2022 - 07:08 UTC

Resolved
This incident is now resolved. The root cause will be provided later. The preliminary root cause provided by Microsoft is as follows:

Due to a sudden change in demand a single storage scale unit in the region reached an operational threshold, and automated throttling became active to ensure the health of existing resources on this scale unit. This throttling would have caused some requests to experience latency, or fail, and this would have had knock-on impact for other services with dependencies on this cluster.

All signatures were imprinted normally during the incident.
Posted Dec 17, 2021 - 12:03 UTC
Investigating
We are currently investigating a problem of degraded availability of signature management apps. Some users might be unable to edit or create signature rules. All other services are operational. Signatures are imprinted as normal. We're waiting for Microsoft to provide more info about the problem.
Posted Dec 17, 2021 - 10:09 UTC
This incident affected: Germany West Central (Signature management (app.codetwo.com)), UK South (Signature management (app.codetwo.com)), West Europe (Signature management (app.codetwo.com)), and North Europe (Signature management (app.codetwo.com)).