On the 26th of September we were alerted to part of our infrastructure performing poorly which impacted services and therefore customer API traffic. We subsequently identified an issue with a monitoring service causing a loop and thereby increasing load, ultimately impacting the resources available for this and other services.

We were then able to determine that a configuration on the cronjob start-up parameters was the root cause. In order to mitigate the issue we added additional resources into the technical stack and isolated the cronjob service to a separate instance as a permanent step to ensure this issue would not reoccur. Customer impact was mitigated 55 minutes after first occurrence.

During our internal incident review, we have identified additional actions for performance, incremental resiliency and speed-to- recovery improvements.

We want to take the opportunity to offer our apologies for the disruption that this incident caused you and your customers.