Write-up
PIS/AIS service is down

On 12th April 2024 at around 06:30 GMT, Yapily suffered a critical incident impacting both AIS and PIS requests. Customers may have experienced intermittent 5xx responses from our API Gateway, until the incident was mitigated at 10:45 GMT.

Incident Summary:

  • Issue: Intermittent AIS and PIS request 5xxs

  • Causes: Outage to one of our content delivery networks (CDN)

  • Resolution: Rolled back the change that introduced the dependency on this CDN

Timeline:

  • Impact Began: April 12, 2024, 06:30 GMT

  • Internal Incident Raised: April 12, 2024, 09:46 GMT

  • Impact Mitigated: April 12, 2024, 10:45 GMT

Root Cause:

The incident was caused by an outage to one of the external content delivery network (CDN) that serves one of our monitoring libraries on our authorisation load balancer.

Mitigating Actions:

Once identified, we rolled back the change that introduced this dependency to restore service. Further, we removed this dependency all together to prevent an incident like this reoccurring.

We want to take the opportunity to offer our apologies for the disruption that this incident caused you and your customers.