#51359 - Degradation to Bacs Simulator
Incident Report for Form3 Staging
Postmortem

On the 8th of August, Form3 observed a degradation on the Bacs Simulator.

Investigation identified that large volumes of test payments were being submitted continuously, causing unexpected load on the Bacs Simulator.

This resulted in a number of test submissions becoming stuck in a non-final state, which also affected receipt of inbound test transactions.

Root cause was identified, and recovery actions were taken at 10:41 BST. Once this was completed, a period of monitoring confirmed that there was no further impact to the Bacs Simulator and test transactions were being processed as expected.

Posted Aug 09, 2022 - 11:02 UTC

Resolved
The Bacs Simulator has recovered and normal service has resumed as of 10:41 BST

We will continue to monitor the service to ensure there is no ongoing impact and further details will be shared in the Post Mortem.
Posted Aug 08, 2022 - 09:56 UTC
Identified
We have identified the issue and are putting in a fix in place to ensure any queued payments are processed to final state.

Our Incident Response team continues to progress the recovery activity as a priority.

We will provide an update as soon as more information becomes available.
Posted Aug 08, 2022 - 09:44 UTC
Investigating
We have identified an issue impacting the Bacs simulator.

Majority of outbound payments are in queued for delivery and no inbound payments are being received.

Our engineers are currently investigating and further details will be posted as the investigation progresses
Posted Aug 08, 2022 - 09:16 UTC
This incident affected: Simulators (Bacs - Direct Participant).