PSP outage: failover, unresolved attempts and a safe return

The primary PSP responds again, so the operator restores traffic. A minute later, automation switches back to the backup. While the team works out who controls routing, earlier payment attempts still have unknown outcomes. This illustrative exercise tests three parts of outage response: new attempts, unresolved operations and a controlled return to the primary route.

Walk through a controlled switch

A fictional platform has routes A and B approved for the same payment use case. At 14:05, timeouts on A rise. The team restricts new submissions to A and checks B's eligibility. Order R-902 was already submitted through A without a response, so it remains under reconciliation. R-903 has not been submitted anywhere; after normal checks, it can use B if its parameters are supported.

Copying R-902's idempotency string into B does not establish shared duplicate protection. In the proposed design, one internal attempt journal coordinates the business operation. Permission for another financial action depends on establishing the previous outcome or following a provider-supported safe process. Support can explain the pending state without encouraging uncontrolled customer retries.

Classify the problem before choosing the route

Technical unavailability, an invalid request, additional authentication, and an operation restriction call for different responses. An HTTP code alone does not always establish the business outcome. For example, Stripe advises treating a 500 outcome as indeterminate and resolving the original operation before using a new key. An error response is not evidence that no financial effect occurred.

Give operators three separate groups. New, unsubmitted operations can be assessed for an approved alternative. Submitted operations with unknown outcomes belong in reconciliation. Confirmed refusals need handling according to their reason and method rules. Showing these groups separately helps prevent a single emergency button from applying the same action to fundamentally different problems.

Prepare the alternative before an outage

Maintain a route record covering legal entity, connected account, permitted transaction types, payment methods, currencies, geography, amount limits, refund handling, and support contacts. Establish required customer authorization and checks. If a backup is not activated or its terms do not cover the particular operation, treat it as unavailable.

Pay special attention to differences between a merchant purchase, a bank transfer, and a payout to a P2P-platform participant. Similar interfaces do not make these actions interchangeable. Do not change the transaction purpose, beneficiary, or merchant details to make a route accept it. When no appropriate authorized route exists, a clear pause is preferable to improvisation.

Adyen's refusal reference distinguishes issuer unavailability, authentication problems, suspected fraud, and prohibited transactions. This illustrates why “send every refusal elsewhere” is too crude. Failover must preserve AML/KYC, sanctions, fraud controls, and transaction restrictions. Escalate an ambiguous restriction to the responsible specialist rather than routing around it.

Control the recovery load

The backup still has capacity and contractual limits. Define a maximum share of new traffic, an acceptable queue, and stop conditions. Choose values from the applicable limits and agreed capacity checks. Move a bounded volume of eligible new operations first, inspect their results, and only then increase traffic.

AWS describes how retries amplify overload and how increasing delays with randomized timing, called jitter, can reduce synchronized bursts. For payments, this supports avoiding a simultaneous replay of the entire queue. Give one layer responsibility for retries and inspect whether SDKs add further attempts automatically.

The backup’s real cost appears after the switch

Assess a backup across the entire lifecycle. Who handles a refund against an old payment? How does support find the route when the customer only knows an order number? Which reconciliation picks up a delayed transaction? If those answers depend on one engineer’s memory, the second PSP adds an integration but does not yet provide manageable resilience.

Store the route, account and routing-policy version for every attempt. One order can have several attempts, but their financial effects must remain distinguishable. Reports should separate the initial route from the route of the ultimately confirmed payment. Otherwise the team may compare different customer populations after an outage and attribute the apparent improvement to provider quality.

Returning to the primary route is another release

Do not restore all traffic because one probe succeeded. Use a bounded check on eligible new operations, inspect latency and confirmations, and expand according to pre-agreed conditions. The proportions and observation period depend on the workload; a universal five-error-free-minutes rule is not meaningful for every service.

Agree what happens to automatic failover while people manage recovery. Two mechanisms changing routes simultaneously can create oscillation and make the experiment impossible to interpret. The decision log should identify which mechanism currently controls routing. A useful exercise demonstrates the ability to move to the backup, return safely and resolve the remaining list of uncertain payments.

Record three numbers after the exercise: new operations completed through the backup, old operations still unresolved, and confirmed duplicate financial effects. Strong conversion does not remove the need to own the second group or investigate a non-zero third number.