The webhook arrived twice. The order must still ship once

The endpoint returned HTTP 200, but the customer’s balance never changed. The log says success, the customer is waiting, and the provider may see no reason to retry immediately. The failure sits between two events: accepting a message and completing the business action it describes. Start webhook design at that boundary rather than with a ten-line handler.

Give each question an authoritative source

A payment integration needs to answer several questions. The provider reports its payment operation. Your internal records show which actions the platform has performed. Settlement reports help establish the movement of funds. Assign an authoritative source to each question: use the provider's supported mechanism for payment status, your own records for fulfillment, and reconciliation for settlement. A valid payment notification should not become proof of all three.

Write a short contract for each provider adapter: stable identifiers, possible duplicates, current-state lookup, and allowed transitions. For example, Stripe documents duplicate deliveries and does not guarantee event ordering. Sorting by time alone does not establish that a state transition is valid. Provider-specific details matter more than a single elegant diagram applied to every integration.

Separate receiving an event from acting on it

A useful design starts by authenticating the notification according to its API contract. Store the accepted event in a durable inbox, then perform business processing separately. Send the required acknowledgement within the provider's deadline, after durable intake. If storage fails, do not claim the message has been accepted. Shipping, customer emails, and report generation should stay outside this short intake path.

Authentication also varies by provider. PayPal documents webhook signature verification; its self-verification procedure depends on the original request body. Check whether middleware changes that body before verification. Avoid copying another provider's signature recipe into a new adapter without checking the relevant specification.

Useful inbox fields include provider, account context, event identifier, operation identifier, receipt time, verification result, and processing state. Store complete payloads only when justified, with appropriate access and retention limits. A support agent usually needs a transaction reference and an explanation, not a fresh copy of every personal field in the notification.

Protect both delivery and business effects

An event-identifier check can stop repeat handling of the same delivery. Also protect the business action itself: one payment operation should produce only one authorized balance credit or fulfillment action. Different messages can describe the same business outcome. Design the action marker and internal balance update as a consistent unit so a crash between them cannot quietly permit another credit.

Choose duplicate keys from the actual contract. Adyen describes duplicates sharing eventCode and pspReference, even when other fields differ. Hashing the entire JSON document is therefore no substitute for understanding its format. An excessively broad duplicate key can also hide a legitimate later event, such as a separate partial refund.

Walk one payment through the system

In this illustrative scenario, order M-418 costs €72. Its confirmed-payment event arrives first. A worker locates the order, checks amount, currency, and account, updates the payment, and creates one shipping instruction. A late processing event remains in the audit history but does not reverse that decision. A second success notification creates no further shipment. A refund event follows its own identity and transition rules instead of being discarded because the order was already paid.

If events conflict, place the operation in a review state and retrieve current information using the provider's supported process. Where an appropriate online lookup is unavailable, use its reporting and escalation path. Silence from a webhook endpoint does not establish that a payment failed.

An error queue needs a safe way out

An error queue is useful only if an operator can return an item to processing safely. In the proposed workflow, the record distinguishes an unknown order, an amount mismatch, an unsupported event type and a technical processing failure. The first three need a decision; the last may be resolved by retrying. A single replay-everything button hides this distinction and can turn recovery into another incident.

Before replay, show which actions have already happened: credit, fulfilment, refund and customer notification. Retry the missing authorised step. Restarting the entire workflow after fixing a parser can duplicate an email or shipment even when event ingestion is idempotent. Test those side effects independently, including after a code upgrade. A message marked processed is not enough evidence that every intended downstream action completed.

Watch age as well as success rate

Imagine a queue with 900 fresh events and one item stuck since yesterday. Average processing time may look respectable while that customer still has no result. Track the oldest unresolved item by operation type and count records without an owner. Refunds, fulfilment and informational notifications can justify different delay tolerances.

To test recovery, follow a specific event from receipt to its financial effect. Record the handler version as well as the result. When data formats change, this will help identify which old events remain safe to replay through new code and which need compatible parsing. Retaining every raw payload forever is not a substitute for that design: use a justified set of fields and an approved retention period.

Pick one event left in yesterday’s error queue. If an operator can safely finish the missing action and explain why completed actions will not repeat, recovery exists in practice. If they must edit the order status by hand, the webhook design still depends on guesswork.