Conversion fell while both segments improved: unpacking the payment funnel

Every segment can improve while overall conversion falls. Additional retries can lower attempt success while more orders complete. Before changing routes or checkout, inspect the unit being measured, the customer mix and the time allowed for payments to finish.

When the aggregate blames an innocent integration

Consider a separate hypothetical example. In period one, segment A contributes 900 orders at 90% conversion and B contributes 100 at 60%. There are 810 + 60 = 870 successful orders out of 1,000, giving 87% overall conversion.

In period two, A improves to 92% but contributes only 200 orders. B improves to 65% and contributes 800. Successes now total 184 + 520 = 704, or 70.4% overall. Both segments improved while the aggregate fell by 16.6 percentage points because the traffic mix changed.

To investigate, compare the periods using the same segment weights. At the old 90% and 10% shares, the new rates would produce 89.3%: 0.9 × 92% + 0.1 × 65%. This is a standardised analytical calculation, not the second period’s observed conversion. Label and show both. Actual performance matters to the business; constant composition helps assess changes in the process itself.

One example produces four useful numbers

In a hypothetical store, 1,000 unique orders reach the payment screen. Customers submit at least one payment attempt for 800 orders. Retries bring the total to 940 attempts. Of these, 680 obtain successful authorization and belong to 680 distinct orders. By the end of the observation window, 660 orders have a successful capture.

• Progression to a payment attempt: 800 / 1,000 = 80%.
• Order-level authorization success among attempted orders: 680 / 800 = 85%.
• Attempt-level authorization success: 680 / 940 ≈ 72.3%.
• Captured-order conversion from payment-screen entry: 660 / 1,000 = 66%.

The first measure helps investigate the interface and available methods. The second describes order outcomes after permitted retries. The third describes attempts, and the fourth represents the selected commercial outcome. For a method without separate capture, or for a bank transfer, define the corresponding confirmed outcome rather than forcing card terminology onto the flow.

Keep orders, attempts, and events separate

Your data model needs a stable order identifier, a separate attempt identifier, and an event identifier. One attempt can produce multiple notifications, and one order can have multiple attempts. Mixing these levels changes a percentage without changing customer behavior. For a P2P workflow, a specific transaction request may serve as the business object, provided its boundaries are defined consistently.

Stripe recommends reusing a PaymentIntent when the same interrupted checkout resumes. That helps associate failed attempts with the relevant cart or session. It illustrates linking a business object to a payment process rather than prescribing one universal model. Source: Stripe, Payment Intents.

Instrument the smallest useful event map

Record payment-screen entry, method selection, attempt submission, required customer action, completion of that action, and the confirmed result. Keep event and receipt timestamps, references, amount, currency, method, and integration version. Analytics fields do not need card credentials or unnecessary personal information.

Use the verified server-side result required by the integration as evidence of payment. Reaching the success page is useful interface telemetry, but it is insufficient proof by itself. Conversely, closing the browser does not establish that the payment failed. Test both scenarios before trusting the funnel.

Stripe notes that webhook deliveries can be duplicated and need not arrive in generation order. Event processing should therefore handle duplicates and update state without depending on delivery order. Source: Stripe, Webhooks.

Compare cohorts that had equal time to finish

Define cohort membership, such as orders first reaching payment during a particular day, and give each cohort the same observation window. Choose that window for the payment method and purchase cycle. Until it closes, label the result provisional and show the pending share. A late confirmation should update the original cohort rather than appear as a new success in an unrelated day.

Segmentation needs equal care. Compare like-for-like methods, devices, currencies, and customer types. An overall percentage can fall because a more difficult segment accounts for more traffic, even when each segment performs as before. Check the traffic mix and sample size before attributing a change to a new integration version.

Adyen’s Received payment details report provides information on authorized and refused payments, interactions, and processing responses. It can validate the server-side portion of a funnel, but it cannot describe visitors who never submitted a payment. Source: Adyen, Received payment details report.

Give unfinished payments somewhere to remain unfinished

Do not force provisional outcomes into success or failure. Show pending and unknown outcomes separately. If that share rises today, lower conversion may reflect confirmation delay rather than customers definitively lost. Investigate waiting age and outcomes after equal observation windows.

Before an experiment, write down the change, comparison membership and completion definition. Check that an order cannot switch groups merely because it is retried. Avoid changing the denominator, observation window and interface simultaneously. Even a genuine improvement then becomes difficult to explain.

In the final review, place additional completed-order counts alongside processing cost, refunds and duplicate-charge complaints. Some of those quality measures emerge after the initial payment, so an early positive result remains provisional until the chosen quality review. A useful improvement should survive that review rather than only look attractive on launch day. Preserve the original cohort membership as late information arrives, so the comparison continues to describe the same customers instead of a conveniently changing sample.