Manual fraud review: capacity planning and decision quality

A rule now flags twice as many orders. In a presentation, that looks like stronger protection. In operations, it may mean half the orders cannot be reviewed before shipping. Evaluate a fraud rule together with queue capacity, deadlines and decision quality. Otherwise the risk simply moves from the model into an operator’s workday.

Calculate review capacity

Imagine a team receiving 300 cases per shift. At six minutes of substantive work each, that is 1,800 minutes, or 30 hours. Four reviewers with six available hours each provide 24 hours. The queue grows before allowing for unusually difficult cases. The numbers are illustrative; the method is useful. A rule change needs an estimate of workload and available time as well as expected fraud detection.

Set priorities before the queue fills. Time until irreversible fulfilment, potential loss and the ability to obtain decisive evidence can help organise work. Simply taking the oldest case first may miss an order leaving the warehouse in ten minutes. Prioritisation must not replace mandatory checks or rely unlawfully on protected personal characteristics.

Start with the meaning of a signal

A risk score is a model output, not a finding of guilt. Stripe’s Radar documentation, for example, distinguishes assessed risk from unknown or unevaluated risk and warns that apparently normal payments can still be fraudulent. Stripe explains its risk outcomes.

Keep your own evidence categories equally clear: unusual behaviour, inconsistent information, confirmed customer report and unresolved technical error. These categories deserve different responses. A missing data feed should not silently become a low-risk result; a delayed response should not become proof of fraud.

Consider combinations in context. A large order has a different meaning for a wholesale customer than for a first-time retail buyer. A device change may follow a broken phone. The practical question is which additional check could resolve the uncertainty with the least unnecessary intrusion.

Build a small menu of proportionate actions

Possible actions include allowing the transaction, using supported authentication, placing fulfilment into a time-limited review, or rejecting an attempt when the available evidence justifies that decision. Apply mandatory provider or legal requirements separately; commercial optimisation cannot override them.

For card payments, EMV 3-D Secure supports authentication data exchange between merchants and issuers. Its risk-based flows can authenticate some purchases without an extra customer challenge. That does not make every authenticated purchase risk-free or settle every later dispute. EMVCo’s overview.

Manual review also does not necessarily mean money is waiting untouched. Stripe notes that payments in its review queue are commonly already processed unless capture is deferred. Verify the actual payment state before deciding what “approve” or “cancel” means in your system. Stripe’s review documentation.

A hypothetical review, with a real stopping point

Suppose a fictional customer orders a laptop while travelling, and the order enters review before dispatch. The reviewer sees a consistent order history, a delivery change and no explanation for the new destination. The task is narrowly defined: resolve the delivery change through the established customer-contact process and assess the remaining evidence.

The reviewer should not request a bundle of unrelated identity documents simply because the order feels unusual. Nor should a friendly conversation override contradictory evidence. Record what was checked, the outcome, the decision owner and any remaining limitation. If the information cannot be obtained before the fulfilment deadline, follow the published customer-facing policy.

Give every case an owner and an expiry or escalation condition. An unattended queue creates a hidden customer penalty. A review that finishes after goods have shipped may still inform later controls, but it cannot perform the preventive role the team expected.

Measure both sides of the decision

Imagine a proposed rule flags 1,000 orders. In a fully labelled hypothetical dataset, 40 are fraudulent and 960 are legitimate. Calling the rule “effective” because it detects 40 cases ignores the burden placed on the other customers.

Track confirmed losses alongside review volume, customer abandonment, time to decision and complaints. The share of flagged transactions confirmed as fraud is different from the share of all fraud detected. For blocked transactions, the counterfactual outcome is often unknown, so do not present estimated false positives as observed certainty.

Compare similar customer and product groups when evaluating a change. A seasonal sales surge can change the mix of orders. Document measurement limitations and have a rollback path for a rule that creates disproportionate disruption.

Outcomes arrive after decisions

Today’s confirmed-fraud rate cannot fairly be compared with the final outcome of last year’s orders. The fresh group has not completed a comparable observation period. Show cohort age and outcome maturity in reporting: confirmed cases, still-unknown cases and decisions that prevent observation of the alternative outcome.

Have two reviewers assess the same de-identified training cases. Persistent disagreement may indicate an unclear instruction rather than poor discipline. Identify which evidence would change the decision and what to do when it is unavailable. Good manual review leaves a clear decision record without turning every unusual customer behaviour into a permanent accusation.

Before enabling the next rule, request two estimates together: the risk it is expected to identify and the review hours it requires. Without the second, the team does not yet know whether it can act on the first.