Design Fraud Detection

Fraud detection looks like a textbook classification problem and behaves like nothing in a textbook. The positive class is a fraction of a percent. The labels arrive weeks late. The people on the other side are trying to fool you and change their tactics as soon as you adapt. And the real output is not a probability but a decision: approve, decline, or send to a human.

An airport screens everyone quickly with a scanner, pulls a small share aside for a closer check, and has a fixed number of officers to do the checking. Set the scanner to flag too many people and the closer-check line overflows. Set it to flag too few and risks get through. Fraud review works the same way: the threshold must respect how many cases the team can handle.

• -Decision: approve, decline, ask for extra verification, or queue for manual review.
• -Costs: a missed fraud loses the amount, plus fees. A review costs analyst time. A wrongly declined customer is annoyed, and may leave.
• -Latency: the decision is made while the customer waits, in perhaps 100 to 200 ms.
• -Scale: a few hundred or thousand transactions a second, mostly legitimate.

• -Labels come from chargebacks, confirmed reports, and analyst decisions. Chargebacks take 30 to 90 days, so the freshest data has no labels yet.
• -Imbalance is extreme, often well under 1%. Use class weights or downsampling, look at precision and recall and not at accuracy, and keep calibration in mind if you downsample.
• -Label bias: you only learn the outcome of transactions you approved. Declined ones never reveal whether they were fraud. Approving a small random share lets you measure.
• -Adversaries adapt, so patterns shift quickly. Retrain frequently and monitor drift closely.

• -Velocity: how many attempts, cards, or addresses from this account, device, or IP in the last minute, hour, day.
• -Identity and device: device fingerprint, account age, whether the shipping address matches the billing address.
• -Behavior: typing patterns, how the session reached checkout, order size compared with the account's history.
• -Graph links: accounts that share a device, card, or address with known fraud.

Most real systems use both. Rules handle clear, known patterns, like a blocked country or a known stolen card number, and are fast, explainable, and quick to deploy. A model finds subtle combinations that rules miss, and gives a score. The final decision combines them: rules can override, and a score above a threshold sends the case to review. Reason codes help the analysts, since a score alone does not tell them what to look at.

The block simulates 100,000 transactions with 378 frauds. Each flagged transaction costs $4 to review, each missed fraud costs $150, and the team can review at most 900 transactions a day. Press Run, and compare the cheapest row overall with the cheapest row that the team can actually handle.

The cheapest threshold overall is 0.5, at $11,278, but it flags 1,057 transactions, more than the 900 the team can review. The cheapest row that fits is 0.6, at $14,620, with 430 flagged, 292 of the 378 frauds caught, and a precision of 68%. Going higher still, at 0.8, gives perfect precision but catches only 136 frauds and costs $36,844 in losses. The point is that the threshold comes from the costs and the capacity, not from a default of 0.5. If the team grows, the best threshold changes, and the same model does not need retraining.

Try the threshold yourself in the demo below. The counts are expected values under the same score model as the code, so they sit close to the table above rather than exactly on it. Move the threshold and the team's capacity to find the cheapest setting the queue can handle.

• -Offline: precision-recall curve, recall at a fixed review volume, and dollars saved, on a time-based split with matured labels.
• -Online: shadow mode first, then a gradual rollout, tracking fraud loss, review volume, and the decline rate of good customers.
• -Monitoring: feature drift, the score distribution, and the mix of fraud types, with alerts for sudden spikes.