I set my AI fraud threshold to 0.98. It paid out nothing
I set my AI fraud threshold to 0.98 because it felt safe — and it paid out nothing. Zero.
Here's how to route risky decisions three ways (pay out, block, ask a human) and read both thresholds off a cost table instead of guessing.
A local model, no API key, no third party, returns one probability per payout request. On 100 generated requests, the guessed thresholds switched automation off completely; the cost-table thresholds paid out 22 automatically, blocked 17, sent 61 to a human — with zero wrong decisions at both ends. Plus three gotchas: a model that's down, question wording that flips outcomes, and a "confidence" field that isn't what you think.
Note: the data is generated (real payout data can't be published) — measure on your own data before you trust any threshold.
Code: https://github.com/webmatze/demo-shop (branches: episode-006-before / episode-006-after)
Prerequisites: Rails 8.1, laya-serve, Elbcasts #005 (decision models)
Full show: https://elbcasts.dev
#AI #Rails #FraudDetection