AI Agent Reliability: Why Pass-Once Isn't Enough

Ask most vendors whether their AI agent works and they will show you a demo where it completes the task perfectly, once. Ask a growing body of enterprise research the same question and you get a very different answer: more than 40% of agentic AI projects are on track to be canceled by the end of 2027, not because the underlying model is weak, but because agents that scored well in testing behave unpredictably the moment they run in production at real volume. The gap between those two outcomes has a name now — researchers call it the difference between pass^1 and pass^k, and it is quietly the most important number in enterprise AI that almost nobody outside a handful of eval teams is tracking.

Pass^1 is the number every demo optimizes for: did the agent complete the task correctly one time. Pass^k asks a harder question — if you ran the exact same task k independent times, did it succeed on all of them? An agent booking an appointment, extracting a line item from an invoice, or triaging a support ticket does not run once. It runs hundreds or thousands of times a week, on inputs that vary just enough to occasionally trip something the demo never exercised. An agent that hits 95% on pass^1 can look completely different at pass^4 or pass^10, because small, independent failure probabilities compound fast across repeated runs — and for a workflow executing hundreds of times a day, that compounding is not a theoretical concern, it is next quarter's support queue.

Why This Gap Stays Invisible Until It Doesn't

Most buying decisions for AI agents happen the same way software buying decisions have happened for twenty years: a vendor or an internal team runs a proof of concept, the agent completes a handful of test cases successfully, and the project gets greenlit for production. The problem is that a handful of successful runs is exactly what pass^1 measures, and pass^1 is close to useless as a predictor of production behavior. It tells you the agent is capable of succeeding under the conditions you tested. It tells you nothing about how often it succeeds under the full range of conditions production will actually throw at it — the malformed input, the ambiguous customer phrasing, the edge case that shows up once in every three hundred calls but shows up three hundred times a day at scale.

This is compounded by the fact that public agent benchmarks are increasingly saturated and, in practical terms, gameable — a model can be tuned to look excellent on a known benchmark suite without that translating into consistent performance on a specific business's actual workflow, data shapes, and edge cases. The benchmark score on the vendor's website and the reliability of the agent running your invoice-matching process are only loosely related, and the gap between them is exactly where the canceled-project statistic comes from.

What This Looks Like When It Goes Wrong

A regional insurance brokerage we worked with had piloted an AI agent from a well-regarded vendor to triage inbound claim documents — reading submitted PDFs, extracting policy number, claim type, and incident date, then routing to the right adjuster queue. The pilot ran twenty test documents, all handled correctly, and the team signed off for a full rollout across all four offices. Within three weeks, the error rate had climbed to roughly one in twelve documents — not because the model got worse, but because live submissions included scanned images at an angle, multi-policy household documents, and handwritten annotations that the twenty-document pilot had never exercised. Each individual failure mode was rare. Run across 600 documents a week, "rare" meant fifty misrouted claims a month, each one costing an adjuster time to catch and manually reroute, and a few slipping through far enough to delay a customer's claim by days.

The fix was not a better model — it was building an actual reliability harness before rollout. We assembled a test set of 150 real historical documents specifically chosen to include the edge cases the original twenty had missed, ran the agent against that set five times independently to get a genuine pass^5 number rather than a single pass, and used the failure clusters to add a confidence-based routing rule: anything the agent scored below a defined confidence threshold on got a thirty-second human glance before routing, instead of going straight to an adjuster queue. Everything above the threshold — the large majority of documents — still routed automatically with no added latency. Ninety days after the rework, the true error rate had dropped to under 1%, and the confidence-routing queue was catching fewer than 15 documents a week, each resolved in under a minute.

Building for Pass^k, Not Pass^1

The practical shift is straightforward once you name it: before an agent goes into production, it needs to be tested against a representative sample of real inputs — not curated ones — run multiple independent times, with the failure modes clustered and reviewed rather than averaged away into a single accuracy score. A 95% pass^1 score with failures scattered randomly across input types is a very different risk profile than a 95% pass^1 score with failures clustered entirely in one predictable category, even though the headline number is identical. The second case is actually the good outcome — it means you can build a targeted checkpoint around exactly the inputs that cause trouble, without slowing down the 90%+ of runs that were never going to fail anyway.

This is also why confidence-based routing, rather than blanket human review, is usually the right architecture. Reviewing every single output defeats the point of automating the task in the first place. Reviewing nothing means the rare failure mode ships silently until a customer or a P&L notices. The middle path — instrument the agent to surface its own uncertainty, route only the low-confidence fraction to a human, and measure pass^k on a real sample before trusting the rest — is what separates an agent that looks good in a demo from one that holds up at the volume your business actually runs.

None of this requires exotic tooling. It requires treating agent reliability as something you measure with a number before go-live, not something you discover from a support ticket after go-live. Most vendors will not offer to build this test harness for you, because it is not their job to prove your specific use case is safe at your specific volume — it is the integrator's job, or yours.

If you are evaluating an AI agent for a workflow that will run more than a handful of times a day, ask for its pass^k number on your data before you ask about its feature list. Visit wizeb.com/services/ai-agents to get a reliability audit built around your actual inputs before an agent that looked perfect in the demo starts quietly failing in the one in twenty runs nobody tested for.