Your agent works.
Prove it.

§ 01 · The number on the slide

Somebody runs the new agent through a test set, it gets 92% right, and that number goes on a slide. From that moment it's real. It anchors the roadmap, the launch decision, the exec update. Nobody in the room asks the three questions that actually matter: which run, out of how many, and what did "right" count as.

I went deep on how you actually tell whether an agent works — the studies, the benchmarks, the eval tooling, the practitioner writing buried underneath the vendor noise. I came out convinced that most of the numbers we quote about agents are softer than we think, and that the softness is knowable. We just don't look.

This is the fourth piece in a series about the unglamorous layer beneath agents. The first three were about access, data, and maintenance. This one is about measurement, and it turns out to have the same shape as the others: the interesting problem isn't the model, it's everything we assumed we could skip.

§ 02 · A single run is not a measurement

Here's the fact that reorganized how I think about this.

Run the same agent on the same task twice and you often get two different runs — not two phrasings of the same solution, but different strategies, different tool calls, different outcomes. One study put 60,000 agent runs against a standard coding benchmark and found that single-run success estimates moved by two to six percentage points depending on which run you happened to report. The standard deviation stayed above a point and a half even at temperature zero, the setting that is supposed to make the model deterministic.¹

The mechanism is simple and a little unsettling. Trajectories diverge inside the first few percent of tokens, and small early differences cascade into entirely different solution paths. Another study found that how consistently an agent behaves on a task predicts how well it does. The same agent on identical inputs produced 2.3 to 4.2 distinct action sequences per ten runs; the tasks it approached consistently landed at 82 to 87% accuracy, the inconsistent ones at 41 to 65%.²

A single run tells you the agent can succeed. It doesn't tell you it will.

So a green checkmark from one run isn't a measurement. It's one sample from a distribution you never characterized. When a two or three point improvement shows up between two versions, you genuinely cannot tell, from single runs, whether you improved the agent or just resampled the noise.

§ 03 · The metric that lies without meaning to

Once you accept that each run is a sample, the next question is what you do with a handful of them. This is where the most common quiet dishonesty in the field lives.

Two metrics look almost identical and mean opposite things.

pass@k is the probability the agent succeeds at least once in k tries. It goes up as k grows. It's the right metric when you can retry cheaply and a human picks the winner — code generation where you review three candidates and keep the good one.

pass^k is the probability the agent succeeds on every one of k tries. It goes down as k grows. It's the right metric when your user lives every attempt, which is every customer-facing agent in production.

The arithmetic is unforgiving. An agent that succeeds 75% of the time per attempt is at 42% by pass^3. τ-bench, the benchmark that introduced pass^k, found that even a strong function-calling agent solved under half its tasks and fell below 25% by pass^8 in the retail domain.³ Same agent, same tasks, different survival function.

Reporting pass@k for a customer-facing agent isn't lying. It's publishing a number that is technically true and practically false.

Your user does not get to retry. They don't see the one run in five that worked; they see the run they got. If your metric quietly assumes a human is standing by to pick the best of k, and no human is, you've shipped a number that describes a product you didn't build.

§ 04 · The path is part of the answer

There's a second thing a final-output score cannot see.

An agent can reach the right answer through a route that will not generalize — a lucky guess, a shortcut that happens to work on your test set. And it can do the opposite: produce a fluent, confident report about work it never actually did. One study of coding agents measured exactly that second failure — models that submitted a patch on all five runs and resolved the task on none of them, at rates from 16% to 80% depending on the model.⁴ Both failures look fine if you only grade the destination.

This is the event agent that didn't know its own address, moved up a level. There, the failure was invisible until you looked at what users actually asked. Here, the failure is invisible until you look at what the agent actually did — the trajectory, not just the result. You have to read the path. The output will not tell on itself.

§ 05 · There are three places to measure, and you're using one

Most teams measure an agent once, offline, before launch, and treat that as the answer.

Offline evaluation runs a fixed dataset with known answers in a controlled environment. It's where regression testing lives, and it's necessary. It's also the only place with clean ground truth, which is why everyone stops there.

Online evaluation scores real production traffic, where you rarely have ground truth and have to lean on proxy signals and sampled human review. Monitoring watches the live system for drift, because the agent that measured well in March is grounded on information that has changed by October.

If that last line sounds familiar, it's the argument from the previous essay wearing different clothes. Measurement isn't a gate you pass through once. It's a loop you keep running, for the same reason the data work is a loop: the world underneath the agent doesn't hold still.

§ 06 · The discipline

None of this is exotic. Sample more than once and report the spread, not a point. Pick the metric that matches how your user actually experiences the product. Read trajectories, not just outputs. Measure again after you ship. It isn't sophisticated. It's just work, and it's the work most teams skip, because a single good demo is so much more persuasive than an honest distribution.

I keep landing on the same place across all four of these essays. The gap is never where the demo points. It's in the layer underneath — the access, the data, the maintenance, and now the measurement — that nobody owns until something breaks in public.

A demo is a single run you got to choose. Production is every run, chosen by someone else.

Reliability is the difference between the two. It's the difference between an agent that impressed a room once and an agent people can depend on. And you cannot claim it from one green run. You have to prove it.