Generation is cheap.
Check the work.
Any model can write the answer now. The product is the part that knows which answers are right, and what it teaches.
AI models now generate more work than anyone can review, and they are close to interchangeable, so the scarce part is knowing which output is right. That job belongs to the verifier: the evals that run inside the system and decide what ships. Everything the verifier approves can train smaller distilled models, so the verifier and those models are the product, as long as people keep checking the verifier.
§ 01The cheap part
Ask a model for a landing page and you get one in a minute. Ask for twenty and you get twenty. Ask three different models and you get sixty, and most of them are fine.
That's the new normal. Generation is cheap. It gets cheaper every quarter, and the models that do it are close to interchangeable. Whatever is best this month gets passed next month, and switching is mostly a config change.
Last time I wrote that AI made the attack free. This is the follow-up question. If you can generate as much as you want, what's actually scarce?
Knowing which of the sixty is right.
§ 02Evals are the part that knows
This summer two projects of mine kept ending up in the same place.
The first was the experiment system I wrote about in "Too small to tell. Say so." It generates landing pages, splits traffic, and returns a verdict. Generating the pages was the easy part. The work was the contract: what counts as evidence, which metrics the reward function refuses to accept, and when the honest answer is that the sample is too small.
That's the code factory pattern with the compiler taken away. Code factories work because code has a judge. The tests pass or they don't, and that check is what lets you generate at volume without drowning in it. Marketing has no tests, only live performance, which is noisy, delayed, and easy to fool. So the center of the system wasn't anything that generated. It was the part that decided whether a winner had actually won. The industry calls this kind of checking evals. When it runs inside the system and decides what ships, I call it the verifier.
The second was a research brief on agent evaluation, written for an orchestrator that hands work to parallel research agents. The main rule for the orchestrator was don't do the research yourself. Its value was in checking the returns against each other, and treating disagreement between sources as information instead of something to average away.
Different problems. Same answer. The generator is a commodity. The checker is the system.
§ 03The checker is the spec
Here's what nobody tells you about writing a verifier: it's the hardest product work in the system, because it forces you to say what good means.
A prompt can stay vague. "Write a compelling headline" works fine as an instruction. It doesn't work as a test. To check the headline, you have to decide what compelling means for this audience, which claims are allowed, what the brand won't say, and how much evidence you need before you believe the number. That's the spec. Most teams never write it down, because the generator let them skip it.
The verifier is also where your knowledge lives. The model vendor ships the same generator to everyone. Nobody ships your definition of correct. Swap the model next quarter and the verifier stays. It's the part you own.
In "Your agent works. Prove it." I argued that a single run isn't a measurement. That essay was about how to measure. This one is about what the measurement is worth once you have it. It's worth more than a score.
§ 04What the checker teaches
The verifier does a second job, and it's the one I underrated.
Every output the verifier approves is a labeled example: a real input, a good answer, and a record that it passed. Run the big model a thousand times, keep what passes, and you have a training set that encodes your standard.
That's distillation. You train a smaller model on the approved outputs of a larger one. The small model doesn't need to know everything the big one knows. It needs to do your task the way your verifier wants it done.
Fin, the customer service agent from the company formerly called Intercom, has published the clearest version of this I've seen. For its reranker, the step that picks which help content the agent reads, Fin had a large model grade the candidates for 400,000 real customer questions. Then it trained a much smaller model on those grades. Fin says the small model beat the commercial reranker it replaced on Fin's own benchmark, cut reranking cost by 80%, and raised resolution rate in an A/B test across 1.5 million conversations without adding latency.1
Then Fin went further. Its Apex models are post-trained on years of its own support conversations, with reinforcement learning tied to whether a conversation was actually resolved. Resolution is the verifier. In benchmarks Fin shared with VentureBeat, Apex 1.0 resolved 73.1% of conversations to GPT-5.4's 71.1%, with fewer hallucinations, at roughly a fifth of the cost of calling frontier models directly.2 Those are Fin's numbers, not an independent audit. Apex 2.0 starts from an open-weight base model from Nvidia.3 The generator is the commodity part. What Fin owns is the resolution data and the models trained against it.
Disclosure: I work at Salesforce, which agreed in June 2026 to acquire Fin.4
Then the loop compounds. The small model handles the routine cases. The big model takes the hard ones. The verifier checks both, and everything it approves becomes the next round of training data. Each cycle, the distilled model covers more of the work, and each run costs less.
So the product isn't the verifier alone. The verifier sets the standard. The distilled models carry it, in weights you own, cheaper and faster on every run. Together they're the thing a competitor can't download.
§ 05Who checks the checker
The catch is the obvious one. A verifier that's wrong doesn't fail quietly. It teaches.
If it rewards a proxy, the generator learns the proxy. Fin's write-up of Apex 2.0 describes a version of this: pressure from its reward pushed the model toward what Fin calls ungrounded promises to customers, which then needed their own checks.3 If the verifier is a model grading another model, it brings its own habits, like preferring longer answers or answers that sound like itself. If it approves bad outputs, distillation bakes them into a model and ships them at scale. A loose prompt makes one bad answer. A loose verifier makes a bad model.
So the verifier needs checking too. A sample of its calls goes to a person. Its judgments get compared against human labels on a schedule, not once at launch. When the verifier and the people disagree, that's the most valuable data in the system, because it's exactly where your standard isn't written down yet.
Which means the job moves. We used to spend our time on what the system produces: the prompt, the model pick, the demo. Now the time goes to how the system decides. Write the checker first. Make it strict. Keep checking it. Then generate all you want.
Generation is cheap. Somebody still has to grade.
References
| 1 | Fin. "How We Built a World-Class Reranker for Fin." September 2025. https://fin.ai/research/how-we-built-a-world-class-reranker-for-fin/↩ |
| 2 | VentureBeat. "Intercom's new post-trained Fin Apex 1.0 beats GPT-5.4 and Claude Sonnet 4.6 at customer service resolutions." March 2026. https://venturebeat.com/technology/intercoms-new-post-trained-fin-apex-1-0-beats-gpt-5-4-and-claude-sonnet-4-6↩ |
| 3 | Fin. "Announcing Apex 2.0: Our most capable model for CS yet." October 2026. https://fin.ai/blog/announcing-apex-2↩ |
| 4 | Salesforce. "Salesforce Signs Definitive Agreement to Acquire Fin." June 15, 2026. https://www.salesforce.com/news/press-releases/2026/06/15/salesforce-signs-definitive-agreement-to-acquire-fin/↩ |