Too small to tell.
Say so.

One version of my landing page got a 7.1% signup rate. The other two got zero. In most companies that variant ships on Monday.

It shouldn't. The 7.1% is one person. The zeros are nine and eleven people who didn't sign up. Put those numbers on a slide with a bar chart and you have a winner; put them through a posterior and you have three versions of a page about which you have learned essentially nothing. I know which one I'd have believed a year ago.

I spent a few weeks building a system to make that mistake structurally hard. It generates landing pages, splits traffic behind a single URL, waits for conversions to mature, and reports a verdict against a contract written before any of it ran. I wrote almost none of the code — Claude Code did, while I decided what was worth building and what was worth deleting. Then I pointed it at myself, ran one real experiment on real strangers, and it told me nothing.

That was the good outcome. Here's why.

§ 01 · Every easy number is a proxy

The hard part was never generating pages. Generation is the part that demos well. The hard part was stopping the system from measuring the wrong thing, because in marketing almost everything easy to measure is a proxy for the thing you actually want.

Clicks. Click-through rate. Sessions. Form starts. Scroll depth. Every one of them correlates with revenue, and not one of them is revenue. The danger isn't that they're useless — it's that they're available. They arrive instantly, they move satisfyingly, and the thing you care about arrives days later in a different system, if it arrives at all.

So in this system the exclusion isn't advice. It's a list in code that the reward function refuses to accept as a primary metric. Not a guideline in a document nobody reads at 11pm with a dashboard open, and not a code review someone can wave through. A test fails.

I learned this properly by nearly getting it wrong. I had planned to point the factory at Amazon affiliate products — build pages, send traffic, earn commission. The purchase happens on Amazon, where I can instrument nothing. But the outbound click is right there: instant, precise, mine. It's the perfect metric except for the part where optimising it optimises for people who leave and don't buy. A page can win the click by overpromising and earn nothing at all.

That idea died on arithmetic before I designed a single page. Two cents of revenue per visitor against clicks costing thirty. Detecting a 20% improvement would have needed fifty thousand sessions of an experiment that loses money on every one. Run the unit economics before the design work. Software people know to profile before optimising; it's the same discipline pointed somewhere less familiar.

§ 02 · Decide what counts as evidence before you look

Before I posted anything, I wrote down what the experiment would need in order to say something, and committed it.

Three versions of the page, identical above the signup box, differing only in how the box asked. Primary metric: a stored email address. A winner would require the probability of beating control to exceed 0.95 and that version to have at least two hundred matured sessions. Stop at six hundred sessions or fourteen days, whichever came first. One look at the data, at the stopping point.

The uncomfortable part was in there too. At a 10% baseline, separating these asks by 50% needs roughly 685 sessions per version. One LinkedIn post from a personal account plausibly produces thirty to a hundred and fifty in total. So the pre-registration said, in writing, before launch: this run is expected to be inconclusive, and inconclusive is the expected outcome rather than a failure.

Writing that down is the whole trick. It costs nothing in advance and it is almost impossible afterwards.

Once you've seen 7.1% against 0.0%, no amount of integrity reliably reconstructs what you would have accepted as evidence beforehand. You don't decide to fool yourself. You just look at the data first and then decide what the threshold should be, and those two things happen in an order you never notice.

§ 03 · What the run actually said

Fourteen days, then the time cap. Thirty-four sessions, one signup.

Eleven sessions on control, no signups. Fourteen on the second version, one signup. Nine on the third, none. Verdict: inconclusive — no version separated from control at the pre-registered threshold, against a planned six hundred sessions.

The most interesting number in the whole run is one I'd never have looked for. The version holding the only signup scored 0.448 — worse than a coin flip against a control with zero. That isn't a bug; I checked, twice, because it looked like one.

The signup was five days old and the maturity window is seven. A conversion that hasn't had time to happen can't be evidence, and neither can one that has only just happened, so it isn't counted yet. With zero matured conversions everywhere, those probabilities reflect nothing but sample size — the third version scored highest precisely because it had the fewest sessions and therefore the widest uncertainty.

That is what no data actually looks like from the inside. Not an empty screen. A set of confident-looking decimals that mean nothing, attached to a bar chart that would have read as a clear winner.

And the honest coda: at the 2.9% signup rate I actually observed, separating those asks at 50% would take about 2,600 sessions per version. Around 7,800 in total. I had thirty-four.

§ 04 · The bugs were never in the interesting part

None of the things that broke were the bandit, the assignment maths, or the agents. They were all in the plumbing, and every one was invisible until something real ran through it.

The first email send would have failed for every recipient, and been unrecoverable. The provider sits behind Cloudflare, which rejected the default Python user agent with a 403 before the request reached the API at all. Worse: every failure was being written to the database as sent, so the update could never have gone out again without hand-written SQL. Two bugs that compose into total, silent, permanent loss — and both were invisible until I stopped trusting the configuration and actually sent something to myself.

Fixing one class of bug introduced another. I moved to a URL-based deploy specifically to eliminate hand-copied errors. It worked, and it silently discarded every secret in the request, so the database came up with no password and sat there dead. The fix removed one failure mode and quietly installed a different one in the same commit.

And the one I nearly shipped. Filtering link-preview bots out of an experiment is obvious — LinkedIn fetches your link to build its card, and so does every platform it gets reshared to, and none of them can convert. Less obvious: LinkedIn's in-app browser sends "LinkedInApp" and its preview fetcher sends "LinkedInBot". Match the platform instead of the bot and you delete most of your real readers from the experiment while believing you've just cleaned your data. I wrote that regex, wrote a test that stepped around the problem rather than asserting the right thing, and caught it only on a second look.

There's a pattern in all three.

Each one produced a plausible result rather than an obvious error. Nothing crashed.

That's the failure mode worth building against, because it's the one your tests won't tell you about unless you go looking for it specifically.

§ 05 · One disclosure

Twenty-two minutes into the live run I changed the page. It was showing "0 tests" and "10 commits" for a repository with several hundred of each, because the build counted them by running tools that don't exist in the deploy environment. A page whose entire argument is that its numbers are measured rather than typed was publishing two false ones to every reader arriving from the post.

My own rule said no mid-run content changes. I made the change anyway, and logged it as a deviation rather than editing the rule to match what I'd done. It can't have biased the comparison — that block is identical across all three versions, so every version changed at the same instant — but a pre-registration you quietly revise afterwards isn't one, and the reasoning belongs in the record where someone can disagree with it.

§ 06 · The best thing it produced was a refusal

Two people got the readout. One of them was a stranger who'd signed up five days earlier, having been told they'd get the result whichever way it went. The email says the sample was too small to tell, shows the numbers that would have made a nice chart, and explains why they don't support the chart.

Everything underneath worked. Assignment, propensity logging, bot filtering, the conversion beacon, reward maturation, the send path — all of it ran unattended for two weeks and produced a correct answer to a question the data could not support. The machinery is validated. The marketing question isn't, and can't be, on thirty-four sessions.

I set out to learn whether an agentic system could run a marketing loop end to end. It can. What I didn't expect was that the most valuable thing it did all fortnight was decline to tell me what I wanted to hear.

Any system can produce a number. The useful ones tell you when the number means nothing.