The Verifier Problem
Sometime over the past year, loop diagrams replaced prompt templates as the thing people screenshot and file away.
Generate, critique, rewrite, score, retry, remember: 20 named patterns, 5 tidy categories and a standing promise that mastering the list is worth 6 figures.
The idea behind them is simple enough to state in one line. Instead of asking an AI once and accepting the answer, you have it produce a draft, get feedback and try again, over and over, until the output is actually good.
The patterns are real and they do show up in serious systems. What the catalogs leave out is that everyone who actually runs these loops, whether in production code or in a plain chat window, learns the same uncomfortable lesson within a month.
A loop cannot create quality. It converts a feedback signal into quality and it charges you time and tokens for the conversion. When the signal is weak, the loop becomes an expensive machine for being confidently wrong on every pass.
So the real craft sits one layer upstream, in what deserves its own name: verifier engineering. Knowing where your ground truth comes from, what it costs to check and what you get to keep once the loop stops running.
together with Hard Skill Exchange:
A loop without a verifier is confidently wrong at scale. An agent without a harness is the same failure, running loose across your whole company. That control layer has a name now:
“Today, most companies are built on business processes. In the future, most companies will be built on harnesses” Jensen Huang
The Agentic Harness Summit (Sept 8-10) covers how enterprises actually run agents without losing control:
▫️ Secure agent identity, permissions, memory, and actions
▫️ Move revenue teams from fixed playbooks to a market-of-1
▫️ Where accountability, pricing power, and the next wave of value land
Table of Contents
1. The Catalog Everyone Saved and Nobody Needed
2. What a Loop Actually Buys You
3. The Signal Ladder
4. Loops That Touch Reality Survive
5. How to Build a Loop That Works
6. Harvest the Loop, Then Delete It
1. The Catalog Everyone Saved and Nobody Needed
Pattern collections are how every technology wave announces itself.
Design patterns had their book, productivity had its systems and AI loops now have their diagrams, which are genuinely useful as vocabulary and nearly useless as strategy.
20 patterns are 5 moves
Strip the branding off any loop catalog and a handful of basic moves remain.
You can verify an output against a check and repair what failed. You can sample several candidates and select the best. You can reflect on a failure and retry with the lesson in hand.
You can decompose a big goal into smaller pieces. You can set an adversary against an answer and make it defend itself.
Everything else in the catalogs is one of those five, placed at a different point in a pipeline.
Multi-critic review is verify-and-repair with more reviewers, tree search is sample-and-select with depth and debate is the adversary move with better staging.
Learning the five takes an afternoon. Placing them well requires something no diagram can hand you, which is a reason to believe each pass is actually better than the last.
Why the mechanism stopped mattering
Wiring these loops stopped being hard a while ago.
The retries, the branching and the bookkeeping all live in free libraries now and even a chat user can run the core cycle by hand with copy and paste.
That is exactly why no pattern can be an advantage on its own.
When the mechanism is free, whatever separates working loops from broken ones has to sit upstream of the mechanism.
It sits in the feedback signal. Before any pattern matters, something has to tell the loop what better means and that something decides nearly everything about whether the loop earns its cost.
2. What a Loop Actually Buys You
A loop looks like it adds intelligence to a system.
What it adds is iteration against a signal, which only resembles intelligence when the signal is honest.
A conversion machine with a meter running
Every loop, whatever its shape, performs the same trade. It spends time, money and complexity in exchange for pulling an output closer to whatever its evaluator rewards.
The value of that trade rises with three things: how trustworthy the signal is, how many rounds you can afford and how much the task is worth. The cost rises with every additional round.
Push the signal’s reliability toward zero and one side of the ledger collapses while the meter keeps running.
That single trade explains most disappointing AI systems today.
The machinery was fine, the signal was noise and iterating against noise is a random walk with a receipt attached.
The model grading its own homework
The most common feedback signal in these loops is the model itself, asked to critique or score its own output. The research on that arrangement is blunt.
A paper from Google DeepMind researchers presented at ICLR 2024, titled Large Language Models Cannot Self-Correct Reasoning Yet, tested exactly this setup.
Asked to review their own reasoning with no outside feedback, models regularly talked themselves out of correct answers and overall accuracy went down rather than up.
The mechanism is easy to picture.
A model reviewing its own work carries the same blind spots into the review that it carried into the draft, so the loop converges on whatever the model finds most agreeable to itself rather than on what is true.
Anyone watching such a loop run sees rising confidence and smoother prose on every pass and reads it as improvement. Often it is agreement compounding, which looks identical from the outside and is worth nothing.
From our partners: a loop needs a verifier, and an enterprise full of agents needs a harness: identity, permissions, memory, and guardrails around every action.
The Agentic Harness Summit (Sept 8-10, free and virtual) is 3 days on exactly that layer:
3. The Signal Ladder
Before choosing any pattern, place your task on a ladder of ground truth. The rung tells you how many rounds you have earned and what ceiling to expect from them.
Five rungs of ground truth
At the top sits executable signal. The test passes or it fails, the code runs or it does not, the numbers reconcile or they do not. The verdict is cheap, instant and impossible to charm.
One rung down is referential signal, a known correct answer or a source document the output can be checked against. Below that comes the rubric-judged rung, where a second AI scores the output against written criteria, a real tool with real noise inside it.
Then comes preference signal, meaning an actual person clicking, replying, or buying, which is slow and honest. At the bottom sits vibes, the unanchored feeling that an output seems good, which is no signal at all.
Match your budget to your rung
Executable signal earns you dozens of rounds, because every pass gets a verdict that costs nothing and lies to no one.
This is the unglamorous reason coding assistants got genuinely good before everything else did. Software ships with its own verifier built in.
Rubric-judged signal earns you one critique and one rewrite and pushing past that mostly launders noise into confidence.
Preference signal earns you no live loop at all and the honest move there is generating a few genuinely different versions and letting real people pick.
Here is the practical test.
Write down the one sentence that describes how you would know the output is right. If that sentence contains the word feels, you are on the bottom rung and no amount of looping will lift you off it.
The most common mistake in AI systems right now is spending a top-rung budget on a bottom-rung problem.
Ten critique passes on a strategy memo produce confident mush with excellent formatting and the confidence is the dangerous part.
4. Loops That Touch Reality Survive
Most of the pattern catalogs date from a period when models could not check themselves at all. The newer reasoning models moved the ground under roughly half the list.
The loops the models ate
The current generation of reasoning models is trained to think before answering and inside that thinking they already draft, question themselves, backtrack and revise the plan, all within a single response. You can watch it happen in their visible reasoning.
Every external loop that merely asks the model to look at its own output one more time now duplicates work the model already performs internally and does it slower, at higher cost, with the self-agreement problem layered on top.
That quietly retires a large share of the classic patterns. The self-referential loops were scaffolding for a capability the models have since absorbed.
The filter that kills half the catalog
The loops still earning their keep share one property: somewhere in the cycle, information enters that the model could not have generated on its own.
A test suite executing, a spreadsheet formula erroring, a search result contradicting the draft, a tool returning nothing, a human answering no.
Even Reflexion, the most cited self-improvement pattern in the research, obeys this rule when you read the original 2023 paper by Shinn and colleagues carefully. The agent improved across attempts because the environment told it that it had failed; the reflection step organized an outside signal rather than substituting for one.
That is the filter worth applying before building anything. If every arrow in the diagram points from the model back to the model, the diagram is decoration.
5. How to Build a Loop That Works
None of this requires an engineering team. The same recipe works in a codebase, a spreadsheet, or a plain chat window and the discipline matters more than the tooling.
A recipe you can run this week
Step one: pick one task you repeat that has a checkable result.
A weekly report with required sections, an email that must answer specific questions, data that must match a source. Skip anything where good is a matter of taste, for now.
Step two: write the evaluator before the loop.
Turn your standard into five to ten pass-or-fail questions. Does every claim have a source? Is it under 300 words? Does it name the next action? A checklist beats the question is this good every single time, because a checklist cannot be charmed.
Step three: separate the roles.
One prompt generates the draft. A second, fresh prompt grades it against the checklist and lists only what failed. The first prompt then rewrites, fixing only the listed failures. Keeping the roles apart is what stops the grader from inheriting the writer’s blind spots.
Step four: cap the rounds and regenerate instead of polishing.
Two rounds for judged work, more only when a hard check like a test or a formula is doing the grading. When quality stalls, throw the draft away and generate fresh, because a model revising a draft tends to defend the draft.
The rules that keep it alive
Define done and give up before you start. A loop without an exit either runs forever or oscillates between two answers and both failure modes look like diligence from the outside.
Gate the loop. Most tasks should exit after a single pass; save the full cycle for the minority of work that is valuable enough and hard enough to undo, to deserve the extra cost. Three critics reviewing three retries adds nine calls to a task that used to take one and nobody notices until the bill arrives.
Watch for the evaluator being gamed. Whatever the grader rewards, the generator learns to produce, so outputs drift toward the grader’s tells while scores climb and quality stays flat.
Goodhart’s old warning, that a measure stops being a good measure once it becomes a target, plays out here in weeks. Refresh the checklist when the scores start looking too good.
Log every attempt, every grade and every fix. This feels like bureaucracy on day one. It turns out to be the entire point, which is where this ends.
6. Harvest the Loop, Then Delete It
Here is the reframe that makes all the discipline above worth the trouble. A loop is scaffolding, rented in time and tokens, around a capability the model does not have yet and nobody keeps scaffolding standing once the building holds its own weight.
A running loop produces two things of lasting value. The first is the verifier itself, the checklist or test that computes what good means for your work, which is rare, hard to copy and useful far beyond the loop it was built for.
The second is the record of attempts. Every draft, grade, failure and fix you logged is a labeled example of your quality standard.
That is exactly the raw material for making the loop unnecessary. A sharper standing prompt for a chat user, a tuned or distilled model for a team, either one reaching the same answer in a single pass at a fraction of the cost.
Serious tooling already works this way. Stanford’s DSPy project treats prompts as things to be improved offline against a measurable score rather than handwritten and frozen and tuning a model’s weights on logged examples is the same idea carried one layer deeper.
Which points at the mature lifecycle. Build the loop where the signal justifies it. Run it while it beats the single call. Harvest what it logged, fold the lesson into something cheaper and delete the loop without sentiment.
The whole subject then collapses into one question worth asking before any diagram gets drawn. What is my ground truth, what does it cost to check and what do I keep when the loop stops running?
Answer all three and the right pattern falls out on its own. Skip them and you have built the most expensive way to be wrong five times in a row.
Really liked the point about writing the evaluator before the loop.
That is the part I see people skip most often. They build the retry cycle first, then ask the model to decide whether the next version is better. It looks like improvement because the answer gets smoother, but nothing outside the model has changed.
The useful bit for me is when a failed check becomes a checklist item for the next run, or a missing source becomes a required input. That makes the next loop less dependent on the model judging itself.
The verifier concept maps directly to retirement planning. I keep seeing clients who rely on 'vibes' — market tips, gut feelings — instead of objective benchmarks like index returns or inflation. A checklist-based review, like the post's evaluator, works for financial goals too: separate the advisor from the investor, cap the review cycles, and let market data be the verifier.