Stop Checking Everything AI Produces

Prefer to watch? The whole argument in five and a half minutes — or read on.

Keep a human in the loop. How many times have we all heard that one? Human in the loop, human-centric, human oversight. Read every line before it goes out. Verify every citation. If you've sat through a single presentation on AI in the last two years, you've heard some version of this, probably more than once. It has become the boilerplate, the responsible-sounding thing everyone says about working with AI.

And it's good advice, directionally. I've probably signed off on versions of it myself. But directional is all it is. As stated, the advice is incomplete, and worse than incomplete, it's impractical. It tells you to check everything without telling you how, or when, or against what. And the moment you try to actually do it, you immediately feel why it fails.

Why "check everything" breaks

We reached for AI in the first place because we're overextended, asked to do more than the hours in the day allow, and asked again the next day. AI is a potential solution. It can produce genuinely good work, extraordinarily quickly. So let's just say, for the sake of the math, that AI can 10X your output.

Now lay "check everything" advice back on top of that. If you have to read every line of ten times the material, by hand, how much time have you saved? The real volume of review work is actually more than ten times.

When you make something yourself, you check it as you build it. You write from what you know, or you go find what you don't, and by the time the thing exists you have already vetted it in a hundred small ways. That's baked into the act of creating. AI skips all of that and just assembles the output. And you didn't do the thinking, so checking its work isn't like proofreading your own writing. It's like being handed a stranger's draft, cold, and told to find the flaws in it.

That's hard work, and it's draining. You have to context-switch, set your whole mind to review something fresh with no history, and go on high alert for every small error and quiet mistake buried inside it. Do that for one document and it's fine. Do it for everything you are producing with AI, all day, and it's exhausting in a way the old work never was.

This experience has been studied. In a randomized trial last summer, researchers at METR, an independent group that runs careful evaluations of what AI systems can actually do, had experienced software developers do real work in code they knew well, with and without AI. With the AI, the group came out 19 percent slower, a genuinely surprising result, because those same developers estimated they had been about 20 percent faster. Their time was shifted into reviewing and untangling what the AI had produced, and it made them slower overall.

And you stop checking anyway

When I first started using Claude Code to help build Grantable, I ran it in the most cautious mode. It asked my permission for everything, and I hand-reviewed every request. Dozens and dozens, and I hadn't found anything out of order. Soon I noticed I wasn't reading carefully, and finally barely at all before clicking accept.

This is the terms-of-service problem. Every time we sign up for an app or buy something online, we click "I agree" to pages of terms almost none of us have read. We don't do this because we're trying to be careless, but because we don't have time and often reading them doesn't feel useful. Rote hand-checking of AI output numbs us in the same way. Go long enough without catching an error and you get lulled, and the checking quietly becomes a ritual you perform without really doing. This is the most dangerous outcome of this safety advice we're all being given.

So don't listen to that advice

Let me just say the thing plainly.

"Human in the loop" isn't wrong, but it is not a plan. How many loops, and where in the loop, is the entire question, and "all of them, everywhere" is the one answer we can be sure is wrong.

The people who solved this first

So how do you actually verify work at AI speed and AI scale? For a better answer, let's look at the people who hit this wall first and hardest: the AI companies themselves.

Anthropic, the maker of Claude and one of the leading AI labs in the world, has said that as of this spring more than 80 percent of the code they merge is written by Claude. Their typical engineer now ships roughly eight times as much code per day as in 2024, by their own admittedly crude measure of it.

Once the AI was writing most of the code, writing stopped being the time-consuming part. Human code review became the bottleneck. And the stakes for getting it right are about as high as they come: more than 300,000 businesses run on Claude, including eight of the ten largest companies in America.

Anthropic's engineers suspected an AI that writes code well might also be capable of reviewing code well. We can draw a similar hypothesis: if AI can write a strong needs statement or donor appeal, it can probably review one too.

They were right. It turns out the same large language models that write code well can be set up to be excellent code reviewers, too.

Adversarial review

At the heart of Anthropic's solution is a key concept: adversarial review. It is very important to have separate AI sessions or agents doing the work and checking the work.

Stay in the same conversation and say "find the problems here" and it will find a few. Push it and it'll find a few more. But hand that same job to a wholly different agent, or a totally separate fleet of them, with different goals and no attachment to the work, and it finds dramatically more.

There's research on why the self-check is so weak. A paper presented at NeurIPS, one of the top AI conferences in the world, is titled, plainly, "LLM Evaluators Recognize and Favor Their Own Generations." A model tends to recognize its own work, and to grade it kindly. The researchers explore why, and self-recognition seems to sit at the heart of it, but the why is not what matters for us. What matters is that the bias is real and measurable, and it means the AI that produced the work is a compromised reviewer of it.

So the structure they landed on is not to simply ask for proofreading at the end of an AI session. It's a two-phase approach. One set of agents does the work. A separate set attacks it, with a single focus: find what's wrong.

The two principles.

Get the full field guide — free PDF ↓

The results are striking. According to InfoQ, a long-running news site for professional software engineers, when Anthropic turned this kind of separate, adversarial review on its own changes, the share that surfaced a real problem jumped from 16 percent to 54. Same underlying model. The only difference was who did the checking, and whether they had a stake in the answer.

Narrow jobs, many judges

There are two dials on adversarial review, and they get turned together. The first is focus. A reviewer with one narrow job gives you clean, clear signal. Ask a single session to check everything at once and it meanders, the way any of us do when we're told to "look this over." Give it exactly one assignment — check this draft against the funder's guidelines, nothing else — and the catches come back specific and useful. The narrower the goal, the sharper the catch.

The second dial is coverage. Reviewers are cheap now, so don't stop at one. Run several narrow reviewers side by side. One on compliance. One on the budget math. One on every factual claim. One reading as a skeptical program officer. Each attacks from its own angle, and together they pressure-test the work from more directions than any single reader could hold in mind at once.

Narrow focus buys you quality of signal. Multiple reviewers buy you quantity of coverage. You want both, and that combination is exactly what the AI companies' review fleets are running at industrial scale — dozens of specialists instead of one general proofreader.

Your version: one plan, a few fresh windows

You don't need a fleet of agents or an engineering team to use this, though I'll show that version in a future post. If your whole AI toolkit is a chat window, that's fine. You just need a few of them, and a little discipline about what passes between them.

Here's the move that makes it all work: before you make anything, have the AI write the plan you'll both follow. Say you're writing a letter of inquiry to a funder you've never approached. Open a chat and talk the whole thing through — what you're asking for, what you know about this funder and what you don't, what has to be true for this letter to work, every piece of nuance you'd brief a colleague on. Then ask it to turn all of that into a phased plan, and, this is the important part, have the plan include the opening prompt for each phase and a list of exactly what to paste or attach when you start it.

Then you run the plan, one fresh session per phase.

1. The research session. New chat. Paste the plan's research prompt and point it at the funder — priorities, past grantees, guidelines. Save what comes back as its own document. That's your research brief.
2. The drafting session. New chat. The plan's drafting prompt plus the research brief. Out comes the draft LOI. Save it.
3. The review session. New chat, and this one attacks. Don't hand the job to the drafting chat — that conversation is a wonderful collaborator and a terrible judge, and it's been agreeing with you all afternoon. The fresh window gets the plan's review prompt, the draft, and the funder's guidelines. Nothing about how the draft came to be.

One plan, fresh windows — outputs flow forward, your once-overs at the seams.

Get the full field guide — free PDF ↓

Notice what's actually carrying the quality here. Each session gets exactly what it needs and nothing more. The separation isn't an inconvenience to work around, it's the mechanism itself. What you feed each phase, and what you save out of it, is the entire recipe. Where you keep those outputs can be as simple or as sophisticated as you like — downloading docs and copy-pasting, pointing Copilot or Gemini at your cloud drive, or working out of shared folders with a tool like Claude Code. More on that spectrum another time. Any of them works, as long as each phase's output lands somewhere you can hand to the next.

Here's the whole method as a take-away guide — the part to save.

What comes back will be a mix, some real problems and some dutiful nitpicks, and that's fine. Your first window thinks the draft is ready. Your second just handed you a ranked list of doubts. The gap between them is small, and it is exactly where your judgment belongs. That part nobody can hand off.

Where the human actually belongs

Which brings us back to "human in the loop," now with an actual answer to how many loops and where. Look at the shape of what you just ran. The work moved through phases, and each phase ended with something landing on your desk: a research brief, a draft, a list of findings. Those handoffs are the seams, and the seams are where you belong.

A once-over at each seam is enough. Does the research brief describe the funder you thought you were approaching? Is the draft roughly the letter you'd want your name on? You're not line-editing. You're checking that what feeds the next phase meets your spec, and that takes a minute, not an afternoon. Save the real, careful, word-by-word read for the one thing that's actually leaving the building, after the adversarial pass has already been through it.

And here's the part that should take the last of the weight off. If you keep your sessions — the plan, the threads, the outputs, even the instructions you dictated along the way — then a mistake found late is not a catastrophe. You and a fresh AI session can trace it back through the phases, find where it entered, see what it touched, and replace exactly that piece. An error in the research doesn't force a rewrite of the letter. It gets surgically fixed, because everything that produced the letter is still there to work from. That's the real reason you don't have to inspect every micro-output along the way. Not because mistakes can't happen, but because in an organized workspace, repair is cheap.

So, pulled out of the recipe, the whole method:

• Have the AI draft the plan with you — phases, opening prompts, and what each phase gets fed.
• Run each phase in its own fresh session, with a narrow job, and save every output.
• Never let the maker be the checker. Review is separate, adversarial, and comes from multiple angles.
• Spend your eyes at the seams — quick spec-checks between phases, the hard read on the final product only.
• Keep everything. Organized context turns "start over" into "go fix that one thing."

The whole method on one card.

Get the full field guide — free PDF ↓

What you've just done, by the way, is run a dynamic workflow by hand. You were the orchestrator, and the plan was your score. Teaching AI to run those loops itself is where all of this is heading, and that's a post of its own, coming soon.

None of this is about trusting AI less, or more. It's about being smarter than "check everything," a slogan standing in for a method. The real skill was never reading every line. It's building the work so it checks itself along the way, and knowing the few places your own eyes truly belong.

So I'll ask you what I keep asking myself. How are you actually pressure-testing your AI's work? I'd like to know what's working on other desks.

Receipts

• METR, July 2025 — experienced developers were 19 percent slower with AI while estimating they'd been 20 percent faster; randomized trial, 246 real tasks. The study
• Anthropic, "When AI builds itself," 2026 — more than 80 percent of merged code authored by Claude; roughly 8× the code per engineer per day vs 2024 (a figure they themselves call crude); "human code review became the new bottleneck." Primary source
• Boris Cherny, Head of Claude Code — uninstalled his IDE and no longer writes code by hand; directs and reviews instead. Interview
• Anthropic, Series G announcement, 2026 — 300,000+ business customers; eight of the Fortune 10 are Claude customers. Primary source
• NeurIPS 2024 — "LLM Evaluators Recognize and Favor Their Own Generations": self-preference bias scales with self-recognition. The paper
• InfoQ, April 2026 — adversarial AI review on Anthropic's own changes raised the substantive-catch rate from 16 percent to 54. The report