Never mark your own homework: independent AI reviewers for a solo builder

Fold, my recipe app for vegetarian households with a vegan at the table, was built at a pace no person could review line by line: 1,073 commits from 15 to 30 October (682 code, 240 merge and 151 data saves, across all branches), with one Claude session writing specs and another writing code. By late October the signs of strain were obvious. CI had been red for six days without anyone noticing, a save check had crashed and reported "no errors" for two days, and I was about to invite households outside my family, whose dietary choices and guests Fold would hold. My own product notes rated the assumption "one person with two Claude sessions can keep shipping safely at this pace" as evidenced, but strained. This panel is my answer to that strain.

How it came about

First, a one-off audit (24 October). I asked how to get Claude Code to review the whole codebase, and what best practice was for a quality harness. Claude's answer rejected the obvious approach: a single "review everything and fix it" prompt. Instead the review ran in stages (map, measure, judge, verify, then fix), with a fresh session trying to disprove the top findings using runnable reproductions before anything was changed. It found three Critical security issues, all since fixed: any household's synced data could be read and written by anyone past the gate, there was no server-side household identity, and the allowance could be bypassed five ways.

Then, a standing panel (29 October). On an AI product course I'm taking, I had Claude write five general-purpose reviewer agents: security, privacy, cost, QA and analytics. That session understood agents but not Fold, so I brought them into Fold's own project and asked for them to be reviewed and refitted with full context. The refit added what generic reviewers miss: guests (whose names and dietary needs a household records though they've never signed up for anything), first use as an empty household (the riskiest assumption), and what a browser test can't check on an iPad. It also added a sixth reviewer, because none of the five checked whether the recipe data was true. For a product whose promise is one meal for a mixed table, that's the failure that matters most.

How it works

• Fresh context, so the builder never grades its own work. Each reviewer is a separate agent that starts with no memory of how the change was built, so it hasn't absorbed the reasoning that produced the code, though, built on the same family of models, it can still share their blind spots.
• Report, never fix. Each is instructed never to edit the repository, and the privacy reviewer has no shell at all; security and QA must also leave git status exactly as they found it. Every report goes back to the building session, which fixes the issues and records each fix against the finding it closes. Several reviews are then re-run on the fixes.
• “Nothing wrong” is an allowed answer, and every reviewer must say what it didn’t check. Many reports also list what they checked and found sound alongside their findings, and every reviewer must then say what it didn’t look at. A reviewer rewarded only for finding problems will find problems, which is Goodhart’s law in miniature, so a clean result has to count as a proper outcome. Uncertainty goes in its own place rather than being dressed up as a finding.
• Six domains, each born from something real. Spend and isolation (the 24 October findings); fail-open guards (the save check); outsiders' personal data; free quotas (Actions minutes ran out on 26 October, deploys on 28 October); evidence (my product notes had no success or stop rules); and recipe truth.
• Judgement where it matters. Security and privacy run on Opus; the rest on Sonnet.
• Local, not in CI. They run as Claude Code subagents, so they use none of the Actions minutes or deploys that had already stopped Fold that month. They do draw on the same limited Claude usage as the building, so each run picks a depth: a light diff review for ordinary changes, a full review before milestones such as the first outside household, or checks-only for small fixes.
• Findings have teeth, and a person makes the calls. A Critical or Blocker stops the merge unless I accept the risk in writing, with the reason logged; that rule lives in Fold's instructions to the builder, not in a CI gate. The cost auditor, which first ran on 1 November, has no such grade, so it advises, and if it recommends a cheaper model or a shorter prompt for reading recipes, it must say the change has to be rerun against the allergen test set and the model comparison first. Each privacy review ends by separating what the agent could settle from what needs a human decision, and changes to stored data wait for my word whatever the reviews say.
• Never defer security or privacy on anything that lets an outsider in. Other reviews can be owed to the next session; these can't, because the harm would land on people outside the family.

A real review, abridged

Security reviewer, the change that removed allergy data, 29 October, diff depth

Verdict. Merge after fixing the backup guard and S-02, plus Emily's sign-off as a stored-data migration.

S-02 · Medium · Stripping on start restamped possibly stale items "now".
Why it matters: a device that hadn't synced for a while could overwrite another device's newer change: for example, a guest added on a phone, lost to an old laptop's people list.
Fix: clean the mirror with the stamps it had, and don't restamp; the server cleans its own copies.

Not checked:

• dependencies (unchanged);
• the accounts database, the admin page, spend and the bundle (not touched);
• the app beyond sync;
• the full tests and the real-data test.

Follow-up, the same day, by the building session: the mirror is cleaned with its stamps before comparing, with no restamping. Tested: a stale device doesn't overwrite a newer guest.

Every report has the same shape: findings with evidence and a suggested fix, then what the reviewer didn't check.

What the reviewers caught on day one

On 29 October, after a first full review of the beta's accounts work, four of the six reviewers ran across four pull requests: two rounds on the "Your week" redesign, a set of fixes to Know-how (Fold's cooking-technique pages), and the change that removed allergy data. The cost and recipe reviewers first ran on 1 November.

Bugs that had passed the build and the tests. By the building session's own run, the code built, typechecked and passed its tests, yet the QA reviewer, probing real scenarios, found problems they had missed. A vegan's own dish was left orphaned after their name was unticked and re-ticked, which put them back on a non-vegan dish containing milk: precisely the failure Fold exists to prevent. During the rollout, an older device's shopping list would have added non-vegan ingredients to a dish set to vegan for everyone. None of these was exotic. Each was an interaction between features that I, having designed each feature separately, hadn't thought to test together.

Guards that would never have fired. The security reviewer showed, with a probe, that a new safety check on the nightly backup ("throw if allergies remain") could never trigger, because it searched the data in the wrong form. It wasn't the first fail-open guard: the save check that reported "no errors" whenever it crashed was one of the reasons the panel exists.

Silent data loss. Stripping old data on start-up restamped it as new, so a device that hadn't synced for a while could overwrite a guest added on a phone.

Privacy leaks in the tooling, not just the app. Two one-off debugging scripts meant to write outside the repository could be tricked into writing every household's data inside it, between them through a relative path, a symlink or the Mac's case-insensitive file system. Separately, one-off guests, promised to be "just this meal", would have been copied into saved usual meals.

A broken evidence pipeline. The analytics reviewer found that the redesign had removed the only code that logged the event behind one of Fold's key measures, so that measure would have gone silent with nobody noticing. It also caught moves being double-counted and new events with no test proving they fire.

Small content errors with real consequences. In Know-how, a pattern matching "oil" also matched "Boil", and "flour" matched "cornflour", so a meringue diagram claimed to involve fat and flour. The QA reviewer caught it in a quick checks-only run.

The findings described here were fixed, or closed by a decision I recorded. Findings that are still open aren't described; those from the first full reviews are counted further down.

The finding that changed the product

The most valuable finding wasn't a bug. The privacy reviewer flagged a new line in the interface, "No one's allergies are affected", which made a safety claim when only 134 of 486 ingredients carried allergen data, with no tested reference set behind it. Rewording the line would have been the quick fix. Instead it prompted a product decision: Fold stopped holding allergy information about people altogether, and shows allergens as a label on each recipe, with "check the recipe and labels". The same day, the building session counted the real store and found no allergy data at all, which confirmed what the decision had assumed: no one in the family had added any. The decision log records the condition under which it would be reversed. Removing the risk turned out to be better than trying to manage it.

The questions only a person could answer

Each privacy review ended with a short list of things the agent couldn't settle. Three from day one: Was an irreversible deletion acceptable? It was, because it removed allergy data and the store turned out to hold none. Should guests be nameless by default? Yes: a one-off guest is now "Guest 1", with a name optional, to limit the personal data Fold collects, and only a returning guest needs to be remembered. Does being vegan count as protected belief data? That one isn't for a product decision to settle. In the UK, ethical veganism has been held to be a protected philosophical belief under the Equality Act, and data revealing a philosophical belief is special category data under UK GDPR, so the answer shapes how Fold stores a household's diets. I haven't taken legal advice on it yet; it's on a list of questions for a legal view before Fold opens beyond my family.

The first review that day, a full one of the beta's accounts work, found more. Security found 13 issues (2 High, 5 Medium, 6 Low) and privacy 15 (1 Critical, 4 High, 6 Medium, 4 Low), across isolation between households, fail-open guards, sign-in abuse, and personal data in logs and backups. The one Critical is moot now that allergy profiles are gone, and the rest are waiting for my decisions; fixing them is a condition for inviting friends' households. What the security review checked against stand-ins and found sound can be published now: row-level security on all 14 tables, invite-only on four layers, signed account cookies re-checked against active membership, and no secrets in tracked files.

The limits, and the part that still relies on trust

• What they can't see. The QA reviewer drives Chrome, but Fold's cooks mostly use Safari on an iPad. Each report lists what it couldn't test, and when a change touches what only an iPad can show, the QA reviewer turns it into numbered steps I can check in under five minutes.
• They're not infallible. One reviewer reported a paper-coloured background as a bug; the builder checked the CSS and a fresh screenshot before dismissing it. Findings are evidence to be checked, not verdicts to be obeyed.
• They cost something. Every review draws on the same Claude usage as the building, which is why depth is chosen per run.
• Nothing technical makes them run. The session building a change decides when its own critics run, and "stops the merge" lives only in the project's instructions. It began as a rule rather than a gate. Whether a rule is enough is still open.

What I learned

• Passing tests is not the same as working. Every serious bug on day one passed the build, the typechecker and the tests. The reviewers found them by trying the scenarios a household would actually create.
• Generic reviewers check generic risks. The domain risk (is the vegan route actually vegan?) only appeared when a reviewer with the product's context looked.
• A check that fails open is worse than none. The save check taught me that; the same week the security reviewer found another fail-open guard in brand-new code, which shows the principle has to be checked, not just stated.
• Privacy risk lives in the tooling as well as the product. Debug scripts and backups handle real data too.
• Removing a risk can beat managing it. Dropping allergy data traded a feature for trust.
• Attention is a budget. Depths, owed reviews and local runs exist because my Claude usage, my deploys and my CI minutes are all finite.

If you're working out how to check AI-built work without slowing it down, I'd be glad to compare notes. Get in touch.