Builds / EXP-002

EXP-002● RetiredPrototypeA working prototype with real users in July 2025. It stopped there: releases waited on an evaluation that never ran end to end again, and by 2026 the gap it filled had closed.▸ Under the hood

All tests passing: when checking an AI product outgrows the product

In 2025 I built Launchling, an AI tool that turned a founder's rough idea into a personalised plan, writing the code with Cursor and ChatGPT. The product ran end to end. The evaluation system I built to check it grew until it couldn't finish a single run, and I never had a reliable figure for what a plan cost. Here's what broke, why I'm retiring it rather than rebuilding it, and what I do differently now.

Launchling · Jun 2025–Jul 2026

The question

Could AI help a first-time, non-technical founder turn a rough idea into a plan they'd actually act on, and how would I know whether its advice was any good?

Started
Jun 2025
Ended
Jul 2026
Status
Retired
Readiness
Prototype
Tools
React + Vite · Vercel · OpenAI API · Airtable · Cursor · GitHub Actions · the stack

Why I retired it

The gap it filled has closed: a general-purpose model now turns a rough idea into a decent personalised plan in one prompt. Its evaluation never ran end to end again, so releases waited on it, and I let the launchling.io domain lapse on 13 July 2026.

Under the hood

Timeline. Started in June 2025 as a no-code tool: a Tally form and Zapier, then a branded version in Framer. The React rebuild began on 10 July 2025 and reached its first users within a few days; it was renamed from TinyLaunch to Launchling in mid-July. Evaluation work began on 13 July and ran to 3 August. The coded version came to 437 commits in all, on 22 days. The domain lapsed in July 2026, and the app is no longer reachable.

Stack

  • Front end: React, Vite and TypeScript, with React Router; Tailwind tried first, CSS Modules at launch
  • Back end: Vercel serverless functions in TypeScript and JavaScript, hosted on Vercel
  • AI: OpenAI GPT-4o in two calls per plan, the plan itself and then how the recommended tools work together (switched from GPT-4 on 12 July 2025)
  • Storage: Airtable for plans, feedback, evaluation records and token usage
  • Guardrails: a limit of three plans an hour from each IP address, and a notice on the feedback form saying what the email address would be used for
  • Evaluation: Python (pytest, Pydantic, Jinja2 prompt templates), GPT-4o as the judge, versioned prompts and rubrics, real and synthetic test cases, and GitHub Actions running the evaluation tests on pushes to main and feature branches
  • Email and analytics: a custom-domain address (moved from Resend to Nodemailer) and Google Analytics
  • Built with: Cursor and ChatGPT

Demo. No recording survives; the build log describes the product and the stack.

Launchling was the first product I coded myself with AI tools, and I think it's more useful to me now as a record of what building with AI was like in mid-2025 than as a product. I'm not relaunching it. I'd rather explain honestly what it was, how far I got, what went wrong, and why the gap it filled has since closed, because each of those taught me something I now use every week.

What it was for

Launchling (originally TinyLaunch) helped people with an early idea, particularly non-technical solo founders and people the startup world doesn't usually make room for, turn it into a clear, personalised plan: what to build, for whom, what to test first, and which tools suited their skills, budget and confidence. It also had a Risk Mapper to surface the assumptions most likely to sink an idea. The point wasn't to generate documents faster; it was to take away some of the advantage enjoyed by people who already know how product development works.

How the stack grew

  1. Prove it: a Tally form and Zapier, with AI generating the plan. No code.
  2. Make it feel real: Framer for a branded experience, until I hit the limits of what it could do dynamically.
  3. Own it: a React, Vite and TypeScript front end, with Tailwind tried first and CSS Modules at launch (July 2025), Vercel serverless functions in TypeScript and JavaScript, GPT-4o in two structured steps (the plan, then the tool stack), and Airtable for storage. Built mostly in Cursor.
  4. Make it trustworthy: a separate Python evaluation system (pytest, Pydantic schemas, Jinja2 prompt templates, versioned prompts and rubrics, real and synthetic test cases), writing its results to Airtable and local files.

Every step was a reasonable decision for a solo builder in 2025. The problem is that each one added software I then had to understand and maintain, and by the fourth I wasn't running a React app that called GPT-4o; I was coordinating two languages, several places that each held their own idea of what a plan looked like, versioned prompts, two kinds of test data, logging and several external services.

What worked

I think it's worth saying clearly, because the ending can make it look as though nothing did. The core flow worked: a founder could answer the onboarding questions, GPT-4o generated a plan and tool recommendations, and the result was saved. I tested it with deliberately absurd ideas, French pirate founders included, and designed it to handle them gracefully. Early versions of the evaluation framework ran and scored plans, in two early runs in July 2025: 11 real plans on 13 July (published at the time: personalisation 4.36, tool fit 4.09, help alignment 3.73), then a larger set of scores, committed on 19 July, that included those 11 (overall 3.98, tool appropriateness 4.08, personalisation 4.08, help alignment 3.94). Neither came from the later full system, which never ran end to end again. I never checked the judge against my own scoring, which is now the first thing I'd do. The thinking behind the evaluation, that AI output needs judging on whether it's actually useful to this person and not just on whether the API returned something valid, is the part I'm proudest of, and it still holds.

Where it broke

As the evaluation system grew, it stopped saving its results properly to Airtable, even though the logs showed it running. Over five days at the end of July and start of August 2025 I tried to get a complete, repeatable evaluation run working again, and couldn't. Between 31 July and 3 August it made 258 run folders and saved a result in none of them (the code made a folder whenever it was loaded, so that isn't a count of runs), and its last summary, on 4 August, records no plans evaluated. I had 33 real plans and 96 synthetic ones to evaluate, but every fix exposed the next problem: scripts that couldn't find each other, configuration one part expected and another didn't supply, duplicate modules and circular imports that stopped the tests loading, mocks of Airtable and OpenAI that no longer matched the real thing, and metadata that existed in the logs but vanished before it reached the results.

Looking back, almost none of these were problems with the AI's output; they were broken agreements between parts of my own system. Each AI-suggested fix was locally sensible without anyone, me or the tool, holding a picture of the whole, so the next missing piece surfaced straight away. The pattern I'd recognise now is that I'd built a sophisticated platform for evaluation before I'd proved the simplest version of it: one plan, one rubric, one result, saved somewhere I could read it.

The uncomfortable part is what I was measuring by the end. I'd started out wanting to know whether Launchling's advice was good, and by August I was counting passing tests. In a product team I'd call that substituting activity for evidence of the outcome, and I did it to myself.

I couldn't see what it cost

Separately, and just as stubbornly, Launchling's AI token usage reached Airtable, but never reliably. I spent days on it with both ChatGPT and Cursor, and it never logged reliably. It mattered more than it sounds: I was paying for every generation and every evaluation run myself, and without reliable usage records I couldn't say what a plan cost to produce, what a batch evaluation would cost, or whether a prompt change had made things more expensive. I was building an AI product without dependable sight of either of the two things that decide whether one works: the quality of what it produces and the cost of producing it.

I still don't know the cause. The records that did arrive miscalculated and often restarted from zero, and when a response carried no usage data the logger wrote a stand-in record (2,000 tokens, one cent), so the table mixed real usage with placeholders. It made a second problem worse: Cursor occasionally ran OpenAI calls I hadn't approved, until I stopped it running scripts at all, and I couldn't see what those cost either.

Why it has no place now

The real reason I'm retiring it is that the gap it filled has closed. The capability Launchling packaged, turning a vague idea into a structured, personalised plan, is now something general-purpose models do well on their own. The people I built it for can get most of what it offered by asking Claude or ChatGPT directly. I think the defensible version of the idea would be something quite different, helping a founder establish whether an assessment is true rather than generating one, and that would be a new product rather than a rebuild. I've written about what makes an AI product durable when the models keep improving, in The Moat, Not the Feature; Launchling is my own worked example of the opposite.

What carried forward

Quite a lot. The instinct that AI output needs judging on usefulness, not just validity, became my guide to evaluating AI output, rewritten a year later with everything Launchling taught me. The failure to see cost became a rule for Fold: it must never be loss-making, and it has a dedicated cost reviewer, which first ran on 1 November. The broken agreements between parts of the system are why Fold's reviewers each start from a fresh context, and why each of them has to say what it didn't check.

If you built something with AI in 2025 that you'd now retire, I'd be glad to compare notes. Get in touch.

Then / Now

Aspect2025October 2026
Building the front endFramer, then React by hand with Cursor, many rounds of iterationA Claude agent in the Claude app built emilysas.com, CMS included, in about 42 hours elapsed from first commit to final launch change
Debugging across filesPaste an error into a chat, apply a local fix, meet the next errorOn Fold, an agent reads the repository and runs the type check, the tests and the build before it merges a change
Shared definitionsHeld in several places, which drifted apartFold's six reviewers each start from a fresh context and must say what they didn't check; fixes are logged against findings, some re-reviewed
EvaluationBuilt the platform before proving one case end to endMy rule, only partly applied so far: one case end to end first, then grow it (my guide)
Cost visibilityDays spent; token usage logged, but unreliablyFold logs cost per call, with an allowance and a daily cap that fail closed; costs still come from a price table, not yet reconciled with billing
StorageAirtable in the critical path of every evaluationA real database planned for Fold, with row-level security on all 14 tables; built and tested against stand-ins, not yet switched on
The product itselfA personalised plan, packaged as a productAnyone can get a decent plan from a general-purpose model in one prompt

Who did what

I framed

  • The problem: people with an early idea, especially non-technical solo founders and people the startup world doesn't make room for, stuck before the first step
  • The aim: learn AI product management by building a real product with real users, and help those founders take a first confident step
  • The test of quality: is the advice useful to this person, not just valid output, judged on tool fit, help alignment, clarity and personalisation
  • Breaking it on purpose: absurd, manipulative and unethical ideas, to check it would guide rather than refuse or play along
  • Synthetic test cases of three kinds (variations, personas and edge cases), to see whether a prompt change helped the people I was building for

I set up the team

1 reviewer · 1 builder session · 1 planning session · 1 model in the product · 1 other · 3 people · 6 standing rules · 6 constraints

See the team ↓

I decided

  • To rebuild in React rather than pay to stay on Framer
  • CSS Modules over Tailwind, and GPT-4o over GPT-4
  • To split generation in two: the plan, then how the tools work together
  • To rename TinyLaunch as Launchling
  • To put the Risk Mapper behind quick feedback and an email address, for better signal on what worked
  • To store every choice a user makes as a stable code, never the wording they see
  • GPT-4o as the judge, and freezing the Airtable schema in a file, with a check script I could run locally
  • To stop building the evaluation for a day and give the tools structure first: goals, acceptance criteria, a runbook and a test plan
  • A Safe Mode rule for Cursor, and blocking test runs before any call to OpenAI
  • To stop Cursor running scripts at all, after it made OpenAI calls I hadn't approved
  • To retire it rather than relaunch or rebuild it

Cursor and ChatGPT built

  • Cursor: most of the code, including the front end, the serverless functions, the tests (110 passing across 22 suites by 18 July 2025) and the evaluation pipeline
  • ChatGPT: strategy, Vercel configuration, architecture advice and prompt design, often as the instructions I gave Cursor
  • GPT-4o: every plan and tool explanation from 12 July 2025, and every score in the evaluation
  • Proposed, and I approved: clearer documentation, better architecture, more helpers and stronger test coverage, when I asked Cursor what it needed to work better
  • Cursor's agent: a security assessment and action plan, on 29 July 2025

The team

How the team workedMe at the top, then the people working with me, then lanes for Plans, Builds, Reviews, In the product, Other. Arrows show briefs from me to Plans; specs from Plans to Builds; changes from Builds to Reviews; findings from Reviews to Builds; calls from Builds to In the product; questions only I can answer from Reviews to me. Inside 6 standing rules and 6 constraints.How the team workedMe at the top, then the people working with me, then lanes for Plans, Builds, Reviews, In the product, Other. Arrows show briefs from me to Plans; specs from Plans to Builds; changes from Builds to Reviews; findings from Reviews to Builds; calls from Builds to In the product; questions only I can answer from Reviews to me. Inside 6 standing rules and 6 constraints.

Agents and models

NameRoleModelWhat it was for
ChatGPTPlansChatGPTStrategy, architecture, Vercel configuration and prompt design; often wrote the instructions I gave Cursor
CursorBuildsCursor (model not recorded)In-file code, refactors, tests and the evaluation pipeline; its agent also wrote a security assessment
EvaluatorReviewsGPT-4oScored each plan for tool fit, help alignment and personalisation, plus an overall score, as JSON with reasons
Plan generatorIn the productGPT-4oTwo calls per plan: the plan, then how the recommended tools fit together. GPT-4 until 12 July 2025
Synthetic plansOtherGPT-4oPlans for controlled test cases (variations, personas and edge cases), kept out of the real plans table

People

  • Me Everything outside the code: the idea, the prompts, the evaluation design, testing, and every decision
  • A friend Wrote absurd and adversarial ideas to try to break it
  • Early users Used it from mid-July 2025; 18 of the 33 real plans came from other people

Standing rules

  • Dry run by default: no Airtable writes, shell commands, commits or file changes without my confirmation
  • Cursor doesn't run scripts; I run them in my own terminal
  • The files that generate plans, map fields or write to Airtable change only with my approval
  • Synthetic tests stay out of the real plans table
  • Every choice a user makes is stored as a stable code, never the wording they see
  • Ethical safeguards guide an idea rather than reject it

Constraints and unblocking

  • My own money for every plan and every evaluation run: $12 of OpenAI usage in all
  • Three plans an hour from each IP address in the coded app; Zapier throttled use in the no-code version
  • Framer's free tier, which I quickly outgrew: the features I needed would have meant a paid plan
  • Solo and part-time, with Python that was rusty after years in product
  • Unblocked: a launchling.io domain and a custom-domain email, with DNS, SPF and DKIM set up by hand after the first provider failed
  • Unblocked: my own OpenAI API account, paid as I went

Reviewers

  • Me Read the plans, the scores and the feedback, and decided what changed
  • A friend Adversarial and absurd ideas

Tools

  • Tally and Zapier The first, no-code version
  • Framer A branded version, until its limits and pricing bit
  • Airtable Plans, feedback, evaluation records and token usage
  • Vercel Hosting and the serverless functions
  • GitHub Actions Running the evaluation tests on pushes to main and feature branches
  • Google Analytics Which pages and plan sections people used

Ledger

ItemValueBasisNote
Time10 July to 3 August 2025: 437 commits on 22 daysMeasuredElapsed time, from git history, for the coded version. The no-code version from June isn't in git, and I didn't record hours worked.
Spend$46.98 (about £35)MeasuredFrom my own records: $12 (about £9) of OpenAI usage in July 2025 and $34.98 (about £26) for the launchling.io domain. Nothing else was paid for.
Cost per planUnknownMeasuredUsage was logged, but unreliably: records miscalculated and often restarted from zero. Why.
Results in the run folders, 31 July to 3 August 2025NoneMeasuredThe system made 258 run folders in those days, but it made one whenever its code was loaded, tests included, so they don't count real runs. None holds a result, and the last summary, on 4 August, records no plans evaluated.
Plans33 real, 96 syntheticMeasuredCounted from the database. Of the real plans, 18 came from other people and 15 from me or friends, mostly genuine ideas rather than tests.
Feedback form responses20, of which 19 came from my own email addressesMeasuredFrom the feedback form on every plan, 11 to 18 July 2025, on 18 plans. With only one response from anyone else, they can't be read as early users' verdicts.
Measured
counted or timed directly
Modelled
estimated from a stated method
Upper bound
the most it could be
Estimated
my own estimate, not measured

Time figures say whether they are elapsed or worked. Time between commits is never presented as hours worked.

Non-sterling costs show the amount paid, with an approximate sterling figure at that month's average rate. Rates from HMRC's monthly exchange rates.

Outside the lab

What this would take in an organisation.

In an organisation, the same failure looks like an assurance process that grows faster than the thing it assures: more gates, more tooling, more reporting, until the team is maintaining the governance rather than learning anything from it. What I'd ask of any evaluation or approval process now: can it run end to end on one real case today; does it report quality and cost separately from "the checks passed"; and if it breaks, does it fail in the direction you've chosen?

The original notes

Series: Launchling 2025