Ordinary tests tell you whether software ran, not whether its output was any good, and that gap is where most of the real risk in an AI product sits. This thread follows my attempts, from Launchling's scoring system in 2025, which grew until it couldn't finish a run, to the guide I'd give a team now.
Knowing if it's any good
What does “good” mean for output nobody can mark right or wrong?
evals · LLM-as-judge · faithfulness · golden sets
Builds
- EXP-009Building
Mullo
The best page, the worse journey: when an AI tool's quality isn't worth the effort
An AI that draws my notes as an illustrated page. Against a general-purpose tool it made the better page but took far more effort, so I'm redesigning it around what only it does.
- EXP-002Retired
Launchling
All tests passing: when checking an AI product outgrows the product
An AI tool that turned a founder's rough idea into a plan. The product ran end to end; the evaluation I built to check it grew until it couldn't finish a run.
Notes
- Diverse by prompt, biased by role: how AI illustration passes society's stereotypes onhuman-in-the-loop · trust calibration · provenanceNote
- How I Used AI Pair Programming to Build a Robust Evaluation System (Without Vibe Coding)agentic engineering · multi-agent workflows · coding agents2025 post
- From Side Project to Stress Test: Using AI to Evaluate TinyLaunchevals · LLM-as-judge · faithfulness2025 post