course
Making Friends with Machine Learning
Why it stayed with me
A great way to ease into ML concepts, and really accessible. It brought home the importance of splitting your data, and what can go wrong if you don't. In a previous role, I stepped in to stop exactly that happening, thanks to Cassie hammering it home.
The ideas I kept
ML is a powerful genie, so the danger is the wisher.
ML is a thing labeller: it makes recipes from examples rather than following instructions. When the wish goes wrong, it's the fault of the wisher, so good decision makers are key, and that means a diverse team, not one person.
Start from what you'd like labelled, not from the technology, and remember that simple analytics is often cheaper. Data mining is for inspiration, not decisions.
Decide up front what good looks like.
Before building, write down the outputs, consider the mistakes and decide which are worse. A system with high precision might leave some good items out, but what it returns is high quality. A system with high recall returns most of the good items, along with a lot of duds.
Choose what matters for your case, and set the minimum performance to productionise and to launch before you start.
In the lab: How I evaluate AI output
Split your data, and keep the test pristine.
A student who passes by memorising the answers hasn't learned anything, which is why you split your data: like a professor setting an exam, you need fresh questions to check performance.
Look for what went wrong in a debugging set carved out of the training data, never while validating. Only a pristine test set tells you how well the system really works, so never test on data involved in training or validation.
Test honestly, and lean towards not launching.
Testing asks whether the evidence makes the null hypothesis, the world where you'd be fine with the default action, look ridiculous. Err on the side of not launching unless it does, judged against the tolerance for risk the decision maker set at the start.
If testing fails and you can't collect more data, kill the project. Start with the simplest model too, because the more complex the solution, the more likely it is to overfit.
Plan for it going wrong.
ML is the high-interest credit card of tech debt. The world you trained from is the world you'll serve best, so watch for training-serving skew, retrain offline away from users, and keep re-testing even when you're not retraining.
At scale, outliers will happen, so build safety nets: a policy layer that checks output before it goes out, small releases compared against the next best thing, and a way to revert. Good documentation lets future teams keep it working.
About the speakers
Cassie Kozyrkov Speaker
Cassie Kozyrkov was Google's first Chief Decision Scientist, where she founded the field of Decision Intelligence and personally trained more than 20,000 Googlers in AI and data-driven decision-making. She's the person I turn to on how data, AI and decisions fit together, and I recommend subscribing to her newsletter: Decision Intelligence.