talk
Decision Making in the Age of AI
The ideas I kept
Two paths to trust, and it's safer to have both.
The most dangerous thing about AI is us. To avoid its pitfalls, have two checks and balances: a human in the loop, and testing with safety nets.
Decide how you'll score success before you start.
The Kozyr criteria ask what it's for (how success was defined and scored), what data it's using (how that data was collected and processed), and how we know it works (how and where performance was tested). A fourth asks how you'll build safety nets.
You can't know whether something is good enough to launch if you didn't decide how to score it before you began, and "better than human" isn't enough. As Cassie put it, "the world represented by your data is the only world you're going to succeed in", and it's very different from the world you live in.
Check the output, not just the input.
A policy layer, a bit like etiquette for your system, checks output and stops it being released. That's more reliable than trying to stop the wrong thing being processed in the first place.
Run the same task several times and see if you get agreement, and if the answers are volatile, route the case to a human.
In the lab: Mullo
Borrowing someone else's model is a risk.
If you deploy someone else's model to solve your problem, your vendor may start losing quality, and you need to be able to switch, which is tricky. You also need a plan for model collapse.
Organisations with enough cash flow, running mission-critical systems, should think about getting their own data.
About the speakers
Cassie Kozyrkov Speaker
Cassie Kozyrkov was Google's first Chief Decision Scientist, where she founded the field of Decision Intelligence and personally trained more than 20,000 Googlers in AI and data-driven decision-making. She's the person I turn to on how data, AI and decisions fit together, and I recommend subscribing to her newsletter: Decision Intelligence.