Is It Right? (The Wrong Question to Ask of AI)
The Problem
Software is supposed to do the same thing every time. Same input, same output. So when AI answers the same question two different ways, it feels broken. People respond in one of two ways. Some check everything, by hand, forever. Others decide it's been fine for a while and stop checking at all. Both are managing AI like software, when it now behaves much more like a factory: the output varies. And when something does go wrong, the instinct is to tinker. Change the instructions after a bad week, then change them again. Half the time you're fixing things that weren't broken, and the answers get worse.
The Principle
Ask whether it's within tolerance, and how you know. Nothing is right every time, in a factory or in AI. Engineers who make real things never expected two bolts to come out the same. They decide how much variation they can live with, measure a sample instead of everything, and chart it over time. Walter Shewhart and W. Edwards Deming taught us to tell two kinds of wobble apart. Common cause is the process being a process. Special cause is one specific thing breaking. Treat everyday wobble as a break and you start fiddling. Treat a real break as wobble and you miss it. Seen this way, an eval is just quality inspection. Keep a set of questions with known right answers and run them every time a skill changes. Then sample what the agent does in production, score it against a short checklist, and put the scores on a chart.
5 Steps to Take Action
- 1
Pick one agent
One that talks to customers, runs at real volume and isn't the riskiest. Name an owner.
- 2
Write a short checklist
Five to seven yes-or-no checks a good answer has to pass, short enough to score one in under a minute. Write it with someone from the front line, not just Legal.
- 3
Collect a baseline before you set limits
Sample twenty to fifty answers a day for three or four weeks. An AI judge can help score them, and a person makes the call. Only then draw the center line and the limits.
- 4
Decide what a break sets off
Three dots in a row below the line, or one far outside it, means someone looks within the hour.
- 5
Fix the skill, not just the answer
When the chart catches something, add it to the test questions, so the next time it fails in testing instead of in front of a customer.
💡 Inspired By
Chapter 15 of The Second Factory, "Tolerances." And Walter Shewhart and W. Edwards Deming, who taught manufacturing to tell everyday wobble from a real break.
This is what Analytics Pitfalls & Principles is about: people helping people.
In Practice
Teresa Vang spent twenty-two years making bolts for airplanes, then took a job as the quality engineer at the small factory that makes ClassicMotors' parts. For two years almost nobody listened to her. Every bad week, the machinists adjusted something, then something else, and the parts got worse. She set a bolt on the break-room table. "No two of these are the same. Not one. Ever." Then she drew a chart on a napkin: a line down the middle, limits above and below, dots wobbling between them. "If a dot jumps outside these lines, or seven in a row creep up on one side, something specific has gone wrong. And now you know where to look." Driving home with the napkin on the passenger seat, Nora Berg remembered the head of IT months earlier, frustrated, when the AI first started making things up. "Software's supposed to do the same thing every time. You put the same thing in, you get the same thing out. That's the whole point of software." Teresa had never once expected two bolts to come out the same. The real question was hers. Is it within tolerance? And how do we know?
