Coming next: The Second Factory →

Back to Library
Metrics & Evidence⚠️ Pitfall

Is It Right? (The Wrong Question to Ask of AI)

The Problem

Software is supposed to do the same thing every time. Same input, same output. So when AI answers the same question two different ways, it feels broken. People respond in one of two ways. Some check everything, by hand, forever. Others decide it's been fine for a while and stop checking at all. Both are managing AI like software, when it now behaves much more like a factory: the output varies. And when something does go wrong, the instinct is to tinker. Change the instructions after a bad week, then change them again. Half the time you're fixing things that weren't broken, and the answers get worse.

The Principle

Ask whether it's within tolerance, and how you know. Nothing is right every time, in a factory or in AI. Engineers who make real things never expected two bolts to come out the same. They decide how much variation they can live with, measure a sample instead of everything, and chart it over time. Walter Shewhart and W. Edwards Deming taught us to tell two kinds of wobble apart. Common cause is the process being a process. Special cause is one specific thing breaking. Treat everyday wobble as a break and you start fiddling. Treat a real break as wobble and you miss it. Seen this way, an eval is just quality inspection. Keep a set of questions with known right answers and run them every time a skill changes. Then sample what the agent does in production, score it against a short checklist, and put the scores on a chart.

🎯

5 Steps to Take Action

  1. 1

    Pick one agent

    One that talks to customers, runs at real volume and isn't the riskiest. Name an owner.

  2. 2

    Write a short checklist

    Five to seven yes-or-no checks a good answer has to pass, short enough to score one in under a minute. Write it with someone from the front line, not just Legal.

  3. 3

    Collect a baseline before you set limits

    Sample twenty to fifty answers a day for three or four weeks. An AI judge can help score them, and a person makes the call. Only then draw the center line and the limits.

  4. 4

    Decide what a break sets off

    Three dots in a row below the line, or one far outside it, means someone looks within the hour.

  5. 5

    Fix the skill, not just the answer

    When the chart catches something, add it to the test questions, so the next time it fails in testing instead of in front of a customer.

💡 Inspired By

Chapter 15 of The Second Factory, "Tolerances." And Walter Shewhart and W. Edwards Deming, who taught manufacturing to tell everyday wobble from a real break.

This is what Analytics Pitfalls & Principles is about: people helping people.

In Practice

Teresa Vang spent twenty-two years making bolts for airplanes, then took a job as the quality engineer at the small factory that makes ClassicMotors' parts. For two years almost nobody listened to her. Every bad week, the machinists adjusted something, then something else, and the parts got worse. She set a bolt on the break-room table. "No two of these are the same. Not one. Ever." Then she drew a chart on a napkin: a line down the middle, limits above and below, dots wobbling between them. "If a dot jumps outside these lines, or seven in a row creep up on one side, something specific has gone wrong. And now you know where to look." Driving home with the napkin on the passenger seat, Nora Berg remembered the head of IT months earlier, frustrated, when the AI first started making things up. "Software's supposed to do the same thing every time. You put the same thing in, you get the same thing out. That's the whole point of software." Teresa had never once expected two bolts to come out the same. The real question was hers. Is it within tolerance? And how do we know?

Take it to work

Give this card to your AI

A skill is a file your AI keeps. Add this one and it knows Is It Right? whenever your work runs into it.

Then ask it: “Does my plan fall into Is It Right??”

How to add it
  • Claude (a paid plan): Customize, Skills, +, Upload a skill. Pick the file you downloaded.
  • ChatGPT or Gemini: where your plan has Skills, upload the same file.
  • Microsoft 365 Copilot Cowork: Plugins, Add, Upload, then pick the Cowork file.
  • An AI at work that won’t take a skill? Open this card as text and paste it in.

All the AI skills, including ones for the moment before an AI, data or BI strategy, a dashboard or a readout goes out.

More on Metrics & Evidence

From the next book

In The Second Factory, a quality engineer who spent twenty-two years making airplane bolts sets one on the break-room table and tells Nora that no two are the same.

This card is drawn from The Second Factory, Brian Cariveau’s next novel, about a company learning to work with AI. The opening scene is free to read now.

Heard something like this at work? See the pitfalls behind what else you’ve heard this month.Heard at work: free, 1 minute, no account