Confident, With Formatting (When Nobody Reads What the AI Wrote)
The Problem
The memo is beautifully organized. The spreadsheet's formulas look perfect. The summary sounds exactly like someone who knows. And the memo quotes a refund policy your company has never had. The formulas are off by one row, so every total is wrong. The summary says two parts are the same when every expert in the building knows they aren't. AI doesn't say maybe when it doesn't know. It fills the gap with whatever sounds right, in complete sentences, with formatting. The engineers call it hallucinating. At a parts counter they called it guessing. Most people would catch the mistake if they read the thing. It just reads so well that nobody does, and then someone copies it.
The Principle
Your name is on it, so read it. Checking is faster than doing, and it is never zero. Treat AI like a really fast, really eager new hire who has read everything and done nothing. You wouldn't let a new hire send a memo to the whole company unread. Check numbers against their source. Tell it to say when it isn't sure, and to cite what it used. And make sure you could explain the answer yourself if someone asked you one hard question about it. If the document knows something you don't, you haven't really checked it.
5 Steps to Take Action
- 1
Read it before it goes
Anything with your name on it gets read in full before it leaves, especially the parts that sound most sure of themselves.
- 2
Tell it to say when it isn't sure
Add one line to every skill: if you're not sure, say so, and never guess about the thing that matters most. Ask for the sources behind every answer.
- 3
Check numbers against their source
Spot-check totals and dates by hand. One row out of place makes every total wrong.
- 4
Ask yourself one question
Before you send it, could you answer one hard question about it without the document in front of you? If not, you haven't checked it yet.
- 5
Ask who checked a skill before you copy it
A shared skill spreads its mistakes as fast as its answers.
💡 Inspired By
Chapters 3, 5 and 9 of The Second Factory. And every first AI answer that was wrong in a way that looked right, which is the most dangerous kind of wrong.
This is what Analytics Pitfalls & Principles is about: people helping people.
In Practice
In her first week of asking AI to do everything first, Nora Berg asked it to draft a policy memo on seller refunds. It wrote a confident, well-organized memo that quoted a refund policy ClassicMotors had never had. It had simply invented one that sounded right. The same week, it built her a spreadsheet whose formulas were off by one row. "It's making things up," she told her neighbor. "Confidently. With formatting." He told her to read it before it went out. "You're the human in the loop. Your name's on it." A few weeks later, the AI listened to Elmer Novak, thirty-one years behind a parts counter, for ten minutes and then summarized, brightly, that the 1966 and 1967 Mustang used the same door hinge. Elmer stood up to leave. "It didn't say maybe. It didn't say I think. It said it like it knew." Months later, when the company finally counted its agents, it found nine different skills explaining the refund policy, with three different refund windows. One of them was the policy the AI had invented for Nora in her first week. Someone had copied it.
