Coming next: The Second Factory →

Back to Library
Metrics & Evidence⚠️ Pitfall

Confident, With Formatting (When Nobody Reads What the AI Wrote)

The Problem

The memo is beautifully organized. The spreadsheet's formulas look perfect. The summary sounds exactly like someone who knows. And the memo quotes a refund policy your company has never had. The formulas are off by one row, so every total is wrong. The summary says two parts are the same when every expert in the building knows they aren't. AI doesn't say maybe when it doesn't know. It fills the gap with whatever sounds right, in complete sentences, with formatting. The engineers call it hallucinating. At a parts counter they called it guessing. Most people would catch the mistake if they read the thing. It just reads so well that nobody does, and then someone copies it.

The Principle

Your name is on it, so read it. Checking is faster than doing, and it is never zero. Treat AI like a really fast, really eager new hire who has read everything and done nothing. You wouldn't let a new hire send a memo to the whole company unread. Check numbers against their source. Tell it to say when it isn't sure, and to cite what it used. And make sure you could explain the answer yourself if someone asked you one hard question about it. If the document knows something you don't, you haven't really checked it.

🎯

5 Steps to Take Action

  1. 1

    Read it before it goes

    Anything with your name on it gets read in full before it leaves, especially the parts that sound most sure of themselves.

  2. 2

    Tell it to say when it isn't sure

    Add one line to every skill: if you're not sure, say so, and never guess about the thing that matters most. Ask for the sources behind every answer.

  3. 3

    Check numbers against their source

    Spot-check totals and dates by hand. One row out of place makes every total wrong.

  4. 4

    Ask yourself one question

    Before you send it, could you answer one hard question about it without the document in front of you? If not, you haven't checked it yet.

  5. 5

    Ask who checked a skill before you copy it

    A shared skill spreads its mistakes as fast as its answers.

💡 Inspired By

Chapters 3, 5 and 9 of The Second Factory. And every first AI answer that was wrong in a way that looked right, which is the most dangerous kind of wrong.

This is what Analytics Pitfalls & Principles is about: people helping people.

In Practice

In her first week of asking AI to do everything first, Nora Berg asked it to draft a policy memo on seller refunds. It wrote a confident, well-organized memo that quoted a refund policy ClassicMotors had never had. It had simply invented one that sounded right. The same week, it built her a spreadsheet whose formulas were off by one row. "It's making things up," she told her neighbor. "Confidently. With formatting." He told her to read it before it went out. "You're the human in the loop. Your name's on it." A few weeks later, the AI listened to Elmer Novak, thirty-one years behind a parts counter, for ten minutes and then summarized, brightly, that the 1966 and 1967 Mustang used the same door hinge. Elmer stood up to leave. "It didn't say maybe. It didn't say I think. It said it like it knew." Months later, when the company finally counted its agents, it found nine different skills explaining the refund policy, with three different refund windows. One of them was the policy the AI had invented for Nora in her first week. Someone had copied it.

Take it to work

Give this card to your AI

A skill is a file your AI keeps. Add this one and it knows Confident, With Formatting whenever your work runs into it.

Then ask it: “Does my plan fall into Confident, With Formatting?”

How to add it
  • Claude (a paid plan): Customize, Skills, +, Upload a skill. Pick the file you downloaded.
  • ChatGPT or Gemini: where your plan has Skills, upload the same file.
  • Microsoft 365 Copilot Cowork: Plugins, Add, Upload, then pick the Cowork file.
  • An AI at work that won’t take a skill? Open this card as text and paste it in.

All the AI skills, including ones for the moment before an AI, data or BI strategy, a dashboard or a readout goes out.

More on Metrics & Evidence

From the next book

In The Second Factory, Nora’s first week of asking AI first produces a refund policy her company has never had. Months later, she finds out where it ended up.

This card is drawn from The Second Factory, Brian Cariveau’s next novel, about a company learning to work with AI. The opening scene is free to read now.

Heard something like this at work? See the pitfalls behind what else you’ve heard this month.Heard at work: free, 1 minute, no account