Coming next: The Second Factory →

Back to Library
Data Foundations⚠️ Pitfall

You Can't Test Drive a Clay Car: Why Fake Data Kills Innovation

The Problem

Your team wants to innovate. Build new models. Test AI capabilities. Leadership says: "Great! Set up a sandbox with fake data." Six months later, you've built pipelines that work beautifully on made-up scenarios. Models that perform perfectly on fake problems. Dashboards that look impressive in demos. Then you point it at production data and everything breaks. Why? Because the innovation isn't in the pipeline you built. It's in the **outcome** that pipeline produces. And you can never see real outcomes with fake data. It's like sculpting a sports car out of clay and thinking you get to drive it when you're done.

The Principle

Real innovation requires real data, secured properly. You don't need fake data. You need production data with proper access controls, data masking where necessary, and security that actually works. The risk isn't using production data. The risk is building solutions on fake problems and discovering they don't solve real ones.

🎯

3 Steps to Take Action

  1. 1

    **Build Your Innovation Pipeline on Production Data** Use actual customer behavior, real transaction patterns, genuine edge cases. Implement data masking for PII/sensitive fields where required. Set up proper access controls and audit logging. Test on the data you'll actually use.

  2. 2

    **Stop the Fake Data Theater** Before investing in fake data generation, ask teams: "How will you use this?" If the answer is "sounds cool, but we're good with test subsets," skip it. Fake data takes 6+ months to build and nobody ends up using it. Use anonymized production samples instead.

  3. 3

    **Secure Production, Don't Avoid It** Invest in proper data governance and access controls. Implement column-level encryption and row-level security. Use data masking policies for sensitive fields. If you really need to obfuscate, de-identify the data—it's still real information, still behaves like production, just without the PII. Build secure innovation environments, not fake ones.

💡 Inspired By

Dean Furness, who's been expanding the halo effect for 15 years—sharing insights, challenging assumptions, and helping teams cut through the fake data theater to build real innovation on real problems. This is what analytics teams need: people helping people.

This is what Analytics Pitfalls & Principles is about: people helping people.

In Practice

A Fortune 500 company spent six months building a "sandbox" with fake data for their AI innovation team. They generated synthetic transactions, made-up customer behaviors, perfectly balanced datasets. The models worked beautifully. Accuracy scores above 95%. Leadership was thrilled. Then they pointed the models at production data. Everything collapsed. The fake data had no edge cases, no data quality issues, no real-world chaos. The models had learned to solve problems that didn't exist. They scrapped the entire effort and started over—this time with properly secured production data, masked PII, and real problems. The new models weren't as "accurate" on paper, but they actually worked in the real world. The lesson: You can't test drive a clay car. Innovation needs real data, secured properly.

More on Data Foundations

From the book

In the book, a new hire takes six months to become useful, mostly because of table names nobody can read. And there are five definitions of an active seller.

This card is one of 57 drawn from The Auction Block by Brian Cariveau — the story of an analytics leader with 90 days to fix what the cards describe.

Each leadership pattern falls into a few pitfalls most often. Find your pattern.The Analytics Leadership Mirror: free, 5 minutes, no account