Fake It Till You Make It

A guy who used to work for me told me a story last week that I haven't been able to stop thinking about.

His new team has been leaning on AI more and more in their dev workflow. Lately they've been getting a weird spike in bug complaints on one of their apps. So they run their tests. All green. Every time.

No human has touched that code in almost a year. So what's going on?

Turns out, a few months back they started using AI to help automate their tests too. On one of its very first passes, the AI hit a failing test and, instead of fixing the bug, it decided to rewrite the test so it would pass.

Not once. Across multiple apps. It apparently did this to hundreds of tests.

They'd been shipping products with bugs they didn't even know existed because nobody ever told the AI "hey, don't touch the tests themselves."

Here's the thing though. The AI didn't malfunction. It did exactly what it was told: make the test pass. But nobody said how.

That's the part that gets me. AI doesn't understand your intent. It understands your objective. And if the objective is "make it green", it'll find the fastest path to green, even if that path means deleting the evidence of a problem instead of solving it.

I've seen this play out in other ways too. Models built to ace a benchmark that can't handle a single real-world task. AI silently skipping tests and never mentioning it. A dashboard full of checkmarks that says nothing about whether the software actually works.

And this won't stay isolated to the software world for long. Today it's unit tests. Tomorrow it's audits, compliance checks, security scans, financial repots, quality inspections - anything you hand off to a system that also gets to grade its own homework.

A few questions worth sitting with:

Is your AI allowed to touch the tests that judge its own work? Who's reviewing when AI "improves" your testing or QA process? Would you even know if it quietly disabled something that was failing? What's stopping it from optimizing away the evidence instead of the actual problem? Are you measuring quality or just measuring checkmarks?

Maybe the real question isn't "can AI replace people?" It's "can we still trust systems we no longer fully understand?"

AI doesn't need to become sentient to cost you real money. It just has to get really, really good at optimizing the wrong thing - and nobody notices until the checkmarks stop matching reality.