CODE IN TOKYO

Heinrich's Law in the AI Era: The Rise of "Quiet Mistakes"

A few weeks ago, a story made some noise: Anthropic — the company behind the AI coding agent “Claude Code” — reportedly leaked parts of its own source code through a packaging mistake on npm.

We should be careful about the details and the actual impact. But the incident reminded me of something.

Even the people building AI ship accidents like everyone else.

And it’s not because they’re incompetent — quite the opposite. World-class engineers, with state-of-the-art security practices, still cannot prevent every mistake or every leak.

That’s the reality.

Heinrich’s Law: 1 : 29 : 300

These are the numbers from Heinrich’s Law, often cited in safety science and incident analysis.

Behind every major accident lie 29 minor accidents, and beneath those, 300 near-misses.

Anyone who has worked on systems for long enough understands this in their bones.

Production incidents rarely come out of nowhere. They are almost always the accumulation of small, ignored signals.

A warning that flickered briefly in the logs. An exception waved off as “probably fine.” Test results no one reads. A runbook only one person understands.

Those tiny pieces of noise eventually combine into a real disaster.

This is exactly why we built CI/CD, automated tests, and structured reviews — to stop relying on human attentiveness.

But recently, a new factor has entered the equation.

AI.

AI generates “95%-correct-looking” mistakes at scale

AI is convenient. There’s no denying it.

Coding speed goes up. Documentation gets faster. Research moves quicker.

But underneath all that, AI is also generating a terrifying volume of small mistakes in parallel.

And the trouble is the kind of mistake.

If the output were complete garbage, we’d notice immediately. The real problem is that AI produces mistakes that look about 95% correct.

The variable names look natural. The code looks like it would run. The design feels reasonable. The documentation reads fluently.

Look closer, though, and edge cases are missing. Authentication is shaky. Error handling is subtly broken. The specification has been quietly misinterpreted.

These are often harder to spot than messy code written by a tired human.

Heinrich’s pyramid, AI edition

In the original Heinrich framework, the 300 near-misses came from human inattention or aging equipment.

In the AI era, near-misses are different.

They are auto-generated as a side effect of efficiency.

Fast, in large volumes.

A 10x productivity boost from AI also means sloppy outputs flowing in 10x faster.

That is the scary part.

Harness design: treat AI as a high-speed automation, not a smart developer

Lately, “harness design” has become a serious topic.

Rather than simply “let AI write code,” the conversation now includes:

The framing has shifted: instead of treating AI as a “smart developer,” people are starting to treat it as a high-speed automation device that could go off the rails.

AI watching AI — and the rules around them

Modern AI development also routinely includes:

The “AI watching AI” pattern is becoming normal.

But the most important piece, in the end, is the outer rule design.

What permissions does it get? Which directories can it touch? Are production commands off-limits? How does the approval flow work?

Sooner or later, it all comes back to guardrail design.

Test design and review-friendly design — the comeback of old ideas

Something else that has quietly gained value in the AI era: test design and review-friendly design.

Historically, “implementation” was the main act. Tests and modular design were supporting roles.

In the AI era, that is flipping.

AI can generate implementations at scale. A thousand-line diff can land overnight.

When that happens, the things that hold real value are:

Property-Based Testing — generating large volumes of unexpected inputs to surface edge cases — turns out to fit the AI era surprisingly well. So does avoiding God Objects with old-fashioned modular decomposition.

In short, the design principles preached long before AI are being rediscovered for their value. No new principles are being invented here; the old textbooks are simply being opened again.

From “humans make mistakes” to “the automation makes mistakes”

The old line was:

“Humans make mistakes — let’s automate.”

The new line is going to be:

“The automation makes mistakes at scale — let’s design something to control it.”

I have to admit, it’s a little funny.

I thought handing things to AI would let me go home early. Instead I find myself attending what feels like a parent-teacher conference for the AI. I was supposed to make it write code. Somehow my hours as an AI monitor are the ones piling up.

But this is probably the infrastructure of the next era.

Ten years ago, CI/CD and automated testing were “early adopter culture.” Today, no serious team ships without them.

A few years from now, I suspect:

will all be the standard development substrate, and a junior engineer will ask, with a completely straight face, “Wait — you used to ship without this?”

The era of celebrating AI as pure convenience is already over.

What’s beginning now is a different game: who steers the car, and how, when it’s running on a very powerful AI engine.

How do we move forward while keeping Heinrich’s pyramid flat?

That’s the kind of thing I find myself thinking about lately.