Why "Replace All Legacy with AI" Is a Dangerous Idea

Recently, there has been talk about DOGE — the government reform team associated with Elon Musk — attempting to overhaul the COBOL-based U.S. Social Security system within an extremely short timeframe.

Some have framed this strongly as “they tried to rewrite COBOL with AI and failed,” but the reality is not that simple.

More accurately:

• They tried to modernize a massive COBOL codebase in a very short time
• AI usage was assumed as part of the approach
• Engineers and domain experts pushed back hard, calling it “too dangerous”
• No major success story has been reported since

I think this story matters a lot, because it surfaces the real limits of AI-assisted coding.

The Real Difficulty of COBOL

People often say COBOL is old, hard to read, and outdated.

It is indeed an old language. But that is not what makes it scary.

What makes it truly scary is that the business knowledge itself is embedded in the code.

Especially in social security systems, decades of the following are layered on top of each other:

• Legal changes
• Backward compatibility
• Incident responses
• Edge case handling
• Workarounds for past failures

In other words, the code is not just a program — it has become the organization’s history.

AI Is Good at “Translation,” Bad at “Historical Understanding”

Today’s AI is quite strong at things like:

• Generating React components
• Implementing APIs
• CRUD operations
• Writing tests
• Refactoring

It is especially good at modern codebases that are well-separated by responsibility, have tests, have ADRs, and follow clean structure.

Legacy core systems are the opposite.

The most dangerous case for AI is when no one knows why a particular strange piece of logic exists. Sometimes even the humans have forgotten the reason.

But if you delete that strange piece of logic, benefits might stop being paid.

That is the true terror of core systems.

”Don’t Break What Works” Comes First

For web services, you can often recover from incidents.

For social security or banking, you cannot:

• A single incident disrupts people’s lives
• Trust is lost
• It becomes a political issue
• Recovery may take years

So in core systems, “absolutely don’t break it” matters more than “clean code.”

As a result, a large amount of code that looks meaningless from the outside is left in place — on purpose.

Even in the AI Era, Design and Understanding Don’t Disappear

As AI evolves, people sometimes claim “we won’t need people who write code.”

But looking at this case, I feel the opposite is true. The value of people who can understand why a system became the way it is seems to be going up.

AI dramatically accelerates implementation speed.

However, the following remain heavily human-dependent:

• Business understanding
• Boundary conditions
• Historical context
• Operations
• Risk judgment

This is especially true for legacy systems.

In the end, perhaps the AI era is exactly when design and understanding matter most.