How a Docker Override Almost Wiped Out Our Development Database

The other day, I had one of those moments that makes you break into a cold sweat.

The database on our development server became empty.

Fortunately, we were able to restore it from a backup, so it did not become a major disaster. Still, checking what had happened and restoring the data took a lot of time.

When we traced the cause, it came down to a Docker Compose configuration change.

It Started with Windows Support

There are Windows users on the team.

If you have worked with Docker, you probably know that volume mounts can behave slightly differently between Windows and Linux.

This incident started as a response to that kind of issue.

The Docker Compose configuration was changed so that it would work in a Windows environment.

The person who made the change had no bad intent.

They simply wanted to fix their own environment. So they used an override.

The problem was that the change was committed to Git and then merged into main.

Linux Only Did What It Was Told

CI/CD pulled that configuration and deployed it to the development server.

Then the database container on Linux started looking at an unexpected data directory.

As a result, the database started up as if it were brand new.

Computers do not read the room.

They do not think, “This setting looks like it is for Windows, so maybe I should be careful.”

They simply execute what they are configured to execute.

And then only the humans panic.

At First, I Thought the Override Was the Problem

At first, I thought the problem was simple.

Someone committed a personal override configuration, and that was the mistake.

That was certainly one factor.

But after calming down and thinking it through, I realized the real issue was a little different.

Docker Compose has an override mechanism.

Separating personal settings into a different file is a normal practice.

So why did the incident happen?

The Real Problem Was CI/CD

Looking back, the CI/CD side had not fixed which Compose files it would use.

What would have happened if CI/CD had explicitly specified the file like this?

docker compose -f docker-compose.yml up -d

Even if someone had committed an override file, CI/CD would not have loaded it.

At the very least, the same incident probably would not have happened.

In other words, the problem was not that an override file existed.

The problem was that CI/CD had not clearly fixed which configuration it was supposed to use.

Development Environments Can Still Hurt

When you talk about this kind of issue, some people say:

“It is only a development environment, so why does it matter?”

I used to think that way too.

But once you are on the side handling the incident, your view changes.

You restore the data.

You check the behavior.

You explain the situation to the people involved.

You think about how to prevent it from happening again.

You end up spending hours on work that creates no real value.

It may not be as serious as a production outage, but it still hurts enough.

Lesson Learned

What I learned from this incident is simple.

Flexibility is important in local development.

But in CI/CD, flexibility becomes an enemy.

You should not leave room for configuration to change depending on the person or environment.

The Compose files used by CI/CD should be specified explicitly.

And personal environment settings should be completely separated from that path.

That is all there is to it.

But this is the kind of thing people often do not notice until an incident actually happens.

System failures are often like that.

They do not always come from grand design mistakes or mysterious bugs.

They often begin with a small sentence:

“I changed the configuration a little.”

And after the incident response was over, I found myself thinking:

“Working with Windows people really is scary.”

They are probably thinking the same thing in reverse:

“Mac people are such a pain.”