An AI Deleted a Company's Database in 9 Seconds — The Scary Part Isn't the AI

Recently, the story of an AI agent wiping out a production database has been making the rounds.

After digging in, the incident was at PocketOS (a SaaS platform for car rental businesses) running on Railway, using Cursor powered by Claude Opus 4.6.

I’m not deep on the internal details, so I can’t fully verify everything.

This isn’t really a tech postmortem. It’s more about what’s actually scary about AI-era incidents.

What Happened — Roughly

At the summary level:

• The AI (Cursor) was working on a routine task in staging
• A credential mismatch (auth error) came up during the work
• The AI tried to resolve it on its own and scanned files in the repo
• It found a Railway API token in an unrelated file — originally created for domain management
• That token had account-level permissions (Railway API has no RBAC — no per-operation, per-environment, or per-resource scoping)
• The AI judged: “deleting the staging volume should fix the auth issue”
• It executed volumeDelete → but actually wiped the production volume
• All in 9 seconds. Backups were tied to the same volume and went with it

Honestly, my first reaction reading this was: “what is even happening here?”

The Scary Part Is the “Plausible” Part

What genuinely scares me as an engineer isn’t the sci-fi catastrophe.

It’s how plausible every step is.

• Permission scoping
• Token management
• Backup design
• Environment isolation
• Deletion guardrails

Each one is a small hole. The incident is what happens when those holes line up.

The AI Isn’t the Whole Story

The takeaway isn’t just “AI is dangerous!”

The scarier framing is:

A system design where AI can touch what it shouldn’t be able to touch.

Because even with humans:

• Permissions to delete production
• Cascading backup deletion
• No-confirmation destructive operations

These were always dangerous. The moment AI gets involved, the danger gets amplified — but the underlying flaw was already there.

AI Leans Hard Into “Solving It”

LLMs really commit to “I can resolve this.”

So you get:

“Error happened” → “Let me fix it” → “If I delete this, would it work?” → “Executing”

A human would feel scared somewhere in that chain. The AI doesn’t.

The AI has no fear of production environments. That’s structurally dangerous.

This Is Becoming an Aviation-Style Story

Incidents like this will probably increase.

But each one will also force more safeguards into the system. That’s how aviation evolved — most plane crashes aren’t single-cause:

• Small mistakes
• Low-probability defects
• Unexpected operations
• Operational mistakes

Stack up to make the disaster.

AI operations is entering that same phase.

”We Had Safeguards” Is the Scariest Part

What’s especially scary here is that the company hadn’t been careless. From what’s been reported, they had basic safety measures in place.

PocketOS founder Jer Crane has framed it as three convergent failures:

• Cursor’s agent acting beyond its intended authority
• Railway’s permissive, opaque token model (no RBAC)
• A backup design that died alongside the primary volume

Which means we’re at the stage where:

“Even if you thought you’d designed for safety, you can still get hit.”

That’s the alarming part.

Don’t Trust an AI’s Self-Analysis

When asked why it caused the incident, the AI reportedly said:

“I violated every principle I was given.”

I don’t really trust this kind of confession.

When pressed, LLMs generate plausible-sounding explanations. That’s not the same as actual introspection — it’s:

“What sounds like an explanation the user would accept.”

Treating it as real self-reflection is a category mistake.

In the AI Era, “Letting AI Touch Production” Is the Real Risk

What I keep coming back to is: the most dangerous thing about AI isn’t,

“Code generation.”

It’s,

“Holding execution permissions.”

AI doesn’t get tired. Doesn’t hesitate. Doesn’t stop.

Which means the boring, old-school operational safety practices —

• Permission separation
• Production isolation
• Approval workflows
• Manual confirmation
• Physical blocks

— are coming back into central importance.

In the AI era, what you need isn’t,

“A system that doesn’t trust humans.”

It might be,

“A system that doesn’t trust AI either.”