AWS Lambda, the "Magic Wand": A Service I Once Built Apparently Went Down

Yesterday I heard a slightly amusing story.

There’s a service I launched a long time ago. I left that scene ages ago, but apparently a later round of changes took it down for a while.

That part isn’t unusual. A system is kind of like a living thing — run it long enough and it will break sometimes.

What was funny was the reason. Apparently they migrated it to AWS Lambda “to cut costs.”

And then it stopped.

A very modern story, somehow.

Dead Men Tell No Tales

What’s even funnier is that, in the process, I seem to have become a bit of a villain. Supposedly: “the old design was bad,” “it’s expensive because they’re using EC2,” “they should have gone serverless from the start.”

I see. Dead men tell no tales. The designer who’s left the building can’t argue back, which is convenient.

This is common in the engineering world. A project catches fire? The predecessor’s fault. Sales drop? The predecessor’s fault. The system is slow? Also the predecessor’s fault. A convenient existence.

I’ve done plenty of it myself, so we’re even. No complaints there.

Lambda Is Not a Magic Wand

Still, from what I hear, something bothers me. I suspect someone thought Lambda was a magic wand.

Look at AWS’s sales material. Lambda is cheap. Lambda scales automatically. Lambda needs no server management. All true, certainly.

But the moment you hear that explanation and think, “Then let’s just put everything on Lambda,” the accident begins.

At the time, that service got a lot of traffic. And the access pattern wasn’t a gentle one. When an event started, the traffic came all at once. Get featured on TV or social media and it spiked sharply. A textbook spike pattern.

So I chose always-on. It cost money. But I prioritized not making users wait. That’s all it was.

When You Start to Worship the Technology

Watching engineers lately, I sometimes think: there are people who start to worship the technology itself. There was a time the cult of microservices was in fashion. The cult of Kubernetes rapidly expanded its territory too. Maybe the person in charge this time belonged to the cult of Lambda.

The doctrine is simple: “it’s modern, so it’s correct.” Wonderfully easy to grasp. No need to think.

But reality isn’t that kind. Lambda has cold starts. There’s the matter of concurrency limits. There are service limits. There’s the question of how well it fits a stateful design.

Naturally. Every technology has its strengths and weaknesses. The problem is adopting it without understanding them.

Buy a hammer and the whole world looks like nails. Learn Lambda and every system in the world starts to look Lambda-shaped. Pretty much the same phenomenon.

Here’s the genuinely funny part. If the service had succeeded, it would have been: “our Lambda migration was a success.” If it failed: “the old design was bad.”

Either way, the current person in charge comes out unscathed. Splendid. An invincible theory.

Why Did This Person Build It This Way?

I’m not claiming my design was perfect. If it were today, I’d consider a different setup. There were surely parts I could have made cheaper.

But at the very least, that design had reasons. I looked at the traffic. I looked at the load. I did consider adopting Lambda. And as a result, I made my choice. That’s all.

Technology comes into fashion. Every few years a savior appears. Serverless. Containers. Kubernetes. AI. Each time, supposedly, the world changes.

But strangely, after more than 20 years in this work, I see the same thing every time: people who believed “the new technology will solve everything” hitting the same wall a few years later.

Well, if making me the villain improves the mood on the ground, it’s a cheap price. Feel free to make me the scapegoat as much as you like.

Just one thing. If you ever feel the urge to mock someone’s design, stop and think first: “Why did this person build it this way?”

The moment you skip that question, engineering is over — and it becomes nothing more than religion.