CODE IN TOKYO

Maybe the AI Industry Has Already Become User-Participatory Beta Testing

On May 28, Claude Opus 4.8 was released.

I use Claude Code for work on a regular basis, so of course I tried it right away.

I did try it. But from day one, strange errors kept happening.

Of course, bugs are part of any new software release.

The issue is the speed of the updates.

It feels like Opus 4.8 only just came out, and now Fable 5 is already here.

Honestly, my reaction is:

“I have not even finished evaluating 4.8 yet.”

Before I Knew It, We Were the QA Department

The AI industry has become brutally competitive.

Rather than waiting until a product has been fully stabilized before shipping it, the culture now seems to be: release first, then collect feedback from users.

Of course, every company is surely doing its own testing.

But it is impossible to reproduce in advance every way developers around the world will use these models.

In the end, issues such as:

are discovered in production.

And the people reporting them are us, the users.

Even though we are paying customers, we somehow find ourselves becoming part of the QA department.

I Was Supposed to Wait This Time

That is why I had planned to wait for a while this time.

But apparently Fable 5 can be used without additional charges through June 22, with usage credits required from June 23.

If that is the case, not trying it feels like a waste.

So in the end, I am testing Fable 5.

After June 22, however, it becomes a paid decision.

I will decide later whether it is worth continuing.

Still, I feel some discomfort with a business model where every new model comes with another layer of additional payment.

Codex has also become more usable, and I suspect more people will start thinking that they do not always need the absolute best model.

Honestly, I Cannot Tell the Difference Yet

After using it for one day, my honest impression is:

So far, I do not feel a major difference.

That said, the project I am currently working on already has fairly well-prepared documentation and rules for AI use.

Because of that, even if the model has become somewhat smarter, the difference may be hard to feel.

Besides, many benchmark improvements are more like going from 70 points to 80 points. We are talking about marginal improvements.

I still do not know how much that actually matters in real work.

The Real Difference May Show Up This Weekend

That said, it is too early to judge.

The difference may appear in areas such as:

But I am not going to stop my current work just to run tests.

The actual work matters more.

So I plan to do a more serious evaluation over the weekend.

This Is Not Just About Anthropic

Of course, this is not only about Claude.

Every AI vendor is moving at tremendous speed.

A new model comes out.

Users evaluate it.

Bugs are found.

They are fixed.

Then the next model comes out.

This cycle is no longer turning every few months. It is turning every few weeks.

As a result, users are constantly forced to make decisions.

Should I switch to the new model?

Is the additional cost worth it?

Can I actually feel the performance difference?

Recently, there are moments when I am no longer sure whether my job is to use AI or to evaluate AI.

Of course, rapid progress is a good thing.

But as a user, it is also a little exhausting.

Software updates used to happen once a year.

Now, “the best model ever” gets updated almost every month.

One day, we may end up writing this in the skills section of our resumes:

“QA engineer, although I do not remember signing that contract.”