CODE IN TOKYO

The World Where Trusting AI Can Hurt You Has Already Started

At a certain junior high school, a science teacher apparently gave students this assignment:

“You may use the internet, so research what salivary amylase does.”

It is a very modern kind of homework.

In the past, students would probably have used textbooks or the library. Now they have the internet. And naturally, some students use AI.

As a result, about half of the class reportedly wrote almost the same incorrect answer.

Salivary amylase exists in saliva and breaks down starch, helping digestion in the stomach.

At first glance, this sentence does not look especially wrong.

If I did not know the science, I would probably believe it too.

But starch is hardly digested in the stomach.

In other words, the answer is wrong.

What is even more interesting is that the source of the misinformation may have been identified. There is a story that AI may have learned from an explanation on the website of a Japanese seasoning company and then returned it to students as a plausible answer.

AI looks as if it knows everything, but in the end, it answers based on information written by someone else.

If the source is wrong, AI can be wrong too.

The difficult part is that AI answers with great confidence.

It does not say, “I am not completely sure, but…”

Even when the content is wrong, it explains the answer in clean language, with a logical structure, and in a tone that sounds convincing.

It is not surprising that junior high school students would be fooled.

I Cannot Laugh at These Students

I am writing this as if I am above the problem, but I cannot laugh at these students at all.

If anything, I may be in more danger because I use AI every day.

In programming, I already leave a large part of the work to AI.

I ask it about design, have it write code, and use it for refactoring. AI has become much smarter recently, so it often produces decent code even without extremely detailed instructions.

Then humans learn the comfort of taking the easy route.

“Well, if AI says so, it is probably correct.”

That feeling grows little by little.

Tests Help, but They Are Not Enough

In programming, there is still one form of protection.

We can run tests at the end.

If a test fails, we know that something is wrong.

But just because the tests pass, that does not mean the logic is correct.

Maybe the code only slipped through the existing test cases.

Maybe the assumption behind the tests is wrong in the first place.

AI writes a terrible piece of logic. A human merges it without understanding it because the tests passed.

When you think about it, that is a fairly frightening world.

The Ability to Doubt AI

Perhaps what people will need from now on is not only the ability to use AI.

They may need the ability to doubt AI.

Of course, I will still ask AI to write code today.

And I am also using AI to help write this article.

So I probably have no right to laugh at the junior high school students who wrote about salivary amylase.

Contact

I provide system development and technical consulting using AI, AWS, and Claude Code.

Contact form: https://holly-money-e94.notion.site/390ff30cf18c8086a676fe630d171873