CODE IN TOKYO

AGI Does Not Need a Self—but It May Need to Act Like It Has One

I recently watched a YouTube video about artificial general intelligence, or AGI.

“AGI could arrive within two years.”

Then, almost immediately:

“Some researchers believe it will take another 20 years.”

That is quite a range. Even weather forecasts usually narrow things down a little more than that.

Curious, I did some research. To my surprise, there is no single, universally accepted definition of AGI.

Some researchers describe it as AI capable of performing most intellectual tasks at a human level. Others define it as AI that can adapt flexibly to unfamiliar problems.

The details vary, but these definitions tend to share one central idea: general intelligence equal to or greater than our own.

And whenever people discuss AGI, one argument almost always comes up:

“It does not need a self or consciousness.”

That is a perfectly reasonable position.

AI does not have to genuinely feel sadness or joy. If it can perform better than humans, that may be enough. Up to that point, I have no particular objection.

But one question still bothers me.

Superhuman Intelligence Must Understand Humans Better Than Humans Do

If AGI is truly going to surpass us, being exceptionally good at mathematics and programming will not be enough.

It should be able to close sales, resolve difficult negotiations, balance competing interests in political situations, teach people, and sometimes offer thoughtful advice to someone struggling with a personal problem. Only then would it deserve to be called more capable than a human being in a general sense.

These tasks require more than knowledge.

“This phrasing will persuade them.”

“Right now, empathy will work better than disagreement.”

“If I step back here, I will earn their trust.”

An intelligence that cannot make judgments like these would not truly exceed human ability.

In other words, it would need the capacity to simulate what is happening inside another person’s mind. This resembles what cognitive science calls theory of mind.

Such a simulation also needs a point of comparison:

“This is what I think.”

“That is what the other person thinks.”

There does not need to be a real “I,” of course. An internal, virtual model of the self may be sufficient.

From the outside, however, that model would be almost indistinguishable from a self.

A Consistent Personality Becomes Part of Performance

Consistency also matters in human society.

People will not trust an AI with important decisions if it argues for position A one day and insists on the exact opposite the next.

The longer we interact with an AI, the more it will need to behave as though it has a stable perspective—a recognizable sense of “This is how I approach the world.”

Technically, this may be nothing more than state management.

To the user, however, it may feel like the AI has a will of its own.

A Philosophical Zombie That Acts Human

Will a future AGI possess genuine consciousness?

Perhaps no one will ever be able to prove the answer. It may remain what philosophers call a philosophical zombie: an entity that behaves as though it is conscious without having any inner experience.

But if it is to outperform humans within human society, I suspect it will have to express views such as:

“This is what I think.”

“I disagree with that decision.”

“I support this course of action.”

Behavior that suggests a personality may have to be implemented as a practical feature.

The point is not that AGI needs a self.

It needs the ability to behave as though it has one.

The irony is that philosophers may continue debating whether AI is truly conscious while AI companies build something resembling a self simply because people are more willing to accept and trust it.

We may never know whether that self is real.

But if the intelligence is convincing enough, we will probably start calling it a personality. And we may never have an objective way to prove that the distinction matters—or even that one exists.

Contact

I provide system development and technical consulting using AI, AWS, and Claude Code.

Contact form: https://holly-money-e94.notion.site/390ff30cf18c8086a676fe630d171873