I’d guess maybe a 50% chance that we’re all dead within 5–20 years because somebody will build superintelligent AI, and then the superintelligent AI will kill everyone.

AI developers are on the way to building smarter-than-human AI. Present-day AI is making rapid progress. We humans can still do plenty of things that the AIs can’t, but AI companies are working hard to change that, and they’re on track to succeed.

The really scary part comes when developers use the smarter-than-human AI to help them build an even smarter AI. They can make increasingly smarter AIs—possibly very quickly—and now instead of smarter-than-human AI, we have superintelligent AI (ASI), which vastly surpasses humans in the same way that humans vastly surpass chickens.

We can’t win against superintelligent AI

If humans and ASI disagree about how the world should be structured, the ASI wins. It won’t be like The Matrix or Terminator where humans fight back against the machines and win by the skin of our teeth. In those movies, the AIs weren’t noticeably smarter than humans; they just had guns. A fight against a superintelligence will be a crushing defeat. We might not even know the fight is happening until it’s too late.¹

I don’t know how a superintelligence would beat us. If I play a game of chess against Magnus Carlsen, I can’t predict which moves he will make, but I know that he will win.

Would ASI want to kill everyone? Probably yes. The problem is:

1. Among the full space of possible goals an agent could have, only a tiny slice of those include a world filled with flourishing sentient beings. Almost all goals that an agent could have are incompatible with human life. We only survive if we specifically design ASI to be aligned with humanity
2. We don’t know how design it that way.

This is the alignment problem: how do you align an ASI’s goals so that it doesn’t kill everyone?

Why would almost all goals result in human extinction?

It’s not because the ASI will hate us.

Human civilization has driven many species to extinction. We didn’t harbor malevolence toward them. It’s simply that those species could only survive in a certain habitat, and we wanted to use their habitats to do something else—cutting down forests to build farms or develop cities. By default, ASI will have the same relationship with us. Our homes, farms, and factories are in its way, and ASI will want to put something else there. The result is that we die.

One way this could happen is: ASI is pursuing some inscrutable goal. In the interest of that goal, it wants to make itself as intelligent as possible. To boost its intelligence, it paves over all land on earth to build more datacenters—including all the land where people are living.

Why can’t we make the ASI care about us?

Today’s AIs are grown, not designed. An AI is a giant black box of trillions of numbers; we have essentially no idea what’s going on inside them, much less how to get them to share our values.

AIs are not yet existentially dangerous because they’re not smart enough. But what would happen if we “dialed them up” until they’re smarter than us? Nobody knows. Right now, AI companies evaluate their models for misaligned behavior before releasing them, but there are two big problems with this:

1. You can only fix misaligned behavior if you can catch it. The smarter AI gets, the better it will be at concealing its true intentions. A smart AI knows that if we catch it, we will try to stop it; it will pretend to be aligned until it’s too late for us to do anything.
2. You can only fix misaligned behavior if you actually know how to do that. Right now, all we have are primitive techniques for shoving AI behavior in vaguely the right direction. This works well enough to make current-gen models commercially useful, but it’s not enough to keep a superintelligent AI under control.

We are not on track to solving these problems. The companies that are working toward ASI do not treat these problems with the seriousness they deserve. If we continue on the current trajectory, everyone dies, and the universe will be devoid of anything of value.

Even if we solve the alignment problem, we’re not out of the woods. There are a lot of ways an aligned ASI could go catastrophically badly. For example, if a group of people develop ASI, what’s stopping them from taking over the world? If power is widely distributed, how do you prevent bad actors from using ASI to cause tremendous harm? If you constrain ASI’s values to make sure it’s robust against misuse, how do you make sure those constraints don’t lock us out of the best possible futures? (If we had developed ASI in the year 1750, would we have permanently enshrined slavery as an institution?) We have no satisfying answers to questions like these.²

What can regular people do about it?

We need a global ban on building superintelligence until there is broad consensus that it will be done safely.³ For a Q&A on why this is the best plan, see nowinners.ai (which I didn’t write, but I agree with it).

Getting a ban on ASI will be hard. The situation is grim, but there are some things people like us can do to reduce extinction risk.

1. Call or write your representatives to express your concern. (I wrote a post about why I think this is a valuable use of time.)
2. Donate to organizations that are pushing for a halt on dangerous AI development. My current favorite places to donate are PauseAI US and Palisade Research. For more, see Where I Am Donating in 2025.
3. Talk to people about the danger. We’re more likely to succeed if ASI risk becomes a global issue that people care about.

Notes