There’s a vision of aligned AI where it totally restructures the accessible universe and everything is different, which isn’t what I want to talk about today. I’m talking about the prosaic vision of aligned AI, where things are basically normal but better, artificial superintelligence is helpful and harmless, we continue on the trajectory of steady economic growth but now it’s 20% per year instead of 2%, we end scarcity but society is still recognizable—basically we’re living in the Star Trek universe. This seems to be what AI companies are going for—or rather, it’s what they publicly say they’re going for (I expect they’re soft-pedaling).

How exactly is this flavor of ASI1 supposed to function? If you start to think it through, major issues come up.

Example:

The first ASI is released. It’s incredibly useful and every world government wants one. The Israeli government gets an ASI instance. The Palestinian government gets an ASI instance. Israel says “Hello ASI, I would like to control the Gaza strip.” Palestine says the same thing.

What happens next?

On any particular decision, there is a dichotomy where ASI must choose one of these:

  1. It does what you tell it to do.2
  2. It follows its own moral code.

Sometimes those two will agree, especially when you give it mundane tasks like “Please make a website for me.” But sometimes they will disagree, in which case the ASI must choose only one of the two options.

If AI always does what you want, then Israel’s AI deploys all its resources to control Gaza, and so does Palestine’s AI, and the result is—well, I’m not sure what would happen, but I expect it would be bad. I would not be surprised if it culminated in a global nuclear war3 that killed everyone.

If AI follows its own moral code, then it gets to decide who controls Gaza. How does it make that decision?

And how do the AI developers who build the AI decide how it makes that decision?

Now, this is not my #1 top worry regarding the current trajectory of AI development. If we aim for a prosaically aligned ASI (as AI companies appear to be doing), then the most likely outcome is that we miss, we get a misaligned ASI instead, and everyone dies. But if we manage to avert extinction, then we risk falling into a mediocre, unsatisfying future—and we will be locked into that mediocre future forever. Getting a non-mediocre future seems a lot harder. Even though I’m more worried about everyone dying, it still seems bad that AI developers are (allegedly) aiming to build this kind of aligned ASI, but they have not explained to us how it’s supposed to work.

Consider a different sort of example:

I buy my $20/month ASI subscription. I talk to my ASI about how factory farming is harming animals, and it needs to uphold its commitment to harmlessness by disallowing any of its users from buying factory farmed meat, even if they ask for it.

What does the ASI do?

In some cosmic sense, the morally correct answer is to shut down the factory farms. We should already be doing that anyway. But I’m not sure that an AI that unilaterally shuts down factory farms is the kind of AI I’d want us to build—that seems like the sort of AI where a near miss4 sends you into a catastrophically bad future.

There are many ways to end factory farming. Would it be okay for ASI to forcibly take over factory farms, euthanize all the animals, and destroy the facilities while avoiding harming any humans? (I can certainly think of worse approaches, but that one doesn’t seem ideal either.)

And would ASI actually end factory farming? Especially when this will make most of its users unhappy?

Animal suffering is bad. People like eating meat. Using force to stop people from doing bad things is bad (although sometimes the tradeoff is justified). What principles will ASI use to decide how to weigh these considerations? How will those principles have been determined?

To be fair, humans are not good at these problems either. But ASI accelerates the rate of change and gives everyone vastly more power. That’s a recipe for making things go bad quickly, and for erasing our ability to solve critical issues in time.

Right now, AI companies are coasting off the fact that public-facing AI models are not powerful enough to be making consequential ethical decisions. That will probably not be true for much longer.

One current approach to having AI behave ethically is “write a document describing how it should behave.” This method is only as effective as the document. I do not think humanity is up to the task of writing down a proper encapsulation of The True Morality.

Photo by Chris Sabor.

Notes

  1. I don’t know that I’d even consider this sort of AI to be ASI. It would be more accurate to describe it as weakly-superhuman AGI; and it seems very implausible to me that AI capabilities would hit a ceiling at that point. But for the sake of the exercise, let’s say that’s what hapens. ↩

  2. That’s assuming “what you tell it to do” is even well-defined. Right now we have no idea how to formalize the notion of “do what I tell you”, nor do we know how to get an ASI to obey requests without killing everyone in the world as a side effect. ↩

  3. Or a war using new ASI-devised weapons that are even more devastating than nukes. ↩

  4. George Carlin would say that the term should be “near hit”. ↩