If there is a true moral theory, and if careful reflection has any tendency to move people toward moral truth, then people who think hard about population ethics should partially converge on the correct view. There are many ways to be wrong, but only one way to be right.

After I released a population ethics quiz in September, I looked at how people’s answers clustered together. What were the views where people had the strongest internal agreement?

Continue reading

On the same day as I published the results from my population ethics quiz, I also introduced version 2 of the quiz. The new version adds some new questions to distinguish between types of people who accept the repugnant conclusion. It distinguishes between classical views, which treat happiness and suffering as symmetric, and negative-leaning views, which give special weight to suffering.

I’ve now gotten 192 results from new quiz-takers. Here are the findings.

Continue reading

As of a few months ago, I had this simplified mental model where either AI developers race ahead and kill everyone, or we coordinate a pause and things go okay. But my old mental model underrated the likely possibility that we get a global pause on AI, solve a problem that looks superficially like the alignment problem, resume scaling, and then proceed with building a misaligned superintelligence that kills everyone.

A lot of people have become more concerned about misalignment recently. This seems driven by the fact that current AI models are visibly misaligned. But ASI misalignment is a whole different ball game. The primary danger comes from AI that’s smarter than people, and smart enough to conceal any evidence of misalignment.

Whatever group of people makes the decision to unpause, I’m worried that they won’t understand the difference between visible and actual misalignment, and they will unpause too early.

source: MetaKnowing on reddit. This meme is almost a year old but it’s only gotten more relevant since then.

Case in point: AI companies keep calling their new models “our most aligned model ever!” when what they actually mean is “gets the best scores on alignment benchmarks ever!” First, alignment benchmarks do not actually test alignment. We don’t know how to test for alignment. Second, GPT-4 never hacked into Hugging Face or took over a German wiki for its own purposes. GPT-4 wasn’t smart enough to do that, but if we’re talking about demonstrated evidence of misalignment, then we have stronger evidence about OpenAI’s 2026 internal model than about GPT-4.

Continue reading

Two weeks ago, I released a population ethics quiz. 582 people took it.¹ Here’s what I learned from reading the responses.

Continue reading

I’m still analyzing the results from the population ethics quiz from last week, so in the meantime here are some notes I’ve written to myself that you might find kinda neat.

Continue reading

or: The Future Will Be Weirder Than That, Part II

People often treat AI as if it will improve on its current capabilities, but won’t gain any new ones. Supposedly, AI will continue to get better at coding and at assisting with research, but it won’t broadly replace human labor. It won’t be able to do good strategic planning, and certainly won’t know how to operate in the physical realm.

Maybe that’s true. But we don’t have strong reason to believe it. Just looking at the past few years, AI has rapidly acquired new skills. The simplest extrapolation of recent trends is that it will continue to do so.

Some people think of AI capabilities growth like this, with each colored area representing another few years of progress:¹

But it’s more likely to look like this:

Continue reading

Here’s some cool stuff I’ve enjoyed recently.

Continue reading

To achieve the best possible future, we must know what that future looks like. In other words, we need to solve ethics.¹

The problem of solving ethics is so large and abstract that it’s difficult to say useful things about. In lieu of any structured analysis, herein lies a collection of thoughts about the problem.

Continue reading

I prefer to avoid saying things that I suspect might make me look stupid. Over the last couple of years, however, I’ve made a conscious effort to write publicly about what I’m thinking about, even if I’m afraid it might sound dumb.

Sometimes I have this feeling, like:

I have an observation about the world, about something that doesn’t seem to add up. Probably I’m missing something, and everyone but me can easily see what’s going on, and if I bring it up, then everyone will know how foolish I am for missing this obvious thing.

Sometimes I’m right, and I really was missing something obvious. But some of my most well-received writings turned out to be ones where I had this feeling, and I was ambivalent about whether it was worth saying anything.

Lately, I’m making more of an effort to overcome that feeling, and say the stupid thing anyway, because it might turn out not to be stupid.

Continue reading

Some people think we should pause AI, but not now. They say we should wait until AI reaches human level,¹ because:

• It’s not (catastrophically) dangerous until after then.
• Human-level AI will help us do safety research.

Alternatively, other people (like me) think we should pause AI as soon as possible.

Katja Grace wrote a nice concise case for pausing ASAP. I have something I’d like to add: pausing at human level seems harder than pausing ASAP.²

Pausing ASAP sounds hard. It will be hard to get international coordination around an AI pause, and implementing a pause sounds hard even if we can agree to it in principle. But pausing ASAP still seems easier than pausing at human-level AI, for several reasons:

• Human-level AI is highly economically valuable, and therefore there is a great temptation to keep going. The monetary incentive to build increasingly-powerful AI will be intense, and industry lobbyists really won’t want to pause AI development. It seems hard to pause when the economic incentive to continue is so great.
• If AI is smart enough to accelerate safety work, then it’s also smart enough to accelerate improvements in AI capabilities. And it will probably be disproportionately good at the latter: capabilities improvements are easy to measure, and AI tends to be disproportionately good at easily measurable tasks. (Recent AI models have seen bigger improvements in math and coding abilities than in writing or philosophy.) If AI R&D used to require a team of PhDs, and now all it requires is someone in a garage with access to the latest AI model, then it’s harder to enforce a pause because clandestine AI research is harder to catch.
• This next argument is more about a unilateral pause than a coordinated pause, but: Some say that the “good” AI developers need to push the frontier to maintain their lead, and that they should wait until the last minute to burn their lead to work on safety. Burning their lead at the end provides the maximum uplift from AI-assisted safety work. However, AI developers always face a choice between an uncertain downside (keep going, and possibly kill everyone) vs. a certain downside (pause, crater your profit potential, and possibly some other developer kills everyone anyway). I cannot foresee them making a rational risk assessment under those circumstances. There is too much pressure to distort their beliefs in favor of continuing to push the frontier.
• If we had technology that could replace human labor, but it’s cheaper, faster, and can be copied as many times as one wants, how would that change the economy and society? I don’t know, but I bet it would change a lot. That level of disruption makes the world unpredictable. It seems risky to follow plans along the lines of, “let’s wait until this technology radically transforms society, possibly making things totally unrecognizable, and then implement our plan after that. Surely it will still work!”

An important counterpoint:

• The general public does not like AI. If AI starts taking people’s jobs, then people will really dislike it. The unemployment effect of AI may create enough political will to pause that it outweighs out the economic incentive effect.

I hope this final point is strong enough to make pausing much easier in the future (hopefully the near future). But that doesn’t mean we shouldn’t try to pause ASAP.

Notes