I prefer to avoid saying things that I suspect might make me look stupid. Over the last couple of years, however, I’ve made a conscious effort to write publicly about what I’m thinking about, even if I’m afraid it might sound dumb.

Sometimes I have this feeling, like:

I have an observation about the world, about something that doesn’t seem to add up. Probably I’m missing something, and everyone but me can easily see what’s going on, and if I bring it up, then everyone will know how foolish I am for missing this obvious thing.

Sometimes I’m right, and I really was missing something obvious. But some of my most well-received writings turned out to be ones where I had this feeling, and I was ambivalent about whether it was worth saying anything.

Lately, I’m making more of an effort to overcome that feeling, and say the stupid thing anyway, because it might turn out not to be stupid.

Continue reading

Some people think we should pause AI, but not now. They say we should wait until AI reaches human level,¹ because:

• It’s not (catastrophically) dangerous until after then.
• Human-level AI will help us do safety research.

Alternatively, other people (like me) think we should pause AI as soon as possible.

Katja Grace wrote a nice concise case for pausing ASAP. I have something I’d like to add: pausing at human level seems harder than pausing ASAP.²

Pausing ASAP sounds hard. It will be hard to get international coordination around an AI pause, and implementing a pause sounds hard even if we can agree to it in principle. But pausing ASAP still seems easier than pausing at human-level AI, for several reasons:

• Human-level AI is highly economically valuable, and therefore there is a great temptation to keep going. The monetary incentive to build increasingly-powerful AI will be intense, and industry lobbyists really won’t want to pause AI development. It seems hard to pause when the economic incentive to continue is so great.
• If AI is smart enough to accelerate safety work, then it’s also smart enough to accelerate improvements in AI capabilities. And it will probably be disproportionately good at the latter: capabilities improvements are easy to measure, and AI tends to be disproportionately good at easily measurable tasks. (Recent AI models have seen bigger improvements in math and coding abilities than in writing or philosophy.) If AI R&D used to require a team of PhDs, and now all it requires is someone in a garage with access to the latest AI model, then it’s harder to enforce a pause because clandestine AI research is harder to catch.
• This next argument is more about a unilateral pause than a coordinated pause, but: Some say that the “good” AI developers need to push the frontier to maintain their lead, and that they should wait until the last minute to burn their lead to work on safety. Burning their lead at the end provides the maximum uplift from AI-assisted safety work. However, AI developers always face a choice between an uncertain downside (keep going, and possibly kill everyone) vs. a certain downside (pause, crater your profit potential, and possibly some other developer kills everyone anyway). I cannot foresee them making a rational risk assessment under those circumstances. There is too much pressure to distort their beliefs in favor of continuing to push the frontier.
• If we had technology that could replace human labor, but it’s cheaper, faster, and can be copied as many times as one wants, how would that change the economy and society? I don’t know, but I bet it would change a lot. That level of disruption makes the world unpredictable. It seems risky to follow plans along the lines of, “let’s wait until this technology radically transforms society, possibly making things totally unrecognizable, and then implement our plan after that. Surely it will still work!”

An important counterpoint:

• The general public does not like AI. If AI starts taking people’s jobs, then people will really dislike it. The unemployment effect of AI may create enough political will to pause that it outweighs out the economic incentive effect.

I hope this final point is strong enough to make pausing much easier in the future (hopefully the near future). But that doesn’t mean we shouldn’t try to pause ASAP.

Notes

I continue to believe we should pause frontier AI development. Any discussion of alternative strategies should be thought of as planning for contingencies.

“Super-persuasive” AI is dangerous because a misaligned ASI could persuade humans to help it take over. But setting that aside, even if we manage to make ASI friendly, it may provide super-persuasive but mistaken guidance that permanently sets us down the wrong path.

This post focuses on the danger of an aligned super-persuasive AI that simply comes up with the wrong answers.

Continue reading

An argument goes: If we don’t build aligned artificial superintelligence, we risk driving ourselves extinct for some other reason. We should rush to build ASI quickly, in spite of the risks—the longer we wait, the more vulnerable we are to extinction from a different cause.

Other than ASI, the biggest extinction risk is synthetic biology. Some lab could (accidentally or on purpose) develop a highly transmissible, 100% fatal super-plague that wipes out humanity.

An aligned ASI could stop that from happening by shutting down dangerous biological research, or by developing advanced countermeasures that stop the spread of deadly infections. So the argument goes: We need to build ASI to save us from non-AI extinction risks.

However, that argument doesn’t work. In the near term, AI will make biological risks worse, not better. AI will accelerate scientific research, which will bring us closer to the level of knowledge necessary to build extinction-level pathogens. And in the long term, the way ASI eliminates biological x-risk is by taking control of the world.

Continue reading

Say you work at a private company that gives you stock options or RSUs. How should you value your stock?

• If you have a choice between getting more stock or more cash salary, how do you decide which to get?
• If you have the chance to sell some stock, should you do it?

Stock is risky and inflexible (especially if you work for a private company where you can’t easily sell shares), but you might be able to get it at a discount to its true value. How do you estimate how much it’s worth?

One heuristic you can use is to compare the stock against a risk-matched index fund. What would happen if you used the cash to buy a leveraged index fund with the same level of risk as the company stock? If the leveraged index fund has a higher expected return than the company stock, that means cash is probably better. (The reverse is not necessarily true because company stock can have other downsides, which I will get into later.)

Continue reading

Prior discussion: niplav’s shortform (2025); Planning for Extreme AI Risks (2025) by Joshua Clymer

A frontier AI company (any one, I don’t care which) should close shop and make an announcement along the lines of:

Powerful AI could end the human race. We are too worried that we don’t know how to make this technology safe. We have decided to shut down because we don’t want to be responsible for building the thing that kills us all.

A common refrain among safety-conscious AI developers: “it doesn’t matter if we stop building dangerous AI, because someone else will just build it instead.” Is that really true, though? If a multi-hundred-billion-dollar company comes out and says “We’ve concluded that our product is horribly dangerous, nobody knows how to make it safe, and there’s too high a risk that it leads to human extinction”, this won’t raise any eyebrows? This has no chance of spurring policy-makers into action?

Continue reading

(Spoilers in this post are hidden with spoiler tags.)

What made Project Hail Mary so good? Among other reasons, it’s because the science drove the story, instead of the other way around.

Character-driven stories and hard sci-fi might take up opposite positions in the ancient battle of “people vs. things”; but when they work, they work for fundamentally the same reasons.

In mediocre “people”-focused stories, the plot dictates how characters behave. In great people-focused stories, the characters decide what happens.

In mediocre sci-fi, the plot dictates what science and technology can do. In great sci-fi, the science and technology constrain what routes the plot can take.

Continue reading

Image credit: Jebulon

To prevent superintelligent AI from killing everyone, I would like there to be a strong international agreement banning the development of ASI until it can be proven safe. But that sort of agreement requires a lot of political buy-in and coordination. In the meantime, it may be easier to get light-touch AI safety regulations passed. To what extent do weak regulations decrease extinction risk?

In this post:

• Part I discusses routes by which weak regulations can reduce extinction risk. [More]
• Part II considers some downsides of weak regulations. [More]
• Part III reviews specific categories of weak regulation and how they might reduce risk. [More]

Continue reading

Depth-first plans lay out a path from here to aligned superintelligent AI. We need those kinds of plans. But depth-first plans depend on many assumptions: “We will make AI safe by doing step 1, then step 2, then step 3.” Step 1 only works under condition A, step 2 requires condition B, step 3 requires condition C. If A or B or C is false, the whole plan fails (and there’s a good chance we all die).

Consider Google’s safety plan from April 2025. To my knowledge, this is the best among the frontier AI companies’ plans.¹

Google’s plan depends on a series of conditions:

Continue reading

Three categories of futures, depending on how AI goes:

1. ASI timelines are long.
2. ASI timelines are short, and we’re on track to solving AI alignment.
3. ASI timelines are short, and we’re not on track to solving AI alignment.

If we want to make a good future for all sentient beings, each of these futures has different implications for what we should work on.

Continue reading