I continue to believe we should pause frontier AI development. Any discussion of alternative strategies should be thought of as planning for contingencies.

“Super-persuasive” AI is dangerous because a misaligned ASI could persuade humans to help it take over. But setting that aside, even if we manage to make ASI friendly, it may provide super-persuasive but mistaken guidance that permanently sets us down the wrong path.

This post focuses on the danger of an aligned super-persuasive AI that simply comes up with the wrong answers.

Continue reading

An argument goes: If we don’t build aligned artificial superintelligence, we risk driving ourselves extinct for some other reason. We should rush to build ASI quickly, in spite of the risks—the longer we wait, the more vulnerable we are to extinction from a different cause.

Other than ASI, the biggest extinction risk is synthetic biology. Some lab could (accidentally or on purpose) develop a highly transmissible, 100% fatal super-plague that wipes out humanity.

An aligned ASI could stop that from happening by shutting down dangerous biological research, or by developing advanced countermeasures that stop the spread of deadly infections. So the argument goes: We need to build ASI to save us from non-AI extinction risks.

However, that argument doesn’t work. In the near term, AI will make biological risks worse, not better. AI will accelerate scientific research, which will bring us closer to the level of knowledge necessary to build extinction-level pathogens. And in the long term, the way ASI eliminates biological x-risk is by taking control of the world.

Continue reading

Say you work at a private company that gives you stock options or RSUs. How should you value your stock?

• If you have a choice between getting more stock or more cash salary, how do you decide which to get?
• If you have the chance to sell some stock, should you do it?

Stock is risky and inflexible (especially if you work for a private company where you can’t easily sell shares), but you might be able to get it at a discount to its true value. How do you estimate how much it’s worth?

One heuristic you can use is to compare the stock against a risk-matched index fund. What would happen if you used the cash to buy a leveraged index fund with the same level of risk as the company stock? If the leveraged index fund has a higher expected return than the company stock, that means cash is probably better. (The reverse is not necessarily true because company stock can have other downsides, which I will get into later.)

Continue reading

Image credit: Jebulon

To prevent superintelligent AI from killing everyone, I would like there to be a strong international agreement banning the development of ASI until it can be proven safe. But that sort of agreement requires a lot of political buy-in and coordination. In the meantime, it may be easier to get light-touch AI safety regulations passed. To what extent do weak regulations decrease extinction risk?

In this post:

• Part I discusses routes by which weak regulations can reduce extinction risk. [More]
• Part II considers some downsides of weak regulations. [More]
• Part III reviews specific categories of weak regulation and how they might reduce risk. [More]

Continue reading

Depth-first plans lay out a path from here to aligned superintelligent AI. We need those kinds of plans. But depth-first plans depend on many assumptions: “We will make AI safe by doing step 1, then step 2, then step 3.” Step 1 only works under condition A, step 2 requires condition B, step 3 requires condition C. If A or B or C is false, the whole plan fails (and there’s a good chance we all die).

Consider Google’s safety plan from April 2025. To my knowledge, this is the best among the frontier AI companies’ plans.¹

Google’s plan depends on a series of conditions:

Continue reading

They say exercise improves sleep quality. Is that true for me?

To test this hypothesis, I took my daily calorie expenditures from the Apple Health app and correlated them with that night’s sleep time.¹ I also included caffeine intake as a potential confounding variable.

The hypothesis: when I exercise more, I’ll get better rest that night, and therefore wake up earlier.

The results:²

name	coef	t-stat	p-value
intercept	9.0134	65.072	0.0000
calories	-1.6844	-6.967	0.0000
caffeine	0.4157	9.404	0.0000
R²	0.2409

I sleep 10 minutes less for every additional 100 calories of exercise. Exercise plus caffeine explained 24% of the variance in my sleep time; exercise alone explained 6.6%.

The trend shows up whether or not I have caffeine:

Data are binned into increments of 100 calories. Any bins with fewer than 5 data points are not displayed. Vertical lines show the 95% confidence intervals for each bin.

Continue reading

A few years back, I got a big pile of money from working at a tech startup. I put a lot of that money into a donor-advised fund. Since now I make hardly any money, that DAF might represent the majority of my lifetime donations. How much of my DAF should I donate per year?

In particular, how much should I donate in light of short AI timelines?

I created a simple model to answer this question.

Continue reading

TLDR: I basically don’t.

Continue reading

Previously, I wrote about how investors can simulate leverage via concentrated stock selection. That’s still true as far as I can tell. However, I also wrote something that I now believe to be false: concentrated equal-weighted factor portfolios have alpha on top of value-weighted factor portfolios. The numbers I found before were not wrong per se. However:

• The alpha came primarily from small-cap and micro-cap stocks. That alpha may not be feasible to capture, or it may be defeated by trading costs; and historical estimates of micro-cap returns are biased upward because closing prices do not accurately represent the average investor’s trade price (Blume & Stambaugh (1983)¹).

  When I constructed hypothetical factor portfolios that had high concentration but screened out small-caps, the results did simulate leverage—they had higher returns and volatility than diversified factor portfolios—but alphas were not consistently positive.

• In the United States (where the data goes back the furthest), the alpha only shows up over the full data series (1927–2025). When restricting to 1964 onward, the alphas are close to zero.
• Concentrated value and momentum had positive alpha; but when I tested two new factors, profitability and investment, they each had negative alpha.

Continue reading

If civilization advances its technological capabilities without advancing its wisdom, we may miss out on most of the potential of the long-term future. Unfortunately, it’s likely that that ASI will have a comparative disadvantage at philosophical problems.

You could approximately define philosophy as “the set of problems that are left over after you take all the problems that can be formally studied using known methods and put them into their own fields.” Once a problem becomes well-understood, it ceases to be considered philosophy. Logic, physics, and (more recently) neuroscience used to be philosophy, but now they’re not, because we know how to formally study them.

Our inability to understand philosophical problems means we don’t know how to train AI to be good at them, and we don’t know how to judge whether we’ve trained them well. So we should expect powerful AI to be bad at philosophy relative to other, more measurable skills.

However, there is one type of philosophy that is measurable, while also being extremely important: philosophy proofs.

Continue reading