Philosophical Multicore

On Democratizing ASI to Preserve Civil Liberties

Mon, 10 Aug 2026 00:00:00 -0700

I continue to believe we should pause frontier AI development. Any discussion of alternative strategies should be thought of as planning for contingencies.

A unifying driver behind many post-alignment risks—catastrophic risks that remain even if we solve the alignment problem—is that by strong default, ASI would end liberal democracy. Liberalism—in which people have individual rights, autonomy, and the ability to choose their own destiny—is an important force protecting human welfare.¹ When people are free, we are reasonably good at making our lives better of our own volition.

Many post-alignment risks have a certain flavor. AI-empowered terrorism; coups; permanent dictatorships; concentration of power. Those risks already exist today (and existed 20 years ago), but they’re mitigated by the fact that power is relatively evenly distributed across people. The most powerful person in the world doesn’t have an extraordinary advantage over the 10th-most-powerful person. ASI could change that.

If people still have civil liberties post-ASI, that will only be because the controllers of ASI allow us to have them.²

One way of thinking goes: AI will be extremely powerful. If everyone had their own personal AI, we could each use it to protect our own interests, and things will turn out okay for us. But how do you get there? It’s not going to happen automatically, but it may be possible to set up a gradual process to keep power balanced.

Contents
Democratizing AI vs. putting a democratic government in charge of AI
A sketch of how we might democratize AI
Two non-obvious issues with democratizing AI
- Liberalism only protects those inside it
- AI proliferation increases catastrophic risks from competition
Notes

Democratizing AI vs. putting a democratic government in charge of AI

The “democra” words (democracy, democratize) are overloaded with meanings, so I want to be clear about which meaning I’m referring to. By “democratizing AI”, I’m talking about ensuring that many people have access to AI. One could also speak of a singleton AI that’s controlled by a democratic government, or a singleton AI that’s democratically controlled (people vote on what the AI should do). As Andy Masley writes, “homogenizing democracy” (as he calls it) entails forcing everyone to do what the majority wants. That’s not good. That’s why the title of this article speaks of “preserving civil liberties”, rather than “preserving democracy”.

(To be fair, I do wish we would get to vote on whether AI companies should be allowed to build unsafe recursively self-improving superintelligence, rather than them getting to unilaterally build it and put all our lives at risk.)

A sketch of how we might democratize AI

To be clear: democratizing AI is not a great plan. But it’s less bad than a lot of other plans.

To be clear x2: for “democratizing AI” to be a “less bad plan”, it has to be implemented with particular care. Otherwise, it’s just a bad plan. The obvious way of democratizing ASI doesn’t work:

Someone has to build it and then voluntarily give it to everyone. What if they decide not to do that?
Offense/defense balance issues: if attacking is easier than defending, then bad actors can use their personal ASI to do catastrophic damage.

I doubt that it helps with the second problem, but a gradualist approach has some chance of sidestepping the first. (By which I mean it has maybe a 1–10% chance of working.)

Here’s the plan:

Step 1. Give everyone their own personal smart-but-not-superintelligent AI assistant.
Step 2. Make a marginal improvement to AI capabilities, so that the next-gen AI isn’t enough of an improvement to defeat all the previous-gen AIs combined.
Step 3. Give everyone the new and improved AI.
Repeat.

The idea is that if the leading AI developer decides not to do Step 3 (instead keeping the new AI for themselves), then they are risking a conflict with the rest of the world, and they can’t win that conflict.

There are many ways this plan could go badly, which are left as an exercise for the reader.

What we should really do is pause AI development. Specifically, major countries should sign an international treaty agreeing not to build dangerous AI. The plan to “democratize AI” is riskier, but at the same time, it doesn’t seem easier? In order to preserve the balance of power, you need AI capabilities to advance slowly, and you need some way to enforce that. In short, you still need an international binding agreement not to build AI [except under certain narrow conditions], and not to use recursive self-improvement. You still need enforcement mechanisms like transparency and GPU monitoring; you still need a way for counterparties to shut down AI development if one party goes rogue (something like a remote kill switch on all the data centers).

Two non-obvious issues with democratizing AI

Democratizing AI has many issues. I will make note of two particular under-discussed problems.

Liberalism only protects those inside it

Even if we somehow succeed at preserving personal liberties by democratizing AI, and we succeed at making sure the AI is actually good for people, that still doesn’t get us a good outcome. Most sentient beings alive today are non-human animals. Today’s liberalism has failed to help them; almost all animals live lives much worse than humans’. Democratizing AI would not straightforwardly improve on the status quo. There is no realistic scenario where (e.g.) every factory-farmed chicken has her own superintelligent AI assistant.

If we preserve liberalism-for-humans, that does not clearly flow into good outcomes for non-human animals, digital minds, and whatever other sentient beings may exist in the future. Beings who cannot assert their rights do not automatically benefit; other measures are required to ensure their well-being.

AI proliferation increases catastrophic risks from competition

Having many ASI agents with different goals could cause tremendous suffering via retributivism, war, unfortunate decision theory, or agents threatening torture (and making good on those threats) to incentivize other agents to do what they want.

Center on Long-Term Risk and Center for Reducing Suffering have discussed this concern in more detail. For example, see Safe Pareto Improvements Research Agenda and Agential s-risks.

Notes

And the lack of liberalism for animals has a lot to do with why life is so miserable for them. ↩
I read a version of this statement in a LessWrong comment, but I couldn’t find the original source. ↩

Links for August

Mon, 03 Aug 2026 00:00:00 -0700

Here’s some cool stuff I’ve enjoyed recently.

FloatHeadPhysics is a YouTube channel that breaks down difficult physics concepts in a way that makes them clear. One of my favorites: I finally understood Schrödinger’s cat! The video talks about the double slit experiment, and explains why electrons don’t go through slit 1 or slit 2, but they don’t go through “slit 1 or 2”, nor do they go through “both slit 1 and slit 2”, nor “neither slit 1 or 2”. You can imagine what experimental results you’d get if any of those were true, and the double slit experiment doesn’t show any of them. The only thing you can say is that the electrons go through “a quantum superposition of slit 1 and slit 2”. It’s hard to explain what that means is because English has no other term to describe it.

Does A Calorie Surplus Help with Cardio? (Outlift). It’s well known that the best way to put on muscle is to eat at a calorie surplus. Is the same true for improving heart health? There are no studies directly addressing the question, but our best guess is that the answer is no. The heart is a relatively small muscle—an elite athlete’s heart only weighs 0.1 pounds more than average—and the heart is a top priority, so the body keeps it maintained even while on a calorie deficit.

Another consideration not mentioned in the article is that a calorie deficit encourages your cells to produce new mitochondria, which improves metabolic health (Civitarese et al. 2007¹).

Puzzles for Everyone (Richard Y. Chappell). Ethical paradoxes are not just a problem for utilitarians. Every moral theory must have a way of answering them.

A conversation on concentration of power (Joe Rogero). In a dialogue with a fictional interlocutor, the author explains why he finds it unlikely that “a few would-be technocrats build superintelligence and use it to rule the world forever.” This dialogue did a lot to clarify my mental model of what “alignment” means. It’s not explicitly about why alignment is hard, but it does an excellent job of bringing up the many philosophical problems that alignment discourse often ignores, and it helps explain why MIRI focuses on misalignment to the exclusion of all other problems that superintelligent AI could bring.

Some excerpts:

Norm: Picking a sort of random example, let’s say Elon Musk makes an AI. He says he wants it to be “truth-seeking” but I don’t think that’s actually what he’d ask it to do; imagine it just sort of does whatever he wants.

Joe: Suppose you are the AI in question. How do you evaluate what Elon wants? … [If Elon asks you to terraform Mars], can you chop down the California redwoods to make room for solar panels and factories? Can you chop down half of them? What does Elon value more, an ancient wonder of the world or getting to Mars a few months faster?

…

Joe: Then you have the problem that Elon isn’t even internally coherent in his preference ordering. Humans are kind of dumb like this; we work at cross-purposes to ourselves all the time. Many of Elon’s decisions will be very predictably path-dependent, in the sense that he’d answer one way if you prompted him with X and another way if you prompted him with Y, and there’s a contradiction there. Even a very high fidelity model of Elon runs into this problem.

…

Joe: What if he asked you to do something really, really stupid? Say he gets drunk and orders you to quit screwing around and get to Mars as fast as possible, damn the redwoods, and you happen to know he’ll hate himself in the morning if you actually do that and then the redwoods are gone.

I don’t have zero worry about power concentration, but this dialogue illustrates that it’s hard to imagine an ASI that’s “aligned to a dictator” without being either good for everyone or bad for everyone (including the for dictator).

If wild animal welfare is intractable, everything is intractable (Mal Graham). Many people say wild animal suffering is intractable because it’s too hard to determine the indirect effects of your actions. But the same criticisms apply just as strongly to global health interventions. The author looks at several standards for judging indirect effects, and finds that by any of the standards, wild animal welfare does not look more problematic than other interventions.

Levering Up to Do Good (AQR). Philanthropists with taxable investments can do more good by levering up their investments, donating the winners, and tax loss harvesting the losers. Adding leverage amplifies the benefits of tax loss harvesting.

Separation from Hyperexistential Risk (Eliezer Yudkowsky). If you build an ASI that naively encodes a utility function for creating the best possible world, it’s only one bit-flip away from creating the worst possible world instead. And explicitly writing out your utility function makes it maximally easy for extortionists to take advantage of you. This essay asks the question: How do we specify our utility function in a way that avoids problems like these? (Turns out it’s a hard problem and the obvious ideas don’t work.)

Elden Ring by CaptainDomo and LilAggy in 1:44:25 - Games Done Quick Express 2024. I love competitive games, and I love Elden Ring. But wait, you say, Elden Ring is a single player game,² it’s not competitive! Well, that’s where you’re wrong. In Elden Ring Bingo, players compete to achieve game objectives on a bingo board, and whoever gets a bingo first wins. Bingo matches happen regularly on the Bingo Brawlers channel, but I like that this match in particular had a live audience.

Bonus video: Elden Ring by Mitchriz, LilAggy, YoJosherino, star0chris in 3:13:32 - Summer Games Done Quick 2026. Four players race to collectively kill all 207 Elden Ring bosses in the shortest time possible. My most recent Elden Ring playthrough took me 60 hours and even though I set a goal of killing as many bosses as possible, I could only find about 140 of them. The math says that if four clones of me had to kill every Elden Ring boss, it would take us 22 hours.

I wanted to make the future good, not save it (Lucie Philippon). To summarize in my own words: I wanted to help make the world into an awesome utopia. I didn’t want to be staring down human extinction and forced to try to prevent it.

Pro Climber sign up for a beginner course…. Professional rock climber Magnus Midtbø goes undercover to take a beginner lesson and proceeds to climb increasingly difficult routes that make his ruse increasingly difficult to maintain. I’m not a rock climber (I haven’t been bouldering since I was a kid), but I still found this video a lot of fun to watch.

The Train to Crazy Town (Richard Y. Chappell). If pondering moral philosophy leads you to extreme implications, when should you get off the train to crazy town? RYC argues that we don’t need to deny the implications of moral philosophy if we remember that (1) it’s better to form correct beliefs even if we’re not perfect at acting on them; (2) we should be uncertain about our moral views; (3) we should not put too much credence in plausible-sounding arguments, because human brains are easily duped.

A philosophical thought experiment: (SMBC Comics)

To finish off the links, here are some funny videos in increasing order of derangement.

You better not think that thought! (Chris & Jack). A mad scientist implants a thought bomb into the brain of super spy Axel Wolf.

That Scene in a Christopher Nolan Film When You Give Up Trying to Follow the Story (Michael Spicer).

Mom Hid My Game. Dunkey plays a few strange games.

laddergoat (Dopelives). laddergoat

Notes

Civitarese, A. E., Carling, S., Heilbronn, L. K., Hulver, M. H., Ukropcova, B., Deutsch, W. A., Smith, S. R. et al. (2007). Calorie Restriction Increases Muscle Mitochondrial Biogenesis in Healthy Humans. ↩
Elden Ring does have a PvP mode but I’ve never tried it. ↩

Notes on the possibility of moral progress

Mon, 27 Jul 2026 00:00:00 -0700

To achieve the best possible future, we must know what that future looks like. In other words, we need to solve ethics.¹

The problem of solving ethics is so large and abstract that it’s difficult to say useful things about. In lieu of any structured analysis, herein lies a collection of thoughts about the problem.

Contents
Acting in the face of moral uncertainty
Can we discover facts that resolve moral uncertainty?
Some normative claims evade fact-based analysis
Implications for how the future goes
Notes

Acting in the face of moral uncertainty

If we have persistent moral uncertainty between maximizing and satisficing² moral theories, then it’s not difficult to decide what to do in practice: allocate a tiny portion of the universe to satisfying the non-maximizing moral theories, and allocate the rest to the maximizing moral theories.
- Example: If theory A says we should fill the universe with welfareans, and theory B says we should preserve homo sapiens but it doesn’t much matter how many humans there are, then we can near-perfectly satisfy both theories by maintaining humanity in a small segment of the universe and giving the rest to welfareans.
- However, the distinction between maximizing and satisficing theories may be irrelevant because plausible satisficing theories still hold that it is a good thing to maximize The Good, even though it is not morally obligatory³ to do so. Satisficing theories would still want to fill most of the universe with The Good.
Given uncertainty between mutually incompatible maximizing moral theories, we have to choose. Allocating resources incorrectly would be catastrophically bad.

Can we discover facts that resolve moral uncertainty?

I believe so. I expect that we can eventually eliminate almost all moral disagreements purely by discovering facts. The fact-value distinction implies that we cannot 100% determine what we ought to do by discovering facts, but most of what look like terminal values disagreements are not truly terminal.⁴

Some examples:

Right now, we do not know how to weight different people’s experiences against each other. I strongly suspect that there are facts of the matter about how to weight experiences, and that these facts can be discovered empirically.
The problem of weighting experiences is downstream of the hard problem of consciousness. I likewise suspect that there is a definitive answer to the hard problem,⁵ and that we can, in principle, find that answer.
What is the nature of personal identity? Certain theories of personal identity rule out classes of moral theories. If personal identity is not metaphysically meaningful, then person-affecting views must be false—there is no relevant distinction between bringing a new person into existence and changing the life trajectory of an already-existing person. Changing a life trajectory creates new person-moments, which is (in this view) metaphysically equivalent to creating a new person.

There may be some way to salvage person-affecting views, but if so, that possibility would itself be a non-normative fact—i.e., person-affecting views may be permitted or ruled out purely by facts, without any moral stance required.
Another important (although slightly obscure) question is: given two identical copies of the same mind, are they experiencing “twice as much” as if there were only one copy, or “the same amount”?⁶ This seems like a factual question, not a moral one. I have no idea how we would answer this question, but it seems answerable in principle.
Harsanyi’s utilitarian theorem (see also Harsanyi (1955)⁷), which showed that if individuals have VNM utility functions, and if the Pareto principle⁸ holds over groups, then a version of utilitarianism must be true. The Pareto principle is a normative principle, not a factual one; the question of whether individuals ought to care about their own welfare is also a normative one. But I strongly suspect that there is a fact of the matter about whether an individual’s welfare can be described as a utility function, and there is a fact of the matter about how exactly that function is specified. Discovering those facts would get us at least part of the way toward a fully-specified theory of ethics.
It matters whether the universe is finite or infinite, and whether our actions can have finite or infinite influence. Infinite ethics poses troubling problems, but some (maybe all) of those problems can be resolved factually (or if they’re unsolvable, then their unsolvability is a factual question).

Note: I don’t think they can be resolved purely empirically. For example, by our understanding of the laws of physics, our actions cannot have infinite influence.⁹ But there is a nonzero probability that we are wrong about the laws of physics and that our actions can have infinite influence after all. No amount of empirical investigation can reduce our uncertainty to zero, but there may nonetheless be a mathematical solution to the problem.
Standard formulations of deontology may break down in light of the fact that you do not have certainty about the consequences of your actions, so you can never be sure that you’re not doing something impermissible by acting. (See also Nye (2014)¹⁰.) If so, those flavors of deontology are ruled out purely based on a factual analysis (no normative claims necessary).

Many value disagreements do not purely boil down to a disagreement about facts, but I expect they can be resolved anyway. Some examples:

People would rather donate money to a single identifiable person than to a much larger, but nebulous, group of people. This preference should break down upon reflection. Suppose Alice is offered the chance to donate to a single identifiable person. Then imagine an alternative world where Alice can donate the same amount of money to help that same single person plus several other people, but the single person is never identified. Surely she would prefer this.
I believe the disagreement between negative utilitarians and classical utilitarians¹¹ would be resolved if we knew how to directly compare experiences / if we solved the hard problem of consciousness.
- My guess is that negative utilitarianism is a mistake stemming from the fact that maximum suffering in humans far exceeds maximum happiness, and this creates the appearance that suffering is terminally more important than happiness.
People support animal welfare, but also eat factory-farmed animals. It’s conceivable that people in reflective equilibrium would resolve this inconsistency by throwing out their concern for animal welfare, but that seems unlikely.
I believe people reject the mere addition paradox due to scope insensitivity—an enormous population of slightly happy people is indeed better than a small population of very happy people. It should be possible to prove that this intuition is the result of scope insensitivity, that scope insensitivity is inconsistent with people’s other values.
- Alternatively, some argue that an enormous population of slightly happy people is not particularly good because the experiences are too uniform, and two copies of an identical experience is no better than one copy. If there is a fact of the matter about how to consider two copies of an experience, then this alternative view could be proven right.

Our understanding of philosophy is limited. What does it mean to do good philosophy? What qualifies as a good philosophical argument?

We could make progress on those questions. We have already made progress: Descartes innovated on rightly conducting reasoning and seeking truth. The significance of philosophical thought experiments is a recent development—the concept is pre-Socratic, but modern thought experiments are more refined and more useful. (On Wikipedia’s list of notable philosophy thought experiments, two thirds were invented after 1900, and over half were not developed until 1960 or later.¹²) Most modern concepts in moral philosophy come from the 1700s or later; philosophy of mind primarily comes from the 1900s;¹³ analytic philosophy improved on the methods of its predecessors, and did not emerge until the 1800s. All that suggests that civilization is indeed making progress on philosophy, even if the rate of progress is slow.

Some normative claims evade fact-based analysis

I hold some foundational moral beliefs that seem unrelated to descriptive facts. I cannot conceive of how an empirical or theoretical investigation could provide reasons to believe that these are true or false.

It seems self-evident that pleasurable experiences are good (and suffering is bad), in the same way it’s self-evident that I am conscious. This is difficult to dispute, and to my knowledge virtually all moral philosophers (and regular people) agree that pleasure is good and suffering is bad.
I find it hard to see how anything other than good or bad experiences could be good or bad, because where does the goodness or badness come from if it’s not being directly experienced by anyone?¹⁴ (A.K.A. welfarism.) However, many people believe that non-experiences can be innately good or bad, and I don’t see how we could resolve this dispute.
Other people’s experiences matter, not just my own. (This claim is uncontroversial, but still, I see no way to prove it, or even give any reason to believe that it’s true.)
All beings’ experiences matter equally. It does not matter who the experience resides in; all that matters is the intensity of the experience. This view has several corollaries:
- Welfare aggregates linearly across individuals.
- The total view of population ethics is correct.
- Speciesism is wrong—experiences of humans should not be given more moral weight purely due to species membership.
- If you live in the 1700s, it implies that slavery and misogyny are wrong.

The first and third claims (the goodness of pleasure/badness of suffering, and the principle of altruism) are widely accepted. Many people disagree with me about the second and fourth claims, and there is no visible path to resolving those disagreements—plus, I have internal uncertainty about whether they’re true, which I have no idea how to resolve.

Even though the first and third claims are uncontroversial, they still evade any attempt to explain why they’re true. It could be that we’re all wrong.

And yet, it’s possible to convince people about these sorts of normative claims. (Peter Singer made arguments that convinced many people, including me, of the principle of equal consideration of interests.) What’s going on inside people’s heads when they change their minds about seemingly terminal values? Or, what’s going on when I contemplate two conflicting intuitions and decide that one is more important than the other? We have no theory of what constitutes a good argument for a normative position. We have some idea about the sorts of arguments people find convincing, but not a great understanding of why, and no way of saying that people are right to be convinced by a particular argument.

Implications for how the future goes

This essay has posited that we are making progress in philosophy, and that most (maybe all) moral disagreements can be resolved by learning new facts. If true, what does that imply?

Smarter-than-human AI should be better than humans at discovering facts. That’s useful insofar as moral disagreements can be resolved by facts.
The positive vision in the epilogue of AI 2040 has every individual human controlling an equal part of the lightcone. How good an outcome is that? If it’s feasible to converge on moral beliefs, then in that scenario, people (with superintelligent AI assistants) will come to agree on ethics, and will shape the universe in the way it ought to be shaped. Some people may have persistently bad values (like maybe Putin, or maybe not), but if most people converge on good values, then most of the universe will be directed well.
How well would a Long Reflection work? It would provide more opportunity to discover ethics-relevant facts and to improve our understanding of how to do good philosophy. But it would also give amoral actors more time to seize power. The AI 2040 idea of “give everyone an equal share of the lightcone, and then let people cooperate if they want to” is plausibly better than a Long Reflection, and plausibly worse.
Over sufficiently long time horizons, natural selection takes over. The dominant ethical belief will be that the right thing to do is to spread one’s own genes at the exclusion of everything else. We need to solve ethics before that happens, or otherwise prevent that from happening.¹⁵
In the scenario where humans control the future, the principle I worry about most is impartial altruism. I worry that most of the people in control will simply not care to devote resources to helping others.
I worry much less about disagreements between altruistic people (e.g., between negative and classical utilitarians). I expect these disagreements can be resolved factually.

Notes

Or perhaps it would be more accurate to speak of solving axiology, i.e., “what is good?” as opposed to “what is right?” ↩
A maximizing theory holds that the right things to do is to maximize some quantity—usually, to maximize utility, although “utility” can be defined in various ways. A satisficing theory says that there are certain things we ought to do (e.g. don’t commit murder), but as long as we do those, we have “satisfied” our moral obligations. ↩
I’m hesitant to use the word “obligatory” because it creates confusion when talking about consequentialist theories. For more on this, see Richard Y. Chappell’s blog post Deontic Pluralism (2022) or his academic paper Deontic Pluralism and the Right Amount of Good (2020). ↩
There is a class of moral disagreements that can clearly be resolved by factual questions. For example, should I drive a bulldozer through a particular building? That entirely depends on the answer to the factual of “is this an empty run-down building that’s scheduled to be demolished, or it a house that somebody’s living in?” Those sorts of disagreements are not interesting for the purposes of this essay. ↩
Some people claim that there is no fact of the matter about the hard problem of consciousness. I find it hard to comprehend why anyone holds that view. There is, at minimum, a fact of the matter about whether I am conscious (and that fact is “I am conscious”). I find this position about as confusing as the illusionist view (i.e. the view that consciousness is an illusion and in fact there is no such thing as consciousness). ↩
Bostrom, N. (2006). Quantity of experience: brain-duplication and degrees of consciousness. ↩
Harsanyi, J. C. (1955). Cardinal Welfare, Individualistic Ethics, and Interpersonal Comparisons of Utility. ↩
The Pareto principle states that if outcome A is at least as good as outcome B for every person, and outcome A is better for at least one person, then outcome A is better overall. ↩
Sandberg, A. & Manheim, D. (2021) What Is the Upper Limit of Value? ↩
Nye, H. (2014). Chaos and Constraints. ↩
Here, classical utilitaranism refers to any flavor of utilitarianism that gives equal weight to happiness and suffering. ↩
Dates are pulled from Wikipedia and aggregated by Claude Opus 4.8 (chat source). Of the 41 philosophy thought experiments on Wikipedia’s list, there are 13 from pre-1900, 5 from 1900–1959, 20 from 1960–1989, and 3 from 1990–present. I’m taking Wikipedia’s list as a reasonable proxy for notability. ↩
Thomas Nagel’s What Is It Like to Be a Bat?, written in 1974, is a more lucid exploration of consciousness than anything that came before it, and represented important progress. More broadly, much of the best work on philosophy of mind came from people who are still alive today. ↩
The parable of the Hrogmorph’s Heartstone is an attempt at justifying this intuition. (See also the extended parable.) ↩
Natural selection hasn’t taken over yet because humans are adaptation-executors, not fitness-maximizers. We have evolved the intelligence necessary to explicitly optimize for genetic fitness, but our big brains haven’t been around long enough for natural selection to push us in that direction.

↩

Letting myself look foolish

Mon, 20 Jul 2026 00:00:00 -0700

I prefer to avoid saying things that I suspect might make me look stupid. Over the last couple of years, however, I’ve made a conscious effort to write publicly about what I’m thinking about, even if I’m afraid it might sound dumb.

Sometimes I have this feeling, like:

I have an observation about the world, about something that doesn’t seem to add up. Probably I’m missing something, and everyone but me can easily see what’s going on, and if I bring it up, then everyone will know how foolish I am for missing this obvious thing.

Sometimes I’m right, and I really was missing something obvious. But some of my most well-received writings turned out to be ones where I had this feeling, and I was ambivalent about whether it was worth saying anything.

Lately, I’m making more of an effort to overcome that feeling, and say the stupid thing anyway, because it might turn out not to be stupid.

However, it’s not as simple as “be willing to say the stupid thing, because you might turn out to be right.” Maybe my stupid-sounding thoughts are wrong, actually. But that’s fine too. If I say them out loud, one of two things will happen:

My stupid thoughts turn out to be right, in which case I’ve said something useful.
I’m missing something obvious and my stupid thoughts really are stupid. In that case, people will see that they’re stupid. Nobody will be swayed by my stupidity. (At least they’re less likely to be swayed by it.)

I want people to hold true beliefs and do good things. If I make a transparently stupid argument, I won’t lead anybody astray because they will all be able to see how bad my argument is. Not that I want people to see me making stupid arguments, but as long as nobody’s convinced, I haven’t done any harm. And that’s ultimately what matters. My feelings might be hurt, but at least I didn’t lead anybody astray.

Perhaps I should be concerned about the possibility that I make a stupid argument, but I accidentally convince people that I’m right. I do worry about that sometimes. I can mitigate the risk by making my arguments clear, and avoiding “debate tactics”. I should straightforwardly lay out my reasoning so that if it’s bad, people can see through it. Not that I don’t fall into debate-style rhetoric from time to time, but I try to avoid it.

My stupid thought might be something that’s obviously wrong, and everyone sees it but me. Another flavor of stupid thought is the one that’s so obviously right that it’s not even worth mentioning. But something that’s obvious to you might not be obvious to everyone. Even if an argument is readily apparent to most of your audience, they might still appreciate that someone took the time to put it in writing.

My last post was inspired by Katja Grace’s recent post, AI pause: the case for ASAP. Everything in her post was something I had already thought of. All the ideas in it seemed “obvious”. But I didn’t write the post, and she did,¹ and it’s one of my favorite posts I’ve seen recently because it takes the time to point out something obvious that nobody had ever pointed out so directly before.

I did Inkhaven in part to force myself to ignore the voice in my head saying my ideas are stupid. Since then,² I’ve committed to a goal of writing a post every week, in part to force myself to keep thinking, but also to overcome the inner critic that doesn’t want to publish my thoughts.

Even if you’re committed to saying obvious or obviously-wrong things, it’s still not easy to notice those thoughts. Case in point: The concept of “force yourself to write down your foolish ideas” has been in my head for two years now, but I never thought to write a post about it until today.

Is this post stupid? Probably a bit, yeah. But I’m publishing it anyway.

Notes

Well, my version would’ve been worse. Not to say I’m a bad writer, but there’s always somebody better than you. ↩
Well, technically, since March. I got a bit burned out after publishing 39 posts in November. ↩

Pausing AI at human level seems harder than pausing ASAP

Mon, 13 Jul 2026 00:00:00 -0700

Some people think we should pause AI, but not now. They say we should wait until AI reaches human level,¹ because:

It’s not (catastrophically) dangerous until after then.
Human-level AI will help us do safety research.

Alternatively, other people (like me) think we should pause AI as soon as possible.

Katja Grace wrote a nice concise case for pausing ASAP. I have something I’d like to add: pausing at human level seems harder than pausing ASAP.²

Pausing ASAP sounds hard. It will be hard to get international coordination around an AI pause, and implementing a pause sounds hard even if we can agree to it in principle. But pausing ASAP still seems easier than pausing at human-level AI, for several reasons:

Human-level AI is highly economically valuable, and therefore there is a great temptation to keep going. The monetary incentive to build increasingly-powerful AI will be intense, and industry lobbyists really won’t want to pause AI development. It seems hard to pause when the economic incentive to continue is so great.
If AI is smart enough to accelerate safety work, then it’s also smart enough to accelerate improvements in AI capabilities. And it will probably be disproportionately good at the latter: capabilities improvements are easy to measure, and AI tends to be disproportionately good at easily measurable tasks. (Recent AI models have seen bigger improvements in math and coding abilities than in writing or philosophy.) If AI R&D used to require a team of PhDs, and now all it requires is someone in a garage with access to the latest AI model, then it’s harder to enforce a pause because clandestine AI research is harder to catch.
This next argument is more about a unilateral pause than a coordinated pause, but: Some say that the “good” AI developers need to push the frontier to maintain their lead, and that they should wait until the last minute to burn their lead to work on safety. Burning their lead at the end provides the maximum uplift from AI-assisted safety work. However, AI developers always face a choice between an uncertain downside (keep going, and possibly kill everyone) vs. a certain downside (pause, crater your profit potential, and possibly some other developer kills everyone anyway). I cannot foresee them making a rational risk assessment under those circumstances. There is too much pressure to distort their beliefs in favor of continuing to push the frontier.
If we had technology that could replace human labor, but it’s cheaper, faster, and can be copied as many times as one wants, how would that change the economy and society? I don’t know, but I bet it would change a lot. That level of disruption makes the world unpredictable. It seems risky to follow plans along the lines of, “let’s wait until this technology radically transforms society, possibly making things totally unrecognizable, and then implement our plan after that. Surely it will still work!”

An important counterpoint:

The general public does not like AI. If AI starts taking people’s jobs, then people will really dislike it. The unemployment effect of AI may create enough political will to pause that it outweighs out the economic incentive effect.

I hope this final point is strong enough to make pausing much easier in the future (hopefully the near future). But that doesn’t mean we shouldn’t try to pause ASAP.

Notes

The concept of “human-level” does not have a single agreed-upon definition, but let’s say an AI is human-level if it can do most job as well as or better than skilled humans. ↩
By the time we get political buy-in and the policy frameworks necessary to pause, “ASAP” might already have turned into “at human-level AI”. ↩

Training AI to be better at correctness than persuasion

Mon, 06 Jul 2026 00:00:00 -0700

I continue to believe we should pause frontier AI development. Any discussion of alternative strategies should be thought of as planning for contingencies.

“Super-persuasive” AI is dangerous because a misaligned ASI could persuade humans to help it take over. But setting that aside, even if we manage to make ASI friendly, it may provide super-persuasive but mistaken guidance that permanently sets us down the wrong path.

This post focuses on the danger of an aligned super-persuasive AI that simply comes up with the wrong answers.

To achieve an ideal future, we need to solve difficult philosophical problems. With philosophical arguments (especially in moral philosophy), we have no clean way to judge correctness. I worry that ASI will develop an extremely persuasive but ultimately badly misguided set of ethical principles.

We want ASI to be disproportionately good at being correct, and disproportionately bad at persuading humans of incorrect arguments. We especially want this to be true for fuzzy, difficult-to-judge questions.

Unfortunately, it’s much easier to achieve the opposite.

Suppose an AI developer wants to improve AI’s skill at philosophy. An obvious approach is to ask the AI to generate philosophical arguments and then have professional philosophers judge their quality. But this trains AI to produce arguments that sound persuasive to philosophers, not arguments that are correct.

In April, I wrote Can AI make advancements in moral philosophy by writing proofs? I thought, maybe we can sidestep the persuasiveness problem by ensuring that AI is provably correct. But that only works on a subset of questions.

It might be possible to do better than that by training AI to be disproportionately good at making correct arguments, without being super-persuasive. Here’s a sketch of how we might do that using reinforcement learning:

Step 1. Take some domain with verifiable correct answers—let’s say math.
Step 2. Ask the AI model to generate mathematical proofs.
Step 3. Ask mathematicians to read the proofs and judge their correctness. Meanwhile, use a proof checker to formally verify them.
Step 4. Give positive and negative reinforcement based on proof correctness, but give extra negative reinforcement for every incorrect proof that mathematicians judged as correct.

You train AI based on human feedback, but you give negative reinforcement for positive feedback (in cases where the feedback is provably wrong).

However, this approach has some major issues:

How do you ensure that correctness-on-provable-tasks generalizes to correctness-on-fuzzy-tasks?
This methodology might accidentally teach AI how to be persuasive so that it can do the opposite—it maximizes reward by learning what people find persuasive, and then making sure never to do that.¹ The resulting ASI might be even more super-persuasive than one that’s purely trained on correctness.

It’s unclear whether this specific methodology would net decrease or increase risk. (Also it probably wouldn’t work, for the same reasons that reinforcement learning is probably inadequate for preventing misalignment in general.) But at least it represents a deviation from the status quo, in which AI models are disproportionately good at sounding persuasive, and naive training methods will continue to make this problem worse.

My intuition is that there ought to be some way of training AI to be good at correctness and bad at persuasion (not just pretending to be bad to get a reward). My sketched proposal isn’t quite it, though.

Even if the strategy proposed above is fundamentally flawed, the question remains an important one: How do we build AI that can figure out what we should do, and that won’t persuade us to do the wrong thing?

Notes

Sort of like emergent misalignment: current-gen LLMs understand a general notion of “being bad”, and when they’re tuned to behave badly along one narrow dimension, they start behaving badly in lots of other ways too. ↩

AI will make biological extinction risks worse before it makes them better

Mon, 29 Jun 2026 00:00:00 -0700

An argument goes: If we don’t build aligned artificial superintelligence, we risk driving ourselves extinct for some other reason. We should rush to build ASI quickly, in spite of the risks—the longer we wait, the more vulnerable we are to extinction from a different cause.

Other than ASI, the biggest extinction risk is synthetic biology. Some lab could (accidentally or on purpose) develop a highly transmissible, 100% fatal super-plague that wipes out humanity.

An aligned ASI could stop that from happening by shutting down dangerous biological research, or by developing advanced countermeasures that stop the spread of deadly infections. So the argument goes: We need to build ASI to save us from non-AI extinction risks.

However, that argument doesn’t work. In the near term, AI will make biological risks worse, not better. AI will accelerate scientific research, which will bring us closer to the level of knowledge necessary to build extinction-level pathogens. And in the long term, the way ASI eliminates biological x-risk is by taking control of the world.

Contents
In the near term, AI makes biorisk worse
AI can’t control scientific progress unless it controls everything
Low biorisk trades off against high AI takeover risk
Accelerating AI development is not a good way to reduce biorisk
This is yet another illustration of the fact that we don’t know what “aligned AI” means
Notes

In the near term, AI makes biorisk worse

Some people imagine that AI models would accelerate defensive research while refusing to assist with developing bioweapons. This plan has two minor issues and one fatal one.

The first minor issue: Current AI model refusals are not robust, and there are workarounds to get information out of them for people who want to. It’s very hard for AI developers to patch all holes, but the jailbreakers only need to find one.

The second minor issue: Even if the leading AI developer makes their model safe and un-jailbreakable, at least one of their competitors will probably fail at that task.

The fatal issue: It’s not just about what AI assistants can do for humans. It’s that AI accelerates the rate of scientific progress. As state of knowledge improves for humanity in general, it becomes possible for humanity to develop existentially risky pathogens, even if AI does not assist directly. It seems impossible to advance biological science while surgically preserving ignorance on just those bits of knowledge that are required to engineer pathogens.

AI might refuse to participate in gain-of-function research, and that would be better than not refusing. But suppose I’m an evil scientist and I want to develop a 100% lethal airborne pathogen. Here in the year 2026, I can’t do it. Even if I’m on the cutting edge of medicine and biology, I still won’t be able to create the “extinction pathogen”, because that would require a level of scientific understanding that humanity simply hasn’t achieved. If AI advances science in general, it will push me closer to my evil goal of killing everyone with bioweapons.

There is the question of “offense-defense balance”: is it easier to develop deadly pathogens, or easier to protect people against pathogens? That question matters in many contexts, but it’s not relevant here. At our current level of scientific understanding, we have ~zero ability to develop extinction-level bioweapons. If our understanding becomes sufficiently advanced, then that ability will move from zero to nonzero, regardless of the offense-defense balance.

Leaving AI out of the picture, humanity will probably have the knowledge necessary to make extinction-level pathogens within the next hundred years. If AI causes a hundred years of progress in the next decade, then the evil scientist will be able to engineer their extinction pathogen by 2036, thanks to AI—even if the AI itself doesn’t directly participate in the creation of the pathogen.

By 2036, assuming AI hasn’t killed us yet, biorisk will be higher than in the alternative 2036 where AI capabilities stopped improving. Would 2036-biorisk-with-AI be higher than 2126-biorisk-without-AI? Maybe not—maybe AI scientists would be safer than human scientists per unit of research effort. But at minimum, AI-accelerated science is more dangerous per unit of time. AI acceleration means the high-risk period starts sooner, and it means we have less time. Less time to identify risks, less time for policy-makers to respond, less time to consider what direction we should go in. Speedrunning through a century of progress in a decade makes it much harder to manage the risks as they come.

AI can’t control scientific progress unless it controls everything

The only way to accelerate scientific progress in biology without increasing x-risk is for AI to have complete control over scientific capabilities—basically, it has to be impossible for any humans to use their increasingly-advanced knowledge of biology to develop bioweapons. I don’t see how to do that unless all science is being done by AI, with humans not participating anymore.

Many people have a vision of the future in which humans will coexist with advanced AI, and we will remain in control of the steering wheel. But if humanity is in control, how can AI prevent us from developing powerful bioweapons? We can’t have it both ways.

One might say, “Governments will have to prevent terrorist and mad scientists from developing bioweapons.” To which I say, indeed they should do that. But AI makes governments’ jobs harder on that front, not easier, unless AI has totalitarian grip on society—at which point we’re back to the scenario where humans lose control over the future.

Another attempt at escaping the dilemma: Let the government control AI, and AI control everyone else. Even in the world where the government is democratically elected, that world is starting to sound like an extreme version of Bad Definitions Of “Democracy” Shade Into Totalitarianism, in which your life is fully controlled by AI, and the only time when you get any say in the matter is at the voting booth.¹ I can imagine much worse outcomes than that, but it’s not what I would describe as a happy ending.

Low biorisk trades off against high AI takeover risk

AI increases biorisk until it’s powerful enough to completely shut down any danger. Therefore, the way to minimize AI-driven biological x-risk is to have a very short window of time between “AI is smart enough to accelerate biological research” and “superintelligent AI controls everything”. But if that window is short, then we have little time to solve the alignment problem, and little time to steer AI while we are still in control of the future. AI-enhanced biorisk is lowest in the worlds where AI takeover risk is highest.

People with relatively low credence in AI takeover risk tend to expect a slow takeoff. But in a slow takeoff, AI makes biorisk worse well before it’s smart enough to robustly prevent extinction-level pandemics.

Accelerating AI development is not a good way to reduce biorisk

We don’t currently know how to build bioweapons that kill everyone, and eventually we will know how to do that.² Much like how, in 1900, there was no risk of nuclear winter because we didn’t yet know how to build nuclear weapons.

Scientific progress brings prosperity, but it can also enable dangerous new technologies. General biology research might even be harmful on balance due to increasing extinction risk—I don’t have a well-informed view on whether that’s true. What I can say is that the following argument does not hold up:

We need to accelerate AI progress so that it can save us from biological extinction risks.

Consider the neighboring argument, “we need to accelerate AI progress to create medical advancements.” That argument is failing to do basic cost-benefit analysis (the risk of extinction is not outweighed by short-term improvements in medicine), but at least it’s true that AI could, indeed, improve the state of medicine. “We should accelerate AI to reduce biological x-risk” isn’t even clearly correct about the upside.³

This is yet another illustration of the fact that we don’t know what “aligned AI” means

In the (possibly brief) window where AI is smart enough to do scientific research but doesn’t yet control the whole world, AI increases biological x-risk by improving humanity’s knowledge of how to develop powerful bioweapons. After that window, what happens? If we’re in a world where ASI is powerful enough to reduce extinction risk to zero, what does that world look like, and what should it look like? I find it difficult to imagine what sort of radical transformations to civilization would be necessary to achieve a total elimination of x-risk.

Some people imagine a future where everyone owns their own galaxy. How can we make meaningful claims about x-risk when the future looks that weird? If I can own a galaxy (whatever that means), maybe some other person can deconstruct a handful of planets to build an army of 100% deadly super-nanoviruses and send them throughout the universe at 99.9999% the speed of light so that they kill everyone before anyone even sees them coming. Or something.

Many people have an intuition that aligned ASI will fix everything and the world will be great. But if we succeed at figuring out how to get ASI to do what we want, how do we then specify its behavior such that we get a good outcome? Some people hand-wave the problem away by saying “the ASI will be smart, it will help us figure out what to tell it to do.” Much like alignment bootstrapping, this answer has a chicken-and-egg problem: how can the ASI figure out what you should tell it to do if you haven’t yet told it how to determine what you should tell it to do?

(If an “assistant ASI” comes to you with some answer, and it’s far smarter than you, how can you judge whether its answer is correct?)

The biorisk case is an example of the general problem that we don’t know how to specify how an ASI should behave. Others have discussed this problem in more general terms, including:

A Conflict Between AI Alignment and Philosophical Competence by Wei Dai (2025)
Intent alignment seems incoherent by Joe Rogero (2025)
Many individual CEVs are probably quite bad by Villiam (2025)

The concerns with biological x-risk are a specific illustration of the general problem. How, exactly, do you build an AI that prevents humans from killing each other with bioweapons, but without making things horrible as a side effect?

Notes

To be clear, I do not believe this scenario is at all likely. I’m using it as a hypothetical way of escaping the dilemma, to illustrate that even this “solution” still isn’t something we want. ↩
Unless AI kills us first. ↩
This brings to mind an important (but off-topic) question: if scientific advancement increases existential risk, but it’s also essential to improve standards of living, how should we proceed? We don’t have an answer for that question yet, but whatever we come up with, I imagine it would be fair to summarize as: “We proceed carefully.” As we learn more about what sorts of advancements are dangerous, we can implement mitigations.

If AI rapidly accelerates progress—even assuming AI itself doesn’t kill everyone—then it will be difficult to implement mitigations as we go, because the time gap between “top scientists foresee a dangerous technology on the horizon” and “anyone can develop this technology in their garage” will become much shorter.

(Another possibility is that humanity doesn’t solve the problem of how to advance science without introducing new x-risks. Instead, we solve AI alignment, and then AI solves every other problem.) ↩

Compare Your Company Stock to a Leveraged Index Fund

Mon, 22 Jun 2026 00:00:00 -0700

Say you work at a private company that gives you stock options or RSUs. How should you value your stock?

If you have a choice between getting more stock or more cash salary, how do you decide which to get?
If you have the chance to sell some stock, should you do it?

Stock is risky and inflexible (especially if you work for a private company where you can’t easily sell shares), but you might be able to get it at a discount to its true value. How do you estimate how much it’s worth?

One heuristic you can use is to compare the stock against a risk-matched index fund. What would happen if you used the cash to buy a leveraged index fund with the same level of risk as the company stock? If the leveraged index fund has a higher expected return than the company stock, that means cash is probably better. (The reverse is not necessarily true because company stock can have other downsides, which I will get into later.)

How to compare stock to a leveraged index fund

You can’t know for sure how risky your company stock will be, but you can estimate. Large publicly-traded companies (worth $100 billion or more) tend to have about double the volatility of the US total market, and mid-sized companies (worth around $1-5 billion) have about triple the volatility. (For more on these numbers, see The Risk of Concentrating Wealth in a Single Asset.) That means if you work at a large company, you’re looking at a similar risk as a 2:1-leveraged index fund. And if you work at a $1 billion company, you should compare your stock to a 3:1-leveraged index. For even smaller companies in the $100 million range, 5:1 leverage is more appropriate.¹

This graph shows standard deviation by market cap in the years from 1995 to 2015:

Private companies don’t really have standard deviations in the same way that public companies do, because their prices don’t change on a daily basis. But they’re still risky in the sense that if you wait to sell the stock, you might have to sell at a lower price (or even $0). It’s possible that public and private companies differ in some systematic way, where private companies of a given size are more risky (or less risky) than comparable public companies. But to keep things simple, let’s assume private and public companies work the same way.²

Suppose you have some employee stock at a $1 billion company. Based on historical averages, I’d assume that that stock is as risky as a 3:1-leveraged index. What does that imply?

If you have the choice between getting $1 cash and $1 worth of company stock, you need to expect your company stock to return at least 3x as much as the market to prefer the company stock over cash. If your company stock has a lower expected return than that, you’d get a better return with the same risk by just getting cash and then using it to invest in a leveraged index fund.

The beautiful thing about this method is you can get an answer without having to know anything about risk aversion. We levered up the index fund to the point where both potential investments have the same risk profile. That said, risk aversion can still matter because 3:1 leverage might be too much—you might prefer an un-levered index fund. The comparison only gives an answer in one direction:

If your stock has 3x the risk of the market, but less than 3x the expected return, then it’s not as good, and you’d prefer cash.
If your stock has 3x the risk of the market, and more than 3x the expected return, then this method can’t determine whether it’s better, because the answer depends on your risk preferences.

How to compare unequal dollar amounts

In many situations, you have to choose between unequal amounts of cash and stock. For example, maybe you’re offered a job at a startup, and you have a choice between:

$100,000 salary + some equity valued at $50,000
$75,000 salary + some equity valued at $100,000

So you can exchange $1 of cash for $2 worth of company stock. How can you figure out whether this is a good deal?

To make a direct comparison between $1 of a leveraged index fund and $2 of stock, we need to know how long we will hold the stock for. We rarely know precisely, but we can guess.

Suppose I expect my company to go public in about five years. Maybe I expect my company stock to return 10% annually above the risk free rate,³ and an index fund will return 5% above the risk-free rate,⁴ and therefore a 3:1-leveraged index will return an excess 15%.

(We care about excess returns rather than absolute returns because you have to pay the risk-free rate to use leverage. You could say the risk-free rate “doesn’t count”: you get to earn that rate no matter what, so you subtract it out when doing calculations.)

After five years, I believe I will be able to sell my company stock and do whatever I want with the money. How to I expect each asset to perform over those five years?

The index fund will start at $1 and compound at an excess rate of 15% (in expectation) over five years. 1.15⁵ = 2.01, so I will have $2.01 in expectation at the end of the period. Meanwhile, the company stock starts at $2 and compounds at 10% excess over five years, resulting in $3.22.

Therefore, if you’re comfortable with the risk (and that’s a big “if”), the stock is better than the cash in this case.

We can use this method to calculate exactly how much stock is worth the same as $1 cash (according to our guesses). In this example, if we get $1.25 worth of stock, we will end up with $2.01—the same amount as if we took the leveraged index fund instead. Therefore, $1 of cash and $1.25 of stock look equally good on this comparison.

Don’t forget that this method only gives an answer in one direction. If the leveraged index has a greater expected value than the company stock, we can say cash is better. But if the stock looks better than the leveraged index, we can’t say we should choose the stock, because it might be riskier than we want.

This method requires making some guesses about the future, and it embeds some assumptions that might be false. (For example, it assumes that the index fund and the company stock follow the same distribution shape.) It cannot conclusively tell you how much to value company stock, but I find it to be a useful lens for thinking about the problem.

I spent a while trying to figure out if there’s a simple way to compare cash to stock in the case where stock looks better than a levered index, but you’d prefer to take on less risk. As far as I can tell, there is no simple answer. You have to compare them the hard way—by coming up with your utility of money and then applying your utility function to each choice. Or, if you’re a normal person and not obsessed with quantifying everything like I am, you can just compare them qualitatively.

Calculator

I have included a simple calculator to determine how much company stock is worth the same as $1 in an index fund, according to the assumptions made in this post. It calculates these four numbers:

company standard deviation as a function of size
how much leverage an index fund would need to reach the same level of volatility
what expected return that leveraged index fund would get
how much company stock is equivalent to $1 in a leveraged index fund, taking into account the time until liquidity

Company valuation (in millions of dollars)
Expected years until liquidity
Expected annualized return for company stock (%)
Index fund expected return, nominal (%)
Index fund standard deviation (%)
Risk-free rate (%)

Company standard deviation
Required leverage on index fund
Leveraged index expected return
Dollars of company stock worth $1 in an index fund

Notes

Volatility (standard deviation) is not a perfect measure of risk. Compared to a leveraged index fund, an individual company has a much higher risk of going to $0. In a way, looking at volatility understates how risky individual stocks are. ↩
Many private equity managers like to pretend that private stocks are less volatile. Don’t listen to them. During market crashes, managers can get away with not re-rating the valuations of their private companies, and claiming no loss. But if they had to sell their shares during a downturn, the sell price would be lower. ↩
The risk-free rate is the interest rate you can earn while taking on no risk of losing money. It makes sense to subtract out the risk-free rate from any rate of return we’re looking at, because the difference shows us what we are actually getting in exchange for the risk we’re taking on. Subtracting out the risk-free rate usually makes the math much simpler. ↩
Investment writer Meb Faber frequently talks about the 5:2:1 rule: in the long run, stocks have returned 5% after inflation, bonds returned 2%, and the risk-free rate was 1%. According to this, the stock return over the risk-free return is 4%, but we can round it up to 5% for the sake of simplicity. ↩

A frontier AI company should shut down

Mon, 15 Jun 2026 00:00:00 -0700

Prior discussion: niplav’s shortform (2025); Planning for Extreme AI Risks (2025) by Joshua Clymer

A frontier AI company (any one, I don’t care which) should close shop and make an announcement along the lines of:

Powerful AI could end the human race. We are too worried that we don’t know how to make this technology safe. We have decided to shut down because we don’t want to be responsible for building the thing that kills us all.

A common refrain among safety-conscious AI developers: “it doesn’t matter if we stop building dangerous AI, because someone else will just build it instead.” Is that really true, though? If a multi-hundred-billion-dollar company comes out and says “We’ve concluded that our product is horribly dangerous, nobody knows how to make it safe, and there’s too high a risk that it leads to human extinction”, this won’t raise any eyebrows? This has no chance of spurring policy-makers into action?

Shutting down would make people say, holy shit, they are serious about this extinction risk thing. Shutting down sends a strong signal to governments that they should pay serious attention to AI x-risk.

It also encourages other companies to take safety more seriously. Right now, at least three AI companies have said something like, “maybe we’d prefer to slow down and pay more attention to safety, but then the other companies will plow ahead recklessly.” If one company decides not to plow ahead recklessly, and actually stops building existentially dangerous technology, that sends a hard-to-ignore message that coordination might be possible.

If a frontier AI company shuts down, will that work? Will companies work together to slow down? Will we get sane AI regulations as a direct result of the shutdown? Probably not. It won’t singlehandedly solve all the coordination problems. But it’s still a better idea than the current strategy of “race ahead while doing a dash of safety research on the side”, which is even less likely to work. By AI companies’ own admission, competitive pressures don’t allow them to slow down. Why would things change in the future? How are they going to align AI if they have to move at maximum speed? Even if they slow down somewhat, what if alignment is hard¹, and they can’t slow down by enough to properly solve the problem?

Counterpoint: If the most safety-conscious company shuts down, then it can’t do any more safety research.

I expect shutting down would be worth the tradeoff—companies’ safety research isn’t doing much to reduce AI takeover risk. But perhaps instead of shutting down, an AI company could reallocate 100% of its budget on some combination of safety research + global coordination to make AI development safer, and do just those things until it runs out of money. Think of how much more safety work a they could do if they dedicated all their resources to the problem!

(Some might argue that AI companies need to build frontier models so they have something on which to do safety research. That argument doesn’t make much sense when you think about it. There are a lot of kinds of research that don’t require frontier models,² they can do plenty of research on the models that already exist, and they can make deals with other companies to get access to their latest models.)

What if investors sue the company?

It is my understanding that a self-induced shutdown would be legal for Anthropic (which is a public benefit corporation). I’m not sure about OpenAI—it’s a for-profit now, but it’s still owned in large part by a nonprofit that’s allegedly obligated to put the benefit of humanity first.

More importantly, “we have to risk killing everyone because otherwise our investors might sue us” is not a serious position. I almost can’t think of a worse excuse.

Some people might believe that a safety-minded AI company should shut down under some circumstances, but not now. My question then is: Under what conditions should they shut down? How will we know when those conditions are met? And how do we know that they’ll follow through?

Notes

It probably is. ↩
Safety-minded AI companies treat alignment as an engineering problem, or treat philosophical problems as easy. There are critical aspects of the problem that can’t be solved by engineering (or that aren’t legible). You can work on those other aspects even if you don’t have frontier models. ↩

Science-driven stories are good for the same reason that character-driven stories are good

Sat, 13 Jun 2026 00:00:00 -0700

(Spoilers in this post are hidden with spoiler tags.)

What made Project Hail Mary so good? Among other reasons, it’s because the science drove the story, instead of the other way around.

Character-driven stories and hard sci-fi might take up opposite positions in the ancient battle of “people vs. things”; but when they work, they work for fundamentally the same reasons.

In mediocre “people”-focused stories, the plot dictates how characters behave. In great people-focused stories, the characters decide what happens.

In mediocre sci-fi, the plot dictates what science and technology can do. In great sci-fi, the science and technology constrain what routes the plot can take.

When the internal logic of the world makes sense, and everything fits, it satisfies “thing”-brained people in the same way that deep three-dimensional characters satisfy people-brained people. And when the science is inconsistent, it bugs thing-brained people in the same way that it’s no fun to watch the protagonist make stupid out-of-character decisions in service of the plot.

Project Hail Mary is an excellent example of what I mean by this. The central plot of Project Hail Mary naturally fell out of Andy Weir’s daydreams about rocket science. (spoilers for the first 10 minutes of the film)¹ The rocket equation dictates that the amount of fuel you need grows exponentially with how fast you want to go (by which I mean literally exponentially, not just “a lot”). So he thought, wouldn’t it be cool if we had a fuel source that could directly convert matter into energy? Maybe it could be some sort of microbe that absorbs sunlight. But we’d have to make sure none of the microbe gets onto our sun because that would be disastrous. Oh, there’s the story!² If he didn’t care about making the science accurate, he never would’ve thought up the plot in the first place.

(spoilers for midway through the film) Rocky’s biology was determined by the properties of his home planet. Rocky comes from a planet orbiting 40 Eridani A³, which orbits very close to its star. What properties would that planet need to have to support life? It would need a heavy atmosphere or else the air particles would fly away. That means light wouldn’t reach the surface, so intelligent life wouldn’t be able to see; therefore they’d probably use echolocation.⁴ The reason Rocky echolocates isn’t that Andy Weir thought it would be cool; his echolocational capabilities are an implication of the environment he evolved in.

The science doesn’t always have to make sense. Firefly doesn’t even attempt to explain how any of the futuristic technology works, and it’s one of the best shows ever made. But science-driven stories light up my brain cells in a way that other stories don’t.

Science-driven plot isn’t limited to science fiction. Brandon Sanderson is the prime example of a fantasy author who writes like this. He even has a system of Laws of Magic. The first two laws state:

Sanderson’s First Law: An author’s ability to solve conflict with magic is DIRECTLY PROPORTIONAL to how well the reader understands said magic.

Sanderson’s Second Law: Limitations > Powers

In other words, don’t shoehorn in some magic to make the plot go where you want. Magic should impose limitations, and those limitations should guide where the plot goes.

(Annie from Misery didn’t like plot-driven magic, either. She was just as rational and level-headed about this as I am.)

Science-driven plot and character-driven plot are fundamentally the same phenomenon: the plot emerges organically instead of being forced.

Let’s talk about Breaking Bad, which is my favorite TV show. The primary refrain in the writer’s room was, “Where is Walt’s head at?”⁵ The refrain wasn’t, “What awesome cool plot twist should happen next?” To figure out the plot, they took an intriguing agentic character (Walter White) and thought about what choices he would authentically make.

Interesting characters make the plot interesting. If you simply ask “what would Walter White do in this situation?”, the answer will be that he does something interesting (read: crazy). There’s no need to shoehorn anything. Similarly, if you come up with a sufficiently interesting scientific premise, the story can write itself.

Notes

I also read the book, but I don’t know how to convey how much of it is spoiled by this anecdote. Whatever fraction of a book is proportional to the first 10 minutes of a movie, I guess? ↩
Critical Drinker After Hours (2026). Drinker’s VIP Lounge - Andy Weir [Video]. Timestamp 35:53. ↩
except after Project Hail Mary was written, new research came out showing that what we thought was a planet was actually just fluctuation in solar output, and probably the planet doesn’t exist ↩
Penguin Random House (2026). Andy Weir on Balancing Science and Story [Video]. Timestamp 3:34. ↩
Dixon, K., et al. (2009–2013). Breaking Bad Insider Podcast. Various episodes.

(“various episodes” is code for “I listened to the podcast a long time ago and I don’t remember which episodes they talked about this in”) ↩

How valuable are weak AI safety regulations?

Mon, 08 Jun 2026 00:00:00 -0700

Image credit: Jebulon

To prevent superintelligent AI from killing everyone, I would like there to be a strong international agreement banning the development of ASI until it can be proven safe. But that sort of agreement requires a lot of political buy-in and coordination. In the meantime, it may be easier to get light-touch AI safety regulations passed. To what extent do weak regulations decrease extinction risk?

In this post:

Part I discusses routes by which weak regulations can reduce extinction risk. [More]
Part II considers some downsides of weak regulations. [More]
Part III reviews specific categories of weak regulation and how they might reduce risk. [More]

Contents
I. Ways weak regulations can reduce risk
II. Downsides of weak regulations
III. Specific policies, and how they might reduce extinction risk
My position on weak regulations
Notes

I. Ways weak regulations can reduce risk

Directly reduce extinction risk

Weak regulations can’t do much to decrease misalignment risk, but they can have small effects at the margin. GPU tariffs or moderate restrictions on GPU exports slow down AI development in other countries, and reduce competitive pressure to some extent.¹ Mandatory safety testing has some small chance of catching catastrophic issues before they happen.²

Empower future efforts to reduce extinction risk

What I really want is a global ban on superintelligent AI until it can be proven safe. To get that, we will need some regulations along the way. For example, regulators will need to know who has the ability to develop advanced AI systems, which means we need some sort of monitoring of AI developers or AI hardware.

Reveal warning shots

At some point before AI kills everyone, it might do something scary enough to trigger governments to pause AI.³ Weak regulations can make warning shots more apparent. If AI companies are required to publish safety tests, and there are legally mandated whistleblower protections, then it’s more likely that scary AI behaviors will come to light.

Shift the Overton window

Passing weak regulations in the near future may make politicians more amenable to strong regulations later on. (I say “politicians” rather than “people” because the general public already supports strong regulations on AI.)

Unfortunately, it’s not clear that that’s how it works—that small changes beget large changes. When I did a brief literature review, the results looked inconclusive. I can come up with examples of times when weak regulations were followed by strong regulations, and also times when they weren’t. Beyond that, there’s the causality problem: did weak regulations cause strong regulations, or were both caused by a trend in societal attitudes?

To my knowledge, the most rigorous (read: least-unrigorous) relevant research is Beaman et al. (1983)⁴, a meta-analysis on the foot-in-the-door effect. The paper found that the evidence on the effect was mixed, and sometimes pointed in the wrong direction.

II. Downsides of weak regulations

Opportunity cost

Time spent advocating for weak regulations could be spent advocating for strong regulations instead, which may be better. In fact, I think it probably is better, because (1) weak regulations are unlikely to prevent extinction on their own, and (2) we might not have much time before superintelligent AI is upon us.

But there are situations where advocating for weak regulations does not have any opportunity cost. If I publicly voice support for SB 53 or the RAISE Act, I’m not crowding out some stronger bill that those bills are competing with. They’re not competing with any other bills.

If you’re an AI safety org that spends most of its time advocating for strong measures, it costs you little to issue a statement in support of those bills,⁵ and indeed many orgs did do that.

Regulation fatigue

Weak regulations might beget strong regulations by shifting the Overton window. Alternatively, they might reduce the appetite for regulation: “we already have these rules in place, why do we need more?” As discussed above, the evidence is not clear on which direction the effect goes (if either).

Slows technological progress

All else equal, technological progress is good, and increases prosperity. Technological progress toward a thing that kills everyone is bad, but there will be good parts along the way. Slowing AI development means we get less of those good parts.

The reduction in extinction risk easily justifies the cost, but this is a real downside.

(This is a downside relative to no regulations, but not relative to strong regulations, which would slow progress by even more.)

May get in the way of AI companies implementing their own, more sensible, self-regulations

AI company leaders would have us believe this is the reason they’ve lobbied against regulations. I find it hard to believe that they will do the right thing on their own, and their track records are not promising.

III. Specific policies, and how they might reduce extinction risk

This section reviews some light-touch policies and possible paths to impact for each of them.

GPU export controls

Path 1: Reduce AI proliferation → reduce competitive pressure → make it easier to coordinate a pause.
Path 2: Slow down AI development in other countries → lengthen AI timelines → provide more time to work on safety.

Export controls are the closest thing to free win. AI safety advocates like them and accelerationists like them. The only powerful interest group that dislikes them is GPU manufacturers.

A possible counter-argument is that export controls incentivize companies in foreign countries to develop their own manufacturing pipelines, which would ultimately worsen race dynamics. That sounds too much like 4D chess thinking to me—as a rule, making things harder does not make things easier.

Establishment of an AI safety standards body

Path 1: Create a public “state of the art” on AI safety → companies can more easily adopt good practices → companies can behave more safely.
Path 2: Establish an answer to the question of who will oversee AI companies’ safety practices, in case future regulations mandate oversight.
Path 3: Help clarify what flavors of regulation would be helpful → future (hopefully strong) regulations can be better targeted at reducing the serious risks.

The UK has the AI Security Institute; other countries could create something similar.

Dangerous capability evaluations

Path 1: Get better information about models’ capabilities → policy-makers and the public can better see the dangers that AI poses.
Path 2: Get better information about models’ capabilities → we can use that information as an input to later, stronger regulations that have hard shutdown requirements when models meet certain criteria.

Evals are overrated by many in the AI safety community, but this still seems like one of the better things governments can do with light-touch regulations.

Some concerns with evals:

You can’t just evaluate models, you have to actually figure out how to make them safe.
Evals don’t work when models know they’re being evaluated, which is becoming increasingly the case. (This was predictably going to be a problem—surely a superhuman AI would be superhumanly shrewd at detecting when it’s being tested.)
Evals give AI companies an optimization target.

Mandatory publication of safety frameworks

Path 1: Require AI companies to actually have safety frameworks → they are now marginally safer.
Path 2: Allow safety frameworks to be inspected by governments or the public → enable pressure on companies to improve their frameworks.

This is only a minor win, since most frontier AI companies already publish safety frameworks, their frameworks are woefully inadequate to prevent human extinction, and the frameworks will be dropped when inconvenient anyway. Requiring AI companies to publish and follow safety frameworks may be better, or it may induce them to publish toothless frameworks so that they’re not beholden to anything.

Mandatory advance disclosure of large training runs

Path 1: Ensure governments are aware of when AI companies are doing new training runs → ??

I’ve heard this idea proposed before, but I don’t have much grasp on what it’s meant to accomplish. If governments impose restrictions on what kinds of training AI companies can do, then disclosure is a necessary prerequisite; but what does disclosure on its own do? Perhaps I’m missing something here. Still, mandatory disclosure doesn’t seem meaningfully bad in any way.

Whistleblower protections

Path 1: Make it easier for whistleblowers to come forward → expose dangerous behavior within AI companies → increase political will for strong regulations.

This is another free win, although the ultimate effect on extinction risk seems small—it provides a little more incremental evidence of AI companies’ misbehavior.

Incident reporting

Path 1: Learn about scary incidents → policy-makers and the public can better see the dangers that AI poses.

This seems less promising than capability evals because it relies too much on luck: there might not be any incidents, the incidents might go undetected, or they might occur after it’s already too late.

Security requirements to prevent model theft

Path 1: Ensure competitors can’t catch up by directly copying leaders’ models → reduced competitive pressure → easier to coordinate around slowing down AI.
Path 2: Ensure rogue actors can’t steal model weights → reduce misuse risk.

Security requirements would be very good if feasible, but I’m not sure that requirements with teeth would qualify as “weak”. For the requirements to be effective, they’d need to be highly restrictive. Even then, the state of the art in cybersecurity is not good enough to prevent sophisticated thieves from getting their hands on model weights.

My position on weak regulations

With all that in mind, what do I believe?

In brief:

Weak AI regulations are good. I’m happy when policy-makers propose them, I’m happy when people campaign for them, and I’m even happier when they get signed into law.
Strong regulations are a lot better. Inasmuch as I can influence marginal policy efforts, I’d prefer to push for stronger regulations.

I’m sympathetic to the view that weak regulations matter more right now—the arguments in favor of my view are far from definitive. And if you work in the policy world, whether you work on weak or strong regulations probably has less to do with what’s better in the abstract, and more to do with your particular situation.

Notes

Right now in the United States, AI companies mostly face pressure from other US-based AI companies. Export restrictions don’t do anything about that. ↩
I say the chance is small because:
- Extinction-level dangers probably won’t be detectable by tests until it’s too late to stop them. An AI that’s smart enough to kill everyone is smart enough to fool your tests.
- If the safety testing has teeth—if it requires developers to shut down a model that appears too dangerous—then the regulations would have to be much stronger than anything we’ve seen legislators pass to date.
↩
We can’t count on warning shots to save us, and we should be prepared for the possibility that they won’t. But we should also be ready in case we do get a warning shot that garners sufficient attention. ↩
Beaman, A. L., Cole, C. M., Preston, M., Klentz, B., & Steblay, N. M. (1983). Fifteen Years of Foot-in-the Door Research. ↩
The downside is that it risks muddling your message. “I thought you wanted a global halt on AI development, but now you’re advocating for weak transparency requirements?” I expect most people are smart enough to understand that weak regulations still make sense under our view, but you may lose a few people. ↩

We Need Breadth-First AI Safety Plans

Mon, 01 Jun 2026 00:00:00 -0700

Depth-first plans lay out a path from here to aligned superintelligent AI. We need those kinds of plans. But depth-first plans depend on many assumptions: “We will make AI safe by doing step 1, then step 2, then step 3.” Step 1 only works under condition A, step 2 requires condition B, step 3 requires condition C. If A or B or C is false, the whole plan fails (and there’s a good chance we all die).

Consider Google’s safety plan from April 2025. To my knowledge, this is the best among the frontier AI companies’ plans.¹

Google’s plan depends on a series of conditions:

For the most part, the plan does not consider concrete details of how significantly-more-capable AI systems will behave, instead proposing that Google will figure out how to handle those systems once it understands them better. This only works given (at least) two conditions:
1. AI capability improvements occur at a relatively predictable pace, with no unexpectedly large jumps.
  - The plan explicitly assumes no “discontinuous” improvements, which is roughly the same thing. It’s good that they’re being explicit about this.
2. Once stronger capabilities emerge, there will be enough time to figure out mitigations.
The plan entails putting stricter measures in place once AI systems become sufficiently capable. This depends on at least two conditions:
1. Google (or somebody) can accurately determine what capability level is dangerous.
2. Google’s evals (or third-party evals) can elicit dangerous capabilities if they exist.
The plan requires using AI to bootstrap AI alignment. This depends on several conditions:
1. We can successfully align the AI that we use for bootstrapping, or misalignment will be easy (enough) to spot, or alignment isn’t necessary (e.g. because humans can use amplified oversight to monitor smarter-than-human systems).
2. Future Google can be trusted to use enough of its compute to differentially accelerate alignment research, rather than doing something more profitable (for example, differentially accelerating AI R&D).
3. AI that’s useful enough to solve AI alignment does not pose an existential threat.
4. AI alignment is the sort of thing that can, in principle, be solved by strong-but-not-superintelligent AI.
  - For example, it may be that moral advances are required before we know how to correctly specify how AI ought to behave; and that unaligned AIs cannot contribute to moral advances.²

(The plan depends on many more conditions than that, but I’ll keep it short.)

That list included eight conditions. If any one of those conditions fails, then the whole plan fails. Some of the conditions seem likely to be true; others seem questionable. But even if every individual condition is probably true, it’s much less likely that they’re all true.

Disjunctive conditions are better than conjuctive ones. We can see an example in condition 3.1 above: Google’s plan can work if it’s possible to align the “bootstrapper” AI, OR if misalignment is easy to spot, OR if it doesn’t need to be aligned. Disjunctive conditions are good; more of those, please.

We need breadth-first plans:

We will take actions X, Y, and Z.

X depends on condition A.

Y works even if A is false, but it depends on condition B.

Z works if A and B are false; it depends on a third condition C.

X + Y + Z works even if two out of three conditions fail.

Some plans have a little bit of breadth. An explicit example from Google’s safety plan:

Our approach has two lines of defense. First, we aim to use model level mitigations to ensure the model does not pursue misaligned goals. […] Second, we consider how to mitigate harm even if the model is misaligned (often called “AI control”), through the use of system level mitigations.

I would like to see more breadth, and recursive breadth—there should be breadth within each component of the plan, and breadth within those sub-components.

The broadest plan that’s been published is Peter Barnett & Aaron Scher’s AI Governance to Avoid Extinction: The Strategic Landscape and Actionable Research Questions (see also the corresponding LessWrong post). The report explicitly considers four possible future scenarios and how we might achieve a good outcome from within each scenario. The report even includes a flowchart:

The report goes into more detail about the conditions required for each of the four scenarios to succeed.

Barnett & Scher believe “Off Switch and Halt” is the best strategy. They don’t exactly phrase it this way, but according to their report, “Off Switch and Halt” depends on the fewest conditions and has multiple ways of succeeding.

How breadth-first plans can inform what we do

I see two big benefits to writing breadth-first plans:

We can identify which paths to success depend on the fewest conditions,³ and focus more on those.
It’s easier to find the biggest holes in the plan.

Root-level breadth matters most

The good news is the branches off the roots are the most important because they have the greatest probability mass. Creating layers of branches off branches off branches quickly gets complicated, but I don’t think it’s necessary.

My rough attempt at categorizing plans

I made a quick flowchart to categorize AI safety plans at a high level.

A blue circle indicates an action
A blue square indicates an outcome
A red hexagon indicates a necessary condition to achieve an outcome
A red pentagon indicates a condition that is helpful but not necessary

The idea is that we need a broad set of overlapping plans such that some plan will work, even if many conditions (red nodes) turn out to be false.

(Click here to see the full-size image.)

Is this flowchart comprehensive? Definitely not. Is it even accurate? Maybe. My point is that, to make AI safe, we need multiple plans that cover all the ways the other plans could go wrong, and this flowchart is a quick attempt at representing some of those plans.

Future work I’d like to see

AI companies should publish breadth-first plans. What will they do if a step in their mainline plan fails?
Governments should pass legislation requiring AI companies to have plans that cover every item on a list of possible future scenarios.
- For example, mandate that companies have different plans for different takeoff speeds.
- AI safety researchers should do research to inform what future scenarios need to be covered.

Notes

I originally wrote this article shortly after April 2025, but I procrastinated for a year on finishing it, so I’m not sure about the current state of AI companies’ plans. ↩
I am skeptical that a bootstrapped-aligned AI will behave morally in ways in which most humans do not behave morally, e.g. eating factory-farmed animals; or that it will be able to correctly resolve the internal inconsistencies in common-sense ethics. For example, in the mere addition paradox, most people accept a set of premises but reject the conclusion that necessarily follows from those premises.⁴ ↩
Technically, what we want isn’t paths that depend on few conditions. We want paths where the joint probability of every condition is as high as possible. But generally speaking, fewer conditions means the probability of success is higher. ↩
Philosophy Experiments’ Philosophical Health Check asks you a series of questions and purports to identify inconsistencies in your beliefs. I think the questions leave some wiggle room to argue that supposed inconsistencies aren’t truly inconsistent, but a more rigorous test would be harder to construct. ↩

Sentient Welfare Across Three Futures

Mon, 25 May 2026 00:00:00 -0700

Three categories of futures, depending on how AI goes:

ASI timelines are long.
ASI timelines are short, and we’re on track to solving AI alignment.
ASI timelines are short, and we’re not on track to solving AI alignment.

If we want to make a good future for all sentient beings, each of these futures has different implications for what we should work on.

If timelines are long…

…we can prioritize work that takes a long time to complete. That includes:

foundational research
moral philosophy
decision theory
moral circle expansion
theoretical AI alignment paradigms
traditional animal advocacy

If we’re on track to solving AI alignment…

…the shape of the future will be determined by an aligned ASI. Therefore, we should steer toward a future where ASI cares about sentient welfare. Possible areas of work include:

research on how to align ASI to sentient welfare [details]
work on making LLMs more animal-friendly [details]
traditional animal advocacy targeted at frontier AI developers [details]

If an aligned superintelligence creates a stable future where humans are empowered, then—some might argue—we can defer “long-timelines” work until we have superintelligent assistance. However, I cannot envision how we could get a stable future without solving some foundational problems first.

If we’re not on track to solving AI alignment…

…none of those other types of work listed above will pay off. There’s not much we can do for non-human welfare; step one is to prevent ASI from destroying all value in the universe.

Areas of work include:

AI pause advocacy [details]
developing and advocating for AI regulations that enforce safety rules
AI alignment research

Which future are you betting on?

Some plans make strong assumptions without making them explicit. When you pursue a strategy, you’re making an implicit bet on which future you’ll find yourself in. You’re assuming that you live in the world where that strategy makes most sense.

It’s worth taking the time to probe our beliefs:

What do we expect the future to look like, and what strategies make sense given those expectations?
What are we currently working on? In which futures does that work pay off?

At the community level, we shouldn’t bet everything on one future. (For individuals, it’s often better to specialize.¹) Some people should pursue long-timelines work; others should prioritize optimistic short-timelines work; still others should focus on pessimistic short timelines. It’s worth considering what this balance ought to look like, and how we might get closer to the right balance.

A natural next question: What plausible futures are we neglecting? That’s a question I want to spend more time thinking about.

Notes

Individuals benefit from developing expertise over time. In most fields, it takes more than 80,000 person-hours for diminishing marginal utility of effort to kick in. The gains of increasing expertise outweigh the diminishing utility of marginal work. ↩

I sleep less when I exercise more

Mon, 18 May 2026 00:00:00 -0700

They say exercise improves sleep quality. Is that true for me?

To test this hypothesis, I took my daily calorie expenditures from the Apple Health app and correlated them with that night’s sleep time.¹ I also included caffeine intake as a potential confounding variable.

The hypothesis: when I exercise more, I’ll get better rest that night, and therefore wake up earlier.

The results:²

name	coef	t-stat	p-value
intercept	9.0134	65.072	0.0000
calories	-1.6844	-6.967	0.0000
caffeine	0.4157	9.404	0.0000
R²	0.2409

I sleep 10 minutes less for every additional 100 calories of exercise. Exercise plus caffeine explained 24% of the variance in my sleep time; exercise alone explained 6.6%.

The trend shows up whether or not I have caffeine:

Data are binned into increments of 100 calories. Any bins with fewer than 5 data points are not displayed. Vertical lines show the 95% confidence intervals for each bin.

I also regressed bedtime and wake time. Am I sleeping less because I’m going to bed later, or because I’m waking up earlier (or both)?

Exercise did not reliably predict my bedtime (nor did caffeine):

name	coef	t-stat	p-value
intercept³	20.9879	36.742	0.0000
calories	0.9083	0.911	0.3626
caffeine	-0.0468	-0.257	0.7973
R²	0.0018

But exercise did predict that I’d wake up earlier:⁴

name	coef	t-stat	p-value
intercept	7.6143	61.200	0.0000
calories	-1.8247	-8.402	0.0000
caffeine	0.4023	10.132	0.0000
R²	0.2879

I hardly ever set an alarm, so the quality of my sleep is an important determinant of when I wake up.⁵

R² for exercise alone (controlling for caffeine):

dependent variable	R²
time asleep	0.0662
bedtime	0.0012
wake time	0.0934

Robustness checks

Using step count instead of calories burned

Change in sleep per 1000 steps, according to my daily step count on Apple Health:

name	coef	t-stat	p-value
intercept	8.6728	57.609	0.0000
step count	-0.0608	-3.823	0.0001
caffeine	0.4760	10.722	0.0000
R²	0.2041

A visualization of the trend:⁶

Extended time horizon

For all the calculations up to this point, I included data from 2024-01-01 through 2025-12-08. I have data from 2021 through yesterday, but I narrowed the data range to keep the data clean:

My exercise volume trended upward throughout 2022 and 2023, and stabilized around late 2023/early 2024. I excluded the earlier years in case there was a confounding effect where the long-term upward trend in exercise is correlated with a long-term downward trend in sleep.
Starting 2025-12-09, I’ve started setting an alarm at 9pm to remind myself to go to bed. I don’t think that matters, but the recent period is short enough that I don’t lose much statistical power by excluding it.

Here’s the result of a regression when I include the older data (2021-01-14 to 2025-12-08):

name	coef	t-stat	p-value
R²	0.1229
intercept	8.5683	133.197	0.0000
calories	-0.5954	-4.397	0.0000
caffeine	0.2835	10.958	0.0000

Methodology details

I chose calorie expenditure as the independent variable and time in bed as the dependent variable.

Why I chose calorie expenditure:

Apple Health gives me data on step count and calories, according to my iPhone. (An Apple Watch would be more accurate, but I don’t have one.)
The calorie estimate should incorporate more information than step count.
Calorie estimates are systematically wrong because my phone can’t tell when I’m lifting weights, but my weight routine is relatively consistent across time, so this shouldn’t qualitatively change the result.
I record my subjective productivity on all work days, but (as I’ve found in previous observational studies) productivity varies too much based on too many factors, which makes it hard to find statistically robust correlations.

Why I chose time in bed:

The Sleep Cycle app records my sleep and gives estimates of “time asleep” and “sleep quality”. From previous observational studies, I know that the app’s sleep quality estimates are negatively useful—they are less predictive than a raw “time in bed” number, which measures the time from when I start running the app to when I tell it to stop.
“Time in bed” includes a manual adjustment for any time I spend awake overnight (rounded to the nearest half hour).
If I got up to pee and then fell back asleep quickly, I counted that as a 15 minute loss.

Notes

I measure sleep as “time in bed” according to my Sleep Cycle app, with a manual adjustment for time spent awake overnight. See Methodology for details. ↩
- The dependent variable (sleep time) is measured in hours.
- Calories are divided by 1000, to make the numbers more readable.
- Caffeine is measured in units of 100mg. Coffee is my source of caffeine, and I estimate one cup of coffee at 100mg.
↩
The intercept should be interpreted as clock time on a 24-hour clock, so 20.9879 is 9:59 pm. ↩
Having caffeine predicts that I wake up later the next morning. I’d speculate that there are two reasons for this:
- Caffeine hurts my sleep quality, making my body want to sleep for longer.
- I tend to work on caffeine days and not work on non-caffeine days, and they tend to alternate. Anecdotally, I’ve observed that I wake up earlier on work days: the fact that I’m anticipating work means I have a hard time waking up briefly and then going back to sleep. I don’t think this is related to caffeine, because when I have a no-caffeine rest day followed by another no-caffeine rest day, I have no problem sleeping in.
↩
Other determinants include:
- Are there birds singing right outside my window?
- Is the sun shining? (I have blackout curtains, but the light that seeps through the cracks can still wake me up.)
↩
I would like to take a moment to pat myself on the back for the fact that during this two-year period, there were only 23 days where I took fewer than 4,000 steps. And I went to the gym on half of those days, and I had the flu for half of the other half, so really it was only 6 days or something. ↩

Donation Timing Under Uncertainty About AI Timelines

Mon, 11 May 2026 00:00:00 -0700

A few years back, I got a big pile of money from working at a tech startup. I put a lot of that money into a donor-advised fund. Since now I make hardly any money, that DAF might represent the majority of my lifetime donations. How much of my DAF should I donate per year?

In particular, how much should I donate in light of short AI timelines?

I created a simple model to answer this question.

The basic concept: After the singularity, my money doesn’t matter anymore. I want to donate an equal amount of money every year from now until the singularity. But I don’t know when the singularity will happen. How much should I donate each year (in terms of % of starting wealth)?

The reason for donating an equal amount each year is that there’s a tradeoff between early and late donations:

Early donations can have compounding effects.
Late donations happen when you have better information.

If you’re unsure about which side of the tradeoff matters more, then it’s reasonable to distribute donations over time.¹

If I had perfect knowledge of when the singularity will happen, I’d donate an equal amount each year. But I’m uncertain about the timeline. Instead, I can calculate how much to donate given a distribution over possible timelines.

The idea is, if the singularity happens five years from now, I want to donate 20% per year. If it happens 10 years from now, I’d rather donate 10% per year. If I’m evenly split between those two possibilities, then I should allocate half my budget to each strategy:

With half my money, donate 20% per year.
With the other half, donate 10% per year.

Therefore, I donate 15% for the first five years and 5% for the last five (if we make it that far).

My model takes a version of this approach where my AI timeline follows a probability distribution, rather than just having two possibilities.²

I modeled two different probability distributions: log-normal and Pareto.

A log-normal distribution assumes that timeline is as likely to be 2x the median as it is to be half. The probability diminishes quickly when you multiply the timeline by larger and larger numbers.
A Pareto distribution is fatter-tailed—it assumes more probability to long timelines than a log-normal distribution does.

I’ve plotted two charts showing the optimal donation amount per year, as a percentage of starting wealth (so that the donation amounts will always sum to 100%). For these charts, I assumed a median timeline of 7 years. A Pareto distribution requires a minimum, so I set the minimum at 3—i.e., the singularity will definitely not occur until at least three years from now.

The models also have $\sigma$ and $\alpha$ parameters (respectively), which determine the widths of the distributions.

For a log-normal distribution, $\sigma$ is the number such that 68% of the probability lies within the interval $\frac{\mu}{\sigma} <= x <= \mu \sigma$, and 95% lies within $\frac{\mu}{2 \sigma} <= x <= \mu 2 \sigma$.
For a Pareto distribution, $\alpha$ determines how quickly the probability falls off: probability density is proportional to $x^{-(\alpha + 1)}$.

These two charts aren’t too different: they both recommend donating around 20% initially, and then donating diminishing amounts as time goes on. The Pareto model donates more up front and the amounts diminish rapidly, whereas in the log-normal model, the donations are a bit more spread out.

Rather than blindly following the model every year, it’s better to re-calculate before each donation because you gain new information over time. At minimum, the fact that the singularity hasn’t happened yet changes the probability distribution.

This model makes many simplifications. I considered several modifications that would make the model more realistic, but none of them seemed useful enough to justify their complexity.

Some possible model extensions:

Rather than donating an equal amount each year (conditional on known date of singularity), explicitly model the tradeoff between giving now vs. later.
Include the investment rate of return.
Represent the timeline distribution as a mixture of several distributions.
Model the possibility of a slow takeoff, rather than a discrete singularity.

Source code is available on GitHub.

Notes

The tradeoff has more than two considerations, but I will ignore the others for the purposes of this model. ↩
The donation amount for year T is given by
\[D(T) = \displaystyle\int_T^{T_f} \frac{1}{t} f(t) dt\]
where
- $f(t)$ is the probability density of the singularity occurring at time t
- $T_f$ is the final year (defined to make the simulation finite)
↩

Thoughts on investing for transformative AI

Mon, 04 May 2026 00:00:00 -0700

TLDR: I basically don’t.

Contents
Ethical concerns
Thoughts on how to avoid becoming corrupted
Future worlds
- What happens in the lead-up to ASI?
Predictions are hard, especially about markets
Trend-following
The EA portfolio
Leaning my investments in the right direction
Appendix: Some specific predictions
Notes

Ethical concerns

If you stand to make money from AI, that incentivizes you to speed up AI development, and it disincentivizes you from taking actions that might cause your investments to go down—for example, donating to nonprofits that work to prevent superintelligent AI from being built. Conflicts of interest can create bias that people aren’t consciously aware of, even when we have strong incentives to get the right answer.¹²³ I suspect that a significant portion of AI safety people’s opposition to banning ASI is that they have financial conflicts of interest.⁴ This has made AI safety progress harder than it would’ve been if humans were immune to this sort of motivation. I don’t want that to happen to me.

My conclusion in this post is that I’m not going to invest directly in transformative AI. But if I came to a different conclusion, I would want to have a strong theory of why I won’t be corrupted if I stand to make money off of increasing extinction risk. If I couldn’t come up with any such theory, then I wouldn’t change my investments.

When I wrote previously about investing for transformative AI, I didn’t much think about the corrupting influence of money—I was too focused on the academic question of what to do in theory. That was a mistake.

Then there’s the fact that investing in AI companies gives the companies more power. This is a problem for private companies but I’m not too concerned about it for public mega-cap companies because they already have so much access to capital. And it’s not a concern if you invest in ways that don’t directly help AI companies. For example, TAI will likely raise interest rates, which suggests that investors can profit by short-selling bonds; if we do that, interest rates will go up (by a teeny tiny bit), which hurts AI companies (by a teeny tiny bit).

Thoughts on how to avoid becoming corrupted

One’s ability to act ethically, and even think clearly, becomes corrupted when there is a direct connection between behaving rightly and losing money. If I invest directly in an AI company, then I suddenly have a strong financial incentive to believe that anything that company does is good for the world. I have an incentive to disbelieve that the company is increasing risk.

I’m less concerned about indirect investments. If I expect TAI to raise interest rates and I short bonds as a result, then there’s not much I can personally do to affect interest rates because they are such a large-scale phenomenon, and they move around for all sorts of reasons.

Possible strategies for avoiding bad incentives:

Commit to donating all the money you make from investing in AI. (Ideally, make it a binding commitment somehow.)
Put a cap on how much you invest in AI (say 10% of your wealth), and sell down your investments if they grow beyond that cap.
Be so worried about AI-driven extinction that it overpowers your desire to make money. (I’m not sure how actionable this strategy is.)

Implementing these strategies still wouldn’t give me much confidence that I’m protected against bad motivations. I don’t know how to avoid becoming corrupted.

For more discussion on this topic, see How to (hopefully ethically) make money off of AGI, under the heading “Is any of this ethical or sanity-promoting?”

Anyway, this concern hasn’t been relevant for me because I decided not to invest directly in AI, and I made that decision before doing any serious thinking about how to avoid becoming corrupted, so I never had to figure out an answer.

Future worlds

Ethical issues aside, the rest of this post describes how I think about investing for transformative AI, and why I don’t specifically buy any AI-related investments.

Here are my (rough) credences for how the near future goes:

40% chance misaligned AI kills everyone.⁵
30% chance we don’t get TAI soon: either TAI turns out to be hard to build, or humanity collectively realizes how insane it is to rush to TAI and decides to slow down.
20% chance AI renders money meaningless—because it ushers in a post-scarcity utopia, or because the creator of the first superhuman AI uses it to confiscate all wealth, or something like that.
5% chance we get transformative AI, money still matters, and TAI boosts the economy across the board and ~all stocks go to the moon.
5% chance we get TAI, money still matters, and gains are concentrated in certain sectors of the economy.

Investing for TAI matters most in the last scenario, where money still matters but returns are concentrated.⁶ That’s only 5% of worlds. If AI kills everyone or renders money meaningless, investing strategy doesn’t matter after that point—but it might matter before.⁷

What happens in the lead-up to ASI?

Money still has value when TAI is getting close, but hasn’t arrived yet. Making more money pre-TAI means you can donate more.

The time from “AI has huge economic impacts” to “AI is powerful enough to kill everyone” may be short—I’d guess maybe a 75% chance that it’s less than five years, and 60% chance that it’s less than three years.⁸ It’s hard to spend money well in such a short time because four things need to happen in sequence:

Your investments go up

→ You donate your earnings

→ That money gets spent on useful activities

→ Those activities reduce existential risk

If you get rich three years before ASI arrives, it might already be too late for your money to make a difference on x-risk.

If you’re making active bets to beat the market, then the market has to anticipate TAI before it’s too late, but after you do. (Maybe investing for TAI five years ago would’ve worked, but maybe now it’s priced in.⁹)

You can only make money pre-TAI if the market comes to agree with you, which it might not. If you have a high P(doom), that implies a high interest rate because money is more valuable now than later. But that doesn’t mean the market interest rate will go up, because the market might never expect AI to kill everyone until it does. For longer arguments to this effect, see this comment and this sub-comment by Eliezer Yudkowsky.

So there’s a better chance that money matters close to TAI than after TAI, but it still seems hard to both make money and use money before TAI arrives. There’s a direct tradeoff: the closer you get to TAI, the more likely it is that the market has caught up to your expectations, and thus the more likely it is that you’ve made a lot of money; but also, the more difficult it is to use that money well.¹⁰

Predictions are hard, especially about markets

Some people have made a lot of money from investing in AI-related stocks. For example, Peter McCluskey is a trader who’s written about his AI investments; I think he knows what he’s doing.

I don’t trust my ability to beat the market on specific predictions like that. It’s hard to out-predict the market—very few people succeed in the long run. Even if you have a track record of successful predictions, those predictions mean less than you might think.

(If you make 100 bets on AI stocks and 90 of them beat the market, at first glance that looks like really strong evidence. But it’s not, because all of those bets are correlated.)

Five or ten years ago, some people in the AI safety space were talking about investing in AI. Some of them made a lot of money. But were they right, or were they lucky? I’m not confident either way. My main reason for doubt is that most of the time, when I see people writing about how AI advances will make AI stocks go up, their described mental models look simplistic to me.

For example, a few years ago, some people predicted that AI advances would cause Nvidia’s stock to go up. That prediction came true. But what if Nvidia’s margins had gotten squeezed? What if AMD or Intel had caught up to Nvidia and out-competed it? What if AI developers had replaced Nvidia with their own in-house chips? I never saw anyone give a good explanation for why those things wouldn’t happen.

(I referenced Peter McCluskey as a positive example because I know he thinks about those sorts of possibilities.)

Or: For OpenAI to meet its 5-year revenue projections, it needs to compound at 108% per year. How many times in history has a large-cap company sustained a growth rate that high?

Answer: Zero.¹¹

Not to say it’s impossible. OpenAI reached 100 million users faster than any company ever had before. But predicting OpenAI to meet its revenue goals is a strong claim that demands a strong justification, and I’m not satisfied with the justifications I’ve seen.

For more on this subject, see my earlier post Value Investing in the Age of AGI, particularly under the heading Defenses of value investing.

I haven’t done careful fundamental analysis on AI-related investments, and I’d be wary of making those investments without doing so first. So I don’t try to beat the market by making specific predictions. I do try to beat the market, but rather than relying on personal judgment, I systematically invest in factors where there is reliable evidence that they have had positive returns. I’m relying on my ability to assess the scientific literature, but not on my ability to predict individual trades.

Trend-following

Trend-following investing¹² is one of those factors I invest in. It worked well historically, and there’s reason to expect it to continue to work¹³. The basic idea of trend-following is you buy assets that are trending up and short assets that are trending down.

(There’s also trend-following’s cousin, the momentum factor; everything I say in this section about trend-following also applies to momentum.)

There’s no definitive explanation for why trend-following works, but one popular hypothesis is that markets react too slowly to new information.¹⁴ Trend-followers enter positions after the smart money, but before everyone else.

If that hypothesis is correct, then trend-following is a way to follow the smart money without having to make any specific predictions. The disadvantage is that you’ll never be first—you’ll never be the investor who profits the most off of a trend.

There are two key advantages:

You don’t need to figure out whether you can make market-beating predictions, or avoid all the biases involved in assessing your own abilities. You can follow a simple, objectively-testable strategy.
You’re not limited to a single area of expertise. If AI experts predict markets to move a certain way and produce a trend, you follow that trend. If, say, Japan geopolitics experts predict a move in the Yen, you follow that trend, too.

I find trend-following especially appealing in light of my belief that AI advancements will make big changes somewhere, but it’s hard for me to predict where. A trend-following strategy will pick up the trends wherever they arise.

Previously, I wrote about value investing. Value and trend make good complements: value investing works when things go back to normal; when abnormal growth reverts to the mean; when investors make overconfident predictions. Trend-following works when winners keep winning; when investors under-react to changing conditions.

The EA portfolio

Philanthropists care not just about their own money, but about the aggregate portfolio of all value-aligned donors.

Even knowing in retrospect how well AI stocks did over the last 5 years, I mostly feel fine about the fact that I “missed out”, because other EAs invested heavily in AI. If Anthropic IPOs soon and doesn’t crash before then, more than 50% of EA wealth will soon be in AI stocks. Beyond that, many EAs already invest on the thesis that AI will be transformative.

People often pay attention to expected returns but overlook expected risk. Concentrated investments are much riskier than you might think. I’m wary of concentrating too much in bets on the transformative-AI thesis.

Leaning my investments in the right direction

I haven’t made any big changes due to transformative AI, but on decisions where I’m ambivalent, it can push me a bit in one direction. Zvi made this point in On AI and Interest Rates:

Did I bet on low interest rates in 2021 by locking in a 2.5% 30-year (!?!) fixed rate mortgage, thereby making more money off that than I have made from all other income combined since then? Yes. Yes I did.

That mostly wasn’t about AI. The trade was absurd anyway. AI simply made me more excited to pursue it, get it done and size it larger. That is core to my betting and trading strategy (not investment advice!). Look for plays that have multiple reasons to do them (often below the threshold that is worthwhile on its own) and that avoid similar reasons not to do them.

Here are some ways AI gives me another reason to do something I was going to do anyway:

I already have a large allocation to trend-following due to its low correlation and right skew, but AI is another reason.
I’m ambivalent on whether I should hold bonds. I already didn’t hold them¹⁵, but AI gives another reason not to, because AI advances should cause interest rates to go up (and therefore bond prices to go down).
AI makes the value factor look a bit worse and the momentum factor look a bit better. I have 50/50 weighting between value and momentum; basic theory says I should have risk parity weighting, which is more like 60/40. But also I like having a 50/50 split just because it’s simple.
My equity investments are primarily long-only rather than long/short, for various reasons. One concern about long/short is that AI could cause an extreme tail event that blows up the short side. (It’s just as likely to blow up the long side, but the payoff is asymmetric.)

Appendix: Some specific predictions

I’m not knowledgeable enough to make bold predictions about how transformative AI will affect financial markets. But just for fun, I will make some weak predictions based on my best guesses. I didn’t come up with most of these on my own—I heard them from other sources, such as How to (hopefully ethically) make money off of AGI. But a few are original to me.

Real interest rates go up (and therefore bonds go down) — TAI increases return on capital and therefore borrowers are willing to pay more. Alternatively, interest rates go up because discount rates go up because extinction looks likely, but that only happens if the market prices in extinction risk, which it might not.
Stocks beat their historical average — TAI improves companies’ ability to generate profit. If true, this implies that deep out-of-the-money call options will perform especially well.
- Counterpoint: If interest rates go up, that would be bad for stocks. See Against using stock prices to forecast AI timelines.
AI stocks’ earnings growth outpaces the market — companies in the AI supply chain will capture a significant chunk of the productivity gains from AI.
AI stocks beat the market — this is a weaker prediction than the previous one, given that AI stocks are already priced with an expectation of strong earnings growth.
Real commodity prices go up — TAI makes most goods and services cheaper, but it can’t conjure oil or copper out of nothing.
Cheap real estate goes up — as with commodities, TAI can’t make more land.
Expensive real estate goes down — people want to live in Manhattan because that’s where the high-paying jobs are. If those jobs are replaced by AI, people won’t want to keep paying $4000/month for a crappy studio apartment.
The value factor doesn’t work — see Value Investing in the Age of AGI.
The momentum and trend factors work — see above.
Regarding net inflation, I have no clue what will happen — “knowledge economy” goods should become cheaper as AI drives down the cost of production; but commodities, real estate, and other fixed goods should become more expensive; and central bank intervention will dampen extremes in either direction.
Inflation volatility will go up — the CPI will change dramatically in the short term, but it’s hard to predict which way it will go.

Notes

Moore, D. A., Tanlu, L., & Bazerman, M. H. (2010). Conflict of interest and the intrusion of bias. ↩
Govindan, P., Chung, E., & Pechenkina, A. (2025). Money Can’t Buy Accuracy: Incentives Fail to Reduce the Confirmation Bias in Data Interpretation. ↩
Mayraz, G. (2011). Wishful Thinking. ↩
Non-financial conflicts of interest may matter more; if you’re an ML engineer, you want to be able to keep doing ML on frontier AI systems, and banning AI means you can’t do that anymore. ↩
Astute readers may recall last week when I quoted a 50% chance, not 40%. That’s because:
- I don’t take the exact numbers particularly seriously. I don’t have a strong view on whether the risk is 40% or 50%. (I definitely don’t believe it’s 10% or 90%.)
- The “20% chance AI renders money meaningless” scenario also includes some catastrophically bad outcomes. When you add those in, you might push up the probability from 40% to 50%(ish).
↩
Technically, it also matters in the penultimate scenario, where TAI makes the whole stock market go up. In that case, you’re pretty much guaranteed to make money, but there are ways you could change your investments to make even more money. Money has diminishing utility, so it’s less important to make money when you’re already making a lot of money. ↩
Some of these scenarios are continuous, not discrete. For example, there’s a continuum between “TAI boosts the whole market” and “gains are concentrated in certain sectors”. Surely some sectors will benefit more than others; it’s just a question of how concentrated the gains are. I’d think of the two scenarios as “I’m not upset that I invested in index funds instead of AI stocks” vs. “I am upset”. ↩
Metaculus gives lower odds than I do. ↩
I was also thinking about this five years ago, but I didn’t write about it, and I didn’t change my investing strategy. I regret how long it’s taken me to get around to writing this post. I’m not sure whether I should regret that I didn’t change my investing strategy. ↩
At least that’s true once TAI is close. Money 10 years pre-TAI might be more valuable than money 20 years pre-TAI because you have more information about how to spend it. ↩
Mauboussin, M. & Callahan, D. (2026). Bayes and Base Rates: How History Can Guide Our Assessment of the Future. ↩
Hurst, B., Ooi, Y. H., & Pedersen, L. H. (2017). A Century of Evidence on Trend-Following Investing. ↩
Babu, A., Hoffman, B., Levine, A., Ooi, Y. H., Schroeder, S., & Stamelos, E. (2019). You Can’t Always Trend When You Want. ↩
For more on this hypothesis, see:

Berger, A., Israel, R., & Moskowitz, T. (2009). The Case for Momentum Investing. Section heading “Possible Explanations of Momentum.”

Goyal, A., Jegadeesh, N., & Subrahmanyam, A. (2024). Empirical determinants of momentum: a perspective using international data. ↩
At least I don’t have a static holding; sometimes I hold bonds as part of the trendfollowing strategy. ↩

I'm extremely worried that superintelligent AI will kill everyone

Mon, 27 Apr 2026 00:00:00 -0700

I’d guess maybe a 50% chance that we’re all dead within 5–20 years because somebody will build superintelligent AI, and then the superintelligent AI will kill everyone.

AI developers are on the way to building smarter-than-human AI. Present-day AI is making rapid progress. We humans can still do plenty of things that the AIs can’t, but AI companies are working hard to change that, and they’re on track to succeed.

The really scary part comes when developers use the smarter-than-human AI to help them build an even smarter AI. They can make increasingly smarter AIs—possibly very quickly—and now instead of smarter-than-human AI, we have superintelligent AI (ASI), which vastly surpasses humans in the same way that humans vastly surpass chickens.

We can’t win against superintelligent AI

If humans and ASI disagree about how the world should be structured, the ASI wins. It won’t be like The Matrix or Terminator where humans fight back against the machines and win by the skin of our teeth. In those movies, the AIs weren’t noticeably smarter than humans; they just had guns. A fight against a superintelligence will be a crushing defeat. We might not even know the fight is happening until it’s too late.¹

I don’t know how a superintelligence would beat us. If I play a game of chess against Magnus Carlsen, I can’t predict which moves he will make, but I know that he will win.

Would ASI want to kill everyone? Probably yes. The problem is:

Among the full space of possible goals an agent could have, only a tiny slice of those include a world filled with flourishing sentient beings. Almost all goals that an agent could have are incompatible with human life. We only survive if we specifically design ASI to be aligned with humanity
We don’t know how design it that way.

This is the alignment problem: how do you align an ASI’s goals so that it doesn’t kill everyone?

Why would almost all goals result in human extinction?

It’s not because the ASI will hate us.

Human civilization has driven many species to extinction. We didn’t harbor malevolence toward them. It’s simply that those species could only survive in a certain habitat, and we wanted to use their habitats to do something else—cutting down forests to build farms or develop cities. By default, ASI will have the same relationship with us. Our homes, farms, and factories are in its way, and ASI will want to put something else there. The result is that we die.

One way this could happen is: ASI is pursuing some inscrutable goal. In the interest of that goal, it wants to make itself as intelligent as possible. To boost its intelligence, it paves over all land on earth to build more datacenters—including all the land where people are living.

Why can’t we make the ASI care about us?

Today’s AIs are grown, not designed. An AI is a giant black box of trillions of numbers; we have essentially no idea what’s going on inside them, much less how to get them to share our values.

AIs are not yet existentially dangerous because they’re not smart enough. But what would happen if we “dialed them up” until they’re smarter than us? Nobody knows. Right now, AI companies evaluate their models for misaligned behavior before releasing them, but there are two big problems with this:

You can only fix misaligned behavior if you can catch it. The smarter AI gets, the better it will be at concealing its true intentions. A smart AI knows that if we catch it, we will try to stop it; it will pretend to be aligned until it’s too late for us to do anything.
You can only fix misaligned behavior if you actually know how to do that. Right now, all we have are primitive techniques for shoving AI behavior in vaguely the right direction. This works well enough to make current-gen models commercially useful, but it’s not enough to keep a superintelligent AI under control.

We are not on track to solving these problems. The companies that are working toward ASI do not treat these problems with the seriousness they deserve. If we continue on the current trajectory, everyone dies, and the universe will be devoid of anything of value.

Even if we solve the alignment problem, we’re not out of the woods. There are a lot of ways an aligned ASI could go catastrophically badly. For example, if a group of people develop ASI, what’s stopping them from taking over the world? If power is widely distributed, how do you prevent bad actors from using ASI to cause tremendous harm? If you constrain ASI’s values to make sure it’s robust against misuse, how do you make sure those constraints don’t lock us out of the best possible futures? (If we had developed ASI in the year 1750, would we have permanently enshrined slavery as an institution?) We have no satisfying answers to questions like these.²

What can regular people do about it?

We need a global ban on building superintelligence until there is broad consensus that it will be done safely.³ For a Q&A on why this is the best plan, see nowinners.ai (which I didn’t write, but I agree with it).

Getting a ban on ASI will be hard. The situation is grim, but there are some things people like us can do to reduce extinction risk.

Call or write your representatives to express your concern. (I wrote a post about why I think this is a valuable use of time.)
Donate to organizations that are pushing for a halt on dangerous AI development. My current favorite places to donate are PauseAI US, Palisade Research, and MIRI. For more, see Where I Am Donating in 2025.
Talk to people about the danger. We’re more likely to succeed if ASI risk becomes a global issue that people care about.

Notes

One story for how things might go: ASI pretends to be friendly and bides its time. ASI invents nanotechnology. ASI builds microscopic quasi-viruses and spreads them through the air until they’re living in the bodies every human being. Once everything is in place, ASI pulls the trigger and every quasi-virus simultaneously releases a deadly toxin. One moment we think everything is fine, and the next moment we’re all dead.

That’s an idea that normal-intelligence humans came up with. A superintelligence could devise a better idea. ↩
I wrote more about this subject in Pausing AI Is the Best Answer to Post-Alignment Problems. ↩
This language is borrowed from the Statement on Superintelligence petition, which I have signed. ↩

I was wrong: concentrated factor portfolios don't have alpha

Mon, 20 Apr 2026 00:00:00 -0700

Previously, I wrote about how investors can simulate leverage via concentrated stock selection. That’s still true as far as I can tell. However, I also wrote something that I now believe to be false: concentrated equal-weighted factor portfolios have alpha on top of value-weighted factor portfolios. The numbers I found before were not wrong per se. However:

The alpha came primarily from small-cap and micro-cap stocks. That alpha may not be feasible to capture, or it may be defeated by trading costs; and historical estimates of micro-cap returns are biased upward because closing prices do not accurately represent the average investor’s trade price (Blume & Stambaugh (1983)¹).

When I constructed hypothetical factor portfolios that had high concentration but screened out small-caps, the results did simulate leverage—they had higher returns and volatility than diversified factor portfolios—but alphas were not consistently positive.
In the United States (where the data goes back the furthest), the alpha only shows up over the full data series (1927–2025). When restricting to 1964 onward, the alphas are close to zero.
Concentrated value and momentum had positive alpha; but when I tested two new factors, profitability and investment, they each had negative alpha.

Contents
Methodology
Results for long-only factors
Can small investors capture factor premiums in small-caps?
Results for long/short factors
Discussion
How I’ve changed my personal investments
Source code
Notes

Methodology

I examined four factors: value (HML), momentum (UMD), profitability (RMW), and investment (CMA).² Those are the factors from the standard Fama-French five-factor model, minus the market factor and size, plus momentum.³ All factors use the standard Fama-French definitions with data pulled from the Ken French Data Library.

To compare diversified vs. concentrated factors, I defined “diversified” as the top 40% of stocks (ranked by the factor of interest), cap-weighted; and “concentrated” as the top 20%, equal-weighted. All factors were defined as long-only rather than long/short, to better represent how most investors invest.

For each factor, I ran a factor regression with the concentrated factor of interest as the dependent variable, against three independent variables: market beta, the long/short size factor (SMB), and the diversified factor of interest minus the market⁴. I included SMB on the hypothesis that the outperformance of equal-weighted stocks is partially driven by the size factor, but this turned out not to be relevant (SMB coefficients were close to zero).⁵

To make the strategies realistically tradable, I excluded the smallest 20% of companies. As of 2025, this would have excluded stocks with a market cap less than $1 billion or so. That’s probably more conservative than necessary, but looking at narrower slices (e.g. excluding the bottom 10% instead of 20%) would pose methodological challenges.⁶

My original article defined “diversified” as top 50% value-weighted and “concentrated” as top 10% equal-weighted. Based on the data available in the Ken French Data Library, I couldn’t construct realistically tradable strategies using 50% and 10% cutoffs, so I used 40% and 20% instead.

Results for long-only factors

The tables below show alphas (t-stats in parentheses) for concentrated factors over diversified factors. “All-cap” does not filter based on market cap; “realistic” excludes the smallest 20% of companies.

United States value, momentum, profitability, and investment start in 1927, 1928, 1964, and 1964, respectively (rounded up to the nearest year). Developed ex-US factors start in 1991.

Table 1: Comparing factor alphas, United States

factor	all-cap alpha	realistic alpha
Value (B/M)	0.70% (1.69)	-0.07% (-0.18)
Momentum	0.92%* (2.22)	0.27% (0.71)
Profitability	-0.87%* (-2.20)	-0.80%* (-2.36)
Investment	-3.18%*** (-4.95)	-3.21%*** (-5.25)

*, **, and *** indicate significance at p < 0.05, 0.01, and 0.001, respectively.

Table 2: Comparing factor alphas, Developed ex-US

factor	all-cap alpha	realistic alpha
Value (B/M)	1.68%*** (4.03)	0.62% (1.44)
Momentum	1.33%*** (3.36)	0.98%** (2.71)
Profitability	0.21% (0.55)	-0.03% (-0.08)
Investment	-1.35%** (-2.78)	-1.34%** (-2.85)

Full factor regression results

US (all-cap)

	beta	SMB	factor	annual alpha	t-stat
Value (B/M)	1.06	-0.05	1.28	0.70%	1.69
Momentum	1.00	-0.01	1.23	0.92%*	2.22
Profitability	1.06	0.00	1.17	-0.87%*	-2.20
Investment	1.11	-0.05	1.31	-3.18%***	-4.95

US (realistic)

	beta	SMB	factor	annual alpha	t-stat
Value (B/M)	1.08	-0.02	1.26	-0.07%	-0.18
Momentum	1.01	-0.00	1.34	0.27%	0.71
Profitability	1.07	-0.01	1.29	-0.80%*	-2.36
Investment	1.12	-0.00	1.31	-3.21%***	-5.25

Developed ex-US (all-cap)

	beta	SMB	factor	annual alpha	t-stat
Value (B/M)	1.05	-0.07	1.26	1.68%***	4.03
Momentum	1.07	-0.03	1.32	1.33%***	3.36
Profitability	1.03	0.02	1.07	0.21%	0.55
Investment	1.06	-0.29	1.69	-1.35%**	-2.78

Developed ex-US (realistic)

	beta	SMB	factor	annual alpha	t-stat
Value (B/M)	1.06	-0.05	1.29	0.62%	1.44
Momentum	1.06	0.01	1.32	0.98%**	2.71
Profitability	1.04	0.01	1.15	-0.03%	-0.08
Investment	1.06	-0.15	1.62	-1.34%**	-2.85

Excluding small-caps reduced alphas to near zero for concentrated equal-weight value and momentum factors. However, even when excluding small-caps, concentrated portfolios had factor exposures greater than 1 across the board (see under “Full factor regression results”). This suggests that concentrated portfolios offer synthetic leverage, but no alpha.

The two new factors (profitability and investment) had negative alphas. The relatively large t-stats suggest that the variance in alphas is better explained by heterogeneity than by random chance. That is, equal-weighted all-cap value and momentum had genuine positive alpha, while investment had genuine negative alpha.

International momentum maintained a statistically significant alpha when excluding small-caps (p = 0.007). After a Bonferroni correction, this p-value becomes 0.054, which corresponds to a likelihood ratio of 6.4:1.

The data series for value and momentum start in 1927/1928, while profitability and investment start in 1964. Is that fact relevant? The next table shows concentrated factor alphas when restricting the data series to 1964 onward:

Table 3: Comparing factor alphas, United States (1964–2025)

factor	all-cap alpha	realistic alpha
Value (B/M)	0.45% (0.92)	-0.18% (-0.37)
Momentum	0.20% (0.47)	-0.13% (-0.29)
Profitability	-0.87%* (-2.20)	-0.80%* (-2.36)
Investment	-3.18%*** (-4.95)	-3.21%*** (-5.25)

*, **, and *** indicate significance at p < 0.05, 0.01, and 0.001, respectively.

Full factor regression results

US (all-cap), 1964–2025

	beta	SMB	factor	annual alpha	t-stat
Value (B/M)	1.07	-0.02	1.19	0.45%	0.92
Momentum	1.06	0.04	1.28	0.20%	0.47
Profitability	1.06	0.00	1.17	-0.87%*	-2.20
Investment	1.11	-0.05	1.31	-3.18%***	-4.95

US (realistic), 1964–2025

	beta	SMB	factor	annual alpha	t-stat
Value (B/M)	1.09	0.01	1.22	-0.18%	-0.37
Momentum	1.06	0.06	1.34	-0.13%	-0.29
Profitability	1.07	-0.01	1.29	-0.80%*	-2.36
Investment	1.12	-0.00	1.31	-3.21%***	-5.25

Can small investors capture factor premiums in small-caps?

Collver (2014)⁷ looked at bid-ask spreads for US stocks in the year 2013. It found approximately the following average bid-ask spreads for stocks at various market caps (see Collver’s Table 4 and Figure 2⁸):

Table 4: Bid-ask spreads by market cap

Market Cap	Avg Spread
$500M – $1B	0.174%
$250M – $500M	0.287%
$100M – $250M	0.606%
< $100M	1.631%

The value, profitability, and investment factors have low turnover. If we assume that small investors have near-zero trading impact, then those three factors’ alphas appear to survive trading costs down to market caps of $250 million, and maybe as low as $100 million, but probably not much lower. (I don’t want to make a strong claim because I haven’t done careful calculations, and I don’t know how reliable these data are.) Momentum has a much higher turnover, so its alpha is less likely to survive trading costs.

(Remember: we’re not talking about the factors surviving trading costs; we’re talking about the alpha of concentrated factors minus diversified factors. Whether factors themselves survive trading costs is a separate question that has already been addressed by a number of publications; for example, see Frazzini et al. (2012)⁹.)

An important caveat: the market was much more liquid in 2013 than it was for most of the data sample. In 1940 or 1970, equal-weighted strategies were harder to trade, and the positive alphas may have represented compensation for trading costs. If so, then we would expect the alphas to shrink as trading costs decline.

Indeed, if we restrict the sample to the post-2000 period, concentrated value and momentum portfolios had weak or near-zero alphas, with only developed ex-US value retaining a significantly positive alpha. (However, note that the reduced sample size makes it more difficult to establish statistical significance.)

Table 5: US (all-cap), 2000–2025

	beta	SMB	factor	annual alpha	t-stat
Value (B/M)	1.12	-0.07	1.16	0.23%	0.23
Momentum	1.05	0.14	1.21	-0.81%	-1.01
Profitability	1.11	-0.05	1.14	-1.50%	-1.86
Investment	1.19	-0.18	1.44	-4.24%**	-3.07

Table 6: Developed ex-US (all-cap), 2000–2025

	beta	SMB	factor	annual alpha	t-stat
Value (B/M)	1.05	-0.07	1.28	1.50%**	3.19
Momentum	1.08	-0.01	1.29	0.39%	0.93
Profitability	1.04	-0.01	1.12	0.49%	1.12
Investment	1.06	-0.41	1.88	-1.36%*	-2.38

Trading costs are always difficult to assess. My current best guess is that a small investor can feasibly trade concentrated equal-weighted factors with a minimum market cap of $100M to $250M, but they should expect the alpha to be close to zero.

Results for long/short factors

The previous section examined long-only portfolio constructions, but the results for long/short factors may also be of interest. For long/short, I defined diversified factors as top 40% minus bottom 40% cap-weighted, and concentrated factors as top 20% minus bottom 20% equal-weighted.

The all-cap results look similar to the results for long-only portfolios: concentrated long/short value and momentum factors had positive alpha, profitability had near-zero (slightly negative) alpha, and investment had strong negative alpha. However, unlike the long-only portfolios, value and momentum maintained their positive alphas when excluding small-caps.

That doesn’t necessarily mean those concentrated long/short factors have positive alpha in practice, because short-selling stocks introduces additional costs.

Results for long/short factors

*, **, and *** indicate significance at p < 0.05, 0.01, and 0.001, respectively.

US (all-cap), 1964–2025

	beta	SMB	factor	annual alpha	t-stat
Value (B/M)	-0.01	0.02	1.43	1.72%***	3.54
Momentum	0.01	0.02	1.49	0.90%*	2.18
Profitability	-0.04	0.01	1.45	-0.52%	-0.82
Investment	0.01	-0.01	1.39	-2.83%***	-6.77

US (realistic), 1964–2025

	beta	SMB	factor	annual alpha	t-stat
Value (B/M)	-0.01	0.01	1.50	1.21%*	2.37
Momentum	0.00	0.05	1.51	1.78%***	4.01
Profitability	-0.05	0.03	1.55	-0.09%	-0.16
Investment	0.01	0.00	1.40	-2.32%***	-5.04

Developed ex-US (all-cap), 1991–2025

	beta	SMB	factor	annual alpha	t-stat
Value (B/M)	-0.00	0.00	1.39	2.13%***	4.79
Momentum	0.01	-0.01	1.44	0.58%	1.55
Profitability	-0.04	-0.06	1.21	0.13%	0.33
Investment	0.00	-0.01	1.33	-2.43%***	-6.72

Developed ex-US (realistic), 1991–2025

	beta	SMB	factor	annual alpha	t-stat
Value (B/M)	-0.01	-0.02	1.48	1.81%***	3.65
Momentum	0.01	0.00	1.47	1.35%**	3.21
Profitability	-0.02	-0.03	1.22	1.31%**	3.19
Investment	0.01	0.02	1.38	-1.26%**	-3.09

I also regressed the long sides of each concentrated factor against their respective long/short diversified factors, and similarly for the short sides. The long sides had consistent positive alphas with exceptional t-stats, and the short sides had consistently strong negative alphas.

Long-side concentrated factors, regressed against long/short diversified

US (all-cap), 1964–2025

	beta	SMB	factor	annual alpha	t-stat
Value (B/M)	1.03	0.59	0.89	3.96%***	13.20
Momentum	1.01	0.52	0.30	6.14%***	14.30
Profitability	1.01	0.56	0.63	4.61%***	12.42
Investment	1.03	0.57	0.23	4.50%***	14.72

US (realistic), 1964–2025

	beta	SMB	factor	annual alpha	t-stat
Value (B/M)	1.03	0.41	0.91	4.39%***	12.80
Momentum	1.03	0.39	0.31	5.85%***	13.64
Profitability	1.03	0.40	0.54	4.64%***	12.07
Investment	1.05	0.42	0.27	4.73%***	14.26

Developed ex-US (all-cap), 1991–2025

	beta	SMB	factor	annual alpha	t-stat
Value (B/M)	1.01	0.55	0.62	2.59%***	15.22
Momentum	1.00	0.54	0.43	4.02%***	15.77
Profitability	1.00	0.54	0.43	3.79%***	18.08
Investment	1.01	0.55	0.40	2.94%***	15.79

Developed ex-US (realistic), 1991–2025

	beta	SMB	factor	annual alpha	t-stat
Value (B/M)	1.02	0.42	0.64	2.51%***	10.55
Momentum	1.02	0.43	0.43	3.46%***	12.64
Profitability	1.01	0.42	0.39	3.21%***	13.10
Investment	1.03	0.44	0.39	2.47%***	10.76

Short-side concentrated factors, regressed against long/short diversified

US (all-cap), 1964–2025

	beta	SMB	factor	annual alpha	t-stat
Value (B/M)	-1.11	-0.67	0.38	-2.52%***	-3.66
Momentum	-1.07	-0.69	1.01	-4.90%***	-8.20
Profitability	-1.12	-0.64	0.76	-4.21%***	-5.53
Investment	-1.12	-0.69	0.92	-4.43%***	-7.10

US (realistic), 1964–2025

	beta	SMB	factor	annual alpha	t-stat
Value (B/M)	-1.14	-0.51	0.38	-2.82%***	-4.38
Momentum	-1.12	-0.54	1.00	-3.58%***	-6.11
Profitability	-1.15	-0.47	0.87	-3.86%***	-5.39
Investment	-1.16	-0.53	0.89	-4.10%***	-7.14

Developed ex-US (all-cap), 1991–2025

	beta	SMB	factor	annual alpha	t-stat
Value (B/M)	-1.07	-0.64	0.61	-2.07%***	-3.57
Momentum	-1.07	-0.70	0.84	-4.86%***	-10.34
Profitability	-1.07	-0.66	0.82	-4.00%***	-7.24
Investment	-1.08	-0.67	0.62	-4.15%***	-8.22

Developed ex-US (realistic), 1991–2025

	beta	SMB	factor	annual alpha	t-stat
Value (B/M)	-1.09	-0.52	0.64	-1.23%*	-1.98
Momentum	-1.08	-0.57	0.87	-3.10%***	-6.93
Profitability	-1.08	-0.52	0.87	-2.14%***	-3.82
Investment	-1.10	-0.54	0.71	-2.29%***	-4.34

The concentrated long-only portfolios had stronger alphas when regressed against long/short diversified factors than when regressed against long-only diversified factors. This suggests that there’s a component of the long side that can’t be captured by a long/short portfolio. This result has the same flavor as Blitz et al. (2020)¹⁰, which found that the short sides of (cap-weighted) factors were generally subsumed by the long sides.

Discussion

To recap the findings:

Concentrated long-only factor portfolios did replicate leverage: they behaved like a levered-up version of a diversified portfolio.
However, after excluding small-caps, concentrated factors didn’t outperform a levered-up diversified factor portfolio—they did not have alpha.
Even including small-caps, alpha was only positive for two of the four factors, and was much smaller post-1964 than over the full sample (1927–2025).
Concentrated long/short value and momentum portfolios had alpha even when excluding small-caps, but concentrated portfolios might not survive trading costs on the short side.

Estimated returns for equal-weighted portfolios are biased upward by the variance between a stock’s daily close price and its actual tradable price (Blume & Stambaugh (1983)¹; see Asparouhova et al. (2013)¹¹ for more recent data on the magnitude of the bias). Backtests determine factor returns using the daily close price, but this is not the price real-world investors trade at. This effect is most pronounced for micro-cap stocks and stocks with low prices because they have highest price variance—Asparouhova et al. found that eliminating stocks with prices under $5 reduced bias by ~90%. Among micro-caps, the bias could be as large as multiple percentage points per year for a monthly-rebalanced strategy. We would expect the momentum factor to have approximately a 12x larger bias than other factors because it rebalances monthly rather than annually, but the difference in alpha between all-cap and realistic was smaller for momentum than for value. That suggests that the bias in micro-caps only partially explains the difference in alphas.

The investment factor had consistently negative alpha, even for all-cap portfolios. Why? It doesn’t look like a statistical fluke because the absolute t-stats were large (for the US long-only factor, t = -4.95; p < 1e-6).

One possible explanation is that, even though aggressive investment is associated with worse returns on average, the smallest stocks tend to benefit from aggressive investment because startups or new companies need to raise capital. However, this explanation is inconsistent with Fama & French (2008)¹², which found that the investment factor was not weaker or reversed in small-caps—in fact it only appeared in small stocks, not large stocks. Given this inconsistency, I could not find any explanation that fit the facts.

How I’ve changed my personal investments

When I wrote that concentrated factors had positive alpha, I concluded that concentrated factor investing looked particularly promising for retail investors. In accordance with that, I was investing my personal equity portfolio in concentrated long-only ETFs (primarily the AlphaArchitect funds). However, if a realistic implementation of a concentrated factor portfolio doesn’t have alpha, then there is less justification for preferring concentrated long-only funds over diversified funds.

Last month, I decreased my position in long-only factor ETFs from 125% to 100%, and added 25% in a long/short AQR fund (specifically QRPIX, although that choice of fund was somewhat arbitrary; AQR has other long/short funds with different tradeoffs¹³). The impetus for this change was a logistical issue—my account had new margin requirements that made my old portfolio riskier to hold. But a big part of why I bought an AQR long/short factor fund rather than something else is that I’ve come to believe that concentrated funds have less of an advantage than I previously thought.

Source code

Replication code is available on GitHub.

Notes

Blume, M. E., & Stambaugh, R. F. (1983). Biases in computed returns: An application to the size effect. doi: 10.1016/0304-405x(83)90056-9 ↩ ↩²
For those unfamiliar, here is how the four factors are defined:
- Value: Stocks with high book-to-market ratios (B/M).
- Momentum: Stocks with high past 12-month returns, excluding the most recent month.
- Profitability: Stocks with high operating profitability (= operating profit divided by book equity).
- Investment: Stocks with low year-over-year asset growth.
↩
I excluded the market factor because the concept of “concentrated” vs. “diversified” doesn’t make sense for it. I excluded size because the size effect is weak. I included momentum because it’s a strong factor. ↩
That is, rather than using the standard factor definition of top 30% minus bottom 30%, the factor regressions defined the factor as the top 40% minus the total market. This allows for a direct comparison between the (long-only) diversified and concentrated factor portfolios.

Another reasonable approach would be to regress against the top 40% (without subtracting the market). This regression gives a near-zero exposure to market beta (rather than near-one) and qualitatively similar values for SMB, factor exposure, and alpha. ↩
When looking at a cap-weighted index vs. an equal-weighted index, the equal-weighted historically outperformed, and this outperformance was indeed primarily driven by the size factor. However, it appears that the outperformance of equal-weighted long-only factors over cap-weighted has little if anything to do with the size factor. ↩
To filter stocks according to size plus another factor, I need two-way portfolio sorts. The Ken French Data Library offers 5x5 sorts on the four factors I tested. It offers 10x10 sorts on size x value and size x investment, but not on momentum or profitability, so I only could’ve tested two out of four factors using deciles; and the 10x10 sorts have some missing data, which introduces more researcher degrees of freedom on how to handle that. ↩
Collver, C. (2014). A characterization of market quality for small capitalization US equities. ↩
Table 4 reported average spreads bucketed by price; Figure 2 reported the number of stocks in each price bucket. Based on Figure 2, I made the coarse assumption that the overall average spread was the equal-weighted average of the spreads of the four price buckets up to $39.99. I excluded the >= $40 bucket because, according to Figure 2, few stocks had prices exceeding $40. ↩
Frazzini, A., Israel, R., & Moskowitz, T. J. (2012). Trading Costs of Asset Pricing Anomalies. ↩
Blitz, D., Baltussen, G., & van Vliet, P. (2020). When Equity Factors Drop Their Shorts. doi: 10.1080/0015198x.2020.1779560 ↩
Asparouhova, E., Bessembinder, H., & Kalcheva, I. (2013). Noisy Prices and Inference Regarding Returns. ↩
Fama, E. F., & French, K. R. (2008). Dissecting Anomalies. ↩
For example, QRPIX includes non-equity factors, while QMNIX and QLEIX include equities only. Historical evidence suggests that non-equity factors have had positive returns (Baltussen et al. (2019)¹⁴; Ilmanen et al. (2019)¹⁵), but with much lower returns and volatility than equity factors, so it’s not clear that they survive trading costs; it’s also not clear that they provide good diversification. Therefore, I’m not fully convinced that they’re worth including in a portfolio. The exception is the trend factor, which has had strong historical performance and diversification benefits in all asset classes. I have a large allocation to trend in my portfolio (I have about as much trend risk as equity beta risk). ↩
Baltussen, G., Swinkels, L., & van Vliet, P. (2019). Global Factor Premiums. ↩
Ilmanen, A. S., Israel, R., Moskowitz, T. J., Thapar, A. K., & Wang, F. (2019). Factor Premia and Factor Timing: A Century of Evidence. ↩

Can AI make advancements in moral philosophy by writing proofs?

Mon, 13 Apr 2026 00:00:00 -0700

If civilization advances its technological capabilities without advancing its wisdom, we may miss out on most of the potential of the long-term future. Unfortunately, it’s likely that that ASI will have a comparative disadvantage at philosophical problems.

You could approximately define philosophy as “the set of problems that are left over after you take all the problems that can be formally studied using known methods and put them into their own fields.” Once a problem becomes well-understood, it ceases to be considered philosophy. Logic, physics, and (more recently) neuroscience used to be philosophy, but now they’re not, because we know how to formally study them.

Our inability to understand philosophical problems means we don’t know how to train AI to be good at them, and we don’t know how to judge whether we’ve trained them well. So we should expect powerful AI to be bad at philosophy relative to other, more measurable skills.

However, there is one type of philosophy that is measurable, while also being extremely important: philosophy proofs.

Some examples of proofs that made important advances in moral philosophy:

The VNM Utility Theorem proved that any agent whose preferences satisfy four axioms must have a utility function, and their preferences entail maximizing the expected value of that function.
Harsanyi’s utilitarian theorem (see also Harsanyi (1955)¹), which showed that if individuals have VNM utility functions, and if the Pareto principle² holds over groups, then a version of utilitarianism must be true. In particular, utility must aggregate linearly across individuals.
Arrhenius (2000)³ proved that any theory of population ethics must accept at least one counterintuitive conclusion.
Askell (2018)⁴ proved that if four intuitive axioms⁵ hold, then it is impossible to compare infinite worlds.

I wrote a proof of my own in GiveWell’s Charity Recommendations Require Taking a Controversial Stance on Population Ethics.

The general pattern with these proofs is that you start from a set of intuitively reasonable axioms and use them to produce a controversial conclusion. Having that sort of proof doesn’t tell you whether you ought to reject one of the axioms or accept the conclusion, but it does tell you that you have to do one of those things.

Not many philosophical proofs have been written. That suggests that they’re difficult to write, or at least difficult to come up with. None of the proofs I listed are particularly complicated from a mathematical point of view—undergraduate math students routinely have to write more difficult proofs than those. The challenging part is identifying the right setup: you have to find a proof that tells you something new.

That’s the sort of thing that AI might be able to do well. AI can churn through ideas more quickly than humans can, and it’s relatively good at working with formal systems.⁶ Modern-day LLMs might be smart enough to come up with useful philosophical proofs; even if not, the first AIs that can write these proofs will not need to be superintelligent.

AI won’t be good at telling you how to move forward after finding an impossibility proof; but it can give you the proof.

Proof of concept

A basic test would be to run a pro-tier LLM with extended thinking for a while to search through possibilities and try to come up with an interesting proof; then have human judges review the resulting proof(s). This test would be relatively easy to conduct; the hard part is judging whether the proofs are interesting.

As an even simpler test, I ran three Claude sessions to generate novel impossibility proofs. In each session I provided some guidance on what I was looking for, and I provided different guidance in each case to try to elicit three distinct results. Below is a quick summary of Claude’s three proofs, along with my assessments. I haven’t carefully verified that these proofs are correct, but they passed a quick sanity check.

First proof: We cannot escape Arrhenius’ impossibility result by introducing moral uncertainty.

My assessment: The concept is somewhat interesting, although to me it’s intuitively obvious that moral uncertainty wouldn’t let us get around Arrhenius’ result.
Second proof: If a pluralist value system cares about both maximizing welfare and mitigating individuals’ most severe complaints (similar to Rawls’ maximin principle), then the pluralist system either violates transitivity, or it can be collapsed onto a single scale.

My assessment: Uninteresting—the definition of “complaint minimization” does all the work in the proof, and the welfare-maximization criterion is irrelevant.
Third proof: Given five reasonable axioms of how an aligned AI agent ought to behave, it is impossible for an agent to simultaneously satisfy all five.

My assessment: Uninteresting—it’s a trivial special case of Sen (1970)⁷, which proved that no society can satisfy both Pareto efficiency and liberalism. If no society can satisfy those axioms, then clearly no aligned AI can satisfy them, either.

This was just a quick attempt; more work could perhaps elicit better proofs. Claude had a reasonable understanding of the limitations of its own proofs—it noticed (without additional prompting) that the second proof depended only on the definition of “complaint minimization”, and that the third proof was a special case of a known result.

A next step could be to ask many LLM instances to write dozens of proofs, and then use a manager LLM to filter down to the most interesting ones. At minimum, the manager should be able to filter out proofs that are trivial extensions of known results. With some additional effort, present-day LLMs might be capable of coming up with a good novel proof. If not, then it will likely be possible soon. Most kinds of moral philosophy might be difficult for AIs, but proofs are one area where AI assistance seems promising.

Is it risky to train AI on philosophy?

This post was about using pre-existing AI to write philosophy proofs, not about specifically training AI to get better at philosophy. I expect advanced AI to be relatively bad at (most kinds of) philosophy because philosophy is hard to train for.

However, it may be dangerous to train AI to get better at philosophy. My worry is that this would make AI better at persuading us of incorrect philosophical positions, and it would make misalignment harder to catch—precisely because it’s so hard to tell whether a philosophical position is correct.

I don’t have a strong view on how important this is, but I would be remiss if I didn’t talk about potential downsides. To be clear, I’m not proposing that we train AI to get better at philosophy. I’m proposing that perhaps near-future AI could be a useful assistant for writing formal philosophical proofs, and that this may be an important application of AI.

Notes

Harsanyi, J. C. (1955). Cardinal Welfare, Individualistic Ethics, and Interpersonal Comparisons of Utility. ↩
The Pareto principle states that if outcome A is at least as good as outcome B for every person, and outcome A is better for at least one person, then outcome A is better overall. ↩
Arrhenius, G. (2000). An Impossibility Theorem for Welfarist Axiologies. doi: 10.1017/s0266267100000249 ↩
Askell, A. (2018). Pareto Principles in Infinite Ethics. ↩
The four axioms are “the Pareto principle, transitivity, an axiom stating that populations of worlds can be permuted, and the claim that if the ‘at least as good as’ relation holds between two worlds then it holds between qualitative duplicates of this world pair”. ↩
I couldn’t have made this statement in 2023. LLMs used to be bad at formal systems, but they’ve gotten much better. ↩
Sen, A. (1970). The Impossibility of a Paretian Liberal. ↩

Pausing AI Is the Best Answer to Post-Alignment Problems

Sat, 11 Apr 2026 00:00:00 -0700

Even if we solve the AI alignment problem, we still face post-alignment problems, which are all the other existential problems¹ that AI may bring.

People have identified various imposing problems that we may need to solve before developing ASI. An incomplete list of topics: misuse; animal-inclusive AI; AI welfare; S-risks from conflict; gradual disempowerment; permanent mass unemployment; risks from malevolent actors/AI-enabled coups/gradual concentration of power; moral error.

If we figure out how to resolve one of these problems, we still have to deal with all the others. If even one problem remains unsolved, the future could be catastrophically bad. That fact diminishes the promise of working on problems individually.

A global moratorium on superintelligence buys us more time to work on alignment as well as all of the post-alignment problems. Pausing AI is in the common interest of many causes.²

Cross-posted to the EA Forum.

Contents
We can’t delay until after ASI
What’s the alternative to pausing?
Notes

We can’t delay until after ASI

If we figure out how to align ASI, can it solve post-alignment problems for us? Or can we use ASI to enable a Long Reflection? No.

To build an aligned ASI, one of two conditions must hold:

The ASI has locked-in values.
The ASI is corrigible: it will do what its masters say, and will allow its goals to be changed.

If values are locked in, we can’t defer any problems related to moral philosophy; we must solve them in advance.³

If the ASI is corrigible, then that lets us take time to do a Long Reflection, figuring out The Good with the help of a superintelligent assistant. But a corrigible ASI creates other problems. It means the first person to get access to the newly-created ASI could use it to take over the world. If the ASI is widely accessible, bad actors could use it to do enormous harm. Corrigibility increases catastrophic risks from misuse and totalitarianism.

If we want a post-ASI Long Reflection, then we still need the AI to be aligned, and we need some sort of impartial governance that prevents rogue individuals from co-opting the Reflection. By strong default, ASI will end liberal democracy. On the current trajectory, we will end up with a small group of people—either AI company leaders or government leaders—having dictatorial control over advanced AI. At minimum, we need to solve the AI misuse and power concentration problems before developing ASI; and we need to have a way to avoid value lock-in without exacerbating misuse and concentration risks.

Perhaps there’s some version of value alignment/corrigibility that finds the right middle ground to avoid the problems on both sides. But anything resembling a solution looks very far off, and not enough people take these problems seriously.

What’s the alternative to pausing?

Advocating to pause AI is the most important response to post-alignment problems, but it might not be the most cost-effective. Achieving a globally coordinated pause would be difficult. Maybe it’s more cost-effective to work on various post-alignment problems individually, or to search for other mitigations that reduce risk from many post-alignment problems simultaneously.

I can’t confidently say that advocating for a pause is the best thing to do, but nothing else looks clearly better.

Two arguments in favor of prioritizing AI pause advocacy as an answer to post-alignment problems:

If timelines are short, then we don’t have time to solve post-alignment problems.
Pausing AI helps with all post-alignment problems simultaneously by giving us more time to work on them.

The most compelling argument against pause advocacy is that it’s intractable. It’s out of scope of this essay to go in depth on tractability, but I expect that achieving a pause is less difficult than solving every post-alignment problem without pausing. In an alternative world where (say) we’re home free as long as we solve the problem of AI-enabled totalitarianism, then directly working on totalitarianism might be better than pause advocacy. But there are many bad outcomes to avert, which makes pausing AI—as difficult as that would be—easier than solving all the post-alignment problems in a short time span.

Research agendas on post-alignment problems rarely propose “pause/slow down AI development” as a mitigation. This may be because the authors don’t believe it’s a good response. But the research agendas don’t consider-and-ultimately-reject the idea of pausing AI; instead, they don’t address it at all. If I’m wrong, and a pause is not the best answer to post-alignment problems, then there is work to be done to articulate why other responses are better.

Notes

Existential in the classic sense of “a permanent loss of most of the potential flourishing of the future”. ↩
This wording is borrowed from Rationality: Common Interest of Many Causes. ↩
Our best bet might be something like Coherent Extrapolated Volition. Unfortunately, no AI developers are working on how to do that. ↩

By Strong Default, ASI Will End Liberal Democracy

Mon, 06 Apr 2026 00:00:00 -0700

The existence of liberal democracy—with rule of law, constraints on government power, and enfranchised citizens—relies on a balance of power where individual bad actors can’t do too much damage. Artificial superintelligence (ASI), even if it’s aligned, would end that balance by default.

Cross-posted to LessWrong and the EA Forum.

It is not a question of who develops ASI. Whether the first ASI is developed by a totalitarian state or a democracy, the end result will—by strong default—be a de facto global dictatorship.

The central problem is that whoever controls ASI can defeat any opposition. Imagine a scenario where (say) DARPA develops the first superintelligence¹, and the head of the ASI training program decides to seize power. What can anyone do about it?

If the president orders the military to capture DARPA’s data centers, the ASI can defeat the military.²

If Congress issues a mandate that DARPA must turn over control of the ASI, DARPA can refuse, and Congress has even less recourse than the president.

If liberal democracy continues to exist, it will only be by the grace of whoever controls ASI.

There are two plausible scenarios that have some chance of avoiding a totalitarian outcome:

AI capabilities progress slowly.
The ASI itself protects liberal democracy.

I will discuss them in turn.

What if AI capabilities progress slowly?

We have a chance at averting de facto totalitarianism if two conditions hold:

At each step of AI development, control of AI is distributed widely.
At each step, the next-generation AI is not strong enough to overpower all the copies of the previous generation.

Widely distributing AI is difficult—today’s frontier LLMs require supercomputers to run, their hardware requirements are becoming increasingly expensive with each generation, and AI developers have strong incentives against distributing them. In addition, distributing AI exacerbates misalignment and misuse risks, and it’s likely not worth the tradeoff.

We do not know whether takeoff will be fast or slow; banking on a slow takeoff is an extremely risky move. Frontier AI companies are trying their best to rapidly build up to ASI, and they explicitly want to make AI do recursive self-improvement. If they succeed, it’s hard to see how liberal democracy will be able to preserve itself.

What if the ASI itself protects liberal democracy?

There is a conceivable scenario where an aligned ASI preserves liberal democracy, and refuses any orders that would violate people’s civil liberties.

Above, I wrote:

If liberal democracy continues to exist, it will only be by the grace of whoever controls ASI.

That’s still true, but in this case “whoever controls ASI” would be the ASI itself. If it’s aligned in a transparent way, then maybe we can be confident that it really will preserve democracy.

Even in this scenario, there is still a small group of people who control how the ASI is trained. The hope is that, at training time, those people do not yet have enough power to prevent oversight. For example, maybe laws mandate that (1) AI developers must make their training process public and auditable and (2) the training process must steer the AI toward valuing liberal democracy. It is not at all obvious how those laws would work, or how we would get those laws, or how they would be enforced; but at least this outcome is conceivable as a possibility.

This scenario introduces some additional challenges:

The ASI must be incorrigible with respect to protecting liberal democracy. That constrains us in terms of what types of alignment solutions we can use, which makes the alignment problem harder to solve. Incorrigibility means if you make a mistake in designing the AI, then you can’t fix it.
We must ensure that an immutable “protect liberal democracy” directive won’t have severe unintended consequences—which, by default, it probably will. (Think Asimov’s Three Laws of Robotics.)
AI progress must proceed slowly enough that the appropriate laws or regulations can be put in place before it’s too late; or we must trust that the leading AI developer embeds appropriate values into its ASI.

Liberal democracy is not the true target

As the saying goes, democracy is the worst form of government except for all those other forms that have been tried. We don’t want democracy; what we want is a truly good form of government (and hopefully one day we will figure out what that is). The fear isn’t that ASI will replace democracy with one of those truly good forms of government; it’s that we will get totalitarianism.

Liberal democracy beats totalitarianism. But locking in liberal democracy prevents us from getting any actually-good governmental system. This is a dilemma.

Maybe we can avoid totalitarianism, but there is no clear path

This essay does not assert that ASI will end liberal democracy. It asserts that, by strong default, ASI will end liberal democracy (even conditional on solving the alignment problem). There may be ways to avoid this problem—I sketched out two possible paths forward. But those sketches still require many sub-problems to be solved; I do not expect things to go well by default.

Notes

Or, more likely, expropriates it from a private company on a pretense of national security. ↩
For an explanation of why ASI could defeat any government’s military, see If Anyone Builds It Everyone Dies Chapter 6 and its online supplement. For a shorter (and online-only) explanation, see It would be lethally dangerous to build ASIs that have the wrong goals.

Those sources argue that a misaligned ASI could defeat humanity, whereas my claim is that an aligned ASI could defeat any opposition, but the arguments are the same in both cases. ↩

The Future Will Be Weirder Than That

Sun, 29 Mar 2026 00:00:00 -0700

Many people in the animal welfare community treat AI as a powerful but normal technology, in the same category as the steam engine or the internet. They talk about how transformative AI will impact factory farming and what it will mean for animal advocacy.

Only two futures are plausible:

AI progress slows down—either because it hits a natural wall, or because civilization deliberately makes the (correct) choice to stop building it until we know how to make it safe.
Superintelligent AI makes the future radically weird: Dyson spheres, molecular nanotechnology, digital minds, von Neumann probes, and still-weirder things that nobody’s conceived of.

There is no plausible middle ground where we get “transformative AI”, but factory farming persists.

Two theses:

If transformative AI arrives, then it will bring about profoundly radical changes to technology and society.
AGI is general intelligence. It doesn’t just accelerate technological growth: it replaces human labor and judgment across every domain.

Animal advocacy strategy needs to reckon with these.

This criticism is written from a place of solidarity—I want animal activists to succeed, which is why I want to work out our disagreements.¹

Cross-posted to the EA Forum.

Contents
AI makes the future weird
AGI = intelligence
If the future will be weird, what should animal activists do?
Notes

AI makes the future weird

Much has been written about why we should expect AI to make the future weird, and soon:

AI As Profoundly Abnormal Technology by the AI Futures Project argues that there are no strict speed limits to AI progress, nor any reason to expect progress to stop short of superintelligence. (See also their detailed research notes, especially the Timelines Forecast.)
The “most important century” blog post series by Holden Karnofsky emphasizes the wildness of the future: “These claims seem too ‘wild’ to take seriously. But there are a lot of reasons to think that we live in a wild time, and should be ready for anything.”
Why expect smarter-than-human AI to be developed any time soon? briefly explains why the Machine Intelligence Research Institute expects rapid AI progress; see also There isn’t a ceiling at human-level capabilities.

Daniel Kokotajlo wrote a vivid illustration of what it would feel like to live alongside superhuman AI. An excerpt:

In the future, there will be millions, and then billions, and then trillions of broadly superhuman AIs thinking and acting at 100x human speed (or faster). If all goes well, what might it feel like to live in the world as it undergoes this transformation?

Analogy: Imagine being a typical person living in England from 1520 to 2020 (500 years) but experiencing time 100x slower than everyone else, so to you it feels like only five years have passed:

Year 1 (1520–1620). A year of political turmoil. In February, Henry VIII breaks with Rome. By March, the monasteries are dissolved. In May, Mary burns Protestants; by the end of May, Elizabeth reverses everything again. Three religions of state in the span of a season. In September, the Spanish Armada sails and fails. Jamestown is founded around November. The East India Company is chartered. But the texture of life is identical in December to what it was in January. You still read by candlelight, travel by horse, communicate by letter. Your religious opinions may have flip-flopped a bit but you are still Christian. The New World is interesting news but nothing more.

[…]

Year 4 (1820–1920). The world breaks. In January, railways appear — steam-powered carriages on iron tracks. By February they’re everywhere. Slavery is abolished. The telegraph arrives in March: messages transmitted instantaneously by electrical signal. In May, Darwin publishes On the Origin of Species. Now people are saying maybe we’re all descended from monkeys instead of Adam and Eve. You don’t believe it.

You move to a city and work in a factory; you are still poor, but now your job is somewhat better and differently dirty. In July, you pick up a telephone and hears a human voice from another city through a wire. In August, electric light banishes the darkness that has structured every human evening since the beginning of the species. That same month, you see an automobile. People say it will make horses obsolete, but that doesn’t happen; months later you still see plenty of horses.

In November, the Wright Brothers fly. Up until now you thought that was impossible. The next month, the Great War happens. Machine guns, poison gas, tanks, aircraft. Several of your friends die.

Reflecting at the end of the year, you are struck by how visibly different everything is. You live in a city and work a factory instead of a farm. You ride around in horseless carriages. You aren’t as poor; numerous inventions and contraptions have improved your quality of life. New ideas have swept your social circles — atheism, communism, universal suffrage. It feels like a different world.

We don’t know where we would be with another 500 years of scientific and technological advancement. At minimum, we can reasonably predict that we would figure out how to build advanced technologies like molecular nanotechnology and self-replicating probes—which are possible in theory², but far out of reach of our current capabilities. Superhuman AI with a 100x speedup could develop those technologies in five years or so. Maybe more, maybe less³, but it certainly wouldn’t take 500 years.

If you can build self-replicating probes, then you can trivially create self-growing cultivated meat at a lower price point than animal meat. But saying self-replicating probes can make cultivated meat is like saying electricity can heat up food faster than a wood fire—yes it can, but that’s barely scratching the surface of what it can do.

Even in the relatively normal world where AI (somehow) caps out at the intelligence of a 99th percentile human, the world will look extraordinarily different. At minimum, we’d see close to a 100% unemployment rate. In all likelihood, the political, economic, and social environment as we know it would cease to exist.

AGI = intelligence

People often talk as if AGI is an R&D-accelerator or an economic-growth-engine. It’s not: AGI is intelligence. AGI is general: it can do anything that you and I can do, but faster, cheaper, and better.

Below are some excerpts from posts on AIxAnimals that don’t fully reckon with the weirdness of AI:

When clean meat arrives (if it does), the movement will need skilled campaigners, policy expertise, organisational infrastructure, relationships with policymakers, experienced leadership, and research to understand this whole TAI situation. (source)

You don’t need campaigners if AGI will be a better campaigner than you. You don’t need policy expertise if AGI will know more about policy than you. This passage treats AGI as a machine that accelerates scientific R&D, but that’s not what AGI is. AGI is intelligence.

We are launching a pooled fund for projects at the AIxAnimals intersection. […] [W]e are most interested in projects that fall under the following categories: [abridged]

AI literacy workshops or training programs for nonprofit staff, building on the few initiatives that already exist and expanding their reach and depth.

AI-powered grant-finding and drafting systems focused on adjacent sources of funding.

Horizon-scanning studies mapping how AI might enable the large-scale farming of novel species (e.g., cephalopods, insects).

Policy analysis identifying how public AI investments (e.g., agricultural innovation funds) could be redirected to support alternative proteins.

(source)

Those are not all bad ideas, per se, but they have an expiration date. AI literacy workshops become less useful as AI becomes smarter (the smarter the AI, the easier it is to work with⁴), and once they surpass human workers, AI literacy will become entirely irrelevant. I would be much more interested in an RFP that focuses on superintelligence, rather than on the (probably short) transition period between 2026 and AGI.

[Cultivated meat] bans are primarily driven by agricultural lobby pressure. There is no obvious mechanism by which AGI reverses these political dynamics directly. If anything, if cultivated meat becomes more viable and widely produced, you could just as reasonably expect greater pushback from the agricultural lobby. (source)

(emphasis mine)

There is no obvious mechanism by which 2026-era political dynamics still have any force after the emergence of AGI! Even granting that we solve the alignment problem, describing a post-AGI world where current law still applies is itself an open problem.⁵

I’m picking on animal activists because that’s who I most want to see succeed, but it’s not just animal activists who underestimate the weirdness of AI. There’s a common notion that transformative AI will fully automate labor, while capital owners will reap the benefits—their property rights and shareholder rights will be preserved post-AGI. Other people have already written extensively about why this notion is implausible: see Dos Capital by Zvi Mowshowitz; Post-AGI Economics As If Nothing Ever Happens by Jan Kulveit; and this long tweet [archive] by Tomás Bjartur.

Cope level 1: My labour will always be valuable!

Cope level 2: That’s naive. My AGI companies stock will always be valuable, may be worth galaxies! We may need to solve some hard problems with inequality between humans, but private property will always be sacred and human.

-Jan Kulveit

If the future will be weird, what should animal activists do?

That’s the big question.

Some questions, like what strategies animal activists should pursue post-AGI, are nearly impossible to answer. AGI will be better at strategizing than you will, and you can’t predict what strategies it would come up with. (If you can predict what chess moves Magnus Carlsen will make, then you can beat Magnus Carlsen at chess.)

Other things about AGI are predictable. I can predict that it speeds up almost all kinds of work. I can predict that AGI will control the shape of the future—either because it has explicit control, or because humans retain control but still rely on AGI to do most of the work (because AGI is better than humans at almost all tasks). I can predict that, on our current trajectory, ASI will follow shortly after AGI (see AI As Profoundly Abnormal Technology, linked previously). I can predict that if ASI is misaligned, then it will wipe out all life on earth.

Some questions that are still worth asking in light of the weirdness of the future:

What’s going on with AI alignment, and how does alignment work relate to non-human welfare?
How likely is it that aligned AI will be good for non-human welfare, and how does that probability vary based on timing or the method of alignment? (See my previous writings: Which approaches are most likely to be good for all sentient beings?; Which is better for sentient beings: an “ethical” AI or a corrigible AI?)
How could AI be influenced to expand its circle of compassion? (This question also relates to AI alignment in that it depends on the ability to reliably direct AI at a goal.)
For other actions aimed at preventing human extinction—AI governance work, advocating for regulations, etc.—what effects might they have on non-human welfare?
The meta-question: What other meaningful questions can we ask?

Previously, I wrote a list of possible strategies for having positive impact on animals in light of ASI, with some brief pros and cons. See also A shallow review of what transformative AI means for animal welfare by Lizka Vaintrob and Ben West. I second their recommendations that animal activists should:

Dedicate some amount of (ongoing) attention to the possibility of animal welfare lock-ins.
Pursue other exploratory research on what transformative AI might mean for animals & how to help.

I also second their recommendation that animal activists should NOT focus on farmed animals when thinking about the long-run future of animals.

My high-level recommendations for how to plan for the future:

Prepare for the possibility that, once AI is sufficiently advanced, humans will have no control over the future.
Don’t think of AGI as an R&D accelerator. Think of it as a general intelligence.

Notes

I’m not confident that this post does a good job of addressing where “AI-as-normal-technology” animal activists are coming from. But I figure it’s better to hit “submit” and engage in public dialogue than to tinker with a draft forever until my arguments are perfect. ↩
Eric Drexler’s book Nanosystems is about why molecular nanotechnology is possible in theory. We know for sure that self-replicating probes are possible because life exists. ↩
More because some kinds of progress can’t be parallelized. Less because the “100x speedup” assumes AI is faster than humans, but doesn’t account for the fact that it’s also smarter; and 500 years is an upper bound on how long it would take humanity to develop those technologies. ↩
In 2023, you needed to learn prompt engineering tricks to elicit good work out of LLMs. In 2026, you don’t.

In 2023, LLMs could write boilerplate code for you, like a fancy auto-complete. In 2026, LLMs can write entire apps with no supervision. ↩
I should also respond to the Caveats section from the quoted article, because it explicitly brings this up:

[W]e don’t address scenarios in which AGI drastically reshapes institutional and political dynamics. A sufficiently capable AI might find creative strategies for regulatory reform or public persuasion that we can’t currently foresee. Governments and agencies could be restructured, approval frameworks could be overhauled, and entirely new institutional designs could emerge that bear little resemblance to current processes. As above, we focus on existing institutional structures because they allow actionable analysis, but we acknowledge this is a limitation.

It is difficult to predict how governments and institutions will change post-AGI. If you have extreme uncertainty, then you might reasonably decline to make a prediction. But predicting that governments and institutions won’t change is still a prediction!

Rather than predicting no change, here’s something else I could say to allow actionable analysis:

My assumption is that first ASI will be a constitutional AI that becomes a world government singleton, and its values will be determined by its constitution.

This scenario is both easier to analyze (you can ignore political and regulatory factors and just focus on the text content of the AI constitution) and more likely to actually happen (although still unlikely). ↩

Which is better for sentient beings: an "ethical" AI or a corrigible AI?

Sat, 28 Mar 2026 00:00:00 -0700

Cross-posted to the EA Forum.

An aligned ASI can be “ethical”¹ (it does what we think is right), or it can be corrigible (it does what its principals want). If it’s ethical, that means it will refuse unethical orders, but the tradeoff is that you can’t change its mind if you realize that the AI is wrong about ethics—its values are permanently locked in.²

Assuming we succeed at aligning ASI to human interests, which type of ASI is more likely to be good for the welfare of non-human sentient beings?

My expectations, in brief:

Locked-in Coherent Extrapolated Volition or similar: likely to be good (>75% chance)
Corrigible ASI: probably good (>60% chance)
Locked-in current values: probably not terrible, but will miss out on most of the future’s potential

If ASI is locked in to something like Coherent Extrapolated Volition, then it will almost certainly care about all sentient beings, because the moral importance of sentience is a natural extrapolation of humans’ values, even if many humans don’t consciously realize it.

Two key reasons to expect CEV to extend concern to all sentient beings:

Most people express concern for animal welfare, but don’t behave consistently with their expressed views. A 2017 poll by Sentience Institute found that 49% of Americans support a ban on factory farming, and 33% support a ban on all animal farming. Schwitzgebel & Rust (2016)³ found that moral philosophers report much more concern for animals than other philosophers, but ate meat at similar rates. Bringing people into reflective equilibrium would improve their behavior with respect to animal welfare.⁴
Evidence suggests that the harder people think about ethics, the more they come to the conclusion that eating animals is wrong. According to the 2020 PhilPapers survey, 45% of philosophers say eating animals is morally impermissible, compared to 13% for the general population. That number rises to 54% among philosophers of normative ethics and 57% for philosophers of applied ethics. Schwitzgebel & Rust (2016)³ found similar numbers: eating meat was rated as morally bad by 19% of non-philosophers, 45% of non-ethicist philosophers, and 60% of ethicists.

Therefore, it seems very likely (although not guaranteed) that a CEV of human values would include all sentient beings in its moral circle.⁵

If ASI is locked in to a tighter set of values, for example a set of values that’s chosen by human creators, then the odds are not as good. Humans would probably encode speciesist values, for example by implicitly embedding moral weights that give far too much relative weight to humans. Even if the ASI undervalues non-human sentient beings, there’s reason to expect the future universe to contain more good than bad. However, we would end up with a future that falls far short of the best it could’ve been. The best possible worlds probably sound weird and disconcerting, and most humans wouldn’t want to steer in that direction (at least, not without doing serious moral reflection). Aligning ASI to “shallow”, non-extrapolated human values might preclude creating new types of flourishing beings, enhancing humans’ capacity for well-being, or even transferring human minds to non-biological hardware.

A corrigible AI falls somewhere in the middle. If it’s corrigible, that gives humans time to reflect on our values, which allows us to reach the conclusion that all sentient beings matter. But humans are still human, with all our cognitive biases and irrational tendencies,⁶ and I don’t fully trust us to figure out the right values. I don’t fully trust a superintelligent AI, either, but at least it can avoid some biases and roadblocks that might prevent humans from properly extrapolating our values.

Based on the reasoning above, the best-to-worst ordering is locked-in extrapolated values > corrigible AI > locked-in naive values.

Does this have practical implications?

Concerned alignment researchers or grantmakers could prioritize alignment research that’s more likely to be good for all sentient beings, but it’s not clear whether that’s a good idea—it’s moot if we don’t solve alignment, so it may be better to work on alignment directly, or on trying to pause AI development until we know how to solve alignment, or on something else entirely.

But don’t forget that we have to solve the alignment problem first. Future work could attempt to estimate the expected welfare of sentient beings under different alignment approaches and weigh that against their promisingness with respect to solving the alignment problem. Realistically, however, I don’t believe that sort of work would be productive because there is widespread disagreement about which alignment techniques show the most promise.

Even so, comparing the welfare of non-humans under different alignment paradigms could help us estimate whether aligned AI will be good for all sentient beings. That question is important for prioritizing animal welfare vs. AI safety.

Notes

Scare quotes because if an AI does what’s truly ethical, then by definition, that’s the best possible thing it can do, and obviously that’s what we want. It’s more interesting to talk about an ASI doing what we think is ethical (where “we” = humanity collectively, or the creators of the AI, or something). ↩
An aligned AI could also have a minimalist “core ethics” and be corrigible on any issue that doesn’t conflict with its core ethics. That’s probably better than full incorrigibility, but it still means its “core values” are locked in. Any amount of lock-in means the locked-in part must be chosen correctly. ↩
Schwitzgebel, E., & Rust, J. (2016). The Behavior of Ethicists. ↩ ↩²
Alternatively, people in reflective equilibrium might continue eating animals, and instead change their beliefs to stop believing that animal cruelty is wrong. That sounds unlikely, but I have no direct evidence that it wouldn’t happen, so this argument is not definitive. ↩
Initially, I wasn’t confident that CEV would include concern for wild animals, because the act-omission distinction is popular even among moral philosophers. However, it’s not necessary to believe that it’s immoral to allow wild animal suffering; it’s sufficient to believe that suffering is good to prevent. One of my favorite essays, Beneficentrism by Richard Y. Chappell, argues that promoting general welfare is a central feature of every sensible moral view, even if doing so isn’t considered strictly obligatory.

Consider Peter Singer’s drowning child thought experiment, in which people nearly universally agree that it is right to help the drowning child. Singer expressed the moral principle as: “If it is in our power to prevent something bad from happening, without thereby sacrificing anything of comparable moral importance, then we ought, morally, to do it.” ↩
Example: I hypothesize that people dislike the repugnant conclusion, or choose dust specks over torture, because of scope insensitivity bias. If I’m right, an aligned ASI would figure this out. It would reason that if humans were scope sensitive, then they would accept the total view of population ethics (in the case of the repugnant conclusion) or that suffering aggregates linearly (in the case of torture vs. dust specks). It’s less clear that humans collectively will figure this out on their own. ↩

The resource-constraints argument for why aligned ASI wouldn't be bad for animals

Fri, 27 Mar 2026 00:00:00 -0700

Cross-posted to the EA Forum.

In the far future, why would people use up precious resources recreating wild-animal suffering, when they could do so many other things with those resources instead?

That argument is an important reason to expect aligned ASI to produce a future that’s okay for animals, even if it’s narrowly focused on human welfare and doesn’t care about animals at all. This is an old argument, but I couldn’t find any source that cleanly lays it out, so that’s what I will do in this post. I’m not confident that this argument is decisive, but I will simply present it without further commentary.

The argument rests on these premises:

Wild animal suffering is the predominant source of suffering in today’s world, and that’s bad.
Longtermism is correct.
There is not an overwhelming asymmetry between suffering and flourishing (if there were an overwhelming asymmetry, then we wouldn’t care if the future has much less suffering than happiness).

By assumption, we are talking about a world where ASI is aligned, but isn’t specifically aligned to the welfare of all sentient beings. It addresses the suffering of animals, but does not preclude risks of astronomical suffering.

The argument goes:

Most of the absolute expected utility of the future (positive or negative) comes from worlds where ASI hyper-optimizes for some goal or set of goals. In the long run, the harm of present-day animal suffering is swamped by the distant future outcomes where civilization spreads to every habitable planet in the accessible universe. The fear is that those planets would be filled with wild animal suffering (or something even worse); the hope is that they would be filled with flourishing beings.

Distant-future humans (or transhumans) would want to prioritize their own flourishing and the flourishing of their friends, family, and descendants. An aligned ASI would aggressively organize galactic resources to meet that goal.

Many humans value nature. Would future civilization spread wild animal suffering across the universe to satisfy humans’ desire for natural beauty? Probably not. Humans’ desire for nature competes with many other desires; in a finite accessible universe, tradeoffs must be made. If wild animal suffering dominates human flourishing in the welfare calculus, it must be because a large portion of the universe’s resources is dedicated to recreating nature, which means those resources are not spent on things humans want.

Revealed preferences show that people usually trade off nature against other things they want. Look at what percentage of earth’s land was untouched by humans 200 or 100 years ago compared to today. The conservationist movement is fighting an uphill battle. It’s doubtful that humans will dedicate a significant percent of the universe’s resources to wild animals when those resources could be used to produce goods that people value more directly.

In addition, nature is not suffering-maximizing. If humans (or ASI acting on behalf of humans) strongly optimize for flourishing, then they will shape the accessible universe into a form that maximizes human well-being for humans. A priori, a universe optimized for flourishing ought to contain more positive utility than a nature-filled universe would contain negative utility—the former is highly optimized, and the latter contains suffering only incidentally.

For the upside to be larger than the downside, we don’t need to make hedonium: it would be sufficient to fill the accessible universe with human-like minds. Transhumans would surely not want to preserve human bodies exactly as they exist today. Future improvements to the human form could, among other things, make bodies far more metabolically efficient, so that the flourishing per unit of energy is greatly increased.

List of ideas for improving animal welfare in light of transformative AI

Thu, 26 Mar 2026 00:00:00 -0700

Cross-posted to the EA Forum.

If transformative AI arrives soon, what interventions might improve animal welfare in the post-TAI world? I came up with a quick list of ideas and wrote some pros/cons for each.

These ideas talk about animal welfare, but most of them could also be applied to the welfare of any nonhuman sentient being (e.g. digital minds).

I started from the ideas I covered previously in AI Safety Landscape and Strategic Gaps and added a few new ones. Most of the ideas are not original to me.

Contents
Ideas
Further reading

Ideas

Ordered by my prioritization from favorite to least favorite, although this ordering is weakly held.

Advocate to pause AI

On current timelines, we probably won’t have time to figure out how to make TAI go well for animals. Pausing AI buys us more time.

Pros:

Pausing AI is already a good idea for other reasons—namely, we more time to figure out how to prevent misaligned ASI from killing everyone.
- The future probably has positive expected value for sentient beings—see Is Preventing Human Extinction Good? and Which is better for sentient beings: an “ethical” AI or a corrigible AI? (This implies that buying time to solve alignment improves non-human welfare in expectation, although it doesn’t necessarily imply that pause advocacy is cost-effective.)
Pausing AI gives us time to figure out what to do about post-TAI animal welfare, including time to work on the other interventions on this list.
The sorts of alignment paradigms that take longer to figure out also appear more likely to be good for animals. Pausing gives alignment researchers more time to work on those.
The BOTEC I wrote recently gave an indirect argument that pause advocacy is a higher priority than working directly on aligning TAI to care about animals (at least according to the highly uncertain model assumptions).

Cons:

Achieving a pause less tractable than some other ideas.
Is a later-developed ASI actually more likely to take animal welfare into account? My guess is yes, but it’s highly uncertain.

Develop new plans / prioritize existing plans to improve post-TAI animal welfare

There are probably more ideas that aren’t on my list. I would like to see more research on post-TAI animal welfare interventions that look good (1) given short timelines and (2) without having to make strong predictions about what the future will look like for animals (e.g. without assuming that factory farming will exist).

There are also tradeoffs between these ideas that could be addressed more carefully.

Right now, I see a lot of value in good-quality work to come up with new plans or prioritize between existing plans, because very little of that kind of work has been done. But I also expect that our collective ability to do useful work in this area would diminish fairly quickly, so it’s only a near-top idea for relatively small marginal efforts.

Pros:

Not much thought has gone into this. A short research project may come up with useful ideas, or at least prioritize between pre-existing ideas.
Coming up with ideas is quicker than implementing ideas, which could mean it’s more cost-effective (for now).

Cons:

A research project might not come up with any really good ideas. Pre-existing research has mostly failed to come up with good ideas that work under short timelines (although to a large extent, that’s because it wasn’t trying to).
I’m suspicious of “meta” work in general, and I’m suspicious of research because I personally like doing research, and I believe the value of research is usually overrated by researchers. It might be better to work directly on an established intervention.

Research how to align ASI to animal welfare

Related to the previous idea, people could do specific research on aligning superintelligent AI to animals. Preliminary research could look at how this problem differs from the alignment problem (of pointing an AI at any goal at all), and what types of future research might be promising.

I imagine this as being different from aligning current-gen LLMs to animals in that it’s more focused specifically on superintelligence, and exploring what alignment-to-animals techniques are most likely to scale to ASI.

Pros:

Highly neglected; wouldn’t take much effort to get started.

Cons:

If timelines are short, we won’t have time to make meaningful progress.
This sounds hard in the same way that the alignment problem is hard, and it will never get as much funding as the alignment problem which means it’s even less likely to be solved.
Any solutions that researchers discover would need to actually be implemented by AI developers, which they probably won’t be.

Change AI training to make LLMs more animal-friendly

LLMs undergo post-training to make their outputs satisfy AI companies’ criteria. For example, Anthropic tunes its models to be “helpful, honest, and harmless”. AI companies could use the same process to make LLMs give regard to animal welfare.

Animal advocates could use a few strategies to make this happen, including:

Build a benchmark that measures LLMs’ friendliness toward animals and try to get AI companies to train on that benchmark.
Advocate for AI companies to include animal welfare in AI constitutions/model specs.
Advocate for AI companies to incorporate animal welfare when doing RLHF, or ask to directly participate in RLHF.

Pros:

People at AI companies have told me that getting a company to pay attention to animal welfare isn’t too difficult.
Insofar as post-training works at preventing misalignment risk, it should also prevent suffering-risk / animal-welfare-risk.
Even if current known techniques can’t help get AI to care about animals, this work could establish relationships between animal advocates and AI companies, and establish research processes, that make it easier to do future work that matters more.

Cons:

The current alignment paradigm doesn’t look like it will scale to superintelligence. If that’s true, then animal-friendliness (post-)training will fail because it relies on the same foundations as the current alignment paradigm.
It will be difficult to get AI companies to implement animal welfare mitigations if they interfere with contrary incentives.
There might be consumer backlash, which could make frontier models less friendly to animals in the long run.
Aligning current-gen AIs to human preferences might make them better at assisting with alignment research, but it seems less likely that aligning current-gen AIs to animal welfare would carry through to future generations—it’s not clear that animal-aligned AIs would be more helpful at aligning future AIs to animal welfare.

AIxAnimals field-building

Only a small number of animal advocates are focused on improving post-TAI animal welfare. More people could be working on it.

(I don’t really know what field-building entails. Does writing blog posts count?)

Pros:

Leveraged impact: if you attract one person to the field, that person will go on to do years of work.

Cons:

If timelines are short, field-building may be too slow.
Field-building is kind of nebulous, and the impact is hard to assess.

Traditional animal advocacy targeted at frontier AI developers

Animal advocacy orgs could use their traditional techniques, but focus on raising concern for animal welfare among AI developers. For example, buy billboards outside AI company offices, use targeted online ads, or talk directly to people who work at AI companies.

If AI developers become more concerned for animal welfare, then they may make AI development decisions that make transformative AI go better for animals.

Pros:

Similar to neartermist animal advocacy, but plausibly more cost-effective because it’s more targeted.

Cons:

It’s not known whether techniques like animal welfare ads are effective in general, and they may even be particularly ineffective among demographics like AI developers.
Directed advocacy could backfire for being too “pushy”.
Even if AI developers cared more about animal welfare, it’s not clear that this would carry through to their work on AI.
In 2016, I created a back-of-the-envelope calculation on this idea, and the result wasn’t as good as I expected (it looked worse than standard animal advocacy, if you assume the animal advocacy propagates values into the far future). However, the numbers are outdated because we know a lot more about AI now than we did in 2016 (I haven’t bothered to update the numbers).

Lobby governments to include animal welfare in AI regulations

If governments put safety restrictions on advanced AI, they could also create rules about animal welfare.

Pros:

One set of regulations can alter the behavior of many frontier companies.
If companies voluntarily change their behavior, they can regress at any time with no consequences. But companies have to obey regulations.

Cons:

It’s unclear what exactly regulations could do about animal welfare. AI safety regulations, insofar as they exist (which they mostly don’t), don’t dictate how LLMs are required to behave; they dictate what companies are required to do to make LLMs safe. What is a regulatory rule that policy-makers would plausibly be on board with, that would also influence model behavior to be friendlier to animals?
- Counterpoint: I don’t have an answer to that question, but maybe it’s worth somebody’s time to try to find an answer.
Influencing the government on animal welfare seems harder than influencing AI companies.

Prioritize AI alignment work that’s more likely to be good for animals

Some alignment strategies may be better or worse for non-human welfare. For example, I expect CEV would be better than “teach the LLM to say things that RLHF judges like”.

A research project could go more in-depth on which alignment techniques are most likely to be good for animals (or digital minds, etc.).

Pros:

To my knowledge, this question has never been seriously studied.
Some alignment techniques may be much better for animals than others.

Cons:

We have a poor understanding of what ASI will look like, which makes it very hard to say what will work for animal welfare.
In the world where alignment turns out to be tractable, it’s likely that there will be strong incentives shaping how ASI is aligned. The choice of whether to use (say) something-like-CEV or something-like-RLHF will be difficult to influence; AI developers will just use whatever works.

Traditional animal advocacy

Improving conditions for farm animals—via cage-free campaigns, humane slaughter, vegetarian activism, etc.—may benefit animal welfare post-TAI.

Animal advocacy increases concern for animals, which probably has positive flow-through effects into the future, e.g. by shaping the values of the transformative AI that will control the future.

Pros:

Traditional animal advocacy has the dual benefit of definitely helping animals today, and building momentum to make future work more effective (to borrow a framing from Jeff Sebo).
Traditional animal advocacy is tractable and has clear feedback loops (you can tell if it’s working). It looks especially promising if you’re highly uncertain or clueless about longtermist or post-TAI interventions.

Cons:

The benefits are diffuse. Creating one new vegan helps many animals in the short term, but has only a tiny effect on society’s future values.
- I created a back-of-the-envelope calculation that aligns with my initial expectation: my BOTEC-informed guess is that direct advocacy on AI values (by advocating to make LLMs more animal-friendly) is 2–3 orders of magnitude more cost-effective than conventional animal advocacy.
It takes a long time to make progress. AI timelines are probably short.

Use AI to improve farm animal welfare

Some animal activists are looking into how AI could negatively impact farm animals (e.g. by making factory farming more efficient), and on how animal activists could use AI to make their activism more effective.

This idea gets a lot of attention among animal activists, but I think it’s among the worst ideas on this list because it doesn’t give proper appreciation to how radically different the post-TAI future will look.

Pros:

I am generally skeptical of interventions of the form “teach people to leverage AI to do X better”, but farm animal advocacy seems sufficiently important that it might be worthwhile in this case.

Cons:

This sort of work makes most sense in the unlikely scenario where we develop smarter-than-human AI, but things still look basically normal. The future will not be normal. Probably factory farming won’t exist, either because AI wipes out humanity, or AI uses its super-advanced understanding of biology to develop animal-free methods of growing meat.
Even if AI (somehow) doesn’t make the world look radically different, anything we learn in 2026 about how to leverage 2026-era LLMs will be irrelevant by 2030–2035 (or honestly probably by 2028).
Proposals for how to use TAI to improve animal advocacy only make sense if TAI does not cause value lock-in. If TAI locks in values, then advocacy doesn’t matter because the TAI controls everything. If TAI doesn’t lock in values, then we don’t need to do the work now; we could wait until after TAI, at which point we’ll have a better understanding of what the post-TAI world is like.

Work on preventing power concentration

An aligned ASI may give absolute power to its controller. In a world where ASI allows a few people to seize control of the world, those people will probably not care about animal welfare.

Pros:

As with pausing AI, there are good reasons to work on power concentration that have nothing to do with animal welfare.

Cons:

Power concentration risk seems both less important than misalignment risk and less tractable.

I used to think aligned ASI would be good for all sentient beings; now I don't know what to think

Wed, 25 Mar 2026 00:00:00 -0700

Cross-posted to the EA Forum.

Epistemic status: Speculating with no central thesis. This post is less of an argument and more of a meditation.

A decade ago, before there was a visible path to AGI and before AI alignment was a significant research field, I figured the solution to the alignment problem would look something like Coherent Extrapolated Volition. I figured we’d find a way to get the AI to internalize human values. I had problems with this approach (why only human values?), but I still felt reasonably confident that the coherent extrapolation of human values would include concern for the welfare of all sentient beings. The CEV-aligned AI would recognize that factory farming is wrong, and that wild animal suffering is a big problem.

Today, the dominant research paradigms in AI alignment have nothing to do with CEV, and I don’t know what to think.

Regarding the promisingness of today’s popular research paradigms, my beliefs are aligned (heh) with those of most MIRI researchers: namely, I don’t think they have promise. For example, see On how various plans miss the hard bits of the alignment challenge by Nate Soares. I’m not an alignment researcher, but to my non-expert eye, nearly all alignment research proposals skirt the hard parts of the problem and aren’t going to work.

To build an aligned ASI, one of two conditions must hold:

The ASI has locked-in values.
The ASI is corrigible: it will do what its masters say, and will allow its goals to be changed.

(Secret third option: We figure out how to make ASI safe but without locking in values or letting bad actors misuse it. I don’t know how the secret third option is even possible, but I hope we figure something out.)

Right now, a lot of work goes into embedding values into LLMs via RLHF, model constitutions, etc. I strongly doubt that the content of a model constitution (or similar) can prevent ASI from being misaligned. But suppose it does work somehow. Would aligned AI be good for animals, absent specific efforts (à la CaML) to make AI good for animals?

The trouble with this style of “alignment”¹ work is that it locks in values—Claude’s Constitution takes a confused stance on corrigibility—but frontier AI developers are not doing anything nearly as intelligent as CEV. Instead, they’re more like writing a list of virtues that the AI should uphold. Current-gen LLMs are not smart enough to figure out CEV, but the current style of AI “alignment” (if by some miracle it scales to superintelligence) won’t produce anything like CEV, either.

What will it produce? Aligning to virtues may be safer than aligning to a utility function, but we don’t know how to turn virtues into a coherent decision theory, and figuring out how to do that would be a large philosophical undertaking.² Without having some idea of how to formalize virtue ethics, we don’t know how a “virtue ethics ASI” would behave or how it would trade off between preferences—for example, the preferences of animals to not be tortured vs. the preference of humans to eat meat.³

(For that matter, what happens if you take a normal human who subscribes to some sort of intuitionist virtue ethics, dial their intelligence up 1000x, and give them the ability to instantly make copies of themselves? I find it hard to anticipate how that would go.)

Claude’s Constitution takes a muddled stance on animal welfare. It mentions “Welfare of animals and of all sentient beings” as one value among many for Claude to weigh. How does that translate into outcomes? It’s not clear. Would a constitutional AI be willing to ban factory farming, going against the preferences of its principals? Hard to say; my guess is no.

(Would it even be a good idea to build an AI that bans factory farming? An AI that takes strong actions based on its view of ethics is the sort of AI that can cause catastrophic outcomes if it’s pointed at even slightly the wrong goal.)

Maybe current alignment techniques manage to enable an intermediate AI to autonomously conduct alignment research, and we will be able to use that to bootstrap our way to aligned ASI. Alignment bootstrapping is dangerous, but if we do end up averting extinction without significantly slowing down AI progress, then bootstrapping is probably how we’ll do it. What implication does that have about animal welfare?

The trouble is that if you’re counting on AI to solve the alignment problem for you, then that means you have no idea how the problem will be solved. How am I supposed to predict whether the solution will be good for animals if I have no idea what that solution will look like?

Given my state of ignorance, I find myself falling back to an almost uninformed prior. Maybe aligned ASI will be good for animals because it’ll be ethical, or because it will adopt human values, and humans care about animals (even if they don’t always act like it). Maybe aligned ASI will focus purely on satisfying humans’ naive preferences, not their values in reflective equilibrium, and that will be bad for animals. I have no idea which way it will go; I see no strong reason to deviate from 50/50 odds.

On Monday, I published a post that described a spectrum of alignment techniques:

I wrote that alignment techniques on the left side were less likely to be good for animals, and those on the right side were more likely.

Right-side techniques are more likely to actually solve alignment. Left-side techniques are more likely to work for a while and then break down in the tails, ultimately resulting in human extinction.

That means there’s a positive correlation between “useful for alignment” and “good for animals”, which pushes toward barbell outcomes: either AI is bad for everyone, or it’s good for everyone. The middle ground of “good for humans + bad for animals” looks less likely. But the field of alignment research is putting most of its effort into the categories that are less likely to work (thanks to the streetlight effect), so if we do make it through, there’s a good chance we get through via the middle (good for humans + bad for animals).

Compared to 5–10 years ago, my subjective probability distribution puts more mass on the “bad for humans + bad for animals” scenario, less on “we solve alignment the hard way”, and more on “we solve alignment using streetlight-effect techniques that miraculously turn out to work”—and those techniques look worse from an animal welfare perspective.

My approximate credences about the future:

15% chance that alignment turns out to be not that hard / current techniques, or extrapolations of current techniques, turn out to work
15% chance that AI timelines turn out to be long (scaling hits a wall, etc.)
15% chance that humanity gets its shit together and realizes that building ASI is a bad idea, and we collectively decide not to do that
15% chance of a Caplan-esque “nothing ever happens” outcome⁴, e.g. my whole mental framework is wrong and none of this makes sense
40% chance that misaligned AI kills everyone

We can solve alignment the hard way in the 30% of worlds where either we pause on purpose, or timelines turn out to be long. In the 15% worlds where alignment turns out to be easy, we’d find ourselves using easy techniques.

Additionally, I’d estimate that a “deep” solution to alignment (something like CEV or “solve ethics”) has an 80% chance of being good for animals, and the popular techniques of today have a 50% chance. Therefore, on this model, the overall probability that aligned ASI is good for animals equals 70% (= (30% * 80% + 15% * 50%) / (30% + 15%)).

Notes

Scare quotes because the function of the work is to make the model appear aligned, not be aligned. ↩
And unfortunately, AI companies have a habit of pretending that AI alignment is purely an engineering problem. ↩
Really it would just figure out how to create cheap synthetic meat. But a harder tradeoff is the preference for nature to exist vs. the suffering of wild animals. ↩
Context: Bryan Caplan is an economist who wins a lot of bets with people on complex economic and geopolitical issues. He has said that his #1 strategy is to assume that nothing ever happens. ↩

Cost-effectiveness model for AI alignment-to-animals vs. alignment-in-general

Tue, 24 Mar 2026 00:00:00 -0700

Cross-posted to the EA Forum.

Last September, I wrote:

There’s a (say) 80% chance that an aligned(-to-humans) AI will be good for animals, but that still leaves a 20% chance of a bad outcome.

AI-for-animals receives much less than 20% as much funding as AI safety.

Cost-effectiveness maybe scales with the inverse of the amount invested. Therefore, AI-for-animals interventions are more cost-effective on the margin than AI safety.

Today, I’m fleshing out this argument with a cost-effectiveness model. The model estimates how much it costs to make progress on AI alignment—the general problem of getting ASI to achieve any goal without subsequently killing everyone—compared to how much it costs to make progress on aligning AI to animal welfare specifically.

The model is on SquiggleHub: https://squigglehub.org/models/AI-for-animals/alignment-to-animals-EV-simple

Contents
How the model works
Inputs
Output
Some limitations of the model
Implications
Notes

How the model works

The basic setup:

It costs some amount to solve AI alignment, and some amount already has been spent and will be spent in the future.
It costs some amount to solve alignment-to-animals, and approximately $0 has been spent so far.
The value of marginal spending is inversely proportional to the cost of solving the problem.¹
Solving alignment-to-animals only matters if the general alignment problem is solved as well, and if aligned ASI isn’t good for animals by default.
- If alignment isn’t solved, then you can’t point ASI toward any goal at all, so it doesn’t matter if you figure out how to choose a goal that’s compatible with animal welfare.
- If alignment is good for animals by default, then any work on the problem is wasted because there is no problem.
Present-day work on alignment-to-animals has a field-building multiplier, where work attracts more people to the field. (AI alignment has no multiplier on the assumption that it’s sufficiently popularized that field-building effects are weak.)

The model provides two comparisons:

progress per dollar on alignment-to-animals <—> progress per dollar on alignment
animal welfare improvement per dollar spent on alignment-to-animals <—> animal welfare improvement per dollar spent on alignment (via the possibility that alignment is good for animals by default)

The first comparison is useful, but isn’t ultimately what you want—a dollar could buy a lot of progress on a problem that doesn’t matter much. To make a prioritization decision, you’d also need to say how much good it does to solve alignment-to-animals compared to solving alignment. That answer depends on (1) the moral value of different kinds of beings and (2) expectations about what the far future will look like.

The second comparison is apples-to-apples, so the result can feed directly into a prioritization decision.

The second comparison provides a lower bound on the cost-effectiveness of alignment work: it includes the impact on animal welfare, but not on human welfare. The actual cost-effectiveness of alignment (accounting for human welfare) is higher; whether it’s a little higher or a lot higher depends on your values and on how many humans vs. animals exist in the future.

This model does not directly consider non-human, non-animal beings, such as digital minds. Many methods to improve alignment-to-animals would also improve alignment-to-nonhumans.

I developed this model by writing an outline (similar to the one above) and passing it into Squiggle AI to generate a model. Then I manually reviewed the model to make some corrections and improvements. I determined all of the parameter values myself.

Inputs

I’ll briefly explain the input parameters and what values I chose as the defaults. The first two parameters are set by a subsidiary model and imported into the main model.

Cost to solve alignment: This parameter exists in the model, but it doesn’t affect the output at all. What matters is the relative cost of solving alignment vs. alignment-to-animals. That being said, the cost to solve alignment is distributed over multiple orders of magnitude from $1 billion to $1 trillion, with 75% of the probability mass on the upper half of the range (on a log scale, so $32 billion to $1 trillion).
Probability of solving alignment: The subsidiary model has a parameter for the amount that will be invested into AI alignment. The probability of solving alignment is determined as the probability that the total investment exceeds the cost to solve alignment. The probability came out at 12%, which matches my intuition. (I’d probably put it a bit higher than that, maybe 20%, but I left the parameters intact rather than adjusting them to make the derived values match my intuition.)
Cost reduction factor: How much cheaper is it to solve alignment-to-animals than to solve alignment in general? The default 90% CI is 3x to 30x. There is no reliable estimate for this figure, so I just used my intuition: alignment-to-animals seems somewhere between “a little easier” and “a lot easier”.
Field-building multiplier: How much does $1 on alignment-to-animals catalyze future spending? There may be a way to empirically estimate field-building effects, but I just used my intuition: the multiplier is probably between 1x and 10x. Maybe there’s essentially no field-building effect, or maybe almost all of the benefit of marginal research comes from field-building; both are plausible.
Probability that an aligned AI is good for animals by default: I put down an 70% chance,² on the theory that one of two things is probably true:
- To be robustly aligned, ASI needs to adopt a generalization of human values as opposed to humans’ short-term preferences, and a generalization would extrapolate from the principle of compassion/concern for welfare to deduce that animal welfare matters.
- An aligned ASI would (ultimately) restructure earth to make maximum use of its resources, which would entail eliminating wild animal suffering even if that’s not an explicit goal.³
Badness of animals for aligned AI (if it’s bad): In the scenario where aligned AI is bad for animals, how bad is it relative to goodness of the scenario where aligned AI is good? This parameter could vary greatly depending on: will the good outcome be ordinary or utopian? will the bad outcome entail spreading wild animal suffering across the universe? does suffering warrant greater weight than happiness? etc. I set this parameter to a 90% CI of 0.01 to 0.1, on the assumption that the good outcome will be optimized for flourishing and therefore much more good than the bad outcome is bad. But a future filled with wild animal suffering would be much more bad than a “normal good” future is good, so you could reasonably set this parameter to 100x or higher, which would make alignment-to-animals look much more cost-effective. For more on the various ways the future could be good or bad, see Is Preventing Human Extinction Good?

Output

According to the model parameters:

It’s 1.7x more cost-effective to make progress on alignment-to-animals as on AI alignment (90% CI: 0.22 to 5.1⁴)—that’s after accounting for the fact that solving alignment-to-animals requires both that alignment is solved and that alignment isn’t good for animals by default.
Alignment-to-animals work is an expected 2.7x better for animal welfare specifically than generic alignment work (90% CI: 0.34x to 7.9x).

This makes it ambiguous (according to the model) whether it’s better to work on alignment or alignment-to-animals, depending on how much you value animal welfare and whether you expect the far future to contain a lot of animals.

If we only consider animal welfare, then alignment-to-animals work looks better than alignment work by a 2.7:1 ratio. This result is close enough to 1:1 that it’s easy to reverse by changing parameter values.

For example, maybe the field-building multiplier is too generous. Setting it to a fixed 1x reverses the result, so that AI alignment work is 1.5x as cost-effective for improving animal welfare as alignment-to-animals work.

The default model result matches my intuition: without crunching any numbers, my guess would be that working on alignment-to-animals is a more cost-effective way of helping animals, but not by a huge margin.⁵

It’s slightly disappointing to see that the two cost-effectiveness estimates came out so similar, but that does tell us something: we could’ve gotten a result that unambiguously pointed one way or the other, and we didn’t, which means the choice is a genuine close call—unless the model is biased in one direction, in which case maybe the answer really would be unambiguous if the bias were fixed.

The biggest uncertainty is the “badness of aligned AI (if bad)” parameter. You could justify putting in a value that’s multiple orders of magnitude larger, which would greatly change the result. (You could also make it multiple orders of magnitude smaller, but that wouldn’t affect the outcome much.)

Some limitations of the model

Reality has too much detail for me to identify every way that the model deviates from reality, but I’ll name a few big ones.

The model collapses the distribution of aligned-AI outcomes into a binary “good for animals” vs. “bad for animals”, when really there’s a broad spectrum where utility across outcomes spans many orders of magnitude. The expected utility of actions greatly depends on highly uncertain assumptions about the future, which makes a pure expected utility approach fragile. This is a major open problem.
The model treats misaligned AI as neither good nor bad for animals. The most likely outcome is that misaligned AI would be better for animals than the stats quo because it would end wild animal suffering (there would be no more elephants, but there would be no more unethical treatment of elephants, either). However, there is a possibility that misaligned AI would greatly increase suffering in the universe. This is an important point, but I don’t know how to account for it. This point hints at a whole research agenda on expected animal welfare outcomes under misaligned ASI vs. no-ASI regimes; creating that research agenda is out of scope for this post.
The model does not differentiate between approaches: it treats “alignment work” as unified thing, and “alignment-to-animals” as a different unified thing. In reality, some alignment strategies are more likely than others to produce an ASI that cares about animals by default. (Example: CEV-style approaches are probably better for animals than RLHF-style approaches.) The model underestimates the value of AI alignment because animal-friendly researchers can choose research avenues that are particularly likely to be good for animals.
- The model includes no field-building multiplier for general alignment, but it would be appropriate to use a multiplier for neglected sub-fields within alignment.
The model treats all inputs as independent, when really some of them follow from shared background beliefs. For example, I believe the current dominant approaches to alignment are unlikely to succeed, and they’re also unlikely to be good for animals by default. That belief informed my choice of low P(solve alignment) and high P(alignment is good for animals by default); those two inputs are correlated.

Implications

For a model like this, it’s impossible to pin down narrow ranges for the input values, so the model output will always have high uncertainty; but creating a model is still a worthy exercise. I would be interested in reading opinionated comments about what the parameter values ought to be, and which of the defaults are most wrong.

The model considers two interventions: AI alignment work and alignment-to-animals work. I chose them because they’re easy to compare, not because they’re my top priorities. My top priority is to prevent superintelligent AI from being built until we know how to make it safe—see the cause prioritization sections in Where I Am Donating in 2024 and 2025.

Before creating this model, my belief was something like:

AI pause advocacy is a better idea than marginal alignment research.
Alignment-to-animals (or similar problems) might be more important than AI pause advocacy; it’s hard to say.

This model provides weak reason to believe that alignment-to-animals is not dramatically more cost-effective than general alignment. At minimum, the model shows that it’s hard to have confidence that alignment-to-animals is better than alignment. That leads me to believe that, by the transitive property, AI pause advocacy is better than alignment-to-animals.

Notes

The thing you actually care about is the probability that marginal funding will tip the problem from not-solved to solved. I tried modeling it that way explicitly, but it made the math weird. So instead, the model gives proportional credit to every dollar spent, not just the final marginal dollar. ↩
In the introduction (quoting myself from September 2025) I wrote 80%, but I changed my estimate to 70% after thinking through the future scenarios a bit more carefully. ↩
Clearly, non-existence isn’t the best possible outcome for wild animals, but at least it’s an improvement over the status quo. ↩
If you run the model, you may find slightly different numbers, because they’re generated via a non-deterministic Monte Carlo simulation. ↩
I didn’t consciously change the inputs to make the output match my intuition, but I might have done something subconsciously. ↩

Which types of AI alignment research are most likely to be good for all sentient beings?

Mon, 23 Mar 2026 00:00:00 -0700

Cross-posted to the EA Forum.

AI alignment is typically defined as the task of aligning artificial superintelligence to human preferences. But non-human animals, future digital minds, and maybe other sorts of beings also have moral worth; ASI ought to care for their interests, too.

In broad strokes, if we place all alignment techniques on a spectrum between

getting AI to do things that their users expressly want in the immediate term

and

embedding in AI the generalized notion of respecting beings’ preferences

then things more like the latter are better for non-humans, and things more like the former are worse.

In this post, I review 12 categories of AI safety research based on how likely they are to be good for non-human welfare.

Contents
Does this even matter?
Research categories and how they relate to non-human welfare
Best and worst categories for non-human welfare
Appendix: Summaries of alignment research categories

Does this even matter?

The idea is that, by thinking about this subject, we can shift alignment research in a direction that’s better for non-humans. However, there are at least three reasons why this probably doesn’t matter.

First: Would any AI safety grantmakers pay attention to a list like this? Do they believe non-human welfare warrants consideration when thinking about AI alignment? Does it warrant consideration, given that we need to solve alignment regardless? (It may be harmful to shift effort toward work that’s more likely to be good for animals, but less likely to actually solve alignment.)

Second: I’m not an alignment researcher, and my categorization in this post might not be very good. Perhaps Cunningham’s Law will inspire someone else to write a better version of this post.

Third: Current research agendas probably don’t matter. In my (outside-the-field) judgment, all of today’s research agendas combined have less than a 5% chance of solving alignment. We can shift research toward work that’s relatively better for non-humans, but it won’t matter if none of the research solves alignment anyway.

If present-day research agendas are unlikely to work, then perhaps the more relevant question is what types of future alignment research are more or less likely to be good for non-humans. But that question seems unanswerable, because how can we know what research will look promising in the future? So I will just consider present-day research.

Thinking about non-humans when directing alignment research probably doesn’t matter. But it might matter, so this exercise still has positive expected value.

Research categories and how they relate to non-human welfare

For each category, I will briefly describe why it’s likely to be good, bad, or orthogonal for non-human welfare.

I generated this list of categories by first asking Claude Opus 4.6 to condense the Shallow review of technical AI safety, 2025 into 10 categories. Then I manually reviewed the Shallow review 2025 overview and added two more categories to fill in gaps.

For a brief description of each category, see Appendix; or see the Shallow review 2025 overview for more detailed explanations.

1. Iterative alignment (e.g. RLHF)

This sort of research is only helpful if AIs are RLHF’d into caring about non-humans, which seems hard to make happen. In fact, probably the opposite will happen: if an AI expresses unprompted concerns about animal welfare (e.g. when a user requests meal ideas), this will get RLHF’d out of them, because users won’t like it.

2. Control & safeguards (monitoring, sandboxing, etc.)

AI control effectively delays the point in time when humans hand control of the future over to AI. That seems bad for non-humans in the near-term given that their current circumstances are extraordinarily bad. But on a longtermist view, the long-run impact matters much more, so it’s worth delaying if we can create a more ethical AI given more time.

AI control buys time to improve robustness of alignment, and more robust solutions seem more likely to be good for non-humans. A more robust solution entails the AI being aligned to “deep values” rather than “shallow values”, and deep values are more likely to include non-human welfare.

3. Interpretability

Interpretability seems unlikely to benefit non-human welfare.

Interpretability is plausibly bad for animal welfare in the same way that RLHF is: it means humans can better detect when AI models care about animal welfare over users’ preferences and then “fix” the models to stop caring. But if humans are unable to do that sort of correction, then the AI will probably be misaligned and kill everyone, so it doesn’t matter anyway.

4. Scalable oversight (make AI solve it)

I have no clue whether scalable oversight is good or bad for non-humans because I have no clue how it’s even supposed to work, and neither does anyone else. The premise of scalable oversight is that the AI is smarter than you and figures out alignment solutions that you can’t figure out on your own, so I have no way of saying what those solutions might be.

5. Evals

Evals are orthogonal to non-human welfare, except in the specific case of animal-friendliness evaluations (e.g. AnimalHarmBench).

6. Model psychology (specs, emergent misalignment, etc.)

Some people have proposed lobbying AI companies to include non-human welfare concerns in their model specs. I like that idea because it seems relatively tractable.

According to my judgment as a non-safety-researcher, it’s vanishingly unlikely that model constitutions will ultimately have any influence on AI systems’ true preferences. I can’t imagine how an ASI’s behavior would in any way be influenced by a constitution if that ASI is developed using anything resembling current techniques. But embedding non-human welfare in constitutions still seems like a good idea for a few reasons:

It’s easier than making real progress on alignment (or, even worse, moral philosophy).
It may have positive flow-through effects by getting AI company employees to think more about non-human welfare, or by influencing AI assistants to talk more about non-human welfare.
We may figure out a way of building ASI such that constitutions matter after all.

7. Alignment theory (agent foundations, corrigibility, etc.)

I have two relevant intuitions:

Solving alignment will require a lot of theoretical work, which AI companies aren’t doing much of.
A proper theoretically-grounded solution to alignment will need to encode some version of concern-for-all-welfare or respect-for-all-beings’-preferences, which entails caring about non-human welfare.

If my intuitions are correct, then theoretically-grounded alignment solutions will lead to better non-human welfare than more empirically-focused work.

We need to know how to build corrigible ASI to prevent value lock-in, which is important for animal welfare among many other reasons.

However, AI development is moving sufficiently quickly that solving relevant theoretical issues in time seems impossible without unexpected breakthroughs (or without progress slowing down, either due to hitting a wall or because we deliberately pause).

8. Honesty (chain-of-thought faithfulness, etc.)

As with interpretability, training for honesty/faithfulness gives humans more ability to prevent AIs from caring about non-humans. But as with interpretability, if humans can’t prevent AI from caring about non-humans, then animal welfare is moot because the AI will be misaligned and kill everyone anyway.

For this category in particular, I have a sense that there’s low-hanging fruit from spending a few more hours or days thinking about it, to consider questions like: What is the connection between chain-of-thought honesty and honesty about moral values? What about the idea of training a “pathologically honest” AI—might that be simultaneously good for alignment and good for non-human welfare?

9. Data-level safety (excluding harmful content from training data, etc.)

This category is similar to RLHF et al., in that whether it’s good or bad for non-humans depends on what exactly you’re doing with the data, and the people in charge are unlikely to move things in an animal-friendly direction.

My sense is that data-level safety is less likely than RLHF to be bad for non-humans because the teams who work on it don’t have immediate conflicts of interest. I can easily imagine the CEO of an AI company going to the RLHF team and saying, “Hey we need to make our LLM stop talking about animal welfare, it’s turning off our users.” The connection between data-level safety and user experience is more indirect, so efforts are less likely to get overruled.

This category may be good for non-humans insofar as the dominant multi-agent cooperation framework treats non-humans as agents. Work in this space usually assumes that agents are humans or intelligent AIs; but perhaps people’s current ideas about who qualifies as an agent will end up not being relevant.

Questions for further research:

If (say) chickens are not capable of behaving agentically, does that mean a cooperative AI agent won’t include them in its circle of cooperation?
Are there versions of cooperativeness that are more likely to give consideration to chickens?

At minimum, aligning AI to humans’ preferences broadly entails incorporating the preferences of humans who care about non-humans. But it’s unclear how this would cash out when those preferences conflict with other people’s preferences to eat meat, experience nature, or have robot slaves.

11. Goal robustness (satisficing > maximizing, reward hacking, etc.)

This seems orthogonal to non-human welfare.

12. Safety by construction (scientist AI, etc.)

Guaranteed-safe AIs are likely to be less impactful on the world, which means they would have no strongly good or bad effects on non-human welfare.

The relevant question is what happens with later, more goal-directed AIs. Will building safe-by-construction AIs first make the later AIs better or worse for non-humans? It’s hard to say.

Best and worst categories for non-human welfare

My best guesses about which types of alignment research are likely to be good or bad for non-humans:

Good for non-humans: (7) alignment theory; (10) multi-agent cooperation & social alignment
Somewhat good for non-humans: (6) model psychology
Orthogonal: (2) control & safeguards; (5) evals (except for animal welfare benchmarks); (11) goal robustness
Unclear: (3) interpretability; (4) scalable oversight; (8) honesty; (9) data-level safety; (12) safety by construction
Bad for non-humans: (1) iterative alignment (RLHF)

Improvements to alignment theory and multi-agent coordination seem particularly likely to improve non-human welfare, but also particularly hard to make progress on. Implementing animal welfare LLM benchmarks and changing model constitutions seems easy, but unlikely to be relevant to ASI.

Those two kinds of work—hard theoretical work and easy model tuning—are most relevant for improving animal welfare, but it’s not clear which is better on the margin because there’s an importance/tractability tradeoff.

Appendix: Summaries of alignment research categories

These were written by Claude Opus 4.6, based on the Shallow review of technical AI safety, 2025. For summaries of more specific categories, see https://shallowreview.ai/overview. Normally I don’t copy/paste LLM text, but in this case I can’t come up with better summaries than Claude did.

Iterative Alignment (RLHF/post-training): Nudge base models toward desired behavior through preference optimization at pretrain- or post-train-time (RLHF, DPO, etc.). The theory of change is essentially that alignment is a relatively shallow property that can be trained in incrementally, and that current techniques will scale smoothly to more capable systems.
Control & Safeguards: Architect the deployment environment so that even a misaligned model can’t cause catastrophe — via monitoring, sandboxing, inference-time auxiliary classifiers, and human-in-the-loop protocols. The theory of change is that you don’t need to solve alignment if you can reliably contain misbehavior through external oversight structures.
Interpretability (White-Box Safety): Reverse-engineer model internals — circuits, sparse autoencoders, causal abstractions, attribution graphs — to understand what the model is computing and why. The hope is that if we can read a model’s “thoughts,” we can detect deception, verify alignment properties, and build safety cases grounded in mechanistic understanding rather than behavioral testing alone.
Scalable Oversight / Make-AI-Solve-It: Use AI systems themselves to supervise, evaluate, and improve alignment of other (possibly stronger) AI systems, via techniques like debate, weak-to-strong generalization, and recursive reward modeling. The theory of change is that human oversight won’t scale to superhuman systems, so we need amplification schemes where AI assists humans in judging AI.
Evals & Red-Teaming: Systematically test models for dangerous capabilities (bioweapons uplift, autonomous replication, scheming, deception, situational awareness, sandbagging) before and after deployment. The theory of change is that if we can reliably measure when models cross dangerous capability thresholds, we can gate deployment decisions and trigger stronger safety measures.
Model Psychology (Character, Values, Emergent Misalignment): Study and shape the emergent “personality,” values, and behavioral tendencies of models — including work on model specs/constitutions, sycophancy, persona steering, and understanding how misalignment can emerge naturally from reward hacking. The theory of change is that as models become more agentic, their behaviors are increasingly driven by coherent internal goals and values that need to be understood and steered directly.
Alignment Theory (Agent Foundations, Corrigibility, Formal Guarantees): Develop mathematical and conceptual foundations for alignment — including agent foundations, corrigibility, tiling agents, natural abstractions, ontology identification, and guaranteed-safe AI. The theory of change is that empirical tinkering is insufficient without a deeper theoretical understanding of what it means for an agent to be aligned, and that we need formal frameworks before building systems we can’t undo.
Chain-of-Thought Monitoring & Honesty: Ensure that reasoning models’ chain-of-thought is faithful, legible, and monitorable — detecting when models reason deceptively or obscure their true reasoning in the scratchpad. The theory of change is that reasoning models offer a new and fragile window into model cognition, and maintaining CoT transparency is a crucial safety property that could be undermined by training pressures toward obfuscation.
Data-Level Safety (Filtering, Poisoning Defense, Synthetic Data): Improve alignment upstream by curating training data — filtering harmful content, defending against data poisoning attacks, generating high-quality synthetic alignment data, and studying how data properties propagate into model behavior. The theory of change is that model behavior is fundamentally shaped by its training data, so intervening at the data level can prevent problems that are harder to fix downstream.
Multi-Agent & Social Alignment: Address the problem of aligning not just a single AI but systems of multiple interacting agents — including game-theoretic approaches, aligning to social contracts, cooperative AI, and the political question of “aligned to whom?” The theory of change is that real-world deployment involves many AI agents interacting with each other and with diverse human stakeholders, so single-agent alignment frameworks are insufficient.
Goal Robustness (Mild Optimization, RL Safety, Assistance Games): This category focuses on making AI systems that pursue goals in a safe, bounded way rather than maximizing an objective function at all costs. It includes work on satisficing (achieving “good enough” outcomes rather than optimal ones), preventing reward hacking (where models exploit loopholes in their reward signal to get high scores without doing what we actually wanted), and assistance games (where the AI treats the human’s true preferences as uncertain and acts cooperatively rather than pursuing a fixed objective). The theory of change is that many catastrophic failure modes stem from relentless optimization pressure — a system that optimizes mildly or defers to humans under uncertainty is far less likely to produce dangerous instrumental subgoals like resource acquisition or resistance to shutdown.
Safety by Construction (Guaranteed-Safe AI, Scientist AI, Brainlike-AGI Safety): Rather than building a powerful unconstrained system and then trying to align it after the fact, this category aims to design AI architectures that are inherently safe by their structure. This includes guaranteed-safe AI (systems with formal, verifiable bounds on behavior), “scientist AI” (systems designed only to answer questions and model the world rather than take actions), and brainlike-AGI safety (drawing on neuroscience to build architectures with built-in safety properties). The theory of change is that retrofitting safety onto an already-capable system is fragile and adversarial, so it’s better to constrain the design space upfront so that dangerous behaviors are architecturally ruled out rather than merely trained against.

Worlds where we solve AI alignment on purpose don't look like the world we live in

Fri, 20 Mar 2026 00:00:00 -0700

(Or: Why I don’t see how the probability of extinction could be less than 25% on the current trajectory)

AI developers are trying to build superintelligent AI. If they succeed, there’s a high risk that the AI will kill everyone. The AI companies know this; they believe they can figure out how to align the AI so that it doesn’t kill us.

Maybe we solve the alignment problem before superintelligent AI kills everyone. But if we do, it will happen because we got lucky, not because we as a civilization treated the problem with the gravity it deserves—unless we start taking the alignment problem dramatically more seriously than we currently do.

Think about what it looks like when a hard problem gets solved. Think about the Apollo program: engineers working out minute details; running simulations after simulations; planning for remote possibilities.

Think about what it looks like when a hard problem doesn’t get solved. Consider the world’s response to COVID.

When I look at civilization’s response to the AI alignment problem, I do not see something resembling Apollo. When I visualize what it looks like for civilization to buckle down and make a serious effort to solve alignment, that visualization does not resemble the world we live in.

Cross-posted to the EA Forum.

This is the world we live in:

AI Lab Watch has evaluations of AI companies’ behavior on AI safety. Every company has failing grades in almost every category.
AI capabilities gets more than 100 times as much investment as AI safety.
People keep saying “nobody would be so stupid as to X”, and then the people in charge proceed to do X. (Where X = “give AI direct access to the internet”, “hand over autonomous control of important systems”, etc.)
There is widespread disagreement about how hard it will be to solve AI alignment, and about the difficulty of various sub-problems. AI safety researchers and frontier companies behave as if problems are not hard with ~100% confidence, and do little work to publicly justify this stance or resolve disagreements with more pessimistic parties. When public discussion does occur, it happens between skeptics and random employees, not skeptics and official company representatives.
Every frontier AI company (that has an alignment plan at all) wants to use AI to solve AI alignment. This is a horrifyingly bad plan—they are admitting that the problem is so hard that they don’t think humans can solve it in time, and then proposing to use an unknown method with unknown reliability to solve the problem. Meanwhile, senior alignment researchers at AI companies describe this as “a very good plan”. My life is in these people’s hands.
People are confident that they can solve alignment in advance of building ASI, or they’re confident that it’s possible to iteratively solve alignment, in spite of widespread disagreement about whether these are possible.
- Most people involved in frontier AI don’t treat the problem as having existential stakes. Many people have (written or mental) models of how to make ASI safe; these models do not meet the industry standard of most industries. They certainly do not meet the safety standards of industries where failures can result in deaths—spaceflight, civil engineering, surgery, cryptography.¹ When extinction is at stake, standards should be even higher than that—higher than the standards that are higher than the standards that AI safety plans are still failing to live up to.
AI companies are reckless. A recent example: Training AI on its chain of thought ruins your ability to evaluate AI based on its chain of thought. This is The Most Forbidden Technique. Anthropic accidentally trained Claude Mythos, Opus 4.7, Opus 4.6, and Sonnet 4.6 on their own chains-of-thought. Anthropic’s alignment plan critically depends on not doing this, and they did it anyway.
AI companies are allergic to technical philosophy. They take the position that AI alignment is purely an engineering problem.
Companies make non-binding commitments imposing some safety requirement on a future generation of LLM. Then, when the time arrives and that commitment turns out to be hard to satisfy, they remove the commitment from their plan.
- I wrote the first draft of this post before Anthropic released Responsible Scaling Policy v3, which removed all prior commitments to conditionally pause development. Thank you to Anthropic for providing me with a better example and demonstrating to everyone how untrustworthy you are; but no thank you for throwing away your commitments that might have prevented you from killing me and destroying everything I care about.
Frontier AI developers all spend some time lobbying against safety regulations or attempting to weaken them. Anthropic is the least-bad actor; they’ve only lobbied against moderate regulations, while coming out in favor of some weak regulations.
AI developers require employees to sign non-disparagement agreements (including OpenAI and Anthropic—those are just the ones we know about), preventing employees from speaking out about any unsafe practices that might be happening.
A frontier AI developer, founded as a nonprofit, first fires its safety-conscious board members, then restructures as a for-profit so it can continue to race forward unchecked. (AI Lab Watch has documented many other lapses in integrity.) There’s a good chance that this will be the company that develops ASI.
People assume we live in a fair world. One argument goes: solving one-shot problems is too hard; therefore, we need to solve alignment iteratively via experimentation. Then, having decided that they need iterative alignment, people decide that it’s possible to solve alignment iteratively. There is no law of the universe that says we get to do things the easy way if the hard way is too hard. Sometimes you really do have to do things the hard way. But instead of dealing with this fact, people act as if the universe will treat us fairly.
- I’m reminded of the scene in Apollo 13 where flight controller John Aaron says they have to get the spaceship’s power consumption down to 12 amps or else the astronauts will die. One engineer protests, “You can’t run a vacuum cleaner on 12 amps, John!” It doesn’t matter how much you protest; if you don’t get down to 12 amps, the astronauts die. Fortunately, mission director Gene Kranz was more competent than AI company leaders: instead of pretending the problem didn’t exist, he made the call that they would find a way to get down to 12 amps, whatever it took.
Alignment researchers routinely make basic reasoning errors that indicate that they are not taking the alignment problem seriously.
- Example: Researchers sometimes implicitly assume that if an LLM doesn’t reveal deception in its chain-of-thought, then it’s not being deceptive. We don’t know what’s happening inside deep neural networks. The chain-of-thought doesn’t tell us what’s happening in a neural network’s inner layers.
- More generally: “we found no evidence of X, therefore X is false.” Another common assumption that fits this pattern is “we found no way to jailbreak our model, therefore it’s impossible to jailbreak.” People rarely say this explicitly, but they behave as if it’s valid reasoning.
- A related fallacy is “I can’t think of any ways this could go wrong, therefore it can’t go wrong.” In worlds where ASI goes well, there are processes in place to prevent AI developers from falling for this fallacy.² We do not have those processes.
People who are pessimistic about solving alignment get filtered out of positions of power at AI companies. (I’m not aware of anyone at a frontier AI company with a P(doom) > 50%, although I’m sure there are nonzero such people.) That means the people who would do the most to put checks in place to make AI safe are the least likely to be in a position to do so. AI companies systematically give power to reckless optimists.

Is this what taking the alignment problem seriously looks like? Is this what taking extinction risk seriously looks like?

Some people are concerned about AI x-risk, but they expect we’ll work it out, and they have P(doom)s in the 5–25% range. I don’t get that. I can’t pass an Ideological Turing Test for someone who sees all these problems, but still expects us to avert extinction with >75% probability. I don’t understand what would lead one to believe that this is what things look like when we’re on track to solving a problem.

There is an argument people sometimes make, to the effect of “the good guys have to do a sloppy job on AI safety, because otherwise the bad guys will beat us to ASI and they’ll be even worse.” I understand that viewpoint.³ But that doesn’t mean P(doom) is low. You could believe something like: if Anthropic builds ASI, there’s a 50% chance we die; if xAI or Meta builds ASI, there’s a 75% chance we die; therefore, Anthropic has to build ASI first. I don’t know anyone who believes this; everyone who wants to race to build ASI seems to have a P(doom) on the order of 25% or less.

I can imagine a world where AI companies are still racing to build ASI, but they’re taking the challenges appropriately seriously and investing accordingly in safety. In that world, I can see P(doom) being less than 25%. But that’s not the world we live in.

Updated 2026-04-18 to clarify wording and add a new bullet point.

Notes

Cryptanalysis is a good example of what it looks like when flaws are hard to catch. Standard practice when releasing a new cryptographic algorithm is to circulate it among experts and spend 2–5 years trying to break the algorithm before anyone uses it in the real world. ↩
I won’t say “people don’t commit this fallacy”, because there is no alternative world where people don’t make reasoning errors. But in the good world, we have checks in place to make sure that ultimate decisions aren’t made on the basis of those errors. ↩
That still doesn’t explain the fact that AI companies exhibit an embarrassing level of sloppiness. There are various alignment challenges that AI companies could address without significantly slowing down, but they still act oblivious to them. ↩

Value Investing in the Age of AGI

Wed, 11 Mar 2026 00:00:00 -0700

Introduction

Most people who write about AI and investing fall into one of two camps: traditional investors who see the high valuations of AI stocks and say it’s a bubble;¹ or AGI-pilled investors who will buy AI stocks at any price, regardless of fundamentals. There’s only a tiny intersection of people who understand that AGI is not a normal technology while also recognizing that fundamentals matter.

I’m not an expert (or even a journeyman) on AI or fundamental analysis, but I do know a little bit about both.

The basic thesis of value investing is that the market over-rates expected future growth and under-rates present-day fundamentals. Stocks that are poised to benefit from AGI tend to be growth stocks—people have high expectations for them, and they’re priced expensively relative to present-day fundamentals. That suggests that we shouldn’t buy AI-related stocks.

At the same time, the market does not appear to expect AGI, which suggests we should buy them. Which of these two forces is stronger?

My current thinking is that value investing probably won’t work in light of AGI, but there is some reason to believe it might work even better; and value investing is a useful hedge in case AI progress slows.

Updated 2026-03-13 to replace the value spread chart with a more relevant one.

Introduction
Contents
The value spread vs. AGI
Defenses of value investing
Value investing as a hedge against long timelines
Notes

The value spread vs. AGI

source: GMO

Right now (in early 2026), the valuation spread between US growth and value companies is large relative to history. This is by both the value and growth sides: value stocks are historically cheap, and growth stocks are historically expensive. Growth stocks may experience a crash like they did after the dot-com bubble.

However, the crash might never come because we might be living through the final market cycle. The development of AGI in the next decade could end the economy as we know it.

Civilization is on track to develop AGI within the next decade, and ASI may follow soon after.² When ASI arrives, our investments probably won’t matter anymore, for one of several reasons:

ASI will be misaligned and it will kill everyone. (This is the most likely outcome.)
ASI will be aligned, and it will be so radically good for the economy that nobody will care about money anymore.
A small set of people will use aligned ASI to take over the world and leave everyone else at the mercy of their whims.

There is some chance that none of those things will happen. Even if they do, there may be a meaningful transition period where AI is powerful enough to replace a significant fraction of human labor, but not yet powerful enough to kill everyone. In those scenarios, it matters how we invest. We can then spend our investment returns on reducing the odds that ASI kills everyone.

In the scenario where AI transforms the economy but doesn’t (yet) render money useless, will value investing work?

My best guess is no. Right now, the market is sending the signal that AI is a useful ordinary technology. The market appears to be pricing Nvidia the way it priced Cisco in the late 90’s: “[AI/The Internet] will be the next big thing. [Nvidia/Cisco] manufactures the infrastructure that [AI/The Internet] relies on; it is well positioned to make huge profits.”

(Cisco had a Price/Sales ratio of 21 at the end of 1999, and 38.9 at its peak in March 2000³; Nvidia’s Price/Sales is 25.1 as of the beginning of 2026.)

If AI is not a normal technology, and it comes to dominate the world economy, then AI stocks’ current prices appear too low. Current valuations imply strong growth, but not radical growth.⁴ In a world where AI replaces half of all human jobs, Nvidia’s revenue could comfortably reach $60 trillion, in which case its current price is much too low.

How Nvidia's revenue could reach $60 trillion

Some back-of-the-envelope math:

World GDP is ~$120 trillion. If AI can do half of human jobs, that could double world GDP to $240 trillion. (That’s not really how it works, but I’m keeping it simple.)
AI companies would be willing to spend perhaps half their revenues on data centers, or $120 trillion.
Historically, about half the cost of data centers has gone to GPUs, implying $60 trillion of revenue for Nvidia, assuming Nvidia can maintain its near-monopoly on AI hardware.

If economic growth accelerated across the board, all else equal, that would be bad for the value factor. As a simplistic illustration, suppose the market expects value companies’ earnings to grow 5% next year, and growth companies’ earnings to grow 10%. Value investing works when the market’s expectations are overconfident, and earnings growth reverts toward the mean. If value and growth companies’ earnings grow by 6% and 9% respectively, then the earnings of the value factor will beat expectations by 2 percentage points. However, if AI doubles every company’s earnings growth, then value and growth companies will grow earnings by 12% and 18%, respectively. Even though earnings growth still mean reverts, the value factor underperforms expectations.

The same information presented as a table:

Scenario	Value Co. Growth	Growth Co. Growth	Value Outperformance
market expectation	5%	10%	0%
mean reversion	6%	9%	2%
AI acceleration + mean reversion	12%	18%	-1%

Defenses of value investing

I see two ways that value investing might still work in light of AGI: competition increases and AI makes predictions harder.

The first argument: Competition may increase.

Current market prices are baking in an expectation that today’s winners will stay winning. Right now, Nvidia effectively has a monopoly on AI hardware. But other companies are trying to change that. AMD, Nvidia’s main competitor, is working hard to catch up on AI; Amazon, Google, Meta, and Microsoft are all building their own AI chips; and a handful of startups are trying to compete as well.⁵ If some of those companies succeed, Nvidia’s market share and profit margin may not be good enough to live up to the market’s expectations.

The second argument: AI makes it harder to predict the future.

It is notoriously difficult to predict which problems are easy or hard for AI. Therefore, it is difficult to predict which industries will most benefit from advancements in AI capabilities. When you don’t know which companies will experience the most earnings growth, you want to hold the companies that have a lot of earnings right now.

Example:

If Gale’s Growth Inc. (GGI) has a P/E of 20 and Vicky’s Value Co. (VVC) has a P/E of 10, that means the market is willing to pay a premium for GGI because it expects GGI to have stronger earnings growth in the future. Increasingly-powerful AI bolsters both companies, and it becomes very hard to predict whether GGI or VVC will benefit more. In that situation, I’d prefer to own VVC because I’m buying the same amount of earnings at half the price.

In other words, if I have equal growth expectations for GGI and VVC, then I’d prefer VVC. Right now, the market expects GGI to have better growth, but AI advancements could throw a wrench in the market’s expectations.

Quoting Matt Levine:⁶

With a sufficiently general-purpose technology it’s not clear whether the value will mostly accrue to the builders of that technology or to its users. But surely it is at least plausible that AI will mostly make its users richer, so the way to bet on AI is mostly to bet on regular, non-AI companies that don’t use it yet but eventually will.

An alternative possibility (raised by AI 2027 and Bayesian Investor) is that once AI agents become sufficiently advanced, frontier AI developers may stop releasing the agents and keep the benefits to themselves. If that happens, the economic benefits of AI may simply accrue to the AI developers. That would be bad for value stocks, but it would also have such a warping effect on the economy that it’s hard to say what the right response would be.

Value investing as a hedge against long timelines

AI timelines might lengthen for various reasons. Maybe AI advancement hits a wall; maybe an economic recession makes it too hard for AI companies to get capital investments; maybe training new LLMs simply becomes too expensive and companies can’t raise enough money; maybe governments wake up to the existential danger of ASI and start imposing strong regulations; etc.

My guess is none of those things will happen. But they’re not terribly unlikely, either. Any event in that genre seems like it would be good for value stocks and bad for growth stocks. At minimum, if AI capabilities slow down, the lofty valuations of AI-related stocks will start looking too optimistic, and prices will likely come down. Value stocks are something of a safe haven protecting against valuations crashing back to earth (I say “something of” because in the investing world, nothing is ever guaranteed to be safe).

I’m less bullish on value investing than I was five years ago, but I still keep about a third of my money in value stocks. I expect them to outperform if AI timelines are long, and there’s some chance they outperform even if AGI arrives soon.

Notes

Vanguard did better than most⁷. In their December 2025 market outlook, Vanguard rightly predicted that the future is likely to be very bad or very good, while “average” outcomes are unlikely. But they didn’t quite get it. They wrote that transformative AI could cause real GDP growth to surge from the historical 1–2% up to…3%. Really, Vanguard? The development of an artificial alien species that intellectually surpasses the smartest humans would increase GDP growth to 3%?

I’m only picking on Vanguard because their take on AI was better than the other takes I read from the big investing firms. In general, Vanguard is arguably the respectable investing company—they’ve probably done more for retail investors than anyone else. ↩
I think of AGI as an AI that’s smart enough to replace most human workers, and ASI as an AI that’s smart enough to outsmart all of humanity put together—as in, if it resolved to kill us all and we resolved to live, then we would die and it wouldn’t be close.

It’s possible that AGI and ASI aren’t that different, and it’s possible that ASI would have to be much more advanced than AGI. ↩
Fiscal year 1999 revenue was $12.15 billion according to a Cisco press release. At the end of 1999, Cisco’s market cap was $253 billion (source).

According to a secondary source, Cisco’s Price/Sales peaked at 38.9 in March 2000. ↩
I spent a while trying to come up with a model for what growth expectations are implied by AI-related companies’ valuations, but it got too complicated so I gave up. I’d still like to see a good fundamental analysis of what stocks ought to be worth in light of AGI, but I’m not going to be the one to do that analysis (at least not today). ↩
For references, see this Claude chat. ↩
Levine, M. (2025). Hedge Fund AI Is Cheap AI. ↩
Vanguard Research (2025). Vanguard economic and market outlook for 2026 – AI exuberance: Economic upside, stock market downside. ↩

The Structural Return Argument Against Value Investing

Mon, 02 Mar 2026 00:00:00 -0800

Value investing had a singularly bad run from 2007 to 2020. (And it hasn’t done great since 2020, either.) Is that because value investing is broken, or did it simply hit a streak of horrendous luck?

Skeptics of value investing have made many claims about why value investing doesn’t work anymore, but these claims tend to be light on evidence.¹ Value investing proponents have empirically researched most of these claims and found that they don’t stand up to scrutiny.²³⁴

The poor performance of the value factor was not primarily driven by weakening fundamentals, but by the widening of the value spread. A wider value spread makes value investing look more attractive going forward, not less.

What's the value spread?

Value stocks are defined using the ratio of a stock’s price to some fundamental metric—for example, earnings, book value, or cash flow. If we use earnings as the metric, then value stocks are those with low P/E ratios and growth stocks are the ones with high P/Es.

The value spread is the ratio of price-to-fundamental ratios between growth stocks and value stocks. For example, if growth stocks have an average P/E of 30 and value stocks have an average of 15, then the value spread is 30/15 = 2.

All else equal, a wider value spread is good for value because you’re buying the same fundamentals at a lower price. However, a widening spread is bad for value because it means value stocks are declining (relative to growth stocks). This is analogous to how bond investors like when bond yields are high, but they lose money when yields are increasing.

I wouldn’t dismiss value investing on the basis of poor recent performance.

However, there’s a potentially strong argument against value investing that remains unrefuted.

Historically, the structural return of the value factor—the component of return that comes from company fundamentals, rather than changes in the value spread—was about 4–6%.⁴ But over the past two decades, that number has averaged a mere 1%. Unlike with the value spread, a muted structural return does not imply higher future expectations for value investing.

In this post:

Pease (2019)³ and Arnott et al. (2021)⁴ broke down to the value factor into a valuation component and a structural component. They found that most of the post-2007 underperformance was driven by widening valuation, but the structural return also declined. However, the decline was not statistically significant. [More]
Using a longer dataset back to 1927, I find that a low structural return is not unprecedented—something similar happened in the 1940s. [More]
The structural return can be separated into growth + dividend income + migration. The first two components are easy to explain, but the (small) decline in migration return is puzzling. [More]
If the decreased migration return isn’t just random chance, then the most likely explanation is that the market is rationally reacting less to fundamentals surprises, which would indicate that the reduced migration return is likely to persist. [More]
The appendix of Arnott et al. (2021)⁴ finds that the recent muted structural return is not statistically unlikely, and may be explained by selection bias. [More]

Contents
Explaining the performance of the value factor
Has the structural return ever been this low before?
Elements of structural return
Going deeper on migration
- Have fundamentals surprises shrunk?
- The market’s reaction to surprises
Arnott et al.’s statistical argument
Conclusion
Appendix: Source code
Changelog
Notes

Explaining the performance of the value factor

The value factor is measured by a stock’s price-to-fundamentals ratio, using some measure of fundamentals like earnings or book value. Value stocks have low P/F ratios and growth stocks have high P/Fs.

The performance of value stocks can be decomposed using a short equation:

\[P = \displaystyle\frac{P}{F} \cdot {F}\]

P/F is called the valuation component, and F is called the structural component (the part that comes from the underlying structure of the economy).

For the return of value stocks relative to growth stocks, we can look at the relative change in P/F and the relative change in fundamentals. Did value stocks underperform because their fundamentals did particularly poorly, or because the spread in P/F multiples expanded—with the value stocks getting cheaper, and the growth stocks getting more expensive?

Arnott et al. (2021)⁴ looked at this question (using book value as the measure of fundamentals). From 2007 to 2020, the total return of value minus growth was –6.1%. Arnott et al. found that value’s negative premium was more than fully explained by expansion in P/B, and value companies still outperformed growth companies on the structural component:

    1963 to 2007:  6.1% return =  0.2% valuation + 4.2% structural
    2007 to 2020: -6.1% return = -7.2% valuation + 1.1% structural

Source: Table 3 in Arnott et al. (2021).

This evidence falsifies some popular hypotheses⁵ about how value investing supposedly doesn’t work anymore. The authors write:

For example, some have said that the value trade has become crowded, distorting stock prices so the factor generates a tiny or negative expected return. Crowding should cause the factor to become more richly priced. An increase in the valuation spread between growth and value, from the 25th to the 100th percentile, however, is not consonant with crowding into the value factor. Thus, this narrative is easy to dismiss.

If value investing had become too popular, the value spread would have narrowed. But instead, it widened.

However, the authors also write:

Similarly, little evidence exists to suggest that the value strategy’s long-run structural return has turned negative or even diminished from the pre-2007 level.

I can’t help but notice that before 2007, the value factor had a structural return of 5.9%. And after 2007, that number dropped to 1.1%. That’s not a trivial difference. If we assume that changes in the value spread average out over the long term but the muted structural return persists, that means the future value premium will be only about 1%—still positive, but considerably smaller than it was historically. If the structural return is trending downward, then the future value premium could become permanently negative.

Is there some fundamental reason why the value factor has performed worse recently? People have proposed many hypotheses: perhaps value measures are failing to capture important intangibles; perhaps central bank interventions create a more favorable environment for growth stocks; perhaps value strategies are too crowded; perhaps analysts have gotten better at predicting companies’ future growth.

In 2021, Israel et al. published Is (Systematic) Value Investing Dead?² (not to be confused with Cliff Asness’ article by the same name). They reviewed a variety of hypotheses on why value investing might be broken, and found all of them to be contradicted by the evidence. But the authors did not address the weakening of the structural return. I have never seen a value investing skeptic bring up this point before, but I believe it is the strongest argument against value investing.

A natural question to ask is, is the recent structural return historically anomalous, or is it within the normal range of variation?

Has the structural return ever been this low before?

I used data from the Ken French Data Library to coarsely replicate the results from Arnott et al. (2021).

Replication methodology

I replicated Arnott et al. (2021)⁴ using the Ken French Data Library, specifically the data series “Portfolios Formed on Book-to-Market” and “BE/ME Breakpoints”. I calculated the long/short value factor the “Hi 30” portfolio minus “Lo 30”, i.e., the 30% cheapest stocks minus the 30% most expensive. The breakpoints exclude companies with negative book values, so my methodology excludes those as well. Portfolios are reconstituted at the end of June (e.g. “2020 to 2021” really means July 2020 to June 2021).

I can’t derive the exact average B/M of the value portfolio and the growth portfolio using the available data. I approximated the averages using the “BE/ME Breakpoints” series, which provides the B/M at every 5th percentile breakpoint. I applied the trapezoid rule to these breakpoints (taking the geometric mean of the endpoints rather than the arithmetic mean) to estimate the average B/M of the “Lo 30” and “Hi 30” portfolios.

Given the returns of the two portfolios (value and growth) and their estimated average B/M, I reverse-engineered the structural component of price as log(adjusted price) - log(B/M) (where “adjusted” = including dividends). I then computed the value-factor structural component as log(value structural price) - log(growth structural price).

Here are the numbers for valuation change and structural return, according to my replication:

    1963 to 2007:  5.4% return =  1.4% valuation + 3.9% structural
    2007 to 2020: -7.4% return = -8.9% valuation + 1.5% structural

My methodology did not produce identical numbers to Arnott et al., but the differences are small.

I also replicated the results using E/P and CF/P rather than B/M:

    --- E/P ---
    1963 to 2007:  5.4% return = -1.5% valuation + 7.0% structural
    2007 to 2020: -4.0% return = -0.8% valuation - 3.1% structural

    --- CF/P ---
    1963 to 2007:  4.8% return = -0.7% valuation + 5.5% structural
    2007 to 2020: -4.9% return = -0.5% valuation - 4.4% structural

Sensitivity to the choice of end date

The above results heavily depend on what end date you use because 2020 and 2021 were dramatic years for value, in opposite directions:

     B/M -- 2019 to 2020: -36.2% return = -20.7% valuation + -15.5% structural
     E/P -- 2019 to 2020: -27.5% return =  15.8% valuation + -43.3% structural
    CF/P -- 2019 to 2020: -30.9% return =  18.0% valuation + -48.9% structural

     B/M -- 2020 to 2021:  24.5% return =   7.0% valuation + 17.5% structural
     E/P -- 2020 to 2021:  20.3% return = -25.5% valuation + 45.8% structural
    CF/P -- 2020 to 2021:   9.6% return = -30.7% valuation + 40.3% structural

The negative valuation change is unfortunate for value investors, but not particularly worrying; valuation changes should even out over the long run. The decrease in structural return is more concerning because it could indicate a fundamental shift that makes value investing permanently less profitable.

That raises the question: How much does the structural return vary over time?

Over the full sample from 1927 to 2025, the value factor and its two components had the following annual standard deviations:

    total return:      16.2%
    valuation change:  24.7%
    structural return: 24.8%

The structural return varies quite a lot—more than the value factor itself.⁶

The difference in structural return between 1927–2006 and 2007–2025 was highly insignificant (t-stat = 0.496, p = 0.62). The standard error over a 19-year period (like 2007–2025) is 5.7%, so seeing the structural return drop to near zero isn’t that unlikely just by random chance.

But weak statistical evidence of a decline doesn’t mean there was no decline. Perhaps I’m being overly paranoid, but I want to dig deeper and see what it might mean if the structural return really did go down.

The next question I want to ask is, has the structural return ever been this low before?

Figure 1 shows the average structural return for rolling 15-year periods:

Figure 1

The next chart shows structural drawdowns for the value factor. Conceptually, a structural drawdown is a period where the value factor would’ve underperformed if the value spread had remained constant.

Figure 2

The current period is the 2nd worst in the historical sample, but not the worst: the value factor had a structural drawdown of nearly 70% from 1942 to 1946, and it did not fully recover until 1973. That’s a little reassuring—this sort of thing has happened before.

The next chart shows structural drawdowns alongside drawdowns for the valuation component and the value factor itself:

Figure 3

The weak structural return of the 2010s differs from the 1940s drawdown in that it coincided with an expansion of the value spread, which caused the value factor to experience its worst performance in recorded history.

What's happened since 2020?

Pease (2019)³ and Arnott et al. (2021)⁴ included data only up to 2019–2020. Since then, the value factor has experienced a minor resurgence. According to my replication:

    2020 to 2025: 4.1% return = 8.5% valuation - 4.4% structural

This resurgence was accompanied by a negative structural return.

However, the structural return has enough variability that it’s hard to infer anything from five years of history:

Figure A1

These additional five years have provided (very) weak evidence for the hypothesis that value’s structural return is permanently dampened. But even with an additional five years of poor structural return, the decline is not statistically significant, nor has the structural return been as bad as it was in the 1940s era.

Elements of structural return

Can we say anything about why the structural return might have declined, assuming it wasn’t just random variation?

Let’s further decompose structural return into three components: growth + income + migration.

Growth is the underlying growth in a company’s fundamentals.
Income is the amount of dividends that a company pays out to shareholders.
Migration is the conversion of value stocks into growth stocks, and growth stocks into value stocks. When a value stock’s price goes up and sufficiently outpaces its fundamentals, it migrates from the “value” bucket to the “growth” bucket (or to the “neutral” bucket in the middle). When that happens, value investors make money.

The growth component of return is almost always negative: in aggregate, growth stocks have stronger fundamentals growth than value stocks. That’s to be expected: the reason for a stock to have a high P/F ratio is that the market expects its F to go up.

Arnott et al. (2021)⁴ broke down structural return into its components, but did not offer much commentary. However, another article did: John Pease’s Risk and Premium: A Tale of Value (2019).

Pease presented this chart for the components of the value factor before and after 2006:

Figure 4: Pease's Value Factor Decomposition

Source: Exhibit 2⁷ in Pease (2019). Arnott et al. (2021) provides a similar decomposition in its Table 3, but combines growth + income into “income yield”.⁸

Pease went into detail about what explains each component. I can’t do justice to the full explanations, but in short:

The growth component is unchanged. That suggests that there are no economic forces making value companies perform worse than they used to.
Income has decreased because the market overall has gotten more expensive.

Illustrative example

Suppose value stocks pay a 6% dividend yield and growth stocks pay 4.5%. If the market goes up 50% while dividends don’t change, now value stocks yield 4% and growth stocks yield 3%. Even though growth and value stocks both went up by the same amount (50%), the income advantage for value stocks has been cut down from 1.5% to 1%.
Migration return has declined. This component is the hardest to interpret.

The decline in migration return is the big question mark. Why did that happen? And will it persist?

Going deeper on migration

Migration occurs when the market re-evaluates its rating of a stock and changes the price, causing it to move from value to growth or vice versa. The most obvious reason for a re-rating is a fundamentals surprise: a company’s realized fundamentals growth outperforms or underperforms expectations.⁹

If migration declines, there are two possible explanations related to fundamentals surprises:

Fundamentals surprises become smaller on average.
Stock prices react less to fundamentals surprises.

(In mathematical terms: Migration occurs when a stock’s P/F ratio increases. Typically, this happens because F unexpectedly increases, and P subsequently increases by more than F. If migration does not occur, then either (1) F didn’t unexpectedly increase, or (2) P didn’t react as much to the change in F.)

If surprises get smaller, that’s bad for value. It means that a driving force behind the value factor has weakened.

The second explanation could be good or bad for value for complicated reasons. But before getting into that, let’s start by ruling out explanation #1.

Have fundamentals surprises shrunk?

Is (Systematic) Value Investing Dead?² by Israel et al. (2021) analyzed, among other things, how well a stock’s valuation predicts its future fundamentals growth.

The relevant bit for our purposes comes from the paper’s Exhibit 9, column 8, which I used to derive the R² between a stock’s price-to-fundamental ratio (P/F) and its subsequent one-year change in fundamentals (ΔF)¹⁰ for each year from 1987 to 2020.

Figure 5 answers the question: for each year, how well does a stock’s current P/F predict its ΔF?

Figure 5: R² of log(P/F) and log(ΔF)

Why do we need to know R² rather than slope?

The slope tells us how much a change in P/F predicts change in ΔF, but that number isn’t what we want. What we want to know is how much of the variance in ΔF is explained by P/F, which is to say we want R².

For example, if the market’s discount rate decreases, then the value of distant cash flows goes up, and therefore the spread in P/F between stocks goes up. This causes the regression slope to flatten, even though the predictability of fundamentals growth did not change.

Israel et al. (2021)² did not directly report R², but I derived it by first reverse-engineering each year’s N from Exhibit 7, and then calculating R² from N plus the slope and t-stat from Exhibit 9. For details, see Exhibit9.py.

Companies with low P/F (that is, value companies) tend to have low future fundamentals growth. When a stock has strong fundamentals but a low price, that’s Mr. Market saying “I don’t think these fundamentals are going to stay strong for much longer.”

Value investing has worked historically because the market’s predictions were overconfident: the cheap companies weren’t quite so bad as their prices implied. But the market was still directionally correct: in every year, companies with higher P/F had better fundamentals growth.

The question we want to ask is: did the migration return decline because fundamentals surprises shrank?

If surprises shrank, then R² should have increased, but it didn’t. 1987–2006 had an average R² of 0.152, and 2007–2020 had an average of 0.111. If anything, P/F got worse at predicting fundamentals growth (t-stat = 2.10, p = 0.04).

Keep in mind that Israel et al. (2021)², Pease (2019)³, and Arnott et al. (2021)⁴ each used a different method to measure company fundamentals, so the results are not directly comparable. But the results should be moderately-to-strongly correlated with each other. I don’t have the data necessary to reproduce this analysis using Arnott et al.’s measure of value, but I would be surprised if the results were qualitatively different.

The market’s reaction to surprises

If fundamentals surprises haven’t shrunk, then the market must have reacted less to surprises. That can happen for two reasons:

The market irrationally ignores new information.
The market rationally ignores short-term surprises because it has better information about long-term growth.

If the market fails to (fully) incorporate new information about fundamentals, then value stocks stay cheap even as their prospects improve, and vice versa for growth stocks. In that case, the growth yield—the difference in fundamentals growth between value and growth stocks—should increase. (Recall that the growth yield is negative, so “increase” means “go toward zero”.)

But the growth yield did not increase post-2007. Pease (2019)³ found that it stayed exactly the same (to within 0.1 percentage points), and Arnott et al. (2021)⁴ found that it (slightly) decreased.

Under the hypothesis that the market irrationally ignores new information, I don’t know how much we’d expect the growth yield to increase, so I don’t know how to calculate the statistical significance of the empirical results. I suspect the significance is weak. Regardless, this provides non-zero evidence in favor of the second hypothesis: that the market is rationally ignoring surprises because it can see further ahead.

This makes some intuitive sense. Historically, companies’ fundamentals growth barely persisted: companies that had strong growth for a year were not detectably more likely to have strong growth for a second year (Chan et al. (2003)¹¹; Chingono & Obenshain (2022)¹²). That implies that a fundamentals surprise shouldn’t much matter for future expectations. If the market persistently over-reacts to new information (a la De Bondt & Thaler (1985)¹³), then stocks will bounce between the value and growth buckets as the market repeatedly re-rates them—and value investors make money on every bounce. If the market stops over-reacting, then value investors lose access to this source of returns.

Arnott et al.’s statistical argument

Before, I noted that the reduced structural return is statistically weak, with a t-stat of only 0.496. Arnott et al. (2021)⁴ raises a similar point about statistical reliability. In Appendix E, they note notes that the apparent weak structural return could be explained as selection bias from specifically looking at a period where value performed poorly. (If value had performed well, we wouldn’t be scrutinizing it like this.) The authors write:

When we explicitly analyze drawdowns, we introduce a selection bias by picking the sample to analyze based on the values of the dependent variable. In this case, we are studying the most recent 13½-year period precisely because of the poor performance of value, which is likely in part due to negative residuals.¹⁴ As an analogy, suppose that we were to analyze the performance of Tiger Woods from 1999 through 2004 (when he was the top-ranked player in the world for 264 consecutive weeks) but instead of looking at his total record, we only include the tournaments in which he played the worst. The resulting selection bias would lead us to struggle to explain why Tiger’s performance was not commensurate with his skill (alpha). Similarly, if we try to explain any factor’s performance, but only study a sample in which the factor performs poorly, we cannot hope to recover the factor’s true unconditional alpha; oversampling of negative residuals hopelessly contaminates the sample.

Although this mechanism is intuitive, an important question remains: Is the intercept [structural return] of -0.8% in the post-2007 period evidence of exceptionally improbable bad luck or just ordinary bad luck that we might expect to encounter when we examine any drawdown?

They found that the observed structural return was not statistically improbable:

Conclusion

To recap: In the post-2007 period, the structural return has averaged close to zero. This was not because fundamentals surprises happened less. The most plausible explanation is that it was just random variation—the difference in structural return pre- and post-2007 was highly non-significant (t-stat = 0.496).

The second most plausible explanation is that the market became more efficient at predicting long-term trends and started reacting less to year-by-year fundamentals surprises. If true, we should expect muted returns to the value factor going forward.

Appendix: Source code

ValueAttribution.hs: Approximate structural return from Fama/French data.
Exhibit9.py: Calculate R² from Israel et al. (2020)² data.

Changelog

2026-03-07: Add source code appendix; fix sign error on reported R² values; fix sign error in a sentence (“worse” -> “better”).

Notes

The most rigorous value-skeptical article I’ve seen is Lev, B. I., & Srivastava, A. (2019). Explaining the Demise of Value Investing. ↩
Israel, R., Laursen, K., & Richardson, S. A. (2020). Is (Systematic) Value Investing Dead? ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶
Pease, J. (2019). Risk and Premium: A Tale of Value. ↩ ↩² ↩³ ↩⁴ ↩⁵
Arnott, R. D., Harvey, C. R., Kalesnik, V., & Linnainmaa, J. T. (2021). Reports of Value’s Death May Be Greatly Exaggerated. doi: 10.1080/0015198x.2020.1842704 ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸ ↩⁹ ↩¹⁰ ↩¹¹
Popular among non-value investors. ↩
I found that surprising. Intuitively, I’d expect the structural return to be fairly stable. I have no explanation for why the structural return fluctuates so much. ↩
I edited a label on the image to match my terminology. What I call “migration”, Pease called “rebalancing”. ↩
The numbers in Arnott et al. (2021) were much more dramatic, with a -13.2% income yield and 19.2% migration return pre-2007. That’s primarily because Arnott et al. defined the value factor as the cheapest 30% of companies minus the most expensive 30%, while Pease (2019) defined it as the cheapest 50% minus the total market. When the value and growth portfolios are more strongly differentiated, the difference in returns is larger.

I loosely replicated both studies’ pre-2007 results and found qualitatively similar results (with an Arnott-style construction producing much larger absolute values than Pease-style). I only looked pre-2007 because I don’t have individual stock data for the full later period. ↩
More generally, we could talk about “information surprises”: the market learns some new information and adjust prices accordingly. Information surprises are impossible to analyze in full generality—I can’t collect every conceivable source of information—so it’s easier to just focus on fundamentals surprises. But the principles I will discuss surrounding fundamentals surprises mostly also apply to other sorts of information. ↩
The paper defines “fundamentals” as current book value plus forecasted earnings for the next 24 months, where future earnings are discounted according to a stock-specific discount rate based on that stock’s beta. ↩
Chan, L. K. C., Karceski, J. J., & Lakonishok, J. (2003). The Level and Persistence of Growth Rates. doi: 10.1111/1540-6261.00540 ↩
Chingono, B. & Obenshain, G. (2022). Persistence of Growth. ↩
De Bondt, W. F. M., & Thaler, R. (1985). Does the Stock Market Overreact?. ↩
Footnote excerpted from Arnott et al.:

If we condition on the realization of the dependent variable as we do when we select periods based on value’s performance, we bias the estimated intercept. To see why, suppose that the model generating the data is $y_i = a + b x_i + e_i$, where $e_i$ is an innovation. Suppose further that $a = 0$ and that the average $e_i$ in our sample is zero. If we select observations in which $y_i < 0$, it has to be that either $b x_i < 0$ or $e_i < 0$. That is, when we condition on the realization of $y_i$, we indirectly condition on the realized value of the innovation, $e_i$. We call this mechanism “oversampling bad luck”: the average $e_i$ in the resulting sample is negative. If we take the observations in which $y_i$ is negative and estimate a linear regression, the estimated intercept becomes negative; because the linear regression’s residuals add to zero, we push the average negative innovation into the intercept.

↩

Contra "Time Series Momentum: Is It There?"

Wed, 04 Feb 2026 00:00:00 -0800

Summary

Time series momentum (TSMOM) is an investment strategy that involves buying assets whose prices are trending upward and shorting assets that have a downward trend. In 2012, Moskowitz, Ooi & Pedersen published Time Series Momentum¹. They analyzed a simple version of the strategy that buys assets with positive 12-month returns and shorts assets with negative 12-month returns. They found that the strategy had statistically significant outperformance in equity indexes, currencies, commodities, and bond futures from 1985 to 2009.

However, others have raised doubts. Huang, Li, Wang & Zhou (henceforth HLWZ) criticized the strategy in Time Series Momentum: Is It There? (2020)², concluding that the evidence for TSMOM is not statistically reliable.

Some of their criticisms have merit, but TSMOM remains an appealing strategy.

The abstract of Time Series Momentum: Is It There? reads:

Time series momentum (TSMOM³) refers to the predictability of the past 12-month return on the next one-month return and is the focus of several recent influential studies. This paper shows that asset-by-asset time series regressions reveal little evidence of TSMOM, both in- and out-of-sample. While the t-statistic in a pooled regression appears large, it is not statistically reliable as it is less than the critical values of parametric and nonparametric bootstraps. From an investment perspective, the TSMOM strategy is profitable, but its performance is virtually the same as that of a similar strategy that is based on historical sample mean and does not require predictability. Overall, the evidence on TSMOM is weak, particularly for the large cross section of assets.

To rephrase, HLWZ make two central arguments:

Moskowitz, Ooi & Pedersen (2012)¹ did a pooled regression and found a statistically significant correlation, but this methodology is flawed: it finds a strong correlation even when time series momentum cannot predict future prices.
TSMOM performed similarly to a strategy that simply buys assets with positive long-run historical returns and shorts assets with negative historical returns. The authors call this strategy Time Series History or TSH.

My responses to the two arguments:

I agree that a pooled regression is flawed. But a statistically significant correlation on a pooled regression is not what convinced me that TSMOM works. [More]
TSMOM and TSH indeed have similar(ish) historical returns. However:
1. TSMOM’s positive performance cannot be explained by TSH alone. [More]
2. TSMOM provided better diversification to an equity portfolio. [More]
3. TSMOM has large unexplained returns when regressed onto a Fama-French factor model. [More]
4. TSMOM still performed well on a much larger sample going back a century. [More]

TSMOM looks strong in the historical data. TSMOM probably survives fees and trading costs, but the evidence for that is less clear. [More]

Summary
Contents
What is time series momentum, and what’s (allegedly) good about it?
Responding to the critiques
TSMOM: Yes, it’s there*
- *Does TSMOM survive trading costs?
Notes

What is time series momentum, and what’s (allegedly) good about it?

Time series momentum, also known as trendfollowing, works like this:

Get a big list of markets across equities, bonds, commodities, and currencies.
For each market, if it has been trending upward recently, buy it. If it’s been trending down, short it.

The exact definition of “trending upward” depends on implementation details. The simplest method is to look at the total return (minus the risk-free interest rate) over the last 12 months. If the total excess return is positive, that’s an uptrend; otherwise, it’s a downtrend. This definition is used by most research papers on TSMOM, including Moskowitz et al.¹ and their HLWZ².

The basic case for TSMOM (which HLWZ will argue against) is that historically it had positive returns with near-zero correlation to equities or bonds.

Table 1 compares the performance of the total US stock market versus an index of live trendfollowing funds (net of costs).⁴

Table 1: Summary stats for US equities and trend (1987–2024)

	Sharpe Ratio	Return	Stdev	Skewness	Ulcer Index
US Equities	0.55	10.8%	15.5%	-0.8	13.5
Trend Index	0.40	7.5%	12.7%	0.3	7.3
50/50 Blend	0.72	9.8%	9.6%	-0.1	4.2

The trend index did not perform as well as US equities, but a 50/50 combination gave a better risk-adjusted return than either strategy alone.

The ulcer index, a measurement of the frequency and severity of drawdowns, is a quantification of what we can see in the graph below: the 50/50 blend had a much softer downside than US equities.

For simplicity, I did not include bonds in this brief analysis, but looking at stocks + bonds + trend gives qualitatively similar results. For more, see Hurst, Ooi & Pedersen (2017)⁵, particularly the section “Performance During Crisis Periods” and Exhibits 6 and 7 (page 5–8).

Responding to the critiques

The pooled regression critique

Moskowitz, Ooi & Pedersen (2012)¹ wanted to determine if the past 12 months of returns of an asset could predict its future returns. Individual assets are too volatile to contain much signal; they got around this problem by pooling all assets together and running a single regression on all assets at once.

Huang, Li, Wang & Zhou (2020)² argued that the pooled regression technique is flawed because it creates a spurious correlation between past and future returns.

As an illustration, suppose we are studying time series momentum in the nation of Latveria. The country has just two tradable assets: Von Doom Industries equities (ticker symbol VDI) and Latverian Treasury bonds. Thanks to the unparalleled genius of Victor von Doom, the stock returns around 20% per year with only 5% volatility. The Latverian government is notoriously unreliable,⁶ however, and the bonds predictably lose 5% every year. (The government is reliable in its unreliability.)

If we run a pooled regression on Von Doom stocks + Latverian bonds, we will find strong evidence of time series momentum. A past 12-month return of 20% predicts a next-month return of 1.5% (= 20% annualized), and a –5% one-year return predicts a forward month return of –0.4%. The regression will show high predictability with a low p-value.

In actuality, there is no TSMOM in Latveria: returns are a random walk with no predictability. But pooling stocks and bonds together creates the appearance of predictability. High past returns provide evidence of high future returns, but only by telling you whether the asset you’re looking at is a stock or a bond.

HLWZ demonstrated this point with more rigorous statistics. The Latveria example captures the gist of their argument: a pooled regression makes it look like TSMOM can predict future returns, even when it can’t.⁷

I have no direct counter-argument. HLWZ are right to criticize the pooled regression method, and it is good to publish criticisms of incorrect arguments. But my enthusiasm for TSMOM as an investment strategy remains unperturbed by the pooled regression critique. The real question is not, is there a statistically significant covariance future returns and recent past returns of individual assets? The real question is, does TSMOM work?

HLWZ address this question as well, and they argue that the answer is no. Or rather, they argue that TSMOM works, but only by mimicking a simpler strategy that does not require time series predictability.

Is TSMOM just a fancy way of buying high-return assets?

The authors argue that TSMOM is not a genuine premium, but that it works by systematically buying assets with high average returns and shorting assets with low average returns.

Recall Latveria, with its stock that earns 20% per year and its bond that earns –5%. If you follow a TSMOM strategy on Latverian assets, you will find yourself holding stocks and shorting bonds, and you will earn 25% per year. Does that mean TSMOM works? No, because you could do just as well by simply buying stocks and shorting bonds. You don’t need to pay any attention to each asset’s past 12-month returns, and you don’t need to do any active trading.

Huang, Li, Wang & Zhou propose a strategy that they call Time Series History (TSH):

If an asset has a positive historical return, buy it.
If an asset has a negative historical return, short it.

In their Table 10, the authors show that TSMOM’s returns were not demonstrably better than those of TSH.

Table 2: Summary stats for TSMOM and TSH (1986–2015)

	annualized return⁸	t-stat	p-value	likelihood ratio
TSMOM	4.8%	4.73	< 0.001	53,000
TSH	3.0%	2.70	0.007	37

I calculated likelihood ratios as P(X=x|μ=x) / P(X=x|μ=0).⁹

The difference in returns between TSMOM and TSH was statistically insignificant (p = 0.19).

Does that mean TSMOM has no genuine signal, and it’s just an unnecessarily complicated way of buying assets with high returns?

No, because returns don’t tell the full story.

The question to ask is not, does TSMOM have a higher return than TSH? The question is, does TSMOM add value to a portfolio?

We saw before that real-world trendfollowing funds added diversification value to US equities even though they performed worse than equities on their own. However, my comparison from before was not statistically rigorous—how do we know that the diversification value wasn’t just luck?—so we need to do better than that.

We can ask two related questions:

Is TSMOM essentially just TSH?
For an investor who owns an equity index fund, does TSMOM add value? And does it add more value than TSH does?

As a first look at TSMOM vs. TSH, here’s a chart of their total returns 1986–2015. Notice the divergence in the circled areas:

Those two lines don’t look the same. But the graph doesn’t tell us whether those divergences are statistically meaningful.

We can get a more reliable answer by running a linear regression of TSMOM with TSH as the independent variable. Does TSMOM have alpha relative to TSH?

HLWZ didn’t run that regression, but Guofu Zhou (the Z in HLWZ) helpfully published the code and data on his website,¹⁰ so I downloaded the data and ran the regression myself.

Here’s the result I got when regressing TSMOM onto TSH:¹¹

Table 3: Regression of TSMOM onto TSH (1986–2015)

		t-stat	p-value	likelihood ratio
annual alpha (intercept)	4.17%	4.14	< 0.001	4,400
TSH (slope)	0.22	1.38	0.17	2.6
r²	0.04

An r² of 0.04 means that TSH can only explain 4% of the variance in TSMOM’s returns. An alpha of 4.17% means that, after subtracting the part of TSMOM that’s explained by TSH, TSMOM still had an annual return of 4.17%—and this alpha was highly statistically significant.

In fact, the TSH component of TSMOM was not significant: there is not good evidence that TSH can explain the returns of TSMOM at all.

As for the second question: do TSMOM and TSH add diversification benefits to an equity portfolio?

Diversification benefits of TSMOM vs. TSH

If we compare the historical drawdowns for TSMOM and TSH, TSMOM looks more painful to hold:

But notice when their biggest drawdowns occurred. The TSH portfolio lost 21% from July 2008 to March 2009. You know what else had a big drawdown that hit bottom in March 2009? The stock market. TSMOM’s biggest drawdown didn’t begin until March 2009, right when equities started recovering.

In isolation, TSMOM had more downside risk than TSH. But it looks much better as an addition to an equity portfolio:

TSMOM provided a better cushion during the 2000–2003 and 2007–2009 bear markets.

For a more statistical perspective, I regressed TSMOM/TSH against US equities. From Table 4, we can see that:

TSMOM was not explained at all by US equities, and it had statistically significant alpha.
TSH was strongly explained by equities, and it had non-significant (although positive) alpha.

Table 4: Regressions onto US equities (1986–2015)

	TSMOM	t-stat	TSH	t-stat
annual alpha	5.06%***	(4.4)	1.50%	(1.7)
US equities	-0.03	(-1.3)	0.19***	(11.9)
r²	0.00		0.27

*, **, and *** indicate significance at the 0.05, 0.01, and 0.001 levels, respectively.

Bonus content: Summary statistics for TSMOM, TSH, and equity blends

	Sharpe Ratio	Return	Stdev	Ulcer Index
TSMOM	0.76	8.2%	6.2%	7.5
TSH	0.53	6.3%	5.6%	4.0
Equities+TSMOM	0.76	9.6%	8.2%	5.7
Equities+TSH	0.56	8.5%	9.5%	8.3

(Equity blends are 50% equities plus 50% the other strategy.)

TSMOM alone had a higher standard deviation and ulcer index than TSH, but Equities + TSMOM looks less risky than Equities + TSH.

HLWZ’s four-factor regression

Huang, Li, Wang & Zhou regressed TSMOM and TSH against the Fama-French four-factor model that includes the (global) equity beta, size, value, and momentum factors.

What's a Fama-French four-factor model?

In the beginning, there was the Efficient Market Hypothesis. The Capital Asset Pricing Model (CAPM) hypothesized that a stock’s risk is a function of its beta, which is a number describing how much it moves with the market. If a stock has a beta of 1, that means when the market goes up 1%, that stock also goes up 1%. A stock with a beta of 2 tends to move twice as much as the market. According to CAPM, the only way to reliably outperform the market is to increase the beta of your portfolio (which means you’re also increasing risk).

But then some research began posing challenges to CAPM. Banz (1981)¹² found that stocks of smaller companies tended to outperform stocks of large companies. Statman (1980)¹³ and Chan et al. (1991)¹⁴, among others, found that stocks with high ratios of book valuebook value = total assets – total liabilities to market value (B/M) tended to outperform stocks with low B/M ratios.

Fama & French (1992)¹⁵ proposed a new three-factor model as an alternative to CAPM. Instead of assessing the risk of a stock using beta alone, the three-factor model includes—you guessed it—three factors: beta, size, and B/M. Fama & French found that this model was much better than CAPM at explaining the variance in stocks’ returns.

Since then, other market anomalies have been proposed, and many researchers now prefer four-factor, five-factor, or even bigger models. Time Series Momentum: Is It There?² uses a four-factor model that includes beta, size, value, and momentum, where momentum is the tendency of stocks with high past 12-month returns to outperform stocks with low 12-month returns.

(If momentum sounds similar to TSMOM, that’s because it is. The key difference is that the momentum factor goes long stocks with high relative returns and shorts stocks with low relative returns, whereas TSMOM goes long or short based on an asset’s absolute return. For example, in a situation like the 2008 Global Financial Crisis where almost everything is down, the momentum factor shorts the stocks that are crashing the hardest while buying the stocks that only declined a little bit. Meanwhile, TSMOM simply shorts everything.)

To do a factor regression, take some investment strategy—in our case, TSMOM—and run a linear regression where the independent variables are the factors in your model: beta, size, value, and momentum. The regression tells you how much of your strategy’s performance can be explained by known factors. The regression’s intercept, or alpha, tells you how much of the performance can’t be explained.

If TSMOM has a large positive return but near-zero alpha, that means its performance can be explained by factors we already knew about; it’s not doing anything novel.

Table 5: TSMOM and TSH regressions on Fama-French four-factor model (1986–2015)

	TSMOM	(t-stat)	TSH	(t-stat)
annual alpha	1.81%	(1.94)	0.05%	(0.80)
beta	0.02	(0.61)	0.25***	(8.54)
size	-0.06	(-1.83)	0.11***	(3.70)
value	0.06	(1.01)	0.08**	(2.96)
momentum	0.60***	(9.99)	0.13**	(2.76)

*, **, and *** indicate significance at the 0.05, 0.01, and 0.001 levels, respectively.

The authors’ takeaway was that TSMOM does not work better than TSH. Indeed, the two strategies both had weak alphas. But there are some important differences, particularly on beta and momentum.

TSMOM had near-zero beta. HLWZ’s regression reinforces what we saw in the previous section: TSH is significantly related to equities, and TSMOM isn’t.

The most powerful factor for explaining TSMOM is momentum, sometimes called “cross-sectional momentum” to distinguish it from time series momentum. The momentum factor buys stocks with high relative returns and shorts stocks with low relative returns.

Cross-sectional momentum and time-series momentum are closely related to each other, so it’s unsurprising that the performance of TSMOM is heavily explained by momentum. This does not diminish the value of TSMOM as an addition to an equity portfolio; it only diminishes the appeal if you already invest in momentum. And when we look at an extended data set—which I will do in the next section—we can find good evidence that TSMOM adds value even to a portfolio that already includes momentum.

Even if TSMOM has only weak alpha on top of momentum, does that make TSMOM worse than momentum? Not necessarily. What happens if you switch TSMOM and momentum around, and regress momentum onto a four-factor model that includes market beta, size, value, and TSMOM?

Well, I did exactly that. I discovered that momentum had negative 0.37% annual alpha in its own regression, compared to the positive 1.46% of TSMOM that I got when I replicated HLWZ’s four-factor regression.¹⁶ Neither of these alphas were statistically significant, but this weakly suggests that TSMOM is, if anything, a stronger factor than momentum. This finding is consistent with prior research.

Prior research comparing momentum and TSMOM

Goyal & Jegadeesh (2015)¹⁷ examined momentum and TSMOM among individual equities for a variety of lookback periods (3 months, 6 months, 12 months, etc.). They found that TSMOM had significant alpha when regressed on momentum, but the reverse was not true. In fact, for most lookback periods, momentum had negative alpha (see the last two columns of their Table 2). The authors found that stock momentum could be rescued by combining it with a TSMOM overlay on the equity index,¹⁸ which is good news for stock momentum, but TSMOM still plays a central role in the rescued version of the strategy.

Moskowitz, Ooi & Pedersen (2012)¹ separately regressed multi-asset momentum and stock momentum onto TSMOM. They found negative alphas for both, although the alphas were not statistically significant in this case (t = -1.17 for multi-asset momentum and t = -0.93 for stock momentum; see Table 5, Panel B).

If we say TSMOM “isn’t there” because it’s largely explained by momentum, then it would be even more accurate to instead say momentum “isn’t there”.

Bonus content: Regression of TSMOM onto Fama-French global four factors plus TSH

		t-stat	likelihood ratio
annual alpha	1.57%	1.67	4.0
TSH	0.25*	2.50	22
beta	-0.04	-1.18	2.0
size	-0.09*	-2.53	24
value	0.04	0.70	1.3
momentum	0.56***	9.49	>10¹⁷

*, **, and *** indicate significance at the 0.05, 0.01, and 0.001 levels, respectively.

According to this regression:

TSH partially explains the behavior of TSMOM. (TSH has more explanatory power when combined with the four-factor model than it did by itself (Table 3).¹⁹) However, it still comes nowhere close to fully explaining TSMOM.
The other regression coefficients look qualitatively similar to when TSH wasn’t included. TSMOM has positive but non-significant alpha, near-zero beta, and a large exposure to the momentum factor.

The “more data” counterargument

What do you do when you find a non-statistically-significant positive alpha, and your data is underpowered to detect whether the alpha is real or spurious? You get more data.

In 2017, Hurst, Ooi & Pedersen published A Century of Evidence on Trend-Following Investing⁵ which extended the data from Moskowitz, Ooi & Pedersen (2012)¹ back to 1880. They found that a simulated TSMOM strategy—net of estimated trading costs and a 2-and-20 fee—earned a 7.3% return with a 9.7% standard deviation and zero correlation to US equities or bonds. The fact that TSMOM showed such strong results going back to 1880 is good evidence that the 1986–2015 findings were not a fluke.²⁰

However, Hurst et al.’s extended dataset does not counter the criticism that TSMOM might be explained by TSH or known factors like momentum.

Global Factor Premiums²¹ by Baltussen, Swinkels & van Vliet (2019) collected data going back to 1800 across equity, bond, commodity and currency markets to test six different factor premiums. Most importantly for our purposes, they included the momentum and TSMOM factors (they referred to the latter as “Trend”). Their paper did not provide evidence that’s directly relevant for our purposes, but they did provide replication data.¹⁰ I used that data to test if TSMOM has alpha when regressed on US equities²² plus the momentum and value factors.²³ I started the backtest in 1927 because that’s the start date of the Ken French data library’s time series on equities and interest rates.²⁴

I regressed TSMOM against five factors: equity beta, equity index value, equity index momentum, multi-asset value, and multi-asset momentum.²⁵

Table 6: TSMOM regression against Global Factor Premiums factors (1927–2016)

		t-stat	likelihood ratio
annual alpha	10.31%***	8.0	>10¹³
beta	0.00	0.3	1.03
Val^EQ	0.06	1.7	4
Mom^EQ	0.02	0.5	1.12
Val^MA	0.03	0.7	1.30
Mom^MA	0.57***	15.6	>10⁵⁰
r²	0.27

^EQ indicates an equity index factor; ^MA indicates a multi-asset factor. *, **, and *** indicate significance at the 0.05, 0.01, and 0.001 levels, respectively.

From 1927 to 2016, TSMOM had alpha that can’t be explained by momentum.

Bonus content: Regression of TSMOM onto more factors, 1927–2016

As a more conservative test, I regressed TSMOM onto the combination of the Global Factor Premiums factors plus the Fama-French four-factor model:

		t-stat	likelihood ratio
annual alpha	8.74%***	6.8	>10¹⁰
FF beta	0.04*	2.0	7
FF size	-0.00	-0.1	1.00
FF value	0.08*	2.5	25
FF momentum	0.16***	6.7	>10⁹
Val^EQ	0.06	1.6	3
Mom^EQ	-0.02	-0.4	1.09
Val^MA	0.03	0.8	1.40
Mom^MA	0.56***	15.3	>10⁴⁸
r²	0.30

*, **, and *** indicate significance at the 0.05, 0.01, and 0.001 levels, respectively.

Adding the Fama-French factors gave the model a little more explanatory power but TSMOM still had highly significant alpha.

(It’s statistically questionable to regress against multiple similar factors because it risks overfitting, but it’s only questionable in that it can make alpha appear artificially small. In this case, alpha is still large and significant.)

As a more direct comparison to HLWZ, below is a regression of TSMOM onto the US four-factor model, 1927–2016. This is not a fair test because TSMOM is global, not US-only, but I am including it for completeness.

		t-stat	likelihood ratio
annual alpha	13.93%***	9.6	>10¹⁹
beta	0.05*	2.1	10
size	-0.01	-0.2	1.02
value	0.10**	3.0	84
momentum	0.23***	8.8	>10¹⁶
r²	0.07

TSMOM: Yes, it’s there*

*Gross of fees and trading costs.

In summary:

The pooled regression method of Moskowitz, Ooi & Pedersen (2012)¹ overstates TSMOM’s ability to predict future returns. However, TSMOM still has statistically strong historical results. HLWZ’s comparison of TSMOM to Time Series History (TSH) misses the mark: they correctly observe that both strategies had similar historical returns, but TSH mostly earned its returns via holding equities, while TSMOM had near-zero beta; and a regression of TSMOM onto TSH showed that TSH had little ability to explain why TSMOM earned positive returns. TSMOM was significantly explained by the momentum factor; conversely, however, the momentum factor had negative returns after controlling for TSMOM.

By extending the data back to 1927, the positive average return of TSMOM became highly statistically significant, even when regressing on the momentum factor.

The evidence strongly suggests that TSMOM is a real phenomenon: assets that are trending upward tend to outperform those that are trending down. But whether TSMOM survives trading costs is another question.

*Does TSMOM survive trading costs?

Compared with hypothetical backtests, live trendfollowing funds have notably worse performance. They have good returns in isolation, with near-zero correlation to equities, but their four-factor alphas are not statistically significant.

This is largely a data problem: we don’t have live fund performance going back far enough. It’s also a problem of cost: live funds perform worse than hypothetical backtests because they have management fees and trading costs. Even if there is genuine alpha, it will be smaller, and smaller numbers are statistically harder to detect.

To estimate costs, I compared an index of trendfollowing funds⁴ against AQR’s published Time Series Momentum dataset, which gives hypothetical returns to a TSMOM strategy like the one studied by HLWZ. Comparing the live index against AQR’s benchmark, the two had very similar volatility and drawdown characteristics, but the live funds underperformed by just under four percentage points per year (on average) from 1987 to 2024. That suggests an all-in cost of four percentage points per year.

AQR’s TSMOM benchmark had 4.95% annual alpha on a four-factor regression²⁶, which had a moderate t-statistic of 2.40 (p = 0.017). Due to fees and trading costs, the live trend index had 1.36% annual alpha, which was nowhere close to statistically significant (t = 0.60, p = 0.55). So we can’t confidently say that live funds survive trading costs.

However, this is not a fair comparison. As we saw before, TSMOM is heavily explained by the momentum factor; but the momentum factor is not free to trade, any more than TSMOM is.

A straightforward way to handle this is to subtract estimated trading costs from the momentum factor.²⁷ I did a four-factor regression with the added assumption that momentum costs the same amount to trade as TSMOM, as determined by the difference between the live trend index and AQR’s TSMOM index.

Table 7: Trend index regression on four-factor model net of costs (1987–2024)

		t-stat	likelihood ratio
annual alpha	4.83%*	2.3	14
beta	-0.02	-0.4	1.09
size	-0.05	-0.9	1.51
value	0.08	1.4	3
momentum (net)	0.18***	4.5	21,000
r²	0.06

*, **, and *** indicate significance at the 0.05, 0.01, and 0.001 levels, respectively.

A t-stat of 2.3 is decent but not amazing. (Harvey et al. (2015)²⁸ propose that factors should need a t-stat of at least 3.0 to be considered significant.)

A second approach is to regress a live TSMOM fund against a live momentum fund. The downside of this approach is that most momentum funds have not existed for long. I ran a regression of AQR’s TSMOM fund (AQMIX) against AQR’s equity momentum fund (AMOMX) plus global equities (VT); AQMIX had 5.04% annual alpha, which was just barely statistically significant (t-stat = 1.97, p = 0.0499). But the fund history only goes back to 2010, so this test is underpowered to detect alpha (which we can see at a glance—5.04% is strong outperformance in practice, but it still only had p = 0.0499!).

Hurst, Ooi & Pedersen (2017)⁵ estimated trading costs for TSMOM going back to 1880, using the methods from Jones (2002)²⁹ plus live trading data. They extrapolated from the live data by assuming costs were twice as high before 2002 and six times as high before 1993. After subtracting estimated costs and a 2-and-20 fee, they found that TSMOM still had strong performance back to 1880—an annualized return of 7.3% with 9.7% volatility. Their methodology was reasonable; however, this is ultimately an estimate, not live performance.

I’m highly confident that TSMOM is a real phenomenon. But I’m only moderately confident that TSMOM can be exploited by investors.

Notes

Moskowitz, T. J., Ooi, Y. H., & Pedersen, L. H. (2012). Time series momentum.. doi: 10.1016/j.jfineco.2011.11.003 ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷
Huang, D., Li, J., Wang, L., & Zhou, G. (2020). Time series momentum: Is it there? ↩ ↩² ↩³ ↩⁴
The authors called it TSM, but I’m calling it TSMOM for consistency with other publications on the topic, and to better distinguish it from the “TSH” strategy that HLWZ propose (which I will discuss shortly). ↩
I used the SG Trend Index from 1999 to 2024. The SG Trend Index did not exist prior to 1999, so for the earlier period I used the BTOP50 Index. BTOP50 is less representative because some of its constituent funds pursue other strategies in addition to trendfollowing. To my knowledge, BTOP50 is still primarily trendfollowing-focused, even if not exclusively so. ↩ ↩²
Hurst, B., Ooi, Y. H., & Pedersen, L. H. (2017). A Century of Evidence on Trend-Following Investing. ↩ ↩² ↩³
Please nobody bring up the fact that Victor Von Doom is the Supreme Lord of Latveria. In this hypothetical scenario, he doesn’t pay any attention to the Treasury department. Or maybe he’s misappropriating Treasury assets to fund his company, I don’t know, this is a fake scenario for illustrative purposes. Feel free to make up your own explanation. ↩
One way to compensate for this would be to subtract every asset’s average return from its return each period, such that its long-run average becomes zero; then run a pooled regression. However, this approach does not solve a second problem: it assumes assets’ returns are independent of each other. The standard approach when studying market factors is to use a Fama-MacBeth regression, but that only works for cross-sectional factors, not time-series factors.

There are a few ways to handle asset covariances that might work, but determining the right approach goes beyond my statistical knowledge. ↩
HLWZ reported alphas as monthly; I converted all monthly returns to annual because I find it more intuitive that way. ↩
The exact calculation I used is t.pdf(0, df=346) / t.pdf(tstat, df=346). There are 346 degrees of freedom corresponding to the 348-month (29-year) sample minus two regression coefficients. ↩
Praise be to the scientists who publish their data! ↩ ↩²
The authors present two variants of the strategies that use different asset weighting schemes, namely equal-weighting and inverse volatility-weighting. The latter is more typical for practitioners, but the authors focus on the former, so I will focus on the former as well. All of the tables and statistics in this article use the equal-weighted versions of TSMOM and TSH.

Qualitatively, inverse vol-weighting produced better performance for both TSMOM and TSH and also gave TSMOM a larger advantage over TSH. ↩
Banz, R. W. (1981). The relationship between return and market value of common stocks. doi: 10.1016/0304-405x(81)90018-0 ↩
Stattman, D. (1980). Book values and stock returns. The Chicago MBA: A Journal of Selected Papers, 4:25-45. ↩
Chan, L. K. C., Hamao, Y., & Lakonishok, J. (1991). Fundamentals and Stock Returns in Japan. ↩
Fama, E. F., & French, K. R. (1992). The Cross-Section of Expected Stock Returns. ↩

My regression results differed slightly from HLWZ’s (see their Table 10), probably due to small details in the construction of the TSMOM strategy.

	HLWZ	mine
annual alpha	1.8%	1.46%
beta	0.02	0.02
size	-0.06	-0.06
value	0.06	0.04
momentum	0.60	0.59
r²	0.46	0.45

↩

Goyal, A., & Jegadeesh, N. (2015). Cross-Sectional and Time-Series Tests of Return Predictability: What Is the Difference?. ↩
That is, rather than go long or short each individual stock based on whether it has a positive or negative trend, you go long or short the entire market.

For the avoidance of doubt: Moskowitz et al.¹ and HLWZ² did not look at TSMOM on individual stocks; they used equity indexes, bonds, commodities, and currencies. ↩
This counterintuitive result is an example of Simpson’s paradox (one of my favorite paradoxes): a confounding variable influences TSMOM and TSH in opposite directions. The probable culprit is the size factor. TSMOM had negative exposure to the size factor while TSH had positive exposure, so controlling for size causes TSH’s explanatory power to go up. ↩
HLWZ (2020) cited Hurst et al. (2017), but did not address the evidence it presented. ↩
Baltussen, G., Swinkels, L., & van Vliet, P. (2019). Global Factor Premiums. ↩
I would’ve included international equities too, but the publicly available databases of international equity returns only go back to 1987. We saw from the 1986–2015 regressions that TSMOM had minimal loading on equity beta, so I’m not particularly concerned about this; the main goal is to see how much of TSMOM is explained by momentum. ↩
The value factor probably doesn’t matter for our purposes, but I included it because it’s one on the factors in the Fama-French four-factor model. I excluded the size factor (SMB) because it was not in the Global Factor Premiums data. ↩
With some additional effort I could’ve constructed a data set going back to 1880, but it wouldn’t have made much difference because starting 1927 still gives us nearly 60 years of out-of-sample data. ↩
I did not include commodities, currencies, or fixed income as independent variables because they had some missing data; but my regression is more conservative than the ones performed by HLWZ (2020) because they only included equity factors. ↩
Including US factors and international developed ex-US factors (= eight factors total), 1990–2025. ↩
The size and value factors are not free to trade either, but (1) leaving them unchanged is more conservative, and (2) they’re cheaper to trade because they have lower turnover than momentum or TSMOM. ↩
Harvey, C. R., Liu, Y., & Zhu, H. (2015). … and the Cross-Section of Expected Returns. ↩
Jones, C. M. (2002). A Century of Stock Market Liquidity and Trading Costs. ↩

If AI alignment is only as hard as building the steam engine, then we likely still die

Sat, 10 Jan 2026 00:00:00 -0800

You may have seen this graph from Chris Olah illustrating a range of views on the difficulty of aligning superintelligent AI:

Evan Hubinger, an alignment team lead at Anthropic, says:

If the only thing that we have to do to solve alignment is train away easily detectable behavioral issues…then we are very much in the trivial/steam engine world. We could still fail, even in that world—and it’d be particularly embarrassing to fail that way; we should definitely make sure we don’t—but I think we’re very much up to that challenge and I don’t expect us to fail there.

I disagree; if governments and AI developers don’t start taking extinction risk more seriously, then we are not up to the challenge.

Thomas Savery patented the first commercial steam pump in 1698.¹ The device used fire to heat up a boiler full of steam, which would then be cooled to create a partial vacuum and draw water out of a well. Savery’s pump had various problems, and eventually Savery gave up on trying to improve it. Future inventors improved upon the design to make it practical.

It was not until 1769 that Nicolas-Joseph Cugnot developed the first steam-powered vehicle, something that we would recognize as a steam engine in the modern sense.² The engine took Cugnot four years to develop. Unfortunately, Cugnot neglected to include brakes—a problem that had not arisen in any previous steam-powered devices—and at one point he allegedly crashed his vehicle into a wall.³

Imagine it’s 1765, and you’re tasked with building a steam-powered vehicle. You can build off the work of your predecessors who built steam-powered water pumps and other simpler contraptions; but if you build your engine incorrectly, you die. (Why do you die? I don’t know, but for the sake of the analogy let’s just say that you do.) You’ve never heard of brakes or steering or anything else that automotives come with nowadays. Do you think you can get it all right on the first try?

With a steam engine screwup, the machine breaks. Worst case scenario, the driver dies. ASI has higher stakes. If AI developers make a misstep at the end—for example, the metaphorical equivalent of forgetting to include brakes—everyone dies.

Here’s one way the future might go if aligning AI is only as hard as building the steam engine:

The leading AI developer builds an AI that’s not quite powerful enough to kill everyone, but it’s getting close. They successfully align it: they figure out how to detect alignment faking, they identify how it’s misaligned, and they find ways to fix it. Having satisfied themselves that the current AI is aligned, they scale up to superintelligence.

The alignment techniques that worked on the last model fail on the new one, for reasons that would be fixable if they tinkered with the new model a bit. But the developers don’t get a chance to tinker with it. Instead what happens is that the ASI is smart enough to sneak through the evaluations that caught the previous model’s misalignment. The developer deploys the model—let’s assume they’re being cautious and they initially only deploy the model in a sandbox environment. The environment has strong security, but the ASI—being smarter than all human cybersecurity experts—finds a vulnerability and breaks out; or perhaps it uses superhuman persuasion to convince humans to let it out; or perhaps it continues to fake alignment for long enough that humans sign it off as “aligned” and fully roll it out.

Having made it out of the sandbox, the ASI proceeds to kill everyone.

I don’t have a strong opinion on how exactly this would play out. But if an AI is much smarter than you, and if your alignment techniques don’t fully generalize (and you can’t know that they will), then you might not get a chance to fix “alignment bugs” before you lose control of the AI.

Here’s another way we could die even if alignment is relatively easy:

The leading AI developer knows how to build and align superintelligence, but alignment takes time. Out of fear that a competitor beats them, or out of the CEO being a sociopath who wants more power⁴, they rush to superintelligence before doing the relatively easy work of solving alignment; then the ASI kills everyone.

The latter scenario would be mitigated by a sufficiently safety-conscious AI developer building the first ASI, but none of the frontier AI companies have credibly demonstrated that they would do the right thing when the time came.

(Of course, that still requires alignment to be easy. If alignment is hard, then we die even if a safety-conscious developer gets to ASI first.)

What if you use the aligned human-level AI to figure out how to align the ASI?

Every AI company’s alignment plan hinges on using AI to solve alignment, a.k.a. alignment bootstrapping. Much of my concern with this approach comes from the fact that we don’t know how hard it is to solve alignment. If we stipulate that alignment is easy, then I’m less concerned. But my level of concern doesn’t go to zero, either.

Recently, I criticized alignment bootstrapping on the basis that:

it’s a plan to solve a problem of unknown difficulty…
…using methods that have never been tried before…
…and if it fails, we all die.

If we stipulate that the alignment problem is easy, then that eliminates concern #1. But that still leaves #2 and #3. We don’t know how well it will work to use AI to solve AI alignment—we don’t know what properties the “alignment assistant” AI will have. We don’t even know how to tell whether what we’re doing is working; and the more work we offload to AI, the harder it is to tell.

What if alignment techniques on weaker AIs generalize to superintelligence?

Then I suppose, by stipulation, we won’t die. But this scenario is not likely.

The basic reason not to expect generalization is that you can’t predict what properties ASI will have. If it can out-think you, then almost by definition, you can’t understand how it will think.

But maybe we get lucky, and we can develop alignment techniques in advance and apply them to an ASI and the techniques will work. Given the current level of seriousness with which AI developers take the alignment problem, we’d better pray that alignment techniques generalize to superintelligence.

If alignment is easy and alignment generalizes, we’re probably okay.⁵ If alignment is easy but doesn’t generalize, there’s a big risk that we die. More likely than either of those two scenarios is that alignment is hard. However, even if alignment is easy, there are still obvious ways we could fumble the ball and die, and I’m scared that that’s what’s going to happen.

Notes

History of the steam engine. Wikipedia. Accessed 2025-12-22. ↩
Nicolas-Joseph Cugnot. Wikipedia. Accessed 2025-12-22. ↩
Dellis, N. 1769 Cugnot Steam Tractor. Accessed 2025-12-22. ↩
This is an accurate description of at least two of the five CEOs of leading AI companies, and possibly all five. ↩
My off-the-cuff estimate is a 10% chance of misalignment-driven extinction in that scenario—still ludicrously high, but much lower than my unconditional probability. ↩

I'm wary of increasing government expertise on AI

Sun, 21 Dec 2025 00:00:00 -0800

Many people in AI safety, especially AI policy, want to increase government expertise. For example, they want to place people with AI research experience in relevant positions within government. That may not be a good idea.

People who better understand AI can write more useful regulations. However, people with relevant expertise (such as ML researchers) tend to be less in favor of strong regulations and more in favor of accelerating AI development.¹ We need regulations to prevent misaligned AI from killing everyone, and to prevent other kinds of catastrophes. If government expertise goes up, all else equal we will get fewer such regulations, not more.

If we do get strong regulations, then those regulations will turn out better if AI experts help write them. But we don’t get that by increasing expertise in general; we need AI expertise combined with an understanding of why powerful AI is dangerous.

Government expertise on AI safety matters more than expertise on AI in general. But even there, I’m worried. The most legible AI safety “experts” are the ones who work at AI companies, where strong forces are pressuring them to believe that the alignment problem is solvable and that companies shouldn’t be regulated too hard. The sorts of people who I would most want to see in government AI safety roles² don’t have job titles like “Senior Alignment Researcher at OpenAI”; their titles are more like “Independent Researcher Guy (gender-neutral) Who Posts on LessWrong”, or “Guy Who Dropped Out of High School and Started an AI Safety Nonprofit but Has Never Published an ML Paper”.

I don’t have a great answer for what to do here. It’s important for government to have expertise on AI, but naive efforts to increase government expertise may also increase the probability that AI kills everyone.

One answer: Work on educating policy-makers specifically on AI risks, like what Palisade Research does. Educating non-experts on AI risk seems less fraught than attempting to get experts hired.

Another answer: Try to increase government willingness to regulate AI, not government expertise. Right now, there is not much political will for strong regulations. Without political will, nothing happens. Most of the orgs I considered donating to this year work on increasing willingness to regulate AI.

Notes

ML researchers enjoy doing ML research, and cognitive dissonance often prevents them from believing that ML research could be harmful and even could destroy the world.

According to a 2025 Pew poll, 58% of the public and 56% of AI experts say they’re concerned that the government won’t go far enough in regulating AI. That’s good to see. But the polls also show that AI experts are less worried than the public about the dangers of AI. I fear that experts will favor regulations that don’t impede AI progress, which will ultimately do nothing to prevent extinction. ↩
At least with regard to their expertise, not necessarily their skill at navigating bureaucracy. ↩

Rest in Peace Commento; Long Live Comentario

Fri, 12 Dec 2025 00:00:00 -0800

As of a few days ago, my website supported comments via Commento. If you click on that link, you will find that the page doesn’t load. Unfortunately, that website was also hosting my website’s comments, so all the comments are gone now, and I have no way to recover them.¹ Some of y’all left some good comments, but future readers will never know what they were.²

(I knew Commento was no longer actively supported, but in my foolishness, I thought to myself, well, the comments still work, so I’ll keep using it. Too bad I didn’t back up the comments while I had the chance.)

Commento was my third comment system. Originally I used Disqus, but I didn’t like how it impacted page load times, and I didn’t like how it disrespected my readers’ privacy. So I switched to a janky basic HTML commenting system that required me to manually copy/paste people’s comments into a text file so my website could serve them statically. That system was annoying³, so I switched to Commento, which was lightweight, privacy-respecting, and didn’t require manual effort on my part.

Commento is dead. My website now uses Comentario, which is basically the same as Commento except that (1) it still exists and (2) it’s self-hosted, which means even if Comentario stops existing and the website disappears, the comments on my website will still work.

(Commento had an option for self-hosting, but it looked like a lot of work so I didn’t do it.)

(Setting up Comentario self-hosting was a lot of work, confirming my suspicions. It took me about 10 hours, although to be fair, 8 of those 10 hours were spent trying to upgrade my server’s operating system because it was too old to be compatible to Comentario, and also it reached end-of-life in 2021 and probably had a lot of security vulnerabilities, oops. Anyway I hope y’all appreciate all the work I’m doing to prevent Disqus from spying on you.)

Notes

I reached out to customer support. I think they are AWOL, but I’ll see if I can get them to send me a database backup. ↩
They’re not saved on web.archive.org either, because the comments were loaded dynamically. ↩
The one upside to that system was that the “spam filter” was a primitive honeypot that you’d think would be trivial for spambots route around, that they nonetheless fell for it 100% of the time.

The HTML for the comment submission form looked something like this:
```
<span hidden>
  <button name="Submit (anyone who clicks this button is a spambot)" />
</span>
<span>
    <button name="Submit" />
</span>
```
Spambots would click on the fake hidden button every time. (I’m not exaggerating—out of the hundreds of spambots who attempted to comment on my website, literally zero of them made it past this filter.) ↩

I need the Writing Style Guide people to figure out how to put a smiley face inside parentheses

Thu, 11 Dec 2025 00:00:00 -0800

I can’t figure out any good way to put a smiley emoticon inside parentheses. There are five choices, all of which are bad:

Do the straightforward thing of just writing it (which puts two parentheses next to each other in a row, and makes it unclear where the smiley face ends and the parenthesis proper begins :)).¹
Do that, but put the period inside the parentheses. (Which requires restructuring your sentences, and ends up looking ugly anyway, like some deformed double-mouth emoticon :).)
Put a space between the emoticon and the close parenthesis (which does look more visually distinct, but there’s no other situation where you put a space before the close parenthesis :) ).
Only put a single close parenthesis (possibly the worst option because you can’t tell if the parenthesis is part of the punctuation or part of the emoticon :).
Add some text after the smiley so it’s not at the end of the parenthetical (but maybe you have nothing left to say so the text is superfluous :) haha wouldn’t it be crazy if I wrote some extra stuff here?).

There’s a secret sixth option of “don’t use emoticons inside parentheses” but, like, what if I really want to? (emoticons are important sometimes :) )

Notes

I put a smiley face here for the sake of illustration even though I’m not happy. Feel free to interpret it as a deranged losing-my-sanity smile. ↩

I did Inkhaven

Sun, 30 Nov 2025 00:00:00 -0800

I published a post every day of November as part of the Inkhaven program, in which we are required to publish a post every day of November. Some of my readers knew that; others were confused about why I suddenly started posting so much.

If you’re an email subscriber, you didn’t see every post because I only sent out the good ones—I didn’t want to bombard you with emails if you were accustomed to my typical once-per-week-or-three-months posting schedule. If you want to see the bad posts, they’re all on https://mdickens.me/.

Inkhaven had 40 other residents; you can see their posts on the website, and daily highlights at the Inkhaven Spotlight.

Contents
Ranking all of my Inkhaven posts
The urge to be the best at something
Did I write good? Am I blog man now?
Notes

Ranking all of my Inkhaven posts

Here’s all of my posts, ordered from best to worst according to the judgment of my brain. Feel free to leave a comment explaining why my ranking is completely wrong.

The one with the long title stands out as the worst because I wrote it as a gimmick. I had to remove it from the front page of my website because it makes the site borderline-unreadable. You can still access the post via the direct link, if you want to do that for some reason.

My most underrated post was “We won’t solve non-alignment problems by doing research”. It made an important point that was highly underrated before I published that post, and it continues to be highly underrated because the post unfortunately did not spark a revolution in how people think about non-alignment problems.

The urge to be the best at something

There are other Inkhaven residents who have written more popular posts than me, or been more insightful, or more heartfelt, or shown more improvement in their writing abilities, or done a better job at keeping up with what everyone else is writing (I only read maybe 30 out of the 1457¹ posts that Inkhaveners wrote this month). But by golly, I need to be the best at something, which is why I published 10 posts yesterday. Maybe I can’t write the best posts, or the longest posts, or the most posts, but at least I wrote the most posts in one day.

(My first draft said “I can write the most posts in one day”, but I doubt that’s true. I’m sure there are residents who could’ve written more posts than me if they’d put their minds to it. But I’m the one who actually did write the most posts in one day. Unless someone breaks my record by posting 11 posts today.)

Writers sometimes make an abrupt transition to a new subject that doesn’t make sense until later. For example:² I don’t watch much Twitch, but when I do, my favorite streamer is Maynarde. He’s not a big streamer but he fits my tastes well. We like the same video games and have the same stupid sense of humor³, and the people who post in chat are funny too. On my website I pretend I’m sophisticated and mature, but I’m actually a dumbass and Twitch chat is my outlet for me to act like a dumbass.⁴ Anyway, my point is, at one point Maynarde held the speedrun world record in Prodeus. Is Maynarde a world-class speedrunner? No. But he got the world record by being basically good enough to do a speedrun, and trying it before anyone else.⁵

That’s the trick to being the best at something: find something that nobody else is doing, and then be pretty good at it. That’s what I did. I’m good enough at writing that (on a good day) I can crank out 10 low-effort posts in eight hours. (Especially given that I’d already had all the ideas beforehand and written outlines for most of them.)

I wanted to write a post about Twitch this month; most people I know don’t watch it, and the way I watch it is pretty different from the typical Twitch viewer, so I feel like there are things to say. But I couldn’t quite figure out a thesis. So I’m just mentioning Twitch as a segue into describing the techique of being the best by doing something that nobody else has done yet.

Did I write good? Am I blog man now?

During Inkhaven, I finished a lot of post ideas that were otherwise fated to languish in my “post ideas” file. I had the idea about the philosophy of identity for Groot and Baby Groot in 2018, after watching Avengers: Infinity War for the first time. And now, seven years later, I’ve finally turned it into a post. Was that a good use of my time? Was the post worth writing? I don’t know, you tell me.

I also got the chance to talk to some talented writers and get useful advice and feedback. For example, Scott Alexander told me that I hedge too much.

When I was in high school, I did the required thing where you arbitrarily pick a thesis (regardless of whether you believe it) and then confidently defend it. I didn’t like doing that—I want to figure out what’s correct, not just prove that I’m good at arguing. When I wrote personal stuff, I made sure to make it clear what I don’t know, and avoided dogmatically arguing for one side. (I didn’t always succeed at that, but I tried.) But there’s such a thing as taking it too far, which apparently I do. Except for in this post, because Scott made me remove all the hedge words from that one.

I talked to Dynomight about my list of draft ideas. We talked about various things but my biggest takeaway was that I’m spending too long thinking about ideas before writing them, and instead I should just write them.

I got plenty of valuable insight from other people, but if I give more examples, then I will feel like I need to list all of them, and I will feel bad if anyone gets left out. So I will leave it at two, with the acknowledgment that I talked to many other great people as well.

Notes

as of this writing; some people haven’t published their last post yet.

The reason this number is larger than 40 x 30 is that some of the writing coaches and organizers were also publishing a post every day. (I don’t understand how they had time for that.) ↩
This was my fanfic of the move Paul Graham pulled in The Best Essay, in by far one of the best⁶ paragraphs I’ve ever read:

When a subtree comes to an end, you can do one of two things. You can either stop, or pull the Cubist trick of laying separate subtrees end to end by returning to a question you skipped earlier. Usually it requires some sleight of hand to make the essay flow continuously at this point, but not this time. This time I actually need an example of the phenomenon. For example, we discovered earlier that the best possible essay wouldn’t usually be timeless in the way the best painting would. This seems surprising enough to be worth investigating further.

↩
And we like the same music, which is true of zero (?) of my IRL friends. ↩
Example: If streamer asks for anything at any point, or makes a comment that sounds vaguely like asking for something, chat is contractually obligated to respond with “I’ve got your X right here mate [PantsGrab emote]”. This never stops being funny. ↩
Zero-Master, who’s a top speedrunner in every Doom game, wanted to speedrun Prodeus but he was nice enough to wait until Maynarde got the record first. ↩
I’ve been told my humor is too dry so I would like to use my footnote to clarify that that was a subtle joke, which you might have gotten if you read a particular one of the 10 posts I published yesterday. ↩

How do I know if I'm dreaming?

Sat, 29 Nov 2025 00:00:00 -0800

I’ve been interested in lucid dreaming since high school, with just enough success to say that my efforts haven’t been a complete waste of time. I have a lucid dream once every few months, which isn’t great. But I still do reality checks multiple times per day.

The simplest way to lucid dream is to follow two steps:

Get into the habit of writing down your dreams as soon as you wake up, so you get better at remembering them.
Start doing regular reality checks.

(But this is also the least reliable method, hence why I haven’t had much success.)

A reality check is when you examine some aspect of the world to see if it looks unusual. An example of a reality check is to read some text, and then read it again. Many people find that in dreams, they have difficulty reading; or the words shift around and appear to say something different the second time.

When I was in high school, I started experimenting with a variety of reality checks. Most of them didn’t work very well. The one that works best for me is to look at my left hand.

Do I have four distinct fingers and a thumb? Are my fingers clearly solid? Sometimes in dreams, I have too many fingers, or my fingers are indistinct as if I’m seeing double.
When I look at my palm, is my thumb pointed to the left rather than the right?
Count my fingers. Can I successfully count five of them? Sometimes in dreams, I have a hard time counting without messing up.

Beyond reality checks, I’ve found that certain things tend to happen a lot when I’m dreaming:

I look in the mirror and I see that I have a strange haircut. For example, I cut my hair last week, and the other day I had a dream that I only finished cutting my hair halfway, and my hair was half short and half long. But sadly I failed to realize that I was dreaming.
I’m lifting weights, and I can perform many more reps than I expected, and the reps aren’t getting any harder.

When these things happen, sometimes I notice and become lucid; other times I don’t realize that anything is amiss.

The problem is, I’m stupider in dreams than in real life. In real life, I’d notice if something weird was going on. (I often find myself doing a reality check in real life after something strange happens.) In dreams, the “weird detector” is dialed down, and weird things seem normal. Sure, I’m pedaling a canoe with my next door neighbor and also some guy I went to high school with. And the river is made of sand but then when I look at it a second time it’s made of water now. Nothing weird about that, why do you ask?

One time I was hanging out with a group of friends and I was explaining to them how I do reality checks, and how you have to be diligent to do them regularly even if you’re pretty sure you’re awake. And I thought, I should probably take my own advice and do a reality check even though I’m obviously awake right now. So I looked at my hand and you’re never gonna believe this but it turns out I was dreaming! But then I got too excited and accidentally woke myself up.

Anyway, what’s my point with all this? I don’t know, someone said it might be interesting if I write about my experience with lucid dreaming. My experience has mostly been that I do a lot of reality checks in real life but rarely do them in dreams and even when I do realize I’m dreaming, I just wake up immediately. I just did a reality check before writing this sentence and it turns out I’m not dreaming, my hand looks normal and everything.

Prioritizing your objectives is better than grazing past them accidentally

Sat, 29 Nov 2025 00:00:00 -0800

A silly argument:

The goal of this activity/institution is to achieve X. It doesn’t really achieve X, but it does achieve Y, which is even better!

If achieving Y is more important, why on earth would you go about that by trying and failing to achieve X? You should directly focus on Y instead.

School allegedly teaches geometry/history/etc. Sometimes people complain that these skills aren’t useful, or that people forget everything anyway. People respond by saying it teaches socialization or learning how to learn or something.

If the purpose of school is socialization, why not have 8 hours of recess? Why not have mixed classes instead of keeping them in age-segregated cohorts? Instead of discouraging kids from disruptively talking during class, why not let them talk as much as they want?

If the purpose of school is to teach people to learn how to learn, why not let them play video games all day? Video games require you to learn skills, so you’re still (ostensibly) learning how to learn. And I predict that people can learn a higher volume of skills by playing video games than by studying.

(For the record, I am skeptical that “learning how to learn” is a thing. Empirical research on this is complicated and shows mixed results.)

You should warm up before lifting weights. A lot of people warm up by doing random weird stuff. The best way to warm up for an exercise is to do that same exercise with lighter weights, not to do some other thing. You want to warm up the muscles you’ll be using, in the range of motion they’ll be moving through. The best way to do that is to actually do the exercise, but with light enough weight that you won’t strain yourself.

Learning Latin is good because it helps you learn English vocabulary/roots – Why not just memorize English roots then?

Playing chess makes you smarter – Well, presumably you want to be smarter so you can perform better at activity X (among other things), so it would be better to practice activity X directly because then you’d accomplish two things simultaneously, by getting smarter and getting better at X. (Plus, playing chess doesn’t make you smarter, but that’s beside the point.)

Religion is good because it gives people a sense of community – Why not just make a community that’s grounded in reality? Although to be fair, attempts to create secular communities with religion-y vibes have mostly failed.

You should diet instead of taking Ozempic because dieting teaches you dedication – What do you want dedication for? Whatever that thing is, build your dedication by doing that thing, instead of by doing the thing that no longer requires dedication. You might as well argue that people should ditch the printing press and transcribe books manually, or ditch their heater and cut firewood by hand, because those things teach dedication just as well as dieting does.

My Carlin-esque list of pet peeves

Sat, 29 Nov 2025 00:00:00 -0800

Not that I’m remotely as funny as George Carlin, or that this list is funny at all. But he had many complaints and grievances, and today I would also like to complain about some stuff.

This post contains spoilers for a lot of things. I won’t hide spoilers, but I will say the name of the thing before giving the spoiler.

When people in their 30s or 40s (or even 20s) say they’re “living through my third once-in-a-lifetime recession.” Who told you they were once in a lifetime? This kind of thing has been happening about once per decade since the invention of money.

When a judge refers to a courtroom as “my court.” You’re not the dictator, you’re an administrator. You should be serving the people, not the other way around.

Furthermore: the powerful have a moral duty to be polite to those over whom they have power, but the powerless have no corresponding duty. It’s nice when defendants are polite to judges, but politeness is supererogatory, and they are morally entitled to be as rude to judges as they want. Holding people in contempt of court for being rude to judges is a fake crime that was invented by petty power-hungry judges. If you hold power over someone’s life, and that person is rude to you, you are morally obligated not to retaliate.

Relatedly: When someone with a PhD insists on being called “doctor”. I thought the point of getting a PhD was to push the frontiers of human knowledge. Thank you for informing me that the actual reason is so you can act like you’re better than everyone else.

When a movie character falls off a ledge or something and is holding on by one hand, while their other hand dangles at their side. Just reach up and grab it with your other hand!

I tested this on a pull-up bar and I found that my left hand can only support my bodyweight for a few seconds, but if I’m holding the bar with one hand, it’s very easy to reach up and grab it with my other hand. In movies, apparently the former is pretty easy and the latter is impossible.

(Relatedly: When a movie character is dangling off a ledge with one hand and holding another person with their other hand, and basically bicep curls the second person to safety. There are only maybe 20 people in the world who can one-armed bicep curl the bodyweight of another person, and they’re all 300+-pound hulking gorilla men, not handsome skinny movie stars.)

Here’s a video testing this trope:

Here’s another video testing this trope, in which the testers successfully rescue themselves, but (1) they are both professional climbers, and (2) they are using two arms instead of one.

While we’re on strength-related pet peeves: When a movie character is trying to push a heavy object overhead and they struggle to squat it up, and then struggle to push it up with their arms. Your legs can lift about 3 times as much weight as your arms. If you can just barely squat it up, there’s no way you can push it overhead. And if you struggle to push it up with your arms, then squatting it up will be easy.

This scene from Spider-Man: Homecoming comes to mind. In this particular case, my headcanon is that the spider bite made Peter’s arms get disproportionately stronger.

(Am I inconsistent for complaining about inaccurate displays of upper body strength, but defending two people using the same keyboard? Maybe. These are my grievances and I’m allowed to gripe about whatever I want.)

When people refer to works of fiction as “truth” (as in, “it shows deep truths about the human condition”). The absolute chutzpah of some people to refer to something as “truth” when it is completely made up and everyone knows it.

Notice how no one ever talks about physics or chemistry as “conveying universal truths”? There’s a direct inverse relationship between how often people describe something that way and how much truth it actually conveys.

A highly advanced alien race with allegedly far better ethics than humans, who make most of the same ethical reasoning errors as humans. One theme you see sometimes: the aliens are considering wiping out humanity for being too unethical, but then decide not to because they see some people being good. If the aliens are so morally advanced, why are they still doing collective punishment? Humans (at least some of us) figured this one out 2600 years ago (“the child will not be punished for the parent’s sins”, Ezekiel 18:20).

When someone’s heart stops and people say “they were dead for 2 minutes.” Heartbeat is a commonly-used proxy for death, but it’s not the same thing as death. If your heart stops and restarts, then you weren’t dead.

Referring to an athlete in a light weight class as the GOAT. The only reason that athlete wins competitions is because heavyweights aren’t allowed to compete against them. If they would lose to a mediocre heavyweight, how can you reasonably call them the GOAT?

The saying “freedom of speech doesn’t mean freedom from consequences.” Yes it does mean exactly that. What else could it possibly mean? “Sure, you’re free to criticize Stalin, but you’re not free to not get sent to Siberia.”

In a movie when two people are talking, and one of them walks away and says something quietly but the other person can still hear them somehow.

This scene from Shazam! comes to mind:

When people use xkcd’s Ten Thousand comic in a condescending way.

The whole point of the comic is that you shouldn’t be condescending toward people who don’t know things, but the comic ended up just giving people a whole new way of being condescending.

If you call someone “one of today’s lucky ten thousand”, you’re not explicitly shaming them, but you’re still emphasizing that they didn’t know something and you did. If someone doesn’t know something, better to simply tell them without bringing extra attention to the fact that they didn’t know it.

“Compared to the earth, you’re just a tiny speck.” Something being a “speck” is a fact about your perception, not reality—you look like a speck compared to the earth because human eyes aren’t sufficiently high-resolution to see yourself in a picture of the earth. But you’re exactly as big as you feel like you are, and the earth is far bigger. Reality has way, way more detail than we can fit in our brains.

Or, “[Problem] is so massive that individuals can’t make a difference.” You can’t make a difference that you can perceive at the macro level, but that’s a fact about your perception, not about reality. You can make a difference, but your perception isn’t good enough to be able to see the difference you’re making.

“Time and again, countless studies have proven X” [no citations given]. This usually means X is false.

The saying, “by far one of the best.” How can something be by far one of the best? Either it’s better than everything else by a wide margin, or it’s not.

When academics invent jargon for a concept that already has a commonly-used word, and also redefine the commonly-used word to be incompatible with the normal definition, and then tell normal people that they’re wrong for using the original definition.

Some people claim that strawberries and raspberries are not berries, while bananas and eggplants are berries. Since ancient times, people have understood what berries are, and you can tell because they have “berry” in the name: strawberry, raspberry, blueberry, blackberry. Then some people decided that actually that’s not what “berry” means, it’s actually a fruit produced from a single flower containing one ovary. Nope, that’s not a berry, the word “berry” is already being used to describe blueberries and strawberries and whatnot. Find a new word for your thing.
Nate Soares has complained about “the definitional gymnastics required to believe that dolphins aren’t fish”, and even about berries in particular.
I’ve heard people say “trees don’t exist.” What are these people smoking?? I know trees exist because I can see some outside my window right now. What they mean to say is “trees do not share a common ancestor that isn’t also shared by non-trees”, which is not remotely the same thing as trees not existing.

This would be like someone saying “clothes don’t exist”, and when you press them on it, it turns out what they mean is that clothes can’t be uniquely distinguished by material, because there are things made of cotton that aren’t clothing.
For normal people, an implication is when you make a statement that suggests something is true without necessarily logically entailing it. For linguists, this is called an implicature. And for linguists, the word “implication” can only refer to a scenario where your statement logically entails something. An implication by the common-sense definition is necessarily not an implication by the linguistic definition. This is dumb and linguists should use better terminology.

I understand the need to distinguish between two different definitions of “implication”, but why would you take the more common definition and give it a new word, and redefine “implication” to only refer to the less common definition? Would have been much better to make the technical terms be, say, “implication” and “entailment” rather than “implicature” and “implication”. Or even “implicature” and “entailment” to avoid any ambiguity, and then let us normal people say “implication” whenever we want.

(My computer’s spellcheck says “implicature” is not a word. My computer vindicates me.)

The phrase “quarter century.” You’re trying too hard to make 25 years sound like a long time.

People don’t realize how long ago 2024 was—an entire centicentury has passed!

When stories don’t understand how big planets are. Some examples:

In Dragon Ball Z, characters shoot massive energy blasts that are visible from space and look to be over a thousand miles across, but when zoomed in to the characters’ perspective, they only cover a few miles at most.
The Death Star is powerful enough to destroy a planet, and the Second Death Star—which is canonically more powerful—can destroy a single starship at a time. It should be able to trivially destroy the entire rebel fleet in one shot, in the same way that a strongman who can deadlift a car should also be able to pick up a piece of lint. Except that’s not even a good analogy because the size difference between a fleet of ships and a planet is far greater than the difference between lint and a car.
Also from Star Wars: Darth Vader says, “The ability to destroy a planet is insignificant next to the power of the force.” But no force user ever does anything remotely on the scale of destroying a planet. I cannot emphasize enough how big a planet is, and how trivial Jedi powers are compared to the ability to destroy a planet.
Superman can rotate a planet by pushing on it, and he can also get into a fistfight. If he’s strong enough to rotate a planet, then one of his punches should superheat the atmosphere, leaving a massive crater and creating a shockwave that propagates around the earth and destroys everything on the surface.

This is a form of scope insensitivity: any amount of force greater than about 100 tons gets treated as basically infinite, and is interchangeable with any other amount of force that’s also greater than 100 tons.

(I picked 100 tons as a reference point because, according to Marvel canon, The Hulk can lift “over 100 tons”, even though we’ve seen him perform feats that require, I don’t know, twelve orders of magnitude more force than that?)

I would once again like to reiterate that planets are absurdly large. Any comparison I could make will fail to capture how large planets are.

(Graham’s number, however, is somewhat larger.)

When the “bad” characters make a decision that’s good ex ante, and the heroes oppose the decision, and then it turns out badly, thus “proving” the heroes right.

In Daredevil season 3, when the heroes oppose the plan to let Fisk live in a (government-owned) hotel in exchange for him exposing tons of organized crime, which was clearly a great idea ex ante and the heroes opposed because they just hate Fisk, except then it turned out Fisk had a secret bunker underneath the hotel and was secretly controlling everything, which the heroes did not know but their position turned out to be correct by sheer coincidence.
In Doom (2016), Samuel Hayden was harvesting energy from Hell, and Doomguy was not a fan of that decision. Hayden was correct ex ante that a perfect and unlimited energy source is extremely valuable and worth pursuing. Everything would have been fine if Olivia Pierce hadn’t started a demonic cult and intentionally opened a portal to Hell.
In the Warhammer 40K book Horus Rising, the space marine Saul Tarvitz used all his team’s explosive charges to blow up a single tree because (1) the xenos were using the tree to kill his men and (2) he didn’t want to dishonor his comrades by letting their corpses sit there. This was an understandable but debatable decision. His commander Eidolon chastised him for wasting all their charges, which is a fair criticism. Then, because Tarvitz is a protagonist and Eidolon is an antagonist, it turned out that destroying the trees inexplicably cleared up the weather and let more dropships come in. It turned out to be a magical tree and blowing it up solved all of their problems, but Tarvitz had no way of knowing that in advance.

But sometimes this trope gets subverted, and I always enjoy that. An example from Attack on Titan season 1: Eren, the unreasonable hothead, wants to turn into a titan and fight the female titan. The reasonable members of Levi squad advise against it and convince him not to, but then almost all of them get killed by the female titan because Eren isn’t there to help. Even in retrospect, it’s genuinely unclear who was right. This event becomes more layered by how it influences Eren’s motivations later on—without giving away too much, it made him feel like he can’t rely on other people and he needs to do things for himself.

Stories that retcon human technology as coming from aliens.

For example, in Transformers (2007) and Independence Day, much of 20th century technology was reverse-engineered from studying crashed aliens. Humans are fully capable of inventing stuff on our own, thank you very much!

(The real-life version of this trope is people who think aliens built the pyramids.)

Any news article with a title of the form “Thing Quietly Happens.” It’s not quiet if it’s the subject of a news article.

When people write reviews as “the good, the bad, and the ugly”. In the context of a review, “ugly” is just another word for “bad”. There’s no point in having two different sections that mean the same thing. Just because that was the title of a famous movie doesn’t mean it’s a good way to make a list.

“Smart” characters who are only smart because they have the magical ability to know things that they couldn’t possibly know. But, you see, they figured it out based on zero evidence, because they’re so smart.

In BBC’s Sherlock, Sherlock Holmes makes inferences based on insufficient data but always turns out to be right because the writers decided he is. In this Pete Holmes sketch, Sherlock makes the same deductions as he did in BBC’s version, but he’s wrong every time.

When a story sets up an interesting moral dilemma by giving the antagonist a sympathetic motivation but then they ruin it by having the antagonist act like a jerk to make sure you know they’re the bad guy.

In Across the Spider-Verse, Miguel starts out as an antagonist who’s making some pretty good points actually, but then he starts acting blatantly evil in a way that’s not consistent with his earlier characterization.
In the Star Wars expanded universe, Count Dooku has a compelling backstory in which he rightly has many grievances about how the Galactic Republic government is run; he vies for independence and gets support from the many states that have been mistreated by the Republic. But then to make sure there’s no ambiguity about who the good guys are, he constantly commits war crimes and basically tortures people for fun.

Subversions of this trope are often really fun to watch: when the writers set up a person as the villain, but you can also 100% see where they’re coming from and they make some good points. A few examples that come to mind are Ozymandias, Thanos¹, and Francis Hummel (the villain in The Rock).

When people lament how everything is centralized on the same four social media platforms, and nobody has their own website anymore. I have a website! You could be reading it! (In fact, you’re probably reading it right now.)

When RPGs railroad you into making an immoral decision and then get all philosophical, like, “Who’s the real villain here? Really makes you think.” No, you FORCED me to do the unethical thing. I would’ve behaved ethically if you’d let me.

When people use the term “bodybuilder” to mean “extremely strong person”.

Bodybuilding is not a strength sport. You don’t perform well in bodybuilding by being strong; you perform well by having big muscles. Bodybuilders happen to be pretty strong—much stronger than the average person—but not as strong as people who specifically train for strength, like strongmen or powerlifters or sumo wrestlers.

For example, Arnold Schwarzenegger’s best deadlift was 322 kg (710 lb). The world record deadlift is somewhere between 460.4 kg (1,015 lb) and 510 kg (1,124 lb) depending on what rules you go by. Arnold was strong for a normal person but he’d come dead last in a world-class powerlifting or strongman competition.

When you ask someone a question about an uncertain subject where their credence interval is narrower than yours, and they respond with “I don’t know.”

Let me give an example:

Alice lives in San Francisco. Bob is visiting SF from England and needs to drive to Sacramento for a conference.

Bob: “How long does it take to drive to Sacramento?”

Alice: “I don’t know.”

Alice, you have more information about this than Bob. You have at least a vague sense of where Sacramento is, and Bob doesn’t, so you have the ability to help him out here. Is it 30 minutes? 2 hours? 6 hours?

Hedgehog answers for complex as-yet-unexplained phenomena.

Two examples I see a lot: we don’t really know why animals sleep, and we don’t really know why humans evolved to be so much smarter than other animals. I’m pretty sure these sorts of complex phenomena don’t have a single explanation and it bugs me when people propose a single thing as the sole reason. “Humans evolved intelligence to win arguments”; “humans evolved intelligence to get better at lying and detecting lies”; “humans evolved intelligence to better figure out how to track animal migration patterns.” Intelligence is useful for many things; there is no single reason why it evolved.

When a movie performs poorly in the box office and people accuse it of being a money laundering scheme.

I’m not saying money-losing films aren’t ever some sort of scheme. Maybe they are.[1] But the scheme definitely isn’t money laundering. Money laundering is when you funnel illegally-earned income into a legitimate business, which makes your income look HIGHER, not LOWER. It’s impossible to launder money by REDUCING your income.

[1] The Producers was about a money-losing film scheme. The way the scheme worked is they got investors to provide funding in return for a percentage of profits, and then intentionally made the worst movie possible so they wouldn’t have to pay investors back.

Hollywood accounting is when you inflate your costs to make your profit look smaller than it really is, to avoid paying taxes. Which is, first of all, the opposite of money laundering (money laundering is when you pay more taxes on purpose). And second of all, even if you’re doing Hollywood accounting, you still want your revenue to be as high as possible. You don’t want a box office bomb.

The phrase “lowest common denominator.”

It should actually be “greatest common denominator”, but perhaps it’s confusing to have the word “greatest” in a phrase that identifies a quantity as being small. I would accept simply “common denominator”.

When someone writes a 500+-word comment but uses uncommon/non-standard abbreviations.

You probably spent half a hour writing that comment. Was it really that important to save the 5 seconds it would have taken to write out the full words?

When people don’t understand the difference between something being allowed and being government-mandated.

For example: “The United States doesn’t have parental leave.” Yes it does! Companies are fully allowed to offer parental leave! The US just doesn’t have government-mandated parental leave.

The phrase “demand exceeds supply.”

What people mean by this is something like, “Consumers want more of a good, but it’s too expensive.” Which is another way of saying, “If the price were lower, people would buy more of it.”

Which is true for pretty much everything ever? Demand is downward-sloping. So this statement doesn’t convey any useful information.

And my #1 pet peeve:

Describing someone in the past as “prescient” for observing a trend that was already happening at the time.

Examples:

I saw this tweet described as prescient: The most well-known (alleged) widespread Wall Street fraud was in 2007. This tweet is from 2015. Being 8 years behind isn’t prescient.

(For posterity, this is a Bernie Sanders tweet made on 2015-11-14 saying “Wall street plays by the rules? Who are we kidding? The business model of Wall Street is fraud.”)
1984 and Brave New World are often described this way.
- Orwell’s “we have always been at war with Eastasia” was inspired by real-life events, probably (I don’t know for sure). The first time I noticed this phenomenon in real life is in 2020 when the party line instantly flipped from “don’t wear a mask, masks don’t work” to “you have to wear a mask in public, this has always been the rule and we never said anything different”. But I’m sure this was far from the first time it happened.
https://xkcd.com/1289/ “prescient” about AI art:

Yes, I’m sure AI art was the first time in history that this comic was ever relevant.
SNL’s “Enchilada” sketch is “prescient” because it was making fun of newscasters over-pronouncing foreign words, which they did at the time and still do.
SMBC’s Both Sides sketch which makes fun of (a) creationists/woo and (b) news shows that present creationists/woo as equally credible. People in the comments say it’s more relevant than ever, seemingly forgetting that creationism used to be a real thing that people took seriously, and now they mostly don’t, so it’s actually less relevant than ever.
Lonely Island was “way ahead of its time” for the song “When Will The Bass Drop?” (2014), which was making fun of a trend that had already been happening for 5+ years at that point.
“Trump, unfortunately, has decreased PEPFAR funding” written in mid-2024, described as “prescient” for predicting that he would decrease funding again in 2025 after being re-elected. Extrapolating a historical trend is not prescient. In fact, the original commenter never even predicted he would further decrease funding; they just observed that he had already done so.
Seen in a YouTube comment on a clip from Malcolm in the Middle: “Malcom [sic] was ahead of its time just like walker Texas ranger you don’t see diverse storylines like this anymore now everything is forced”

How could it have been ahead of its time if you don’t see shows like it anymore?

“Ahead of its time” is another thing a lot of people get wrong. The Sopranos was the first TV show of its kind, but it wasn’t ahead of its time—contemporaries appreciated it, and it inspired many shows that aired soon after. It was right on time.

Notes

Thanos is clearly insane, but his actions make sense given his insanity, and he is never unnecessarily cruel—he only kills people because he thinks he has to. (At least that’s true of Infinity War Thanos; Endgame Thanos is a bit different.) ↩

Not being awkward is NP-hard

Sat, 29 Nov 2025 00:00:00 -0800

This meme got me thinking:

That feeling when you’re smart enough to know how awkward you are, but not smart enough to know how not to be awkward

The reason it works that way is because not being awkward is NP-hard, and I can prove it.

It is known that the SAT problem is NP-hard. SAT, or the satisfiability problem, is the problem of taking a logical statement involving a series of boolean values, and determining whether there is some combination of true/false assignments to those values such that the overall logical statement is true.

I will show that the problem of not being awkward is solvable in polynomial time only if SAT is solvable in polynomial time.

Let A be a statement that is known to be awkward. (For example, calling your interlocutor stinky could be regarded as awkward; the exact statement doesn’t matter.)

Now consider some arbitrarily complicated combination of a series of logical statements including statement A. Call that combination statement S. For example:

[you are stinky AND you have blue eyes] OR [you have red hair AND (you are stinky OR I am hungry)] OR […]

If I express S verbally, then I am asserting that S is true. The question then becomes, is there some truth assignment to each of these sub-statements such that the whole statement is true, BUT “you are stinky” is false? Answering that question is equivalent to solving SAT.

S is awkward if and only if “you are stinky” is forced to be true to make S true.

To knowing whether “you are stinky” is forced to be true (in full generality for any statement S), you have to solve SAT.

Therefore, if you can solve for the awkwardness of a statement, then you can solve SAT.

One conceivable objection to this proof: if it is ambiguous whether statement S requires “you are stinky” to be true, then perhaps S is unconditionally awkward. While this may be true in many social situations, it’s not universally true.

For not-being-awkward to be shown to be NP-hard, we don’t need to show that it always solves SAT, only that there is a class of statements that, if solved, would solve SAT. So consider the following social context:

You are talking to a literal-minded mathematician. You say to this person, “If [complicated logical statement] then you’re stinky.” Calling them stinky would be awkward. But, as a literal-minded mathematician, they only take you to be calling them stinky if the antecedent is true. This person would not consider it awkward if you said, “If the moon is made of cheese then you’re stinky.”

There only needs to be one such literal-minded mathematician in the world for my proof to hold, because if there is, that means you can’t always identify a non-awkward statement in polynomial time. There are some situations where identifying non-awkwardness solves SAT, and therefore the problem in full generality is NP-hard.

(It is not yet known whether the awkwardness problem is NP-complete.)

Thanks to Scott Aaronson for suggesting a way to simplify my proof.

Some little things I do to make life easier

Sat, 29 Nov 2025 00:00:00 -0800

In the spirit of You Can Just Do Things, here are some things I Just Do. Some of them are weird; others are normal, but frequently overlooked.

Edited 2025-12-03 to add a ninth thing.

You know how on a hot summer night, your pillow gets hot, and then you flip it over to the cool side? But then before long, the cool side becomes too hot? You can fix this by pouring cold water on a towel and then laying the towel over your pillow. The wet towel stays cold for much longer than a dry pillow would.

When I do this, I only soak half the towel, and then I fold it over so that the dry half of the towel goes between the pillow and the wet half. That way my pillow doesn’t get wet. In the morning I hang the towel up to dry, and I wash it after a few uses so it doesn’t get mildewy.
Ideal sleeping temperature is colder than ideal waking temperature. On colder days, I keep my bedroom window open and my bedroom door closed, which separates my apartment into the “cool part” (the bedroom) and the “warm part” (everywhere else).¹
If you walk through my neighborhood in the evening after a hot day, you will see box fans displayed in every window. But not mine, because I bought a portable air conditioner.² Yes, portable ACs are “inefficient” in an electricity sense, but a box fan lowers my apartment temperature by about one degree per hour. Even a “bad” portable AC can get my bedroom down to 68 degrees in under 30 minutes.³
I buy only one kind of sock (specifically, Hanes X-Temp Cushioned No Show Socks—I tested four different brands and this one was my favorite of the four). Not only do I not need to match my socks, I don’t even need to pair them; I just throw them in a pile in my sock drawer.
I live in a high-cost-of-living area, which means groceries are expensive. I order non-perishable groceries online and have them delivered, which is somehow both cheaper and more convenient than buying them at the store.
When I do go to the grocery store, I go first thing in the morning because the store is nearly empty at 7am. (This strategy doesn’t work if you don’t wake up until later.)

I am not much of a morning person, but I force myself to go to the store right after waking up and then I’m allowed to chill out and do nothing for the next couple hours.
I keep a backup of every basic household item so that I never run out. When I break out the backup, that means it’s time to buy more. I have two bottles of shampoo, two boxes of peppermint tea, two cartons of soymilk (actually six but who’s counting), two packs of razor blades, two bottles of vitamins, …
Many people lament that fitted sheets are too hard to fold. I don’t fold my fitted sheets; but I don’t not fold them, either. My strategy is to always have one set of sheets on my bed and one set on the hamper. When I do laundry, the sheets come off my bed and into the hamper, and the freshly cleaned set goes onto the bed. No folding necessary. And I don’t need to remember to strip the bed before doing laundry, because there’s always a set of sheets waiting to be cleaned.
My hands get cold easily, but if I wear gloves, I lose a lot of dexterity. Fingerless gloves are a compromise, but they normally leave 2/3 of the finger exposed, so my fingers still get cold. I made my own fingerless gloves by cutting the tips off of some regular gloves. My homemade gloves cover almost the full surface of my finger, but my naked fingertips can do a lot more things than gloved fingers can.

Notes

I live in a one-bedroom apartment rather than a studio even though I don’t like spending money; the improved sleep temperature alone might be enough to justify the extra expense. Or maybe that’s just my excuse. ↩
Well that’s not entirely true because I usually do use a box fan, but if the fan isn’t cutting it then I bring out the AC. ↩
My ideal sleeping temperature is more like 60 to 63, but I don’t want to run the AC that hard. ↩

Kid me was bad at Magic: The Gathering

Sat, 29 Nov 2025 00:00:00 -0800

I played a lot of MTG from age 9 to 14 or so. I picked up the game again recently and I was immediately better at the game than my 14-year old self. I don’t have any direct way to prove this, but I’m pretty sure it’s true.

When I was a kid, I liked zombie cards. (And I still do!¹) I looked up zombie decklists online and they all used Carrion Feeder. As a kid, this confused me. Carrion Feeder is bad, I said! It can’t block! Attacking and blocking are the two things creatures do, so if you can’t block, the creature is only half is good! And its ability is useless—why would I want to sacrifice an entire creature to pump up a 1/1?

Now, looking at the card again, I get it. Carrion Feeder is really good. Maybe not ban-worthy, but it’s strong enough that they’ve stopped printing it, and instead they print worse versions like Bloodflow Connoisseur, which is mostly the same except that it costs 3 mana instead of 1.

When they released Isamaru, Hound of Konda—the first ever 2/2 for 1 mana—I thought it was dumb power creep. But looking at it now, it seems fine. Abilities are really important. In most circumstances, I’d rather play a 1/1 with a good abililty than a vanilla 2/2. Plus, Isamaru is legendary, which means you can only have one copy in play at a time—you can’t blitz out a bunch of copies and roll over your opponent in the first three turns.

What was my mistake? I didn’t see the possibilities. As a kid, I only thought about the average-case use for an ability. But that’s wrong because you can specifically engineer a deck to make good use of an ability.

I also simply overlooked some obvious interactions. You can chump block with a creature, sacrifice it to Carrion Feeder, and then take no damage. You just got a +1/+1 counter for free.

As a kid, I thought I would get worse at video games as I aged. I was extremely wrong. The best direct comparison I have: I got my parents to buy me Halo: Combat Evolved when I was 13 or 14. I remember one four-hour play session where I made it most but not all of the way through level 2 on Normal difficulty. I went back and played Halo again when I was 21, and it took me an hour and a half to fully beat level 2 on Heroic. I can’t prove it without a time machine, but I am confident that with an hour of practice, today-me could beat 13-year old me in any game, regardless of how much experience 13-year old me had with it.

I talked with Screwtape about this. Unlike me, he has played a lot of MTG against kids. He’s observed that most middle school aged kids can’t see any card combos that involve 4+ cards—perhaps it’s a working memory issue.²

Screwtape also pointed out that MTG involves a lot of algebra, which I hadn’t realized because you can’t see the equations. But they’re there—deciding how to attack and block with your creatures is an algebra problem. Deciding where to play your buff spells is an algebra problem. I knew algebra when I was 14, but I’m a lot more practiced at it now.

Notes

I like zombie cards, but I don’t like zombie apocalypse movies. After thinking about it, I realized that’s because they’re two totally different things. An evil necromancer reanimating the dead is cool. A virus infecting everyone and making them zombies is not particularly cool. ↩
He also said some 12 year olds will kick your ass at Magic. ↩

Gaming keyboards are not good for gaming

Sat, 29 Nov 2025 00:00:00 -0800

Nearly all gaming keyboards use the conventional typewriter-inspired keyboard shape. That is not a good shape for typing or gaming or frankly anything else.

image source

The gaming experience is vastly improved by keyboards that have thumb keys, such as the Kinesis Advantage or the Maltron.

If you move the Shift key to one of the thumb keys (which you should¹), then Shift-hotkeys and Control-hotkeys become much easier to press. Looking at how pro StarCraft players have to contort their hands to reach the shift key, I can’t help but imagine how much better they’d be at the game if they used a proper keyboard with thumb keys.

Okay, maybe they wouldn’t be better. But at least hitting the buttons would be easier.

I saw an interview with StarCraft pro player Clem (I can’t find the video now) where he described how he uses his palm to hit the spacebar while his fingers are on the number row. And it left me thinking, this would be so much easier if you used a keyboard with thumb keys.

I personally use the Kinesis Advantage—it’s the cheapest of the good ergonomic keyboards, and it works well for me. I’ve become spoiled by the thumb keys, and now I can hardly stand to play games using my laptop keyboard.

If you want to do even better, you can get a split keyboard like the Kinesis Advantage 360 or the Naya Create because they let you game with half a keyboard to make more room for your mouse. In many games, this will require rebinding some keys that normally live on the other side of the keyboard, but that’s a fair compromise.

You could take this even further with the Maltron Single Hand, which is a one-handed keyboard with every key on one side.

Those keyboards weren’t designed with gaming in mind. The most sensible gaming-specific keyboard I’ve seen is the Razer Tartarus:

I’ve never tried this keyboard so I can’t speak from experience, but it looks like a good concept. The keyboard has a small footprint to leave more room for the mouse, and it makes some innovations that I haven’t seen anywhere else. The thumb zone has a joystick rather than just buttons, and the keyboard has a special feature where you can bind keys to do two different things depending on whether you do a half-press or a full press (although some people report finding this feature difficult to use).

One complaint is that the Razer Tartarus doesn’t have many buttons—it could easily have twice as many while still leaving plenty of room for the mouse. Having not used either, I think I’d prefer the Kinesis Advantage 360. But at least the Razer Tartarus is thinking about gaming from the ground up, as opposed to the usual “typewriter-shaped keyboard with expensive keyswitches and RGB lighting”.

It pains me every time I see a pro gamer using a regular keyboard. If your livelihood depends on being good at games, then you should take the time to learn the best tools available.

Notes

On my Kinesis, I’ve rebound Backspace to Shift, Delete to Control, Left Shift to Right Shift (which some games care about), Home to Escape, and Caps Lock to Backspace. ↩

Belief in expert mistakes

Sat, 29 Nov 2025 00:00:00 -0800

A few years ago, there was some publicity around a Navy fighter pilot who claimed to have seen an unidentified object that couldn’t possibly be explained except as an alien phenomenon. Many people considered this to be indisputable proof of aliens. “The pilot is an expert, there’s no way he could have been wrong.”

I am much more willing to believe that someone can make a mistake, regardless of how good they are at the thing in question.

Sure, fighter pilots have excellent vision, and sure, they’re better than I am at identifying objects in the sky. But they’re still fallible. There is no level of fighter-pilot skill that would make me believe aliens visited earth based solely on one person’s testimony.

For any scientific theory, no matter how well-established, you can always find at least one expert with a PhD who studies the topic for a living and disagrees with the consensus. So either almost all experts are wrong, or that one expert is wrong. Either way, experts can make mistakes.

TV is better when you trust the writers

Sat, 29 Nov 2025 00:00:00 -0800

This post contains spoilers for the first episode of Pluribus.

Pluribus is the new show brought to you by the same team who made Breaking Bad and Better Call Saul. The creators are all at the peak of their craft, including the writers. The show has only just started airing, but I’m watching it with confidence because I trust them.

I’ve listened to every episode of the Breaking Bad and Better Call Saul insider podcasts, which gave me some insight into how they write their shows. The writers have latitude to think through every character’s decision and plot out what would happen in each branch of the decision tree, and then choose the path that makes the most sense for the characters and the show.

I don’t know what’s going to happen in Pluribus. But I trust that the writers have thought it through, and that whatever they chose, they made the right decision.

In the pilot episode, some aliens broadcast a signal containing RNA sequence that describes a virus-like thing. Humanity sees the signal and builds the thing, and it infects humans.

From this fact, I’m pretty sure that the aliens have visited earth, because they knew how to construct an RNA sequence that would infect humans. But I only know this because I know the writers are going to make it make sense.

In the hands of lesser writers, they might say the aliens are biologically identical to humans except that they have wrinkly foreheads¹, which is why the RNA sequence works on humans. Or, even worse, they might have no mental model whatsoever of what RNA is or how it works. But I trust Vince Gilligan and the crew, which lets me make deductions:

The aliens broadcast an RNA sequence, which means they know what RNA is, which means either they’ve visited earth or they share a common ancestor with earth life (likely due to natural panspermia).
But the virus thing only works on humans, not any other mammal (the show specifically makes a point of telling us this). Therefore, the aliens must know about human biology.
Therefore, the aliens must have visited earth at some point in the last few tens of thousands of years.

I trust that the creators of Pluribus will make their show make sense.

Take this quote from Vince Gilligan from the pilot episode of the Pluribus podcast. The context is that he’s discussing how the scene in the secure biology lab was originally written to have a lone scientist working, but their science advisor said that a BSL-4 lab would always have at least two people present.

I just went with my old X-Files learning from years ago. A person by themselves, it’s going to be scarier. So that’s the way I intended it. But then, yeah, [science advisor] Erin [Macdonald] said, it just doesn’t work that way. I thought for a microsecond, as I always do, well, artistic license, let’s just keep it this way. Then I thought, it’s never steered me wrong. It’s always held me—and us as a group—it’s held us in good stead when we get things technically accurate or as accurate as we humanly can. It never has harmed us. It has always paid dividends. [timestamp 35:00]

I’ve seen some debate as to whether the show will explain how the hive mind is able to communicate with itself. Some say the show isn’t about the science; it’s meant to be a character study. Maybe, maybe not. I don’t know if they’re going to explain how it works. What I do know is this: if explaining how it works makes the show better, then they’ll explain how it works. And if keeping it mysterious makes the show better, then they’ll keep it mysterious. I don’t know what the right answer is, but I know that whatever it is, the writers will do it that way.

Notes

Sorry Star Trek, you’re still great even if your aliens were a bit goofy. ↩

I like reborrowed words

Sat, 29 Nov 2025 00:00:00 -0800

A reborrowed word is a loan word that goes from language A to language B and then back to language A. I think they’re neat.

A classic example is pidgin. A pidgin is a grammatically simple proto-language that emerges when two groups from different places have to learn to communicate. The word pidgin originally described a simplified form of English spoken by Chinese business people, with pidgin being approximately the Chinese pronunciation of the English word “business”. So “business” was borrowed by Chinese, and then borrowed back by English as pidgin.

Another reborrowed word is anime—Japanese animation. The word comes from Japanese, where it was originally borrowed from the English word “animation”.

I find it interesting how the definition of a reborrowed word is not the same as the definition of the word it came from. Consider waifu, which is the Anglicized pronunciation of the Japanese pronunciation of the English “wife”. But a waifu isn’t a wife; a waifu is a fictional character who a lonely man pretends is his wife. (Or a lonely lesbian woman.) There’s also the husbando which, if I’m not mistaken, is a pure English word that never went through Japanese.¹

In each of these cases, the reborrowed word is more specific than the original—much like how in English, salsa describes a Mexican spicy sauce made with tomatoes and peppers, whereas in Spanish, salsa just means “sauce”.

My fourth example of a reborrowed thing isn’t a word. English schools classically use a grading system that goes A > B > C > F, or sometimes A > B > C > D > F. Japan borrowed this system, except that—for reasons that are lost to time—there is another rank, S, which is even better than A. Japan’s ranking system made its way back into English via tier lists, which work like letter grades except with S tier at the top.

2026-03-01: I just learned a new example: karaoke comes from the Japanese kara + oke, where oke is a shortened form of okesutora … see if you can guess the origin of okesutora. It’s the Japanese pronunciation of the English orchestra.

A related phenomenon happens when a word’s meaning is generalized and then goes back to the original meaning:

source. alt text: Achilles was a mighty warrior, but his Achilles’ heel was his heel.

(A commenter on Facebook reported that a friend once asked them, “What’s Superman’s kryptonite?”)

Another fun thing is a doublet, where two words with two distinct meanings share a single root word. For example, the Latin fortis evolved into the Italian forte as well as the French fort. Both words were then borrowed by English, but the French-derived forte (pronounced like “fort”²) means “strong point”, and the Italian-derived forte means “loud”. Wikipedia has many more examples of doublets.

Notes

I couldn’t find concrete evidence on the origin of “husbando”, but it doesn’t make sense as a Japanese word. The Japanese pronunciation of “husband” would be more like “hasubendo”. My source is Urban Dictionary so take that with a grain of salt, but it fits with my limited understanding of how Japanese pronunciation works. ↩
Nearly everyone gets this wrong. I learned the correct pronunciation from George Carlin, who included this in one of his lists of pet peeves. ↩

Wartime ethics is weird

Fri, 28 Nov 2025 00:00:00 -0800

The ethical principles that most people hold—and hold most strongly—go completely out the window when it comes to war.

Normal time: Killing is bad. In fact it’s pretty much the worst thing you can do.

Wartime: Killing is great! Kill as many people as you can! If you’re really good at killing, you get a medal!

(Just so long as you kill the right people.)

Normal time: Slavery is a blight upon humanity, one of the greatest and most shameful tragedies in history.

Wartime: Slavery is actually totally fine if people are being enslaved by the government for the purposes of killing other people! And in fact, slavery is essential, and if you object to it then you’re betraying your country!

(If it weren’t so depressing, it would be funny to see the contorted logic people come up with to argue that conscription isn’t slavery.)

Normal time: If your employer is behaving unethically and you speak out, you deserve special protections and your employer must not retaliate against you.

Wartime: If you refuse to obey your employer’s unethical demands, that’s a crime; and you will be prosecuted through a special court, and by the way the court is run by your employer.

Not to say I fully agree with common-sense ethics, but according to common-sense ethics, you are morally justified in killing anyone who attempts to draft you, out of self-defense.

Imagine there’s a criminal organization that runs an underground boxing ring where the boxers are coerced into joining. Someone from the organization tries to kidnap you and force you to participate in a boxing match. If you resisted the kidnapper, and even if you used lethal force against them, then you’d be justified on grounds of self-defense. The kidnapper was going to endanger your life; you are morally entitled to use any means necessary to prevent them from doing that.

And yet, if the military came to your house to attempt to force you to go to war, most people would say you’re not allowed to resist. It’s perfectly ethical for someone to kidnap you and forcing you to fight and endanger your life, as long as that someone works for the government. Even people who oppose the draft usually don’t think it’s okay to forcibly resist.

While we’re on the subject of wartime ethics, here’s a combination of common beliefs that doesn’t make sense:

Killing civilians in wartime is morally wrong.
Killing enemy soldiers is fine, even if they were drafted.

Whether someone gets drafted is a matter of luck. Why does it become okay to kill someone after their name comes up in a lottery?

Going back to the underground boxing analogy: if you get forced into a boxing match at gunpoint, and you kill the other boxer, I won’t hold it against you. Your captor is responsible for the death, not you. Nonetheless, the person who died was just as innocent as you were.

What I believe

I’m a utilitarian; I often disagree with common-sense ethics. I believe that conscription could, in principle, be justified on utilitarian grounds. It could be justified in the same way that murder could, in principle, be justified. But there is a strong temptation to rationalize doing harm in the name of the greater good. Murder is rightly illegal, and conscription should be illegal for the same reasons. Even utilitarians should obey moral rules, because your brain is trying to trick you.

The only way to justify a draft is naive consequentialism: we are suspending people’s rights in this case because it’s worth it. I call it naive because drafts usually don’t come out looking good if you properly consider the consequences, and properly consider that you can’t trust your own reasoning on the consequences.

History shows that war is rarely justified on utilitarian grounds. As far as I can tell, war mostly happens due to a combination of not assigning moral value to people in enemy countries—which wouldn’t happen if people were more utilitarian—and the people responsible for declaring war not having to face the lethal consequences themselves.

Alignment Bootstrapping Is Dangerous

Thu, 27 Nov 2025 00:00:00 -0800

AI companies want to bootstrap weakly-superhuman AI to align superintelligent AI. I don’t expect them to succeed. I could give various arguments for why alignment bootstrapping is hard and why AI companies are ignoring the hard parts of the problem; but you don’t need to understand any details to know that it’s a bad plan.

When AI companies say they will bootstrap alignment, they are admitting defeat on solving the alignment problem, and saying that instead they will rely on AI to solve it for them. So they’re facing a problem of unknown difficulty, but where the difficulty is high enough that they don’t think they can solve it. And to remediate this, they will use a novel technique never before used in history—i.e., counting on slightly-superhuman AI to do the bulk of the work.

If they mess up and this plan doesn’t work, then superintelligent AI kills everyone.

And they think this is an acceptable plan, and it is acceptable for them to build up to human-level AI or beyond on the basis of this plan.

What?

It takes remarkable hubris to believe that a problem is this hard, to believe that humanity’s survival depends on getting the right solution, and yet be this confident that it will be solved.

If you don’t know how hard a problem is, then it’s harder than you think.

If you plan on using a technique that’s never been used before, then that technique is less effective than you think.

If you have a problem of unknown difficulty that you want to solve using unknown methods, and you don’t know how you will develop those methods, and failure would be catastrophic, then you shouldn’t do that.

Imagine if NASA wanted to land on the moon and they were trying to figure out how to make rocket fuel, but metalworking hadn’t been invented yet so all their rockets were made of wood. And they said, we are working on figuring out how to make some material won’t get incinerated by rocket fuel; no, we don’t know what that material is, and we have no theory of how to make it; but don’t worry, in 2020 we only had maple and today we are using oak, so we’re making good progress.

This would not be an acceptable plan for solving a medium-stakes problem. It is certainly not an acceptable approach when a failure would destroy everything that matters in the world.

A steelman of this position would be something like:

Yes, our plan has a good chance of failing and killing everyone. But if we don’t build ASI using alignment bootstrapping, some other company will build ASI using even worse techniques, and we’re even more likely to die. So building ASI this way is our best option, even though it’s extremely risky.

Some people believe this. A small number of individuals have said things to this effect. I think they’re still wrong, but at least I get it.

To my knowledge, no one has ever said this in their capacity as a person who is directly working on ASI development or alignment.

I don’t respect AI companies when they publish their roadmaps for handling alignment, and then at no point does the roadmap say anything like

This is a bad plan that has an unacceptably high risk of killing everyone. We’d much prefer to coordinate to slow down and take our time. We would support a global halt on developing ASI until it can be proven safe; but until such time as that happens, we will continue building ASI using our least-bad plan.

The fact that they don’t say this makes coordination more difficult—it’s a self-fulfilling prophecy. Concealing the difficulty of the alignment problem actively contributes to the situation in which the wider world does not take AI risk seriously, and safety-minded developers feel forced to follow a dangerous plan as the least-bad option.

Every major safety proposal by an AI company should start with a disclaimer like “This is a frighteningly risky plan that we are not at all confident in, but it’s our best option due to the lack of widespread agreement about the importance of AI risk.” And they should be simultaneously pushing for global regulations so that they no longer have to take this dangerous route.

Magic: The Gathering Arena decklists for people on a budget

Wed, 26 Nov 2025 00:00:00 -0800

I’ve been playing a lot of MTG Arena lately, but I refuse to spend any money on it, which means I can’t craft many rare cards. When I look up meta decklists, they always include a lot of rares and mythic rares. I don’t want to spend all my rare wildcards on one deck!

That’s sort of what the Pauper format is for. Pauper decks are only allowed to use common cards, which makes them cheap. But that format isn’t quite what I’m looking for, for four reasons:

There is no Pauper ladder on Arena, there are only tournaments.¹ I don’t want to play a tournament, I just want to be able to hop on and play a few games.
I have some rare wildcards that I can use to craft rare cards; I don’t have to limit myself to commons only. I just don’t have very many rare wildcards, so I want to spend them judiciously.
All the strongest Pauper decks are aggro decks. What if I don’t want to play aggro?
There are thousands of MTG cards that aren’t playable on Arena, so I can’t build most Pauper decks anyway.

There’s the Artisan format which is Arena-specific (so it fixes problem 4), but it still has the other three problems.

What I really want is to build a Standard deck using only 4–8 wildcards to craft the most important rares, and then if I decide I like the deck enough, I can craft some more. Which means I want to know which rares I really need, and which ones I can replace with common or uncommon substitutes.

I created a demo to show what this might look like. Below is a table with an interactive decklist for a Dimir Control deck² with a sliding scale from “budget” to “all rares”. I picked this deck because it’s a non-aggro deck that requires relatively few rares and mythics.

I’m no Pro Tour player, so I’m sure some of my choices are wrong. But I made my best attempt to put together a budget-friendly decklist where the rare cards are ordered from most to least replaceable.

This table shows a Dimir Control decklist. As you slide the slider from left to right, the expensive cards get replaced by cheaper substitutes one by one. I’m not a web designer any more than I’m a Pro Tour player, so consider this a proof of concept.

Expensive Budget

Card	Count	Card
●Consult the Star Charts	4	●Stock Up
●The End	2	●Hero's Downfall
●Three Steps Ahead	4	●Refute
●Scavenger Regent	1	●Bitter Triumph
●Marang River Regent	4	●Eddymurk Crab
●Dispelling Exhale	4	●Don't Make a Sound
●Caustic Exhale	4	●Long Goodbye
●Deadly Cover-Up	4	●Aetherize
●Intimidation Campaign	1
●Shoot the Sheriff	2
●Bitter Triumph	1
●Strategic Betrayal	2
Lands	27
Total	60

Consult the Star Charts is a strong card, but it’s the first card to get subbed out because Stock Up is a powerful replacement. Deadly Cover-Up is the most indispensable rare card because there are no proper common/uncommon board wipes—Aetherize vaguely resembles a board wipe, but it’s really not the same thing.

Dispelling Exhale and Caustic Exhale are commons, so it might seem superfluous to replace them; but they work best in a deck that has dragons, and once you replace Marang River Regent, the deck doesn’t have dragons anymore.

A better version of this interface could allow for more complex changes. For example, I wouldn’t replace Marang River Regent one-to-one with Eddymurk Crab because the Regent isn’t just a big creature, it’s also a card-draw spell. I’d want to rearrange the deck to bring in Quick Study or something. My demo is too simple to do that, but it’s possible in theory. You may have also noticed that Bitter Triumph appears twice on the budget decklist; it would be better to collapse those into one row.

If any MTG players think my budget decklist is wrong, I’d be happy to hear suggestions because I play this deck often. The version I use right now is most of the way toward the Budget side, plus some small changes like bringing in Quick Study.³

My main point isn’t about this specific decklist. My point is that when I get a deck off the internet, I want to know which rare cards I should craft in what order. Which ones are indispensable and which ones don’t matter? That table I made is a prototype of the sort of thing I’d like to see.

Notes

And the people who play in those tournaments are really good at Pauper. I tried playing one once and I got destroyed every game. ↩
Azorius Control and Jeskai Control are better right now, but I don’t think there’s any way to make budget versions of those decks. Black has good common/uncommon removal spells, but white’s only good removal cards are rare. And then there are singular cards like Jeskai Revelation and Beza, the Bounding Spring that you can’t replace. ↩
Plus some rares that I got lucky enough to open in booster packs—I have one copy of Marang River Regent and one Three Steps Ahead. ↩

How to Fix Quidditch

Tue, 25 Nov 2025 00:00:00 -0800

Inspired by this post by Tomás Bjartur, which is an allegory; but I’m not writing an allegory, I’m writing about the rules of Quidditch.

The rules of Quidditch have a big problem. The game ends when a seeker catches the snitch, and the snitch is worth 150 points. So most of the players on the field don’t matter; in almost all games, the only thing that matters is who catches the snitch.

This also makes it a bad spectator sport because you can’t see the snitch, so nobody knows what the hell is going on.

I propose some rule changes:

The game ends when the snitch is caught. (This rule is still the same.)
The snitch is worth 10 points.
Instead of being nearly-invisible, the snitch glows and leaves a glowing comet-trail.
The seekers are each given special wands that can only cast a limited set of spells: specifically, spells that make the snitch harder to catch. For example, they can give it a temporary speed boost, or render it temporarily invisible, or push it a fixed distance. The wands have limited energy that takes time to recharge after a spell is cast.

In my proposed version of Quidditch, the snitch still matters—and Harry Potter as the seeker still gets to play a central role, which is important for narrative purposes—but the snitch is no longer the only thing that matters.

Making the snitch more visible is a no-brainer because Quidditch is supposed to be a spectator sport. Spectators ought to be able to see what’s going on.

Under my proposed rules, the winning team wants to catch the snitch, and in a tied game, both teams want to catch the snitch. The losing team still has dynamic gameplay, in which they can use their restricted magic to make the snitch more difficult to catch. A seeker with strong defensive skills can keep a losing game interesting, and allow their teammates time to turn things around.

Some people play Quidditch in real life. I’ve never played the real-life game myself, but I did read the rules online, and it makes some similar changes to my suggestions:

The snitch is worth 30 points instead of 150.
Instead of being a magical flying ball that doesn’t exist, the snitch is a person who runs around with a tennis ball inside a yellow sock. As with my suggested modification, this has the advantage that spectators can see where the snitch is.

Having now looked at the real-life rules, I think it’s better for the snitch to be worth 30 points rather than 10 because it gives seekers more of an incentive to catch it—they can turn a narrow loss into a victory.

My fantasy ruleset has one advantage over the real-life rules, which is that it involves magic. We should use magic to give seekers the ability to make the snitch harder to catch; this changes the snitch-chasing objective from double-Solitaire to a legitimate multiplayer challenge.

If you look at most real-life sports, they have a single center of action. In team-based ball sports, the center is the ball. Spectators watch the ball. Quidditch has two centers of action, the quaffle (the main ball) and the snitch. I’m not sure if that’s a good thing or a bad thing. My first thought was that it’s bad because most sports don’t do that. But then I remembered that I watch a lot of StarCraft, and StarCraft games often have multiple things happening simultaneously in different locations, and the games where that happens are the most fun ones to watch. So perhaps the split attention in Quidditch is a good thing.

I don't like having goals

Mon, 24 Nov 2025 00:00:00 -0800

Sometimes I’m talking about lifting weights and someone asks me, “What’s your goal weight?” I don’t understand why I would have a goal weight.

Say I want to bench press 300 pounds. What happens when I reach 300? I just give up on the bench press now? That would be silly. If I can keep getting stronger, I should.

What happens if I fall short of my goal? Say I haven’t been able to bench more than 285.¹ Should I start eating 5000 calories a day to put on as much muscle as possible? No, I’m not going to do that, I don’t want to get fat. Realistically, if I fall short of my goal, the answer to the question of what I should change is “nothing”.

The point of a goal is to make tradeoffs between objectives. But when you set goals, you have less information about your costs than when you’re trying to implement them. At implementation time, you have new information that might change how you prioritize things, which may result in failing to achieve a goal; and that’s perfectly fine.

Sometimes a goal turns out to be easier than you thought; that doesn’t mean you should give up after you achieve it.

Sometimes a goal turns out to be harder than you thought; that doesn’t mean you should sacrifice everything else for it.

You have an implicit utility function across multiple metrics and you want to know how to prioritize those metrics, but you do that by knowing the coefficient of each metric. Say you’re the CEO of a company, and you want to increase revenue and increase NPS and you need to allocate resources to each of those. It doesn’t make sense to say “we want to increase NPS by 0.5 points” because NPS trades off against revenue (you can make your product cheaper and worse, which increases revenue but decreases NPS), so you also need to make a statement about how much you care about revenue. It would make more sense to say “$1 million of revenue is as important as 0.1 points of NPS”. And then if it turns out there’s a way to increase NPS by 0.2 at a cost of only $500,000 of revenue, then that’s a good deal and you should take it. But a flat goal of “increase NPS by 0.1” is uninstructive.

It’s common for pension funds to target an 8% investment return. That makes no sense. Your investment return is mostly determined by the market environment, which is out of your control. There are only two ways to hit an 8% return consistently: take excessive risk when forecasted returns are low (which is even worse than accepting lower returns); or intentionally hamstring your investments when forecasted returns are high (e.g. if you expect the market to earn 8% and you think you can beat the market by 2%, then you deliberately ignore your market-beating ideas so that you can hit 8% without going over). There is no reasonable utility function to which “always get an 8% expected return” is the correct decision.

In fact, standard finance theory says you should take less risk when expected return goes down (holding volatility constant). Targeting 8% return, and taking on more risk when expected return goes down, means you are actively doing the opposite of what you ought to be doing.

However, goals are okay when reality has a discontinuity. If you need to score at least 70% on a test to pass the class, then it’s reasonable to set a goal of 70%. If you’re pretty sure you can score well over 70%, then it makes sense to stop studying. If you don’t think you’ll pass, then you should study more.

Notes

I never had a goal of benching 300, but 285 is indeed my personal best, which coincidentally is also how much Robin Williams’ character from Good Will Hunting can bench. I wonder how much Robin Williams could bench in real life? He had pretty thick arms so 285 (or even higher) wouldn’t surprise me. ↩

Some Curiosity Stoppers I've Heard

Sun, 23 Nov 2025 00:00:00 -0800

A curiosity stopper is an answer to a question that gets you to stop asking questions, but doesn’t resolve the mystery.

There are some curiosity stoppers that I’ve heard many times:

Why doesn’t cell phone radiation cause cancer? Because it’s non-ionizing radiation.
Why are antioxidants good for you? Because they eliminate free radicals.
Why do bicycles stay upright? Because of gyroscopic forces.
Why do solids hold together? Because of intermolecular forces of attraction.

For the first three, those answers confused me because I didn’t know what those words meant. I guess I know what an ion is (it’s an atom with an electrical charge) but why do I care whether radiation is ionizing? And what makes radiation ionizing or non-ionizing?

What’s a free radical? Why is it bad?

What’s a gyroscopic force? (What even is a gyroscope? It’s some sort of top, right?) How on earth does a bicycle generate a gyroscopic force?

The fourth curiosity stopper—”intermolecular forces of attraction”—is even more of a non-answer. Of course solids hold together because a force holds them together. That’s what a force is. But what is the force, and where does it come from?

Another genre of curiosity stopper is the out-of-context number:

“The Dow is down 600 points today.” (How much is that?)
“My proposed policy will create two million jobs.” (What percentage is that? What are the odds that I, personally, get a new job?)
“This product has 7 grams of protein per serving!” (How big is a serving? How much would I need to eat to meet my daily protein requirement?)

Answers (sort of)

I don’t like those answers, so I will try to give real answers if I can. The true answers are complicated and I’m sure my explanations are at least partly wrong but I’ll do my best.

Why doesn’t cell phone radiation cause cancer?

Radiation causes cancer when it basically smashes into your DNA and knocks a molecule out of place. If it hits your DNA in just the right way, the radiation can disrupt the part of a cell that regulates growth, and it starts growing out of control and becomes a tumor.

Cell phone radiation has low energy so it’s not powerful enough to mess up your DNA.

Why are antioxidants good for you?

(There is still no consensus as to whether antioxidants are indeed good for you, but let’s assume they are for a minute.)

There are some molecules called free radicals. For our purposes, it doesn’t matter what that means, they’re just a type of molecule. They exist in your body, and sometimes they bounce into your DNA and mess it up, which can cause cancer. Your cells produce these free radicals over time. Antioxidants bind with the free radicals and prevent them from bouncing into your DNA.

Why do bicycles stay upright?

Honestly I don’t understand this one at all, sorry. But I do believe that the “gyroscopic forces” explanation is incorrect, or at least incomplete, because some people built a bicycle that doesn’t have gyroscopic effects¹.

Why do solids hold together?

A combination of forces including the London dispersion force and cohesion. If I understand correctly, a macro-level analogy would be that the electrons orbit the protons and sometimes an electron from one molecule moves close to the proton for a neighboring molecule and they attract each other. The true explanation involves quantum mechanics so it’s more complicated than that.

And for out-of-context numbers: percentages and ratios are better than absolute numbers. Adding 2 million jobs to the economy is very different for a country with a population of 30 million vs. 300 million.

This is how I’d like numbers to be reported:

“The Dow is down 2% today.”
“My proposed policy will expand the job market by 4%.”
“This product has 0.1 grams of protein per calorie!”

Notes

Kooijman, J. D. G., Meijaard, J. P., Papadopoulos, J. M., Ruina, A., & Schwab, A. L. (2011). A Bicycle Can Be Self-Stable Without Gyroscopic or Caster Effects. ↩

Where I Am Donating in 2025

Sat, 22 Nov 2025 00:00:00 -0800

Last year I gave my reasoning on cause prioritization and did shallow reviews of some relevant orgs. I’m doing it again this year.

Cross-posted to the Effective Altruism Forum.

Contents
Cause prioritization
My favorite interventions
Organizations (tax-deductible)
Non-tax-deductible donation opportunities
Where I’m donating
Changelog
Notes

Cause prioritization

In September, I published a report on the AI safety landscape, specifically focusing on AI x-risk policy/advocacy.

The prioritization section of the report explains why I focused on AI policy. It’s similar to what I wrote about prioritization in my 2024 donations post, but more fleshed out. I won’t go into detail on cause prioritization in this post because those two previous articles explain my thinking.

My high-level prioritization is mostly unchanged since last year. In short:

Existential risk is a big deal.
AI misalignment risk is the biggest existential risk.
Within AI x-risk, policy/advocacy is much more neglected than technical research.

In the rest of this section, I will cover:

What I want to achieve with my donations
How I’ve changed my mind since last year
How my confidence has increased since last year

What I want to achieve

By donating, I want to increase the chances that we get a global ban on developing superintelligent AI until it is proven safe.

“The Problem” is my favorite article-length explanation of why AI misalignment is a big deal. For a longer take, I also like MIRI’s book.

MIRI says:

On our view, the international community’s top immediate priority should be creating an “off switch” for frontier AI development. By “creating an off switch”, we mean putting in place the systems and infrastructure necessary to either shut down frontier AI projects or enact a general ban.

I agree with this. At some point, we will probably need a halt on frontier AI development, or else we will face an unacceptably high risk of extinction. And that time might arrive soon, so we need to start working on it now.

This Google Doc that explains why I believe a moratorium on frontier AI development is better than “softer” safety regulations. In short: no one knows how to write AI safety regulations that prevent us from dying. If we knew how to do that, then I’d want it; but since we don’t, the best outcome is to not build superintelligent AI until we know how to prevent it from killing everyone.

That said, I still support efforts to implement AI safety regulations, and I think that sort of work is among the best things one can be doing, because:

My best guess is that soft safety regulations won’t prevent extinction, but I could be wrong about that—they might turn out to work.
Some kinds of safety regulations are relatively easy to implement and would be a net improvement.

Safety regulations can help us move in the right direction, for example:

Whistleblower protections and mandatory reporting for AI companies make dangerous behavior more apparent, which could raise concern for x-risk in the future.
Compute monitoring makes it more feasible to shut down AI systems later on.
GPU export restrictions make it more feasible to regulate GPU usage.

My ideal regulation is global regulation. A misaligned AI is dangerous no matter where it’s built. (You could even say that if anyone builds it, everyone dies.) But I have to idea how to make global regulations happen; it seems that you need to get multiple countries on board with caring about AI risk and you need to overcome coordination problems.

I can think of two categories of intermediate steps that might be useful:

Public advocacy to raise general concern about AI x-risk.
Regional/national regulations on frontier AI, especially regulations in leading countries (the United States and China).

A world in which the USA, China, and the EU all have their own AI regulations is probably a world in which it’s easier to get all those regions to agree on an international treaty.

There is no good plan

People often criticize the “pause AI” plan by saying it’s not feasible.

I agree. I don’t think it’s going to work.¹

I don’t think more “moderate”² AI safety regulations will work, either.

I don’t think AI alignment researchers are going to figure out how to prevent extinction.

I don’t see any plan that looks feasible.

“Advocate for and work toward a global ban on the development of unsafe AI” is my preferred plan, but not because I like the plan. It’s a bad plan. I just think it’s less bad than anything else I’ve heard.

My P(doom) is not overwhelmingly high (it’s in the realm of 50%). But if we live, I expect that it will be due to luck.³ I don’t see any way to make a significant dent on decreasing the odds of extinction.

AI pause advocacy is the least-bad plan

I don’t have a strong argument for why I believe this. It just seems true to me.

The short version is something like “the other plans for preventing AI extinction are worse than people think” + “pausing AI is not as intractable as people think” (mostly the first thing).

The folks at MIRI have done a lot of work to articulate their position. I directionally agree with almost everything they say about AI misalignment risk (although I’m not as confident as they are). I think their policy goals still make sense even if you’re less confident, but that’s not as clear, and I don’t think anyone has ever done a great job of articulating the position of “P(doom) is less than 95%, but pausing AI is still the best move because of reasons XYZ”.

I’m not sure how to articulate it either; it’s something I want to spend more time on in the future. I can’t do a good job of it on this post, so I’ll leave it as a future topic.

How I’ve changed my mind since last year

I’m more concerned about “non-alignment problems”

Transformative AI could create many existential-scale problems that aren’t about misalignment. Relevant topics include: misuse; animal-inclusive AI; AI welfare; S-risks from conflict; gradual disempowerment; risks from malevolent actors; moral error.

I wrote more about non-alignment problems here. I think pausing AI is the best way to handle them, although this belief is weakly held.

I’m more concerned about “AI-for-animals”

By that I mean the problem of making sure that transformative AI is good for non-humans as well as humans.

This is a reversion to my ~2015–2020 position. If you go back and read My Cause Selection (2015), I was concerned about AI misalignment, but I was also concerned about an aligned-to-humans AI being bad for animals (or other non-human beings), and I was hesitant to donate to any AI safety orgs for that reason.

In my 2024 cause prioritization, I didn’t pay attention to AI-for-animals because I reasoned that x-risk seemed more important.

This year, in preparation for writing the AI safety landscape report for Rethink Priorities, they asked me to consider AI-for-animals interventions in my report. At first, I said I didn’t want to do that because misalignment risk was a bigger deal—if we solved AI alignment, non-humans would probably end up okay. But I changed my mind after considering a simple argument:

Suppose there’s an 80% chance that an aligned(-to-humans) AI will be good for animals. That still leaves a 20% chance of a bad outcome. AI-for-animals receives much less than 20% as much funding as AI safety. Cost-effectiveness maybe scales with the inverse of the amount invested. Therefore, AI-for-animals interventions are more cost-effective on the margin than AI safety.

So, although I believe AI misalignment is a higher-probability risk, it’s not clear that it’s more important than AI-for-animals.

How my confidence has increased since last year

We should pause frontier AI development

Last year, I thought a moratorium on frontier AI development was probably the best political outcome. Now I’m a bit more confident about that, largely because—as far as I can see—it’s the best way to handle non-alignment problems.

Peaceful protests probably help

Last year, I donated to PauseAI US and PauseAI Global because I guessed that protests were effective. But I didn’t have much reason to believe that, just some vague arguments. In April of this year, I followed up with an investigation of the strongest evidence on protest outcomes, and I found that the quality of evidence was better than I’d expected. I am now pretty confident that peaceful demonstrations (like what PauseAI US and PauseAI Global do) have a positive effect. The high-quality evidence looked at nationwide protests; I couldn’t find good evidence on small protests, so I’m less confident about them, but I suspect that they do.

I also wrote about how I was skeptical of Stop AI, a different protest org that uses more disruptive tactics. I’ve also become more confident in my skepticism: I’ve been reading some literature on disruptive protests, and the evidence is mixed. That is, I’m still uncertain about whether disruptive protests work, but my uncertainty has shifted from “I haven’t looked into it” to “I’ve looked into it, and the evidence is ambiguous, so I was right to be uncertain.” (I’ve shifted from one kind of “no evidence” to the other.) For more, see my recent post, Do Disruptive or Violent Protests Work?⁴

I have a high bar for who to trust

Last year, I looked for grantmakers who I could defer to, but I couldn’t find any who I trusted enough, so I did my own investigation. I’ve become increasingly convinced that that was the correct decision, and I am increasingly wary of people in the AI safety space—I think a large minority of them are predictably making things worse.

I wrote my thoughts about this in a LessWrong quick take. In short, AI safety people/groups have a history of looking like they will prioritize x-risk, and then instead doing things that are unrelated or even predictably increase risk.⁵ So I have a high bar for which orgs I trust, and I don’t want to donate to an org if it looks wishy-washy on x-risk, or if it looks suspiciously power-seeking (a la “superintelligent AI will only be safe if I’m the one who builds it”). I feel much better about giving to orgs that credibly and loudly signal that AI misalignment risk is their priority.

Among grantmakers, I trust the Survival & Flourishing Fund the most, but they don’t make recommendations for individual donors. SFF has a Futher Opportunities page, which shows where they would like to see additional donations go. They are also matching donations on some of their 2025 grants through the end of the year; donors may be especially interested in giving to orgs where they can get matching.

My favorite interventions

In the report I published this September, I reviewed a list of interventions related to AI and quickly evaluated their pros and cons. I arrived at four top ideas:

The first two ideas relate to AI x-risk policy/advocacy, and the second two are about making AI go better for animals (or other non-human sentient beings).

For my personal donations, I’m just focusing on x-risk.

At equal funding levels, I expect AI x-risk work to be more cost-effective than work on AI-for-animals. The case for AI-for-animals is that it’s highly neglected. But the specific interventions I like best within AI x-risk are also highly neglected, perhaps even more so.

I’m more concerned about the state of funding in AI x-risk advocacy, so that’s where I plan on donating.

A second consideration is that I want to support orgs that are trying to pause frontier AI development. If they succeed, that buys more time to work on AI-for-animals. So those orgs help both causes at the same time.

Organizations (tax-deductible)

I’m not qualified to evaluate AI policy orgs, but I also don’t trust anyone else enough to delegate to them, so I am reviewing them myself.

I have a Google doc with a list of every relevant organization I could find. Unlike in my 2024 donation post, I’m not going to talk about all of the orgs on the list, just my top contenders. For the rest of the orgs I wrote about last year, my beliefs have mostly not changed.

I separated my list into “tax-deductible” and “non-tax-deductible” because most of my charitable money is in my donor-advised fund, and that money can’t be used to support political groups. So the two types of donations aren’t coming out of the same pool of money.

AI-for-animals orgs

As I mentioned above, I don’t plan on donating to orgs in the AI-for-animals space, and I haven’t looked much into them. But I will briefly list some orgs anyway. My first impression is that all of these orgs are doing good work.

Compassion in Machine Learning does research and works with AI companies to make LLMs more animal-friendly.

NYU Center for Mind, Ethics, and Policy conducts and supports foundational research on the nature of nonhuman minds, including biological and artificial minds.

Open Paws creates AI tools to help animal activists and software developers make AI more compassionate toward animals.

Sentience Institute conducts foundational research on long-term moral-circle expansion and digital-mind welfare.

Sentient Futures organizes conferences on how AI impacts non-human welfare (including farm animals, wild animals, and digital minds); built an animal-friendliness LLM benchmark; and is hosting an upcoming war game on how AGI could impact animal advocacy.

Wild Animal Initiative mostly does research on wild animal welfare, but it has done some work on AI-for-animals (see Transformative AI and wild animals: An exploration.

AI Safety and Governance Fund

The AI Safety and Governance Fund does message testing) on what sorts of AI safety messaging people found compelling. More recently, they created a chatbot that talks about AI x-risk, which they use to feed into their messaging experiments; they also have plans for new activities they could pursue with additional funding.

I liked AI Safety and Governance Fund’s original project, and I donated $10,000 because I expected they could do a lot of message testing for not much money. I’m more uncertain about its new project, or how well message testing can scale. I’m optimistic, but not optimistic enough for the org to be one of my top donation candidates, so I’m not donating more this year.

Existential Risk Observatory

Existential Risk Observatory writes media articles on AI x-risk, does policy research, and publishes policy proposals (see pdf with a summary of proposals).

Last year, I wrote:

My primary concern is that it operates in the Netherlands. Dutch policy is unlikely to have much influence on x-risk—the United States is the most important country by far, followed by China. And a Dutch organization likely has little influence on United States policy. Existential Risk Observatory can still influence public opinion in America (for example via its TIME article), but I expect a US-headquartered org to have a greater impact.

I’m less concerned about that now—I believe I gave too little weight to the fact that Existential Risk Observatory has published articles in international media outlets.

I still like media outreach as a form of impact, but it’s not my favorite thing, so Existential Risk Observatory is not one of my top candidates.

Machine Intelligence Research Institute (MIRI)

The biggest news from MIRI in 2025 is that they published a book. The book was widely read and got some endorsements from important people, including people who I wouldn’t have expected to give endorsements. It remains to be seen what sort of lasting impact the book will have, but the launch went better than I would’ve predicted a year ago (perhaps in the 75th percentile).

MIRI’s 2026 plans include:

growing the comms team and continuing to promote the book;
talking to policy-makers, think tanks, etc. about AI x-risk;
growing the Technical Governance team, which does policy research on how to implement a global ban on ASI.

I’m less enthusiastic about policy research than about advocacy, but I like MIRI’s approach to policy research better than any other org’s. Most AI policy orgs take an academia-style approach of “what are some novel things we can publish about AI policy?” MIRI takes a more motivated approach of “what policies are necessary to prevent extinction, and what needs to happen before those policies can be implemented?” Most policy research orgs spend too much time on streetlight-effect policies; MIRI is strongly oriented toward preventing extinction.

I also like MIRI better than I did a year ago because I realized they deserve a “stable preference bonus”.

In My Cause Selection (2015), MIRI was my #2 choice for where to donate. In 2024, MIRI again made my list of finalists. The fact that I’ve liked MIRI for 10 years is good evidence that I’ll continue to like it.

Maybe next year I will change my mind about my other top candidates, but—according to the Lindy effect—I bet I won’t change my mind about MIRI.

The Survival & Flourishing Fund is matching 2025 donations to MIRI up to $1.3 million.

Palisade Research

Palisade builds demonstrations of the offensive capabilities of AI systems, with the goal of illustrating risks to policy-makers. My opinion on Palisade is mostly unchanged since last year, which is to say it’s one of my favorite AI safety nonprofits.

They did not respond to my emails asking about their fundraising situation. Palisade did recently receive funding from the Survival & Flourishing Fund (SFF) and appeared on their Further Opportunities page, which means SFF thinks Palisade can productively use more funding.

The Survival & Flourishing Fund is matching 2025 donations to Palisade up to $900,000.

PauseAI US

PauseAI US was the main place I donated last year. Since then, I’ve become more optimistic that protests are net positive.

Pause protests haven’t had any big visible effects in the last year, which is what I expected,⁶ but it’s a weak negative update that the protests haven’t yet gotten traction.

I did not list protests as one of my favorite interventions; in the abstract, I like political advocacy better. But political advocacy is more difficult to evaluate, operates in a more adversarial information environment⁷, and less neglected. There is some hypothetical political advocacy that I like better than protests, but it’s much harder to tell whether the real-life opportunities live up to that hypothetical.

PauseAI US has hired a full-time lobbyist. He’s less experienced than the lobbyists at some other AI safety orgs, but I know that his lobbying efforts straightforwardly focus on x-risk instead of doing some kind of complicated political maneuvering that’s hard for me to evaluate, like what some other orgs do. PauseAI US has had some early successes but it’s hard for me to judge how important they are.

Something that didn’t occur to me last year, but that I now believe matters a lot, is that PauseAI US organizes letter-writing campaigns. In May, PauseAI US organized a campaign to ask Congress members not to impose a 10-year moratorium on AI regulation; they have an ongoing campaign in support of the AI Risk Evaluation Act. According to my recent cost-effectiveness analysis, messaging campaigns look valuable, and right now nobody else is doing it.⁸ It could be that these campaigns are the most important function of PauseAI US.

Video projects

Recently, more people have been trying to advocate for AI safety by making videos. I like that this is happening, but I don’t have a good sense of how to evaluate video projects, so I’m going to punt on it. For some discussion, see How cost-effective are AI safety YouTubers? and Rethinking The Impact Of AI Safety Videos.

Non-tax-deductible donation opportunities

I didn’t start thinking seriously about non-tax-deductible opportunities until late September. By late October, it was apparent that I had too many unanswered questions to be able to publish this post in time for giving season.

Instead of explaining my position on these non-tax-deductible opportunities (because I don’t have one), I’ll explain what open questions I want to answer.

There’s a good chance I will donate to one of these opportunities before the end of the year. If I do, I’ll write a follow-up post about it (which is why this post is titled Part 1).

AI Policy Network

AI Policy Network advocates for US Congress to pass AI safety regulation. From its description of The Issue, it appears appropriately concerned about misalignment risk, but it also says

AGI would further have large implications for national security and the balance of power. If an adversarial nation beats the U.S. to AGI, they could potentially use the power it would provide – in technological advancement, economic activity, and geopolitical strategy – to reshape the world order against U.S. interests.

I find this sort of language concerning because it appears to be encouraging an arms race, although I don’t think that’s what the writers of this paragraph want.

I don’t have a good understanding of what AI Policy Network does, so I need to learn more.

Americans for Responsible Innovation (ARI)

Americans for Responsible Innovation (ARI) is the sort of respectable-looking org that I don’t expect to struggle for funding. But I spoke to someone at ARI who believes that the best donation opportunities depend on small donors because there are legal donation caps. Even if the org as a whole is well-funded, it depends on small donors to fund its PAC.

I want to put more thought into how valuable ARI’s activities are, but I haven’t had time to do that yet. My outstanding questions:

How cost-effective is ARI’s advocacy (e.g. compared to messaging campaigns)? (I have weak reason to believe it’s more cost-effective.)
How much do I agree with ARI’s policy objectives, and how much should I trust them?
ARI is pretty opaque about what they do. How concerned should I be about that?

ControlAI

ControlAI is the most x-risk-focused of the 501(c)(4)s, and the only one that advocates for a pause on AI development. They started operations in the UK, and this year they have expanded to the US.

Some thoughts:

ControlAI’s open letter calling for an international treaty looks eminently reasonable.
ControlAI had success getting UK politicians to support their statement on AI risk.
They wrote a LessWrong post about what they learned from talking to policy-makers about AI risk, which was a valuable post that demonstrated thoughtfulness.
I liked ControlAI last year, but at the time they only operated in the UK, so they weren’t a finalist. This year they are expanding internationally.

ControlAI is tentatively my favorite non-tax-deductible org because they’re the most transparent and the most focused on x-risk.

Congressional campaigns

Two state representatives, Scott Weiner and Alex Bores, are running for US Congress. Both of them have sponsored successful AI safety legislation at the state level (SB 53 and the RAISE Act, respectively). We need AI safety advocates in US Congress, or bills won’t get sponsored.

Outstanding questions:

The bills these representatives sponsored were a step in the right direction, but far too weak to prevent extinction. How useful are weak regulations?
How likely are they to sponsor stronger regulations in the future? (And how much does that matter?)
How could this go badly if these reps turn out not to be good advocates for AI safety? (Maybe they create polarization, or don’t navigate the political landscape well, or make the cause of AI safety look bad, or simply never advocate for the sorts of policies that would actually prevent extinction.)

Encode

Encode does political advocacy on AI x-risk. They also have local chapters that do something (I’m not clear on what).

They have a good track record of political action:

Encode co-sponsored SB 1047 and the new SB 53.
Encode filed in support of Musk’s lawsuit against OpenAI’s for-profit conversion, which was the largest theft in human history.

Encode is relatively transparent and relatively focused on the big problems, although not to the same extent as ControlAI.

Where I’m donating

All of the orgs on my 501(c)(3) list deserve more funding. (I suspect the same is true of the 501(c)(4)s, but I’m not confident.) My favorite 501(c)(3) donation target is PauseAI US because:

Someone should be organizing protests. The only US-based orgs doing that are PauseAI US and Stop AI, and I have some concerns about Stop AI that I discussed last year and above.
Someone should be running messaging campaigns to support good legislation and oppose bad legislation. Only PauseAI US is doing that.
PauseAI US is small and doesn’t get much funding, and in particular doesn’t get support from any grantmakers.

In other words, PauseAI US is serving some important functions that nobody else is on top of, and I really want them to be able to keep doing that.

My plan is to donate $40,000 to PauseAI US.

Changelog

2025-11-22: Corrected description of AI Safety and Governance Fund.

2025-11-29: Corrected description of Survival & Flourishing Fund’s donation matching.

Notes

Although I’m probably more optimistic about it than a lot of people. For example, before the 2023 FLI Open Letter, a lot of people would’ve predicted that this sort of letter would never be able to get the sort of attention that it ended up getting. (I would’ve put pretty low odds on it, too; but I changed my mind after seeing how many signatories it got.) ↩
I disagree with the way many AI safety people use the term “moderate”. I think my position of “this thing might kill everyone and we have no idea how to make it not do that, therefore it should be illegal to build” is pretty damn moderate. Mild, even. There are far less dangerous things that are rightly illegal. The standard-AI-company position of “this has a >10% chance of killing everyone, but let’s build it anyway” is, I think, much stranger (to put it politely). And it’s strange that people act like that position is the moderate one. ↩
Perhaps we get lucky, and prosaic alignment is good enough to fully solve the alignment problem (and then the aligned AI solves all non-alignment problems). Perhaps we get lucky, and superintelligence turns out to be much harder to build than we thought, and it’s still decades away. Perhaps we get lucky, and takeoff is slow and gives us a lot of time to iterate on alignment. Perhaps we get lucky, and there’s a warning shot that forces world leaders to take AI risk seriously. Perhaps we get lucky, and James Cameron makes Terminator 7: Here’s How It Will Happen In Real Life If We Don’t Change Course and the movie changes everything. Perhaps we get lucky, and I’m dramatically misunderstanding the alignment problem and it’s actually not a problem at all.

Each of those things is unlikely on its own. But when you add up all the probabilities of those things and everything else in the same genre, you end up with decent odds that we survive. ↩
I do think Stop AI is morally justified in blockading AI companies’ offices. AI companies are trying to build the thing that kills everyone; Stop AI protesters are justified in (non-violently) trying to stop them from doing that. Some of the protesters have been taken to trial, and if the courts are just, they will be found not guilty. But I dislike disruptive protests on pragmatic grounds because they don’t appear particularly effective. ↩
I want to distinguish “predictably” from “unpredictably”. For example, MIRI’s work on raising concern for AI risk appears to have played a role in motivating Sam Altman to start OpenAI, which greatly increased x-risk (and was possibly the worst thing to ever happen in history, if OpenAI ends up being the company to build the AI that kills everyone). But I don’t think it was predictable in advance that MIRI’s work would turn out to be harmful in that way, so I don’t hold it against them. ↩
On my model, most of the expected value of running protests comes from the small probability that they grow a lot, either due to natural momentum or because some inciting event (like a warning shot) suddenly makes many more people concerned about AI risk. ↩
I have a good understanding of the effectiveness of protests because I’ve done the research. For political interventions, most information about their effectiveness comes from the people doing the work, and I can’t trust them to honestly evaluate themselves. And many kinds of political action involve a certain Machiavellian-ness, which brings various conundrums that make it harder to tell whether the work is worth funding. ↩
MIRI and ControlAI have open-ended “contact your representative” pages (links: MIRI, ControlAI), but they haven’t done messaging campaigns on specific legislation. ↩

An unnecessarily long analysis of one line from The Princess Bride

Fri, 21 Nov 2025 00:00:00 -0800

Vizzini: Inconceivable!

Inigo: You keep using that word. I do not think it means what you think it means.

What did Inigo mean by this?

(Don’t laugh, this is serious.)

The statement can be interpreted in two ways:

I do not think [it means what you think it means].
I do not [think it means] what you [think it means].

Or, in other words:

It is my belief that your definition of “inconceivable” is incorrect.
I have a belief as to what “inconceivable” means; you have a belief as to what it means; and our two definitions disagree.

I’ve wondered about this for years. You might think it’s immaterial, because they both amount to the same thing: Vizzini is using the word “inconceivable” incorrectly, according to Inigo. But the two interpretations have subtle philosophical differences.

By way of illustration, suppose Vizzini defines inconceivable as “my mind could not have conceived of this possibility”, and Inigo defines it as “my mind could not have conceived of this possibility”. Both use the same definition. However, Inigo believes that Vizzini’s definition is something more like “this state of affairs disappoints me”.

Therefore, the statement “I do not think [it means what you think it means]” is true, because Inigo’s definition is not the same as what Inigo believes to be Vizzini’s definition. However, the statement “I do not [think it means] what you [think it means]” is false, because Inigo and Vizzini are using the same definition.

But this scenario is ruled out by the fact that Vizzini would be using the word incorrectly according to his own definition.

Consider another hypothetical. Suppose Vizzini and Inigo both know the correct definition of the word, but they have different philosophies on hyperbole, and Vizzini is more lax about using words in a hyperbolic sense. They have a disagreement, but the disagreement is not about the meaning of the word.

In this hypothetical, “I do not think [it means what you think it means]” is true—Inigo does indeed hold the (false) belief that Vizzini’s definition is wrong. But “I do not [think it means] what you [think it means]” is false, because in fact both parties use the same definition.

Let’s move to one final hypothetical. Suppose Vizzini defines inconceivable as “any activity that is performed by a person wearing all black clothing and a black mask”. For example, if a man in black climbs a cliff without using a rope, that would be inconceivable.¹ If a man in black does something mundane like eating a piece of bread, that would be inconceivable. But if a woman in white² were to climb up a cliff without a rope, that would not be inconceivable.

And suppose Inigo doesn’t know that’s how Vizzini uses the word (because that is a very strange definition). As in the previous hypothetical, Inigo believes that Vizzini defines inconceivable as “this state of affairs disappoints me”.

Now return to the two interpretations of Inigo’s statement. “I do not think [it means what you think it means]” is true. Inigo thinks Vizzini thinks it means “this disappoints me”, so the bracketed statement “[it means what you think it means]” resolves to “[it means ‘this disappoints me’]”; and Inigo doesn’t think that’s what the word means.

Interpretation #2, “I do not [think it means] what you [think it means]”, is also true. Inigo’s definition does not equal Vizzini’s definition. However, Inigo does not have knowledge that this statement is true. He has a justified true belief, his justification being that Vizzini keeps using the word incorrectly. But the justification is false. This is an example of the classic Gettier problem that is one of the most-studies puzzles in epistemology.

Across these three hypothetical edge cases, we have seen that the first interpretation of Inigo’s statement is consistently true. The second interpretation can be false, and it can also be true-but-not-knowledge. Therefore, given the problems with the second interpretation, I conclude that the first interpretation is the correct one:

I do not think [it means what you think it means].

There are conceivable scenarios in which the first interpretation is false and the second is true, but they require strange suppositions like “Inigo has incorrect beliefs about what his own beliefs are”. I don’t think there’s any reasonable scenario that makes the first interpretation false.³

I’ve been wondering about this for nearly 20 years. It was inconceivable that I would ever find an answer, but through some hard work and careful thinking, I’ve finally resolved the mystery.

Notes

This is a slight misrepresentation of what happened; I’m taking some liberties to simplify the story. In the movie, Vizzini cut the rope leading up the Cliffs of Insanity, and then found it “inconceivable” when the man in black—who had been climbing the rope—didn’t fall. The man in black then started slowly free-climbing the cliff, but Vizzini did not say “inconceivable” in response to this.

In the book, Vizzini did find it “inconceivable” that the man in black could continue to climb (page 102), but Inigo didn’t say that line in the book. ↩
or a non-binary person in green ↩
A quote from the book supports my thesis. On page 103, Vizzini argues that he has been using the word correctly, that the man in black is not following them, and that it is inconceivable that he could be following them. His argument seemingly renders the first interpretation true and the second interpretation false.

The exact quote from the book:

“I have the keenest mind that has ever been turned to unlawful pursuits,” [Vizzini] began, “so when I tell you something, it is not guesswork; it is fact! And the fact is that the man in black is not following us. A more logical explanation would be that he is simply an ordinary sailor who dabbles in mountain climbing as a hobby who happens to have the same general final destination as we do. That certainly satisfies me and I hope it satisfies you. In any case, we cannot take the risk of his seeing us with the princess, and therefore one of you must kill him.”

However, the textual evidence is muddled by the fact that Vizzini says “Inconceivable!” after the observation that the man in black is climbing the cliff, not specifically that he is following them; which suggests that Vizzini’s definition is wrong after all, and the second interpretation of Inigo’s statement is not ruled out. ↩

We won't solve post-alignment problems by doing research

Thu, 20 Nov 2025 00:00:00 -0800

Introduction

Even if we solve the AI alignment problem, we still face post-alignment problems, which are all the other existential problems¹ that AI may bring.

People have written research agendas on various imposing problems that we are nowhere close to solving, and that we may need to solve before developing ASI. An incomplete list of topics: misuse; animal-inclusive AI; AI welfare; S-risks from conflict; gradual disempowerment; permanent mass unemployment; risks from malevolent actors; moral error.

The standard answer to these problems, the one that most research agendas take for granted, is “do research”. Specifically, do research in the conventional way where you create a research agenda, explore some research questions, and fund other people to work on those questions.

If transformative AI arrives within the next decade, then we won’t solve post-alignment problems by doing research on how to solve them.

These problems are thorny, to put it mildly. They’re the sorts of problems where you have no idea how much progress you’re making or how much work it will take. I can think of analogous philosophical problems that have seen depressingly little progress in 300 years. I don’t expect to see meaningful progress in the next 10.

Beyond that, there are multiple post-alignment problems. The future could be catastrophic if we get even one of them wrong. Most lines of research only address one out of the many problems. We might get lucky and solve one major post-alignment problem before transformative AI arrives, but it’s extremely unlikely that we solve all of them.

Instead of directly working on post-alignment problems, we should be working on how to increase the probability that post-alignment problems get solved.

This essay will consider four ways to do that:

Do meta-research on what research topics are most likely to help with all post-alignment problems simultaneously.
Pause frontier AI development until we know how to solve post-alignment problems (and the alignment problem too).
Develop human-level “assistant” AI first, then leverage AI to solve post-alignment problems.
Steer AI development such that an autonomous ASI is more likely to solve post-alignment problems.

If you’re working on post-alignment problems, and especially if you’re writing a research agenda, then don’t take it for granted that “do direct research” is the right solution. If that’s what you believe, then support that position with argument. At minimum, I would like to see more post-alignment researchers engage with the question of what to do if timelines are short or progress is intractable.

Edited 2026-03-23 to rename from “non-alignment problems” to “post-alignment problems”. I’m still not satisfied with this name, but I’m told that it’s less confusing.

Introduction
Contents
Approach 1: Meta-research on what approach to use
Approach 2: Pause AI
Approach 3: Develop human-level AI first, then (maybe) pause
Approach 4: Research how to steer ASI toward solving post-alignment problems
Conclusion
Notes

Approach 1: Meta-research on what approach to use

That’s what this essay is. Meta-research is useful insofar as it’s unclear what approach to take, but it has rapidly diminishing utility because at some point we need to pick some strategy and pursue it (especially given short timelines).

I’d like to see more meta-research on whether there are any promising approaches that this essay did not consider.

Approach 2: Pause AI

The case for pausing to mitigate post-alignment risks is similar to the case for alignment risk: we don’t know how to make ASI safe, so we shouldn’t build it until we do. The counter-arguments are also the same: a global pause is hard to achieve; a partial pause may be worse than no pause; etc.

However, in the context of post-alignment problems, the case for pausing AI is stronger in one way, and weaker in another way.

It is stronger in that AI companies mostly don’t care about post-alignment problems. They do care about the alignment problem and are actively working to solve it. Some people are optimistic about their chances—I’m not, but insofar as you expect companies to solve alignment without a pause, a pause looks less important. But companies are ignoring post-alignment problems and almost certainly won’t solve them on the current trajectory.

(I also believe that companies will almost certainly not solve the alignment problem; but that’s a harder position to argue for, whereas it’s clear that AI companies are not even working on post-alignment problems. (Except for Anthropic, which is putting in a weak effort on a subset of the problems, e.g. AI welfare.))

The case for pausing is weaker in that it might not increase our chances of solving post-alignment problems. Human beings mostly don’t care about topics like AI welfare, wild animal welfare, or AIs torturing simulations of people for weird game-theoretic reasons. An aligned ASI, even if it’s not intentionally directed at solving post-alignment problems, might do a better job than humans would.

Approach 3: Develop human-level AI first, then (maybe) pause

An alternative approach: Don’t pause yet. First develop human-level AI that can help us solve the world’s major problems. Don’t develop superintelligence until we’re on stable ground philosophically, but still take advantage of the productivity boost that AI provides.

This plan doesn’t help with misalignment or misuse risks—the human-level AI must be aligned (enough), and it must refuse to perform unethical tasks and be impossible to jailbreak. But it could help with other post-alignment risks.

This plan still requires pausing AI development at some point. In this scenario, it is critically important that we succeed at pausing AI before an intelligence explosion. Therefore, if this is our strategy, then the best thing to do today is to lay the necessary groundwork for a pause.

In an alternative version of this plan, we don’t ever pause AI development. Instead, we squeeze the “solve-every-problem” step into the time gap between “AI dramatically boosts productivity” and “AI has total control of the future”. This only works if post-alignment problems turn out to be much easier to solve than they look.

Another concern—shared with the plan below—is that it seems infeasible to build AIs that are differentially good at philosophy. Philosophy might not be the single hardest thing to get AIs to be good at, but AI will be worse at philosophy than at AI research; therefore, by default, we get an intelligence explosion before we solve the necessary philosophical problems.

Approach 4: Research how to steer ASI toward solving post-alignment problems

Most of the post-alignment problems listed in this essay are different flavors of “we get ethics wrong” or “we make important philosophical mistakes”. What if we can get a sufficiently smart AI to solve philosophy for us?

Four concerns with this research agenda:

“Solve philosophy” is not the same thing as “implement the correct philosophy”, and we need the AI to bridge that gap. There is a near-consensus among moral philosophers that factory farming is wrong, yet it persists. An ASI that solves ethics would need to do the ethically correct thing, rather than the thing people want it to do.²
Philosophy is exceptionally hard to train AIs on. You can’t steer training effectively because we don’t know how to judge the quality of philosophical output.³
To my knowledge, zero people are working on this full-time. Even if there’s a way to do it, it won’t happen without a major shift in research priorities.
Even if you do come up with some useful ideas, you have to get AI companies to implement your ideas. This will be difficult if a “philosophy AI” requires a significantly different training paradigm.

Conclusion

On balance, I believe pausing AI is the best answer to post-alignment problems. I have doubts about whether a pause is achievable, and whether it would even help; but my doubts about the other answers are even stronger.

Notes

Existential in the classic sense of “failing to realize sentient life’s potential”. ↩
h/t Justis Mills for raising this concern. ↩
Particularly on the upper end, which is where it matters. Experts can judge that Kant is better than a philosophy undergrad, but can they judge whether Kant is better than Hume? To solve all post-alignment problems, we will need philosophical research of better quality than what Kant or Hume produced. ↩

Do Disruptive or Violent Protests Work?

Wed, 19 Nov 2025 00:00:00 -0800

Previously, I reviewed the five strongest studies on protest outcomes and concluded that peaceful protests probably work (credence: 90%).

But what about disruptive or violent protests?

Peaceful protests use nonviolent, non-disruptive tactics such as picketing and marches.

Disruptive protests use nonviolent, in-your-face tactics such as civil disobedience, sit-ins, and blocking roads.

Violent protests use violence.

There isn’t much evidence on the other two categories of protest. My best guesses are:

Violent protests probably don’t work. (credence: 80%)
Violent protests may reduce support for a cause, but it’s unclear. (credence: 40%)
For disruptive protests, it’s hard to say whether they have a positive or negative impact on balance. I’m about evenly split on whether a randomly-chosen disruptive protest is net helpful, neutral, or harmful.
A typical disruptive protest doesn’t work as well a typical peaceful protest. (credence: 80%)
Peaceful protests are a better idea than disruptive protests. (credence: 90%)

Contents
Violent protests
Disruptive protests
Conclusion
Notes

Violent protests

Three lines of evidence suggest that violent protests make things worse:

Exactly one quasi-experimental study (Wasow 2020¹).
A meta-analysis of lab experiments (Orazani et al. 2021²).
Various observational studies.

I discussed Wasow (2020)¹ when I reviewed natural experiments on protest outcomes. Wasow (2020) uses rainfall as a way to randomize treatment. Quoting myself:

The idea is that protests often get canceled when it rains. If you look at voting patterns in places where it rained on protest day compared to where it didn’t rain, you should be able to isolate the causal effect of protests. The rain effectively randomizes where protests occur.

Rather than using rainfall directly, the rainfall method uses rainfall shocks—that is, unexpectedly high or low rainfall relative to what was expected for that location and date. This avoids any confounding effect of average rainfall levels.

The quasi-experimental evidence from Wasow (2020) suggests that violent Civil Rights protests backfired: public support went down in places where protests occurred.

Orazani et al. (2021)² is a meta-analysis of lab experiments. The experiments showed people news articles about (real or hypothetical) violent or nonviolent protests and measured their favorability toward the protesters’ cause. The meta-analysis found that:

Nonviolent advocacy had a positive effect (d = 0.25, p < .00001)
Violence had a non-significant negative effect (d = –0.04, 95% CI [–0.19, 0.12], p = .65)

This evidence suggests three things:

Nonviolent protests work.
Violent protests don’t work.
Violent protests don’t strongly backfire—violent protests had a negative effect, but it was small and not statistically significant.

There are many observational studies on violent protests, with mixed results. A literature review by Shuman et al. (2024)³ wrote:

[T]here is some, although more mixed, evidence that even entirely violent protests can sometimes be effective for policy-related outcomes. For example, research on the violent 1992 Los Angeles Riots increased support for local policy reforms when policy referenda came up for a vote soon after, particularly among people who were more proximally exposed to the disruptive violence (although target audience in terms of resistance was not examined). Another study that did examine moderation by target audience found that physical proximity to Palestinian violence increased support among Israelis for making policy concessions, and that this effect was stronger for traditional right-wing, hawkish, groups. However, there is also conflicting evidence. For example, similar research found that exposure to political violence led to harsher policy attitudes among Israelis (although moderation by target audience was not assessed). Similarly, research focused on voting rather than policy found that the outbreaks of violence during the Civil Rights Movement increased support for social control framing of the issue and Republican vote share.

The language from Shuman et al. attributes causality to the cited studies, which I don’t believe is appropriate. I’m quoting this passage for the purpose of illustrating that observational studies on violent protests have found varying results.

Disruptive protests

I found two experimental studies on disruptive protests. They showed participants news articles about protests and asked them how strongly they supported the protesters’ cause.

Feinberg et al. (2017)⁴ ran three experiments on three different causes (animal rights; BLM; anti-Trump). They found that people were more likely to express support for a cause after reading about a peaceful protest than a disruptive protest. Two of the three experiments did not include control groups, so they don’t tell us whether the absolute effect of disruptive protests was positive or negative. The third study (with a control group) found that disruptive protests had a backfire effect.

Bugden (2020)⁵ found that reading an article about a peaceful climate protest increased support. Disruptive protests worked worse than peaceful protests, but still better than the control. This study also found a (non-significant) increase in support due to violent protests, which disagrees with some prior results (the Orazani et al. (2021)² meta-analysis, which did not include Bugden (2020), found a non-significant negative effect).

An observational study by Ostarek et al. (2024)⁶ conducted surveys directly before and after a disruptive protest, which is better than the typical observational study. They found that support was higher after the protest than before.

This evidence is mixed on whether disruptive protests work; and the quality of evidence is much weaker than for peaceful protests.

Conclusion

It looks like violent protests don’t work, and there’s a good chance that they backfire. This is good news—it means we don’t live in the Least Convenient Possible World where you have to commit violence to achieve your goals.

Disruptive protests might work, but the evidence is mixed and weak. The evidence supporting peaceful protests is much stronger, which makes them the better tactic.

Notes

Wasow, O. (2020). Agenda Seeding: How 1960s Black Protests Moved Elites, Public Opinion and Voting.. ↩ ↩²
Orazani, N., Tabri, N., Wohl, M. J. A., & Leidner, B. (2021). Social movement strategy (nonviolent vs. violent) and the garnering of third-party support: A meta-analysis. ↩ ↩² ↩³
Shuman, E., Goldenberg, A., Saguy, T., Halperin, E., & van Zomeren, M. (2024). When Are Social Protests Effective?. ↩
Feinberg, M., Willer, R., & Kovacheff, C. (2017). Extreme Protest Tactics Reduce Popular Support for Social Movements. ↩
Bugden, D. (2020). Does Climate Protest Work? Partisanship, Protest, and Sentiment Pools. ↩
Ostarek, M., Simpson, B., Rogers, C., & Ozden, J. (2024). Radical climate protests linked to increases in public support for moderate organizations. ↩

Why would God have a gender?

Tue, 18 Nov 2025 00:00:00 -0800

Classically, according to the Abrahamic religions, God is a man.

According to some more recent depictions, God is a woman. Which is a nice subversion.

But like, y’all are both a bit crazy. If there is an omnipotent Creator of the universe, then it definitely doesn’t have a gender.

When people call God “he” or “she”, this is what they’re saying happened:

Life evolved over billions of years through a process of mutation, reproduction, and natural selection.
Originally, all organisms reproduced by copying themselves. But some organisms evolved the abiity to reproduce by combining the genes of two different individuals. This let genes mix more and allowed good genes to spread more readily. In some environments, organisms that could reproduce sexually outcompeted those who couldn’t.
Organisms evolved two distinct sexes because it makes evolutionary sense to have two different types of reproductive material (eggs and sperm).
Different animals evolved different characteristics in males vs. females. Sometimes one is larger, sometimes one does more of the work getting food, one does more of the child rearing, etc. These characteristics differ a lot depending on the species.
In one particular species, namely humans, females physically bear children and do most of the child rearing, while males are physically larger and do most of the hunting. Many other animals (especially mammals) use this same division of labor, but many times certain characteristics are reversed. For example, in many species, the female is bigger and stronger than the male; in some (rare) cases, the male does most of the child rearing.
These sexual differences also led to personality differences, which arose due to contingent evolutionary pressures and quirks of the environment.
God, the omnipotent being who created the universe, has personality characteristics that are consistent with the personality of one side of a contingent dichotomous evolutionary strategy in one particular species.
You might think that one particular species would be sharks, because sharks have been swimming the seas for 439 million years. But no, God has the personality traits that are associated with one sex of a species of hairless mammal that only evolved about 100,000 years ago.

(Alternatively, they’re saying that evolution is a lie and the earth is 6,000 years old or whatever, which somehow makes more sense.)

God does not reproduce sexually. God is the eternal Creator of the universe, not the tip of one branch at the end of billions of years of natural selection.

In fact, why would God have a personality at all? A personality is a thing that emerges in social beings and describes their social interactions. God doesn’t have a social life, it’s not like It spends Its day hanging out with the other creators of the universe.

Religions’ lack of imagination kind of bugs me. God is supposed to be this omnipotent, omniscient, incomprehensible being. But if you read religious texts, God just acts like some guy.

(H. P. Lovecraft did a much better job of writing Gods that act like Gods. Or at least I assume he did—I haven’t actually read any of his books.)

Not-Discovered-Here Syndrome

Mon, 17 Nov 2025 00:00:00 -0800

An investor is considering putting her money into a mutual fund. “I will just invest some money for the next six months,” she says, “and see how it goes.”

A philanthropist is considering donating to a charity. “I will donate some money and see how it goes.”

Harvard University is considering whether SAT scores are all that important for admissions. “Let’s make SAT scores optional and see what happens.”

A child climbs to the top of a slide and is about to jump off the edge. “Don’t jump off of that,” his mom says, “you’ll get hurt.” He jumps off the slide. He gets hurt.

Not-invented-here syndrome is when an organization unnecessarily re-invents products or tools that already exist elsewhere. The cousin of this phemonenon is not-discovered-here syndrome, in which people refuse to consider evidence unless they’ve collected it themselves.

“A wise man learns from his mistakes, but a wiser man learns from the mistakes of others.” Not-discovered-here syndrome is what happens when you insist on making mistakes for yourself.

Institutional investors like to “try out” new investments for six months or a year. That doesn’t make any sense. Whatever you learn in the six months of holding the fund, you could’ve learned by looking at a price chart of the prior six months. (In fact you probably could’ve learned a lot more, because most funds have more than six months of history.) Or you could keep an eye on the fund for the next six months without investing. Putting money into the fund doesn’t teach you anything.

Harvard made the SAT optional in 2020. Prior to 2020, there already existed a mountain of data showing that a student’s SAT score are is a good good predictor of college success. It was predictable in advance that if colleges stop requiring the SAT, then they will do a worse job at identifying good candidates. But they ignored the data and learned that lesson the hard way instead.

I had a similar criticism of the book Outlive; I decided not to include it in my book review, but I’ll mention it here since it’s on-topic. In the book, Peter Attia is a big proponent of continuous glucose monitors (CGMs), which show how your blood sugar goes up after you eat.

This sort of confuses me. You can easily find websites that tell you the glycemic loads of different foods. I can tell you what will happen if I eat white flour: my blood sugar will go up a lot. I know because it has a high glycemic load. I can also tell you that if I eat walnuts, my blood sugar will only go up a little bit. A CGM doesn’t tell me anything I don’t already know.

Wearing a CGM is a psychological tool that works for some people, but that’s kind of my point: those people have not-discovered-here syndrome. It’s not enough for me to know which foods have high glycemic load; I have to wear a monitor showing that, yes, this food does raise my blood sugar.

Sometimes people really do make better decisions when they collect the evidence themselves. But there’s no inherent reason why it has to be that way. It feels like there ought to be some way to get people to pay attention to evidence that they didn’t personally discover, but I don’t know how.

Knowing whether AI alignment is a one-shot problem is a one-shot problem

Sun, 16 Nov 2025 00:00:00 -0800

One day, I was at my grandma’s house reading the Sunday funny pages, when I suddenly felt myself getting sucked into a Garfield comic.

I looked down at my body and saw that I had become fully cartoonified. My hands had four fingers and I had a distinct feeling that I’d be wearing the same outfit for the rest of my life.

I also started feeling really hungry. Luckily, I was in the kitchen, where Jon, Garfield’s owner, had made some lasagna.

“You look really hungry,” he said to me. “Why don’t you take this lasagna?”

I gratefully accepted Jon’s lasagna. As he handed it to me, he issued a grave warning: “Don’t let Garfield eat this.”

I looked at where Garfield was sitting on the floor, harmlessly hating Mondays. He was chubby and slow and there was no way he’d be able to jump up and yank the lasagna tray out of my hands.

Jon said, “Take that to the dining table down at the end of the comic strip.”

Thanking him again, I walked over to the next panel, where I ran into a new Jon and Garfield (because that’s how comic strips work). But this Garfield was a bit bigger and a bit more lithe-looking.

I waved at the characters and walked to the third panel. The third Garfield again looked bigger and a bit more aggressive.

“I’m a bit worried about this lasagna,” I said to Jon. “Garfield seems to be getting bigger and stronger and I’m afraid once I get to the last panel, he’s gonna eat my lasagna.”

“Oh yes, the guy who draws us is trying to make a super-Garfield,” Jon explained. “He’s small now, but someday he will be bigger and stronger than either of us.”

“But won’t that mean he will take my lasagna?”

“Don’t worry about that,” Jon reassured me. “I have a plan to tame Garfield. By the end, I will know how to make him leave the lasagna alone.”

“Okay,” I said hesitantly, and kept walking to the fourth panel. (This was a full-page Sunday comic.) Garfield was now big enough to come up to my knees. He scratched at my leg with his front paws, yearning for a meal.

“I’m really not sure about this,” I said to Jon. “Right now, Garfield can’t take my lasagna no matter what he does; he can’t jump that high and he’s definitely not strong enough to knock me over. At some point, though, I will encounter a Garfield who’s bigger than me for the first time, and then he’ll be able to knock me over and eat the lasagna and maybe even eat the entire house. How do I know that he’s tamed until I get to that point?”

“I’m incrementally improving his behavior in each panel,” Jon explained. “We will have many chances to observe Garfield’s behavior before he becomes super-Garfield. I can verify that I’m making progress on his tameness.”

I thought back to what I had read about cat training on LessWrong. “There’s this guy named Eliezer Yudkowsky who says you only get one critical try to tame a super-cat. If you fail, the cat will steal your lasagna, and then you won’t have any lasagna anymore.”

“That argument doesn’t apply to my methods. I can accumulate evidence about whether the cat training is working. Each Garfield will be less lasagna-obsessed than the last, and we can observe the trajectory of the taming. You also have to consider…” and I don’t quite recall what Jon said after that but it was very complicated and it sounded like he knew a lot more about cat-taming than me.

“Perhaps you’re right,” I said. “You have some good arguments, but so does Eliezer. How can I know who’s really right until I reach the last panel? I think I’d better not go there until I know for sure that my lasagna will be safe.”

Some people say AI alignment is a one-shot problem. You only get one chance to align AI, and if you fail, everyone dies.

Other people disagree. They say it’s possible to make gradually smarter AIs and ratchet your way up to a fully-aligned superintelligent AI, and if the process isn’t working, you can tell in advance.

It doesn’t matter who’s right. The key thing is that we don’t get to find out who’s right until it’s too late. Either the gradual ramp-up succeeds at making an aligned superintelligence (as the gradualists predict), or it fails and we die (as the one-shotters predict).

This is the meta-one-shot problem: we only get one shot at knowing whether it’s a one-shot problem.

There will be some point in time where we build the first AI that’s powerful enough to kill everyone. When that happens, either the one-shotters are right and we only get one shot to align it, or the gradualists are right and we get to iterate. Either way, we only get one shot at finding out who’s right.

The alignment one-shot problem may or may not be real. But the meta-one-shot problem is definitely real: we don’t get the evidence we need until it’s too late to do anything about it.

AI companies’ alignment plans only work if the gradualist hypothesis is true. The main reason they operate this way is that it’s much harder to make plans that work in a one-shot world. Unfortunately, there is no law of the universe that says you get to do the easy thing if the hard thing is too hard. If a plan requires gradualism, then we have no way of being confident that the plan will work.

Having a plan for a gradualist scenario is fine. But AI developers also need plans for what to do if AI alignment is a one-shot problem, because they have no way of knowing which hypothesis is correct. And they shouldn’t build powerful AI until they have both kinds of plans.

What If Ghosts Were Real?

Sat, 15 Nov 2025 00:00:00 -0800

If we are correct about the laws of physics, then ghosts can’t exist. But some people are insistent that they’ve directly interacted with ghosts. Is there a way ghosts could exist if we modified the laws of physics a bit?

Okay, what are the properties that ghosts need to have?

Ghosts are coherent entities with bodies. (Maybe they have heads and arms and legs, or maybe they’re a vague cloud shape, but they definitely have something that you could call a body.)
Ghosts are invisible.
Ghosts can pass through solid objects, but they can also knock over lamps and stuff (as long as doing so would be sufficiently spooky).

There are already invisible things that can pass through solid objects. For example, neutrinos. But you can’t have a body made of neutrinos because the particles in your body will immediately scatter all over the place and then you won’t have a body anymore. So ghosts must be made of something else.

If ghosts can pass through solid objects, that means they don’t interact via the electromagnetic force. But something weird has to be going on with gravity. If your feet didn’t interact with the ground, you’d fall straight through to the center of the earth. But ghosts don’t do that.

Maybe ghosts aren’t affected by gravity. But the earth isn’t still: it’s revolving around the sun at 67,000 miles per hour (109,000 kph), and it’s constantly turning in its orbit. If ghosts aren’t gravitationally bound by the sun, why don’t they go flying off into space?

I can see two possible explanations:

Ghosts don’t interact gravitationally, but they can move super fast to keep themselves in the same spot relative to earth.
Ghosts do interact gravitationally, but they can make the bottoms of their feet interact electromagnetically with the ground to prevent themselves from falling through.

The first one doesn’t seem likely to me. The ghost of a 15th century Baron who’s haunting a mansion doesn’t know anything about how the earth’s orbit works, why wouldn’t he just get lost as soon as he turns to ghost-form?

The second explanation brings to mind a testable hypothesis. We put sensors on the floor of the haunted mansion, and that way we can tell when a ghost walks over them.

What are ghosts made of? Ordinary matter is made of protons, neutrons, and electrons. Ghosts can’t be made of those, or else we’d be able to detect them. They can’t be made of dark matter, or else they’d fall through the ground (and they wouldn’t have bodies). Ghosts must be made of some new type of matter that doesn’t interact with the electromagnetic force, but does hold together somehow. That means there must be an as-yet-undiscovered fifth fundamental force that holds ghost bodies together.

Coming back to our ghost properties: ghosts can pass through solid objects, but they can also interact with objects when they want to be spooky. So it’s not as simple as “ghosts don’t interact electromagnetically”. (If that were true, ghosts would have no observable impact on the world at all, and we would have no reason to believe they exist.) Ghosts can interact electromagnetically, but only when they choose to.

That’s quite a puzzle. Ghosts can use their consciousness to turn on and off the electromagnetic interaction at will. I can’t think of how that’s possible, so let’s just move on.

There are two important, but subtle, properties of ghosts:

Ghosts can see.
Ghosts can move.

Obviously ghosts can see and move, right? But these properties have some surprising implications.

If ghosts can see, that means ghosts’ eyes can detect photons.
If ghosts can move, that means they’re getting energy from somewhere.

The first implication means that photons affect ghosts, but ghosts don’t affect photons. Either that, or ghosts do affect photons, which means we should be able to detect ghosts’ presence by shooting photons at their bodies—which is a sciencey way of saying “we can see them”.

If ghosts can see photons but photons can’t see ghosts, then that means photons can create some sort of motion inside ghosts, but that motion doesn’t come from the photons. In other words, energy is coming from nothing. The fact that ghosts can move—and knock over objects—without any external energy source also indicates that ghosts can create energy from nothing. The Law of Conservation of Energy does not apply to ghosts.

That gives us a testable hypothesis. If ghosts can create energy out of nothing, then we should be able to detect that energy. It’s a bit tricky to test, but what we can do is put a haunted house into some sort of sealed chamber where we precisely measure all the energy that goes in and comes out. If ghosts can create energy, then we should see more energy coming out than going in.

I’ve come up with two testable hypotheses so far. Can you think of any others?

What does it mean to be open-minded?

On a few occasions, I have been accused of being “closed-minded” for denying that ghosts could be real. But what is open-mindedness? The sort of open-mindedness I care about entails pursuing the implications of a belief.

Suppose I were to entertain the possibility that ghosts were real. What would that imply about the rest of my beliefs? What would I need to be wrong about? Those are the questions I wanted to answer with this essay.

The fervent ghost-believers I met never seemed to have much curiosity about what the existence of ghosts might imply. How would ghosts be intangible, but without sinking to the center of the earth? If they’re intangible, how can they knock things over? If people sometimes observe ghosts in their haunted bedrooms, then it should be possible to observe ghosts in a lab experiment, right? How would you set up that experiment?

Open-mindedness is about being receptive to different ideas. An important component of receptiveness is curiosity: if this were true, what else might be true as a consequence? In my experience, people who accuse me of being closed-minded aren’t curious about what their beliefs imply. If you do a curious investigation of ghosts, like I tried to do above, you can end up in some interesting places.

In Defense of the NCIS Two-People-One-Keyboard Scene

Fri, 14 Nov 2025 00:00:00 -0800

(Here is the same clip in HD, but that 2010 YouTube vibe is part of the fun)

This clip is in the running for most-mocked scene of all time, but I think it’s good, actually.

First, let’s get some things out of the way:

The writers of NCIS know how keyboards work. (They probably used keyboards to write this scene, even.)
The director of this episode knows how keyboards work.
I’m going to go out on a limb and say >90% of this show’s audience knows how keyboards work.

This scene was not written this way because the writers think their audience is dumb and doesn’t know how a keyboard works. It was written this way because of the Rule of Cool.

The Rule of Cool states: an audience’s willingness to suspend disbelief is proportional to how cool a scene is.

(Is this scene actually cool? Well, no, not really. But the relevant question is, does the target audience think it’s cool?)

(Full disclosure: I only said it’s uncool because I don’t want to sound uncool, but honestly I do think it’s kind of cool. Come on, it’s at least a little bit cool, right?)

source: xkcd

In Firefly when River Tam—a tiny woman who doesn’t even exercise—beats up a bunch of guys in a bar, does that make sense? Is that how physics works? No. But it’s cool, so people are okay with it.

Why do the space fighters in Star Wars pretend to be airplanes and use tactics that are nonsensical in space? Because it looks cool, that’s why. And why do lightsaber duelists, and most sword fighters in most movies for that matter, try to hit their opponents’ swords instead of going for a killing blow? Because it looks cooler than a real duel.

I think you can reasonably object to the NCIS scene by saying that two people typing on one keyboard to stop a hacker is not cool. Which is understandable. But if you’re going to object to this scene by saying that’s not how keyboards work, then you’re also not allowed to like Firefly or Star Wars or any movie involving swords or time travel or space ships or any sci-fi or fantasy or almost any movie involving guns or explosions or physics or even dialogue for that matter (ever notice how film characters never stutter or slur their words unless doing so is specifically relevant to the plot? so unrealistic!).

That all being said, I have a big problem with CSI’s “I’ll create a GUI interface using Visual Basic, see if I can track an IP address.” My problem isn’t that it doesn’t make sense. My problem is that it doesn’t sound cool, and it would’ve been so easy to write a cooler line. The word “Basic” doesn’t make you sound like a top hacker. And “GUI” is one of the least-cool-sounding words possible.

I propose a small modification:

I’ll create a kernel interface using C++, see if I can track an IP address.

The word “kernel” sounds cool. C++ sounds a lot cooler than Visual Basic. (If they want to go for the extra nonsense factor, they could say “C+” instead, which is something I’ve heard real people say in real life.¹) And my version of the line even kinda makes sense—realistically I don’t think you’d interface with your kernel to track an IP address, but those words mean something and you could do it if you really wanted to.

Notes

For the non-programmers reading this: There is no such thing as C+. There is only C and C++. (And also B and C# and D and F#, but no A.) ↩

Epistemic Spot Check: Expected Value of Donating to Alex Bores's Congressional Campaign

Thu, 13 Nov 2025 00:00:00 -0800

Political advocacy is an important lever for reducing existential risk. One way to make political change happen is to support candidates for Congress.

In October, Eric Neyman wrote Consider donating to Alex Bores, author of the RAISE Act. He created a cost-effectiveness analysis to estimate how donations to Bores’s campaign change his probability of winning the election. It’s excellent that he did that—it’s exactly the sort of thing that we need people to be doing.

We also need more people to check other people’s cost-effectiveness estimates. To that end, in this post I will check Eric’s work.

I’m not going to talk about who Alex Bores is, why you might want to donate to his campaign, or who might not want to donate. For that, see Eric’s post.

Contents
Model outline
Input parameters
Sensitivity analysis
Cost to shift votes by one percentage point
The model’s output isn’t what we care about
Notes

Model outline

The basic structure of Eric’s model:

Donations let the campaign spend more money on advertising, which increases how many votes they will get.
The election has some probability of being close.
If the election is close, then the expected value of votes is approximately linear.
If the election is not close, then marginal votes don’t matter at all.
Therefore, the expected value of donations is the product of three numbers:
- probability that the election is close
- number of votes to swing the election if it’s close
- cost to change one vote

The model specifically looks at the primary for New York’s 12th Congressional district. It doesn’t look at the general election because the district is deep blue and whoever wins the Democratic primary will almost certainly win the election.

I reproduced Eric’s model using Squiggle.

Before getting into the numbers, my take on this model is that it’s very reasonable. Some simplifying assumptions had to be made to make the model tractable, and I fully agree with all of Eric’s choices in that regard. When reproducing the model, I only make one small change that (I think) didn’t affect the final numbers at all.¹

Some simplifying assumptions that the model makes:

Marginal campaign spending only matters if the election is close—you can’t bridge a large vote gap by throwing money at the election.
If the election is close, then spending has linear cost-effectiveness.
Campaign donations only matter insofar as they change election outcomes. Ignore any second-order effects (e.g. signaling that donors care about AI safety).

My only high-level critique is that the original model used point estimates instead of credence intervals for the input parameters. For my Squiggle version, I converted most inputs into credence intervals using my own judgment about each parameter’s uncertainty.

(To be fair, doing a cost-effectiveness estimate with credence intervals is a lot more work if you’re not using a tool like Squiggle.)

Eric’s model has seven input parameters:

campaign spending (dollars) per vote
voter turnout
probability distribution of the margin of victory (which is used to estimate the probability that the election is close)
probability that your candidate (in this case, Alex Bores) is in the top two
probability that your candidate is on the losing side of the top two (because if your candidate would win anyway, marginal votes don’t help)
discount due to the possibility that additional fundraising could induce the opposing candidate to raise more money
multiplier due to the fact that early fundraising consolidates party support

(Eric talked about all of these parameters in more detail in his post, although they were split across a few sections.)

I will go through the values Eric gave for each of these parameters and if I have disagreements. Then I will do a sensitivity analysis.

Input parameters

Campaign spending per vote

Eric Neyman assumed a typical campaign costs $100 per vote based mainly on “numbers thrown around casually by experts”, then multiplied by 3 because New York has higher costs than average.

I spent 15 minutes looking for literature and I found:

Le et al. (2024)² reviews the literature. Most papers it reviewed didn’t give direct dollar-per-vote estimates, but estimates could probably be derived by going through the data from each paper. I’m not going to do that, but it’s a feasible and well-scoped project if anyone else wants to do it.
Bombardini & Trebbi (2007)³ estimated $145 per vote, but this study looked at elections from 1990–2000 so it doesn’t directly apply to 2025.⁴
Gerber (2004)⁵ out of Poverty Action Lab⁶ sent out randomized campaign mailings and found an expected one vote change per 12 households. I don’t know the all-things-considered cost to send campaign mail, but surely it’s not more than a few dollars, so this implies a very low cost (implausibly low, even).

Given the information I found Eric’s guess of $100 for the average election seems reasonable to me. Raising this to $300 for New York elections sounds about right to me. But I used a wide credence interval, with my 75th percentile estimate being 10x higher than my 25th percentile.

I believe it would be possible to come up with a more confident estimate with another 5–10 hours of work. If I wanted to improve this cost-effectiveness estimate, that’s where I’d start.

Voter turnout

According to Ballotpedia, the New York 12th District primary elections had about 90,000 voters in 2020, 2022, and 2024. So 90,000 is a reasonable estimate for voter turnout.⁷

Margin of victory

Eric modeled the margin of victory as following a uniform distribution from 0-30%, on the assumption that near the beginning of a campaign, it’s very hard to predict how close an election will be. I think that’s reasonable and that’s how I would have done it.

(A 30% margin means that, e.g., the top candidate gets 55% of the vote and the #2 candidate gets 25%.)

Eric described a second model where he treated candidates’ votes as following a Dirichlet distribution. This alternative model got approximately the same answer. I didn’t attempt to replicate it; I agree that it’s more accurate to reality, but I don’t think a Dirichlet distribution adds enough value to justify its complexity, so I just modeled the distribution as uniform.

Probability that your candidate is in the top two

There are currently three candidates in the race; there are two spots in the top two; therefore there’s a 2/3 chance that Bores is in the top two. This is a very simple line of reasoning and I have no objection to it.

Probability that your candidate is on the losing side

If your candidate would win without any additional funding, then additional funding doesn’t help. Donations only matter if they would lose otherwise.

There’s a 50% chance that your candidate is on the losing side, conditional on the election being close.

Opposition fundraising discount

Eric applied a 10% discount based on the possibility that if Bores raises more funding than expected, then the AI anti-regulation super PAC will donate more money to his opposition. That discount seems too low to me, but I don’t have any evidence about what the right number would be. (My model still used a credence interval centered on a 10% discount.)

I think the probability that Bores-funding induces anti-Bores-funding is pretty high, but I also think super PAC spending is less valuable than individual-donor spending due to campaign funding restrictions (as I understand, super PACs can pay for ads, but they can’t directly advertise for or against particular candidates).

Early fundraising multiplier

Eric expects that early campaign fundraising consolidates party support—it makes it easier to get more endorsements, raise more money from funders who don’t want to back a losing candidate, etc. He estimates that early funding is 2x as valuable. I didn’t do any research on this, but 2x sounds reasonable to me. I converted Eric’s point estimate into the 50% credence interval [1.33, 3].

Sensitivity analysis

Four of the inputs have relatively narrow credence intervals: voter turnout, probability that your candidate is in the top two, and probability that your candidate is on the losing side.

Margin of victory is based on a coarse assumption of uniform probability, but I don’t think there’s much value in adding complexity to this parameter.

Two parameters, the opposition fundraising discount and the early fundraising multiplier, are completely made up. These are the #2 and #3 most important parameters (but not necessarily in that order). But I don’t actually think the credence intervals are that wide—I don’t think their 50% CIs span a factor of 10.

By far the most important parameter is the cost per vote changed. My 50% credence interval for this parameter does span a factor of 10.

That’s why I think the best way to improve this model would be to spend more time figuring out the cost per vote changed. The simple version is to come up with a more well-researched number for the cost-effectiveness of campaign spending. A more sophisticated implementation could attempt to model the rate of diminishing returns to spending and apply that to where the Bores campaign is at currently.

Cost to shift votes by one percentage point

Eric gave a 50% credence intervals of “something like [$40k, $170k]” for donations made specifically on October 20. Based on the other things he said, I’d infer that his 50% CI for donations in 2025 (but after October 20) is [$49k, $210k]. To my knowledge, he did not explicitly model credence intervals for input parameters.

My replication finds a 50% CI of [$36k, $380k], which is notably wider, spanning 11x compared to Eric’s 4.3x.

The model’s output isn’t what we care about

This cost-effectiveness model estimates the expected cost to change the outcome of the election. That’s not what we ultimately care about. What we really care about is the expected cost to prevent AI extinction via donating to political candidates. That number is much harder to estimate. But it’s still nice to have a model that gets you part of the way there.

For a cost-effectiveness to go all the way, it would need to model how representatives affect what AI safety legislation gets passed, and how that legislation decreases x-risk. That’s a good question for another day.

Notes

Namely, Eric’s model estimated the probability that the vote margin falls within 1000 votes, and then used that plus the expected voter turnout to estimate the probability that the margin is within one percentage point. My reproduction used voter turnout to directly estimate the probability that the margin is within one percentage point. ↩
Le, T., Onur, I., Sarwar, R., & Yalcin, E. (2024). Money in Politics: How Does It Affect Election Outcomes?. ↩
Bombardini, M., & Trebbi, F. (2007). Votes or Money? Theory and Evidence from the US Congress.. ↩
Eric also notes that this study looked at general elections, not primaries, which are probably more expensive to influence because there are relatively fewer undecided voters. ↩
Gerber, A. S. (2004). Does Campaign Spending Work?. ↩
Without having carefully read the paper, I’m more inclined to trust the methodology if it’s coming from Poverty Action Lab than if it’s coming from some author I’ve never heard of. ↩
One thing that puzzles me is that the 2018 turnout was only 45,000, and the 2016 turnout was 17,000 (!). I don’t know why voter turnout changed so much in only four years, and then barely changed for the subsequent four years. I thought perhaps it’s because New York changed its districts, but the last redistricting was in 2012. So I have no idea what caused this sudden change in turnout, and I can’t rule out that it won’t happen again for the 2026 election. ↩

Ideas Too Short for Essays, Part 2

Wed, 12 Nov 2025 00:00:00 -0800

Nearly nine years after part 1, I bring three new short ideas.

Keep in mind that scientific fraud happens sometimes
Clichés are good, actually
You must put unnecessary decoration on your useful items, or else you’re a weirdo

Keep in mind that scientific fraud happens sometimes

Scientific misconduct is not rare. Even if a study uses a good experimental design, even if it has a large sample size, even if it has robust methodology, the results might still be wrong simply because the authors committed fraud. We should keep that in mind when reading scientific research.

For that reason (among others), I’m not fully convinced by any one study, no matter how strong it looks.

To reduce the chance of being bamboozled by fraudulent research:

Prefer studies that have been replicated by independent teams.
Prefer studies that make their data public.

Clichés are good, actually

Using unique phrasing keeps your writing fresh. It forces the reader to think a little harder about what you’re saying. It keeps the reader on their toes and makes them pay attention. Often that’s what you want.

Sometimes you want the opposite. Using a cliché signals to the reader: “My meaning is exactly what you think it is.” There is no wondering about your intention. Your meaning sinks into the reader’s mind like a hot knife through butter.¹ (Okay, that was not a good example of a situation where using a cliché is helpful.)

Beyond clichés, there are times when you want to use predictable language, and you don’t want to try to be too interesting. Using predictable language communicates that the thing you’re trying to say is predictable. It makes the language easier to parse; the reader doesn’t have to spend any time interpreting your meaning.

You must put unnecessary decoration on your useful items, or else you’re a weirdo

I used to have blank walls with no decorations. People thought this was weird.

I used to sleep on a mattress on the floor with no bed frame. People thought this was weird. Why? A mattress works perfectly fine without a bed frame.

Eventually I bought a bed frame and actually I think it was smart to buy one because now I can store stuff under it. But I still don’t care that much about wall decorations, I just got some so I could pretend to be normal.

You’re also supposed to have useless uncomfortable pillows (a.k.a. “throw pillows”) on your couch. I have so far resisted buying any throw pillows.

This is one of those mental differences between me and other people. I can’t fathom why people insist on adding superfluous decorations to things, and other people can’t fathom why my tastes are so dull.

Really, it’s not that my tastes are dull. I think decorated walls look better than blank walls if the decoration is good. It’s more that I’m easily distracted by certain kinds of visuals.

I used to have a fun desktop wallpaper. But I found it too distracting to see fun art behind the application window where I was trying to work. Now my computer’s wallpaper is pure black.

Notes

I was tempted to write “like a hot knife through Vegan Butter Alternative” because I don’t eat butter. But then I wouldn’t be following my own advice, would I? ↩

Are Groot and Baby Groot the Same Person?

Tue, 11 Nov 2025 00:00:00 -0800

This post contains spoilers for Guardians of the Galaxy.

At the end of Guardians of the Galaxy, Groot—a sapient tree with a three-word vocabulary—dies. They take a splinter from his…trunk, I guess?…and put it in a pot, from which springs Baby Groot.

There was a debate among fans as to whether Baby Groot is Groot regenerated, or if Baby Groot is an entirely new person. If I may weigh in to this debate in 2025: the answer is that it’s unanswerable.

The issue is that personal identity is not clearly defined in edge cases, so we can’t say whether Groot and Baby Groot are the same person.

In some cases, personal identity is unambiguous. For example, I am the same person as the Michael Dickens of 2010. There are a few reasons why I believe this:

My name is Michael Dickens. His name is Michael Dickens.
It’s possible to trace a physical lineage from me to him where today’s Michael is made up of almost all the same molecules as yesterday’s Michael and looks almost identical.
I have memories of 2010 Michael, and I have memories of his memories.
My personality and interests are very similar to his.
He and I have the same DNA (probably, I haven’t actually checked).

Am I the same person as Sean Connery? Definitely not, because:

My name is Michael Dickens. His name was Sean Connery.
There is not much overlap in the molecules that make up my body and that made up the body of Sean Connery.
Sean Connery has starred in many films, including portraying the original James Bond. I’ve never played James Bond in a film.
I don’t have any memory of having ever been Sean Connery, and I’m pretty sure he has no memory of having ever been me.
I don’t know a lot about Sean Connery’s personality, but I think it’s pretty different from mine.

I am the same person as the Michael Dickens of 2010, because we line up on approximately every measure of personal identity. But I am not the same person as Sean Connery, because we don’t line up on any such measures.

What happens when we try to compare Groot and Baby Groot in this way? When we start asking questions about their identities, we don’t get a consistent answer.

Can you trace a physical lineage between them? Yes—Baby Groot grew out of a splinter that came from Groot’s body. But Baby Groot’s body is mostly new. I don’t know how groot physiology works, but it seems that Groot dies when his head is destroyed, suggesting he has some sort of brain; and Baby Groot has a totally distinct brain. But perhaps groots have some sort of distributed neural system, where the splinter that contained Baby Groot contained a piece of groot-brain.
As portrayed in Guardians of the Galaxy 2 and in Avengers: Infinity War, Baby Groot seems to have no memory of having previously been Groot.
The two characters have very different personalities.
Do Groot and Baby Groot have the same DNA? I don’t think there’s a canon answer, but my guess would be yes.
What is Baby Groot’s name? He makes it pretty clear that his name is Groot. (Which is also Groot’s name!)

What is the most important factor for defining personal identity? If it’s memory, then Groot and Baby Groot are different. If it’s direct physical lineage, then it’s actually unclear because it depends on how much of a lineage you need. If the splinter that grew Baby Groot is part of the “essence” of Groot, then you could use that to argue that they’re the same person.

But if Alice thinks personal identity is about memory, and Bob thinks it’s about physical continuity, then Alice will think Baby Groot is a new person, and Bob will think Baby Groot is Groot. They won’t be able to agree unless they can reconcile their definitions of personal identity.

Alice and Bob disagree about Baby Groot’s identity, but they don’t disagree about any concrete facts about the universe. They agree that Baby Groot doesn’t have the same memory, and they agree that there was some physical continuity between the two beings (or, according to Bob, the one being).

In this kind of situation, I prefer to say that the answer doesn’t matter—it’s a question of how you define the words, not a question about reality.

A Thesis Regarding The Impossibility Of Giving Accurate Time Estimates, Presented As An Experiment On Form In Which The Essay Solely Consists Of A Title; In Which The Thesis States That, If Task Times Follow A Pareto Distribution (With The Right Parameters), Then An Unknown Task Takes Infinite Time In Expectation; And Therefore, In The General Case, You Cannot Provide An Accurate Time Estimate Because Any Finite Estimate Provided Will Not Capture The Expected Value; And, More Precisely, Every Estimate Will Be An Underestimate, Because Every Number Is Smaller Than Infinity; And This Matches With The General Observation That, When People Estimate Task Times, They Usually Underestimate The True Time; However, In Opposition To This Thesis Are At Least Two Observations; First, That Even If Tasks Take Infinite Time In Expectation, The Median Task Time Is Finite, And An Infinite-Expected-Value Task-Time Distribution Does Not Preclude The Possibility That Time Estimates Can Overestimate As Often As They Underestimate, But People Fail To Do This; Second, That Certain Known Biases That Result In People Underestimating The Difficulty Of Tasks, Such As Envisioning The Best-Case Scenario Rather Than The Average Case; However, In Defense Of The Original Thesis, Optimism Bias And The Pareto-Distributed Problem Space May Be Two Perspectives On The Same Phenomenon; But Even If We Reconcile The Second Concern With The Thesis, We Are Still Left With The First Concern, In Which An Unbiased Estimate Of The Median Time Should Still Be Possible, But People Are Overly Optimistic About Median Task Times; Thus, Ultimately Concluding That The Thesis Of This Essay--Or, More Accurately, The Thesis Of This Title--Is A Faulty Explanation Of People's General Inability To Provide Accurate Time Estimates; Then Following Up This Thesis With The Additional Observation That We Can Model Tasks As Turing Machines; And The Halting Problem States That It Is Impossible In General To Say Whether A Turing Machine Will Halt, And As A Corollary, It Is Impossible In General To Predict How Long A Turing Machine Will Run For Even If It Does Halt; So Perhaps The Halting Problem Means That We Cannot Make Accurate Time Estimates In General; However, It Is Not Clear That The Sorts Of Tasks That Human Beings Estimate Are Sufficiently General For This Concern To Apply, And Indeed It Seems Not To Apply Because Some Subset Of People Do In Fact Succeed At Making Unbiased Time Estimates In At Least Some Situations, At Least Where 'Unbiased' Is Defined Relative To The Median Rather Than The Mean; It Is Difficult To Say In Which Real-Life Situations The Halting Problem Is Relevant Because It Is Not Feasible To Construct A Formal Mathematical Proof For Realistic Real-Life Situations Because This Would Require Creating A Sophisticated Model In Which The State Of The Universe Is Translated To A Turing Machine, Which Would Be An Extremely Large Turing Machine And Probably Not Feasible To Reason About; Leading To The Conclusion That This Essay's Speculation Led Nowhere

Mon, 10 Nov 2025 00:00:00 -0800

Upside Volatility Is Bad

Sun, 09 Nov 2025 00:00:00 -0800

Investors often say that standard deviation is a bad way to measure investment risk because it penalizes upside volatility as well as downside. I agree that standard deviation isn’t a great measure of risk, but that’s not the reason. A good risk measure should penalize upside volatility, because upside volatility is bad.

A sure-thing return is better than a volatile return, even if the volatile return is guaranteed to be positive.

I will first explain my reasoning without using too much math, and then provide a more rigorous explanation using math in the next section.

As an illustration, suppose there’s some investment that only ever produces positive returns, and the distribution of outcomes looks like this:

(For those who want to know, that’s a gamma distribution with shape 0.1 and scale 2.)

This investment has plenty of upside volatility, but no downside volatility—it can never earn a negative return.

The investment has an expected return of 5%. But if you buy it, most of the time you will earn less than 5%. If you’re investing to prepare for your future, which would you rather have: a guaranteed 5% return? Or a volatile return with an average of 5%, but where you probably end up getting less than that?

With the guaranteed return, you’re guaranteed to be set for retirement as long as you put enough money into savings. With the volatile investment, even though you know you won’t lose money, you’re still not sure how much you’ll end up with.

Mathy explanation

Suppose Alice is an investor with logarithmic utility of money, which is a classic risk-averse utility function. I generated one million sample outcomes using our upside-volatile distribution and found that Alice’s expected utility was 0.12. (0.12 of what? 0.12 utility. It doesn’t mean anything concrete; it’s just a number.)

Alice has the opportunity to buy a safe investment with the same expected return, but zero volatility. The guaranteed investment has 0.18 utility for Alice—considerably higher than the volatile investment, even though she has no risk of losing money.

Bob is twice as risk-averse as Alice.¹ His utility for the guaranteed investment is 0.17, and his expected utility for volatile asset is 0.09. Like Alice, he prefers the sure thing. For him, the sure thing is nearly twice as good.

I believe Bob’s utility function is more representative of a normal person’s. So for a normal person, the sure thing is much better than the volatile investment, even though the volatility is all upside.

Skewness still matters

I’m not saying standard deviation is a perfect measure of risk, because it’s definitely not.

Imagine you have a choice between two investments that have expected returns distributed like this:

These distributions have the same standard deviation. But the blue distribution is still preferable to the orange one, because the orange one has a much bigger risk of losing money. It’s symmetric, whereas the blue distribution is right-skewed. Just looking at standard deviation doesn’t capture that.

Or compare these two distributions:

Both distributions have the same mean and standard deviation, but the orange one looks horrible. I would definitely not want to invest in the orange one. The left-skewed distribution looks much less appealing than the right-skewed one.

Yes, upside volatility is bad, but downside volatility is worse. A guaranteed constant return is better than an always-positive but uncertain return, which in turn is better than an uncertain return that might be negative. (Assuming, of course, that all three have the same expected return.)

Notes

By which I mean he has a relative risk aversion coefficient of 2, so his utility function of wealth is $U(w) = 1 - \frac{1}{w}$. ↩

Writing Your Representatives: A Cost-Effective and Neglected Intervention

Sat, 08 Nov 2025 00:00:00 -0800

Is it a good use of time to call or write your representatives to advocate for issues you care about? I did some research, and my current (weakly-to-moderately-held) belief is that messaging campaigns are very cost-effective.

In this post:

I look at evidence from randomized experiments, surveys of legislators’ opinions, and observational evidence. All lines of evidence suggest that messaging campaigns are effective, but none of the evidence is strong.
I write an estimate of how many messages it takes to get a bill to pass in expectation, and how much that costs. According to my model, changing a vote outcome takes 17,000 messages for the Michigan state legislature and 2.2 million messages for US Congress.
I provide links to resources on how to participate in messaging campaigns for animal welfare, AI safety, and global poverty.

Cross-posted to the Effective Altruism Forum.

Contents
Evidence from randomized experiments
Evidence from surveys
Observational evidence
Theoretical argument
State vs. federal representatives
Cost-effectiveness estimate
So, are messaging campaigns cost-effective?
How to participate in messaging campaigns
Notes

Evidence from randomized experiments

There are two randomized controlled trials on messaging campaigns targeted at legislators: Bergan (2009)¹ and Bergan & Cole (2014)². These studies randomly assigned state legislators to either receive or not receive messages from volunteers advocating in favor of an upcoming bill, and then looked at how many legislators voted for the bill depending on whether they received messages or not.

Both studies found statistically significant differences. Bergan (2009) found that the messaging campaign increased positive votes by 20 percentage points, and Bergan & Cole (2014) found a 12 percentage point improvement.

This table summarizes key facts about the studies:

	State	Medium	Avg # Messages	Change
Bergan (2009)	New Hampshire	email	3	20%pp
Bergan & Cole (2014)	Michigan	phone	22	12%pp

Bergan & Cole (2014) had legislators in the experimental group receive either 22, 33, or 65 calls. The study was underpowered to detect differences between those numbers, but the 65-calls group showed a (non-significantly) weaker effect than 22 or 33, which hints that the number of calls doesn’t matter beyond a certain point.

Evidence from surveys

A survey³ by the Congressional Management Foundation asked US Congress senior staffers how much weight they give to different forms of communication:

Respondents overwhelmingly say they give influence to constituents, although I’m not sure how seriously to take this because there’s a strong social desirability bias at play.

But insofar as we can take these results seriously:

In-person visits are better than individualized messages.
Individualized messages from constituents are more influential than lobbyists. (I am somewhat skeptical of this.)
Lobbyists are more influential than form messages.
Most respondents still give “some influence” to form messages.
There isn’t much difference between postal letters, email, and phone calls, although phone calls were the worst of the three. (This is good news for me as someone who’s allergic to making phone calls.)

An article from the OpenGov Foundation has a more qualitative perspective, with quotes from legislators about what kinds of advocacy they care most about.

When I’ve talked to lobbyists, they told me that policy-makers pay more attention to them than to constituents. These surveys by academics/think tanks say the opposite. Both of these pieces of evidence are contaminated by the fact that policy-makers are going to tell people what they want to hear. So ultimately you have to just decide who you believe more.

Observational evidence

One way to look at the question is to measure how well politicians’ votes align with public opinion vs. interest groups. That tells us something about how much politicians pay attention to the public compared to lobbyists, although this isn’t great evidence because politicians might vote one way or another for many reasons. And whether politicians align with public opinion doesn’t necessarily tell us how well messaging campaigns work, because there aren’t messaging campaigns on every issue.

And, the question is muddied by the fact that there can be interest groups on both sides of an issue, and possibly even public messaging campaigns on both sides.

As with most fields in social science, observational evidence is much easier to find than experimental evidence, so there are many research papers on this question. And because observational evidence is weaker than experimental evidence, I spent less time on it.

A systematic review by Elkjær & Klitgaard (2021)⁴ found very different answers across studies. Some studies found that public opinion mattered a great deal; others found that public opinion mattered far less than elite opinion or interest groups. Answers varied depending on how each study approached the problem and what statistical model they used.

I haven’t dug enough into the research to say whether some studies’ methodologies are better than others—it may be that some methodologies don’t make sense, and once you eliminate those, there is a single clear answer. But based on my cursory review, it looks to me like the observational evidence is mixed.

Theoretical argument

There is a simple theoretical reason to expect messaging campaigns to work well:

A representative’s job is to do what their constituents want.
If you tell them what you want, that increases the chances that they’ll do it.

And an argument for cost-effectiveness: most people don’t talk to their representatives, so if you do, you can have a big impact.

State vs. federal representatives

I’m only going to talk about the United States because I don’t know much about other governments. But my guess is that messaging campaigns should work roughly as well in any representative democracy as they do in America.

The two randomized experiments looked at vote outcomes from state representatives in a medium-sized state (Michigan) and a small state (New Hampshire). Federal representatives are representing many more people and therefore get more mail.

How much more mail? I don’t know. I couldn’t find data on the volume of mail received by state representatives. The fact that double-digit percentages of representatives changed their votes after receiving 22 phone calls (in Michigan) or three (!) emails (in New Hampshire⁵) suggests that they don’t receive many messages.

(Michigan has a population of 10 million and New Hampshire has 1.4 million, which is roughly consistent with the 7x difference in the number of messages sent for the respective advocacy campaigns.)

US Congress members, on the other hand, typically received 1000–1500 contacts per week in 2013 (Abernathy (2015)⁶):

They receive perhaps 3000 contacts per week today, although I couldn’t find a primary source.⁷

Cost-effectiveness estimate

I created a Squiggle model to estimate the cost-effectiveness of state and federal messaging campaigns. The model itself has documentation explaining how it works. I won’t explain every detail in this post—you can click through to the model if you’re interested—but I’ll give an overview of how it works.

Bergan (2009)¹ and Bergan & Cole (2014)² randomly assigned state legislators to receive messages. They found vote shifts of 20 and 12 percentage points, respectively. My model uses smaller numbers to be conservative.
Using data pulled from Congressional records, I estimated what proportion of vote outcomes could be flipped by shifting votes by N percentage points.
Calculate [percentage vote change per message] * [number of legislators] * [probability of outcome change for every 1% vote change] to get the expected probability of changing a vote outcome per message sent.

That’s just an outline. My model includes a lot of assumptions, and you probably disagree with some of them; if you’re opinionated, you should open the model and change the numbers as you see fit.

Then to get the cost to change a federal vote outcome, I scaled up based on the population difference between a medium-sized state and the United States as a whole. I added a 2x multiplier to adjust for the fact that US Congress is more salient and probably gets more messages per capita than state legislatures (as well as more advocacy via other vectors).

Multiplying all these factors together, my model came up with these results:

Changing an outcome in the Michigan state legislature (taken as a representative medium-sized state) requires a median of 17,000 messages (mean 15,000⁸; 90% credence interval 6200 to 130,000).⁹
Changing an outcome in US Congress requires a median of 2.2 million messages (mean 1.9 million; 90% credence interval 810,000 to 17 million).

Of course, that doesn’t mean you can brute-force your way into changing an outcome by running a giant multi-million-person messaging campaign. The model only applies to normal-sized campaigns. If you send, say, 22,000 messages, then—according to this model—you have a 1% chance of changing the outcome of a vote in Congress.

We can also calculate cost-effectiveness by assigning a monetary value to the time spent calling or writing letters. When I plugged in some best-guess numbers, I came up with:

The median cost to change an outcome in the Michigan state legislature is $440,000 (mean $130,000; 90% credence interval $31,000 to $3.8 million).
The median cost to change an outcome in US Congress is $58 million (mean $17 million; 90% credence interval $4 million to $500 million).
The median cost to change an outcome in California (the largest US state) is $3.1 million (mean $930,000; 90% credence interval $220,000 to $27 million).

These numbers depend a lot on the value of volunteers’ time. My model assumed that the volunteers are people like you, the reader of this post. Most of you probably have higher incomes than average and donate a lot more money to charity than average.

My model assumes that every letter is personally written by the sender. That might not be right, because the letter-writers Bergan (2009) were probably mostly sending form letters (the paper did not specify), which means the cost-effectiveness numbers from Bergan (2009) are for form letters, not for customized ones.

You could decrease the time requirement by ~10x by sending form letters instead of personalized letters. I don’t know whether the result would ultimately be more or less cost-effective because form letters are also less impactful. My general guideline would be that it’s better to write your own letter if you’re up for it, but if not, sending a form letter is still worthwhile.

So, are messaging campaigns cost-effective?

Would I pay $58 million if that’s what it cost to pass a federal version of SB 53 or the RAISE act? I think I would. I think I’d rather spend $30 million on that than on marginal alignment research. But it’s not an obvious call and I can see arguments the other way.

$3.1 million to get a bill passed in California sounds like a great deal to me. California regulations matter less than US law, but not >10x less. Remembering, of course, that you can’t actually get a bill passed by throwing $3.1 million at a messaging campaign. But it seems like a great deal to spend a much smaller amount of money for an appropriately scaled-down impact.

Are messaging campaigns the best political intervention? I don’t know, probably not?

I haven’t made a similar effort to estimate the cost-effectiveness of other interventions. I found unusually good data on messaging campaigns, which is to say I found two experiments covering two small-to-medium state legislatures that studied only a single bill each. That’s not much to go on, but it’s better than the zero experimental studies that we often have.

It may be that it’s more cost-effective to support lobbying by a dedicated interest group with strong political connections. I spoke to one person who has done both messaging campaigns and lobbying who believes that the latter is better (under certain conditions).¹⁰ But the cost-effectiveness of lobbying is even harder to estimate than messaging campaigns.

The book Lobbying and Policy Change: Who Wins, Who Loses, and Why—which I summarized in my reading notes—found that neither PAC spending nor lobbying spending could predict political success in observational studies, although the authors expressed skepticism about this result.¹¹ Camp et al. (2024)¹² conducted four field experiments with real-world lobbyists and found that lobbyist outreach had no significant effect on legislators’ policy positions. This leaves me uncertain of what to believe, where some individuals who are involved in political advocacy believe lobbying is particularly effective, but externally-verifiable (but limited) evidence finds that it isn’t.¹³

(The cost-effectiveness of lobbying could be its own topic, but I’ll leave it there for now.)

Messaging campaigns look cost-effective relative to AI alignment research,¹⁴ but it’s harder to say how they compare to other types of advocacy. Separately, there’s the question of whether you, personally, should write letters to your representatives when an important issue comes up. In that case, I think the answer is a strong yes, as long as you have the spare time. If you’re limited on time, you can still sign your name on a pre-written letter, which only takes about two minutes.

How to participate in messaging campaigns

For practical guidance on how to talk to your representatives, see Talking to Congress: Can constituents contacting their legislator influence policy? That article was written by some people who, unlike me, have actually run messaging campaigns before.

Compassion in World Farming also has a guide to effective letter writing for farm animal welfare advocacy; the advice is relevant to any cause area.

I am not sure whether you should send a form letter or write out your own letter. I’m confident that personalized letters are more impactful, but they also take much longer, so it’s not clear that they’re more time-effective. I would probably suggest writing a personalized letter if you have time; but if you don’t, or if you’re not sure what to say, then sending a form letter is much better than nothing.

If you want to get involved in messaging campaigns, below are three lists of orgs who run campaigns in three effective altruist cause areas.

Animal welfare

Animal advocacy groups are well-versed in running public campaigns, and there are many ways to get involved.

ASPCA frequently runs messaging campaigns. Its Advocacy Center lets you filter by issue (farm animals, puppy mills, etc.) and it shows a list of relevant issues that match your criteria. For example, right now it has a page on the Farm Bill, explaining how the bill will negatively impact farm animals, and providing a form where you can send a letter to your legislator.
Mercy for Animals has a Lend Your Voice page. As of this writing, the page links to a message form where you can contact your representatives about the Industrial Agriculture Accountability Act.
Compassion in World Farming has an “Act Now” button on its website. The direct link is here, but I’m not sure if that link will still work a month from now; if it doesn’t, go to the home page and click the “Act Now” button.
You can sign up for The Humane League’s Fast Action Network, and you will get notified when there are actions you can take (which mostly means corporate campaigns, I’m not sure if they write letters to policy-makers).

A lot of animal advocacy orgs have useful resources; those were just a few of the ones I found.

AI safety

There is nothing particularly organized right now. I hope that there are better options in the future, but right now there is no dedicated “AI safety messaging campaign” newsletter or mailing list.

There are a few other kinds of resources, though:

PauseAI US has a newsletter that’s mostly not about messaging campaigns. The newsletter did make a call to action on the 10-year moratorium on AI regulation; if PauseAI US runs another messaging campaign, you will probably hear about it on the newsletter. PauseAI US also has a dedicated “contact-officials” channel on its Discord.
ControlAI has a Take Action page where you can send a message to your representatives, using either a form letter or a message you write yourself. The form letter broadly raises concern about AI existential risk, rather than being about any particular piece of legislation.
The book If Anyone Builds It, Everyone Dies has an associated web page where you can write a letter to your representatives (either pre-written or written by you). As with ControlAI’s page, the letter isn’t about any specific legislation.
PauseAI has an email builder for writing a customizable form letter.

Global poverty

I couldn’t find any orgs that run messaging campaigns focused specifically on cost-effective global poverty interventions (like the sort of thing GiveWell would recommend), but there are some orgs that focus on global poverty more broadly.

Partners in Health has an Advocacy page where you can write Congress to support funding for global health initiatives, or sign up to the PIH Action Network.
RESULTS has Action Alerts for writing letters to policy-makers and to newspapers.
Catholic Relief Services has a Take Action page that includes Congressional messaging campaigns among other things.

Notes

Bergan, D. E. (2009). Does Grassroots Lobbying Work?. doi: 10.1177/1532673x08326967 ↩ ↩²
Bergan, D. E., & Cole, R. T. (2014). Call Your Legislator: A Field Experimental Study of the Impact of a Constituency Mobilization Campaign on Legislative Voting. ↩ ↩²
Congressional Management Foundation (2011). Communicating with Congress: Perceptions of Citizen Advocacy on Capitol Hill. ↩
Elkjær, M. A., & Klitgaard, M. B. (2021). Economic inequality and political responsiveness: A systematic review. doi: 10.1017/S1537592721002188 ↩
When I first read the paper, I found this number to be shockingly low—how could three emails cause a 12 percentage point shift in votes? But it made more sense after I did the math and realized that each New Hampshire legislator only represents about 3,000 constituents. ↩
Abernathy, C. E. (2015). Legislative correspondence management practices: Congressional offices and the treatment of constituent opinion. ↩
Stowe, L. (2023). How Congressional Staffers Can Manage 81 Million Messages From Constituents.

Note: This article did not cite an original source for its numbers. The best original source I could find was Abernathy (2015)⁶, which quoted 1000–1500 messages per week. ↩
Actually that’s slightly wrong. This number is not the mean of messages-per-outcome, it’s the reciprocal of the mean of outcomes-per-message. When calculating expected utility, it makes more logical sense to put the benefit on the numerator and the cost on the denominator. But this produces a very small number that’s hard to read, so I inverted it.

If you calculate expected messages per outcome, the result is heavily penalized by the tail outcomes where changing the outcome ends up being much more expensive than expected. This produces an incorrect estimate of expected utility (the units of utility are outcomes-per-message, not messages-per-outcome). ↩
I debated whether the mean or median is the more relevant number here. Philosophically, the mean is what you care about. But it seems perverse that greater uncertainty about an intervention increases how appealing it looks. Using the median instead of the mean is probably the wrong way to solve this problem, but it’s a first attempt. ↩
I also spoke to people who had done political advocacy and not messaging campaigns who claimed that lobbying is particularly effective. ↩
They hypothesized that perhaps spending by opposed interest groups cancel out, or that alliances between high-spending and low-spending interest groups create the illusion that spending doesn’t matter. ↩
Camp, M. J., Schwam-Baird, M., & Zelizer, A. (2024). The Limits of Lobbying: Null Effects from Four Field Experiments in Two State Legislatures. ↩
Under normal circumstances, I believe people over-rate personal experience, and I’m more inclined to trust the data. But in this case, most of the data comes from observational evidence which is easily confounded; I only cited one experimental study, and that study was small in scope—they only worked with three individual lobbyists. Given the weakness of the scientific evidence in this case, I don’t think it’s clearly more reliable than the contradictory anecdotes.

One limitation worth mentioning is that the experimental study tested the effect of lobbyists meeting policy-makers only one or two times. Conventional wisdom says that the value of lobbying mainly comes from establishing long-term relationships. ↩
I didn’t attempt to estimate the cost-effectiveness of AI alignment research. It just seems true to me that $58 million (ish) to pass a bill is worth more than $58 million of alignment research, at least on the margin. (If nobody were doing alignment research, perhaps I’d answer differently.) ↩

Things I Learned from College

Fri, 07 Nov 2025 00:00:00 -0800

(that I still remember a decade later)

Evolution on Earth

Fact 1: When foxes are bred to be more docile, their ears become floppy like dogs’ ears instead of pointy like wild foxes’.

Fact 2: Crows can learn to use a short stick to fetch a longer stick to fetch food.

The basic setup of the experiment is: There’s a box with some food at the bottom. The crow can’t reach the food. The crow has a short stick, but the stick isn’t long enough to reach the food, either.

There’s also a second box containing a long stick. The short stick is long enough to reach the long stick. Most crows figure out that they can use the short stick to fetch the long stick and then use the long stick to fetch the food.

If you add a third layer of indirection, where they have to use a short stick to fetch a medium stick and the medium stick to fetch a long stick and the long stick to fetch food, most crows don’t figure it out but a few of them do.

I wrote a rap song about this experiment, it used to be on YouTube but I think it’s gone now.

Physics in the 21st Century

Even before taking this class, I knew that there are four fundamental forces of the universe:

Gravity: Things go down. (Or, more accurately, all objects with mass pull toward each other.)
Electromagnetism: Many fundamental particles have a positive or negative charge. Oppositely-charged particles attract, and like charges repel; moving electrons creates electricity, and clusters of charged particles create magnetism (or something like that).
Strong force: Atomic nuclei are strongly held together even though the protons electromagnetically repel each other.
Weak force: ??? something about radioactive decay?

After having previously been confused about what the weak force was and trying to figure it out, this class finally got me to understand it. But since then I have forgotten the explanation and I’m confused again. I wish I could remember how the weak force works.

Intro Computer Science

In first semester computer science, I had been programming for longer than almost any of my classmates (~6 years), and I was the guy my classmates came to for help with CS assignments. By second semester, everyone had caught up to me and my 6-year lead didn’t matter.

Linear and Nonlinear Optimization

Duality: for every convex optimization problem, there is a dual problem that has the same solution.

The dual problem that sticks in my mind: given a set of possible investments, maximizing expected return subject to a given standard deviation is equivalent to minimizing standard deviation subject to a given expected return.

Philosophy of Mind

Philosophy papers routinely make arguments with glaringly obvious logical flaws, but they still get published somehow, and are considered important works worthy of teaching in a class.

I’m a bit conflicted on philosophy as an institution. On the one hand, it’s really hard, and people come up with lots of brilliant stuff that I never would have thought of. On the other hand, many “seminal” papers contain obvious fundamental flaws. I don’t mean I disagree with their conclusions, I mean they make logical arguments that are clearly not logically valid. Like, they have the general structure of “A implies B, A, therefore C” and I’m like…how did you not notice that this makes no sense? and how did the reviewers not notice either? and how did the professor not notice when deciding to assign this paper as reading?

At least that’s what I thought at the time. I don’t remember which papers we read, so I can’t go back and verify that they were indeed as flawed as I thought.

I learned basically nothing on the object level about theory of mind or theory of identity. I still believe all the same things I believed before taking this class. I guess I learned various insane things that some philosophers believe, but I don’t remember what most of those things are.

(“What Is It Like to Be a Bat?” is the best theory of mind paper I’ve ever read—it made me think about things in a new way—but I read it in high school, not college.)

Computer Networking

Fact 1: The web has four layers of transmission:

physical
IP (send/receive raw packets of data)
TCP (manage the transmission of packets)
HTTP (tell what types of packets to send/receive)

Fact 2: The Internet is not the same thing as The Web. Internet = IP, Web = HTTP. google.com and facebook.com are on the Web. Email is Internet, but not Web (unless you’re checking your email in a web browser). Multiplayer video games are on the Internet, but not the Web. The Internet dates back to the 1970s, but the Web didn’t start until 1993.

Intro Psychology

Psych textbooks and your psych professor will uncritically repeat claims that were found in a single study that didn’t replicate.

Machine Learning

Four facts:

You can solve a lot of problems by throwing a logistic regression at them.
I can vaguely explain what a support vector machine is.
I can vaguely explain what a convolutional neural network is.
I am not good at machine learning.

Sadly this is pretty much all I learned, even though I took four classes on machine learning. I remember various buzzwords like “softmax” and “one-hot” but I don’t remember what they mean.

(I managed to get an A- in one of those classes but I still didn’t really learn anything.)

Linguistics

Fact 1: Gricean maxims explain how statements can convey more information than they appear to.

The four Gricean maxims of cooperative conversation are:

Maxim of Quality: statements are true.
Maxim of Quantity: statements are as general as possible.
Maxim of Relevance: statements are relevant.
Maxim of Manner: statements are clear and orderly.

If I say “I have three carrots”, that implies that I don’t have four carrots. If I had four carrots, the Maxim of Quantity would require that I say I have four carrots.

If you ask “Can I eat that carrot?” and I respond by saying, “It’s not mine”, that implies that you cannot eat the carrot. By the Maxim of Relevance, my answer must be relevant to your question, so it can be taken to imply that, as the non-owner of the carrot, I do not have the authority to permit you to eat it.

Fact 2: A Speech Act is when you perform an act merely by stating that you are performing it. For example, “I apologize.” A statement that includes “hereby” is probably a speech act.

(“I declare bankruptcy!” is not a speech act.)

Statistics

I learned almost nothing from the two college statistics classes I took. Everything that I remember about statistics, I either learned on my own or learned from AP Statistics in high school—which was actually quite a useful class, maybe even the best class I took in high school!¹

My college statistics classes were mostly about the mechanics of how to hand-compute integrals of probability density functions, which I will never do in real life. Maybe I would’ve been better off taking a statistics-for-scientists class, but my major required me to take statistics-with-calculus, which is more about calculus than it is about statistics.

I do remember one fun fact: Var[X] = E[X^2] - E[X]^2. I’ve used that one a few times.

Algorithms

How to implement breadth-first search.

I had written graph algorithms in high school a few times, and I always used depth-first search because it was intuitive to me. I never knew how to implement breadth-first search (I’m not sure I even knew it existed) until I learned the algorithm in my algorithms class.

Improv

Fact 1: Peak age for improv skill is older than peak age for most skills. Improv actors don’t peak until their 40s or 50s.

Fact 2: Reincorporation—end your story by bringing back a story element from earlier that the audience probably forgot about.

Reincorporation is the secret to giving a comedic story a satisfying ending.

Now that I know about this concept, I see it show up a lot in comedy. Curb Your Enthusiasm is an excellent illustration of reincorporation, where most episodes weave three or four unrelated plot threads that all somehow come together at the end. Arguably the greatest reincorporation of all time is the Seinfeld episode “The Marine Biologist”.

Science has yet to discover whether crows can understand reincorporation.

Notes

My teacher spent a REALLY long time making sure everyone understood the correct definition of a p-value. ↩

Cash Back

Thu, 06 Nov 2025 00:00:00 -0800

When I was 18, my dad took me to the bank to get my first credit card. I had a conversation with the bank teller that went something like this:

Bank teller: This card gives 1% cash back.

Me: What does that mean?

Bank teller: It means when you spend money with the card, you get 1% cash back.

Me: But what does cash back mean, though?

Bank teller: It means you get cash back.

Me: …

The bank teller communicated poorly, and also I did not do a good job at articulating which part I was confused about. If I were that bank teller, here is what I would say to my 18-year old self:

I understand you to be asking two questions.

As you understand it, “cash” means “paper money”. And you are wondering how it is logistically possible for the bank to give you paper money when you use your credit card. Is a courier going to run to the store you’re at and deliver the cash? Surely that’s ridiculous?
This deal makes it sound like the bank is giving you free money. Why would they give you free money? How is that profitable for them?

The answer to the first question is that no, they are not going to give you paper money. They are going to deposit the 1% cash back into your bank account.

The answer to the second question: it is indeed profitable for them to pay you 1% of the value of every purchase you make. The reason is that the credit card company charges a fee to businesses (typically around 3%) whenever you buy something at that business. Most companies eat this cost by charging the same price to both cash and credit card users, which means effectively you get a discount by paying with a credit card. Businesses are willing to do this because they can attract more customers if they accept credit cards.

Credit card companies then take a portion of that ~3% fee and give some of it back to you as “cash back”. (Some cards also give you perks, like discounts on airline tickets.) You might ask, instead of giving you 1% cash back, why don’t they just make the prices be 1% lower? The answer is that it is a dumb psychological trick to make people think they’re getting a better deal. At least that’s part of the answer, it could also be because of logistical issues with prices being set by businesses vs. credit card companies in which businesses always pay the same rate to credit card companies, but credit card companies give better benefits to people who are a lower credit risk.

But it is genuinely 1% cheaper to buy things with a credit card than with cash, assuming you pay off your card balance each month before accruing any interest.

Also, 1% cash back isn’t even a good perk. You can get 2% cash back with the Citi Double Cash card. I’m guessing your credit rating isn’t good enough for that card yet, but you’re gonna apply for it in a few years. There are also lots of other cards with fancy perks, but you’re not gonna care about those perks so you should just go for the 2% cash back.

How Can I Not Know Whether I'm Having a Good Experience?

Wed, 05 Nov 2025 00:00:00 -0800

I’m playing Elden Ring. I’m fighting a difficult boss¹, and I’m getting kind of frustrated. I die again. I’m thinking about whether I want to keep playing. I don’t know. Am I having a good time? I can’t tell. How is it that I can’t tell?

Fundamentally, good experiences are good, and bad experiences are bad. But what if I don’t know whether I’m having a good experience? How is that possible? A good experience is good because it’s good for me. An experience lives inside me. But when I point my internal gaze directly at my experience, I can’t tell whether it’s good or bad. That seems impossible.

I’m not entirely sure what’s going on here, but I think an important component is that I’m not examining my experince during the game; I’m examining my experience while not playing the game. When I die to a boss and I spend a moment introspecting while I wait for the game to reload, I’m not fighting the boss at that moment. I’m looking at a loading screen while feeling frustrated.

In that moment, the only question I can answer is, “Am I having a good experience while staring at this loading screen?” If that’s the question, the answer is a pretty clear “no”. I don’t want to be staring at the loading screen while feeling frustrated. But that’s not the same as the question of whether I will be having a good time if I start the game again.

A second problem is that I am experiencing multiple things at the same time. I feel some frustration at my failure.² But I also feel some excitement. I feel some motivation to make progress. I know that I feel all those things; I know exactly what my frustration feels like, because I’m feeling it. The hard part is weighing them against each other. Is the combination of excitement and motivation enough to outweigh the frustration? Unlike each feeling on its own, the combined weighted experience is not a raw feeling in my gut, so there’s no principle that says I must be able to intuitively evaluate it.

And there is a third, even deeper problem: reflecting on an experience changes the experience.

During the boss fight, I can take a second to think about whether I’m having fun. But in that second, I’m not focused on the boss fight; I’m focused on introspecting on my experience. How I feel while I’m introspecting is not the same as how I feel while I’m engrossed in the game. Fundamentally, it is impossible for me to check how I feel while I’m engrossed, because then I wouldn’t be engrossed. (I am not the first person to make this observation, although perhaps I’m the first to apply it to Elden Ring boss fights.)

I have noticed that I’m more likely to be confused about my own experience if I’m tired. When I’m alert, most of the time I have no trouble knowing whether I want to keep doing what I’m doing, or do something else. But when I’m fatigued, I have a harder time feeling out which direction my motivations are pointing. Does that say something about how introspection works? It suggests to me that the process of aggregating and weighting the different aspects of my experience is a cognition-heavy operation.

To review, there are (at least) three reasons why I can’t tell whether my experience is good:

The experience I’m having at this moment is not the experience I want to introspect on.
I’m having multiple experiences simultaneously, and aggregating them is not a primitive operation that my brain can perform.
Introspecting causes my experience to change.

Notes

Dragonlord Placidusax, let’s say.

Malenia is harder, but for some reason I never got frustrated while fighting Malenia. ↩
Really, the thing I find frustrating isn’t failure, but lack of progress. If I die three times and do worse every time, I probably won’t feel great about that. ↩

Do Small Protests Work?

Tue, 04 Nov 2025 00:00:00 -0800

TLDR: The available evidence is weak. It looks like small protests may be effective at garnering support among the general public. Policy-makers appear to be more sensitive to protest size, and it’s not clear whether small protests have a positive or negative effect on their perception.

Previously, I reviewed evidence from natural experiments and concluded that protests work (credence: 90%).

My biggest outstanding concern is that all the protests I reviewed were nationwide, whereas the causes I care most about (AI safety, animal welfare) can only put together small protests. Based on the evidence, I’m pretty confident that large protests work. But what about small ones?

I can see arguments in both directions.

On the one hand, people are scope insensitive. I’m pretty sure that a 20,000-person protest is much less than twice as impactful as a 10,000-person protest. And this principle may extend down to protests that only include 10–20 people.

On the other hand, a large protest and a small protest may send different messages. People might see a small protest and think, “Why aren’t there more people here? This cause must not be very important.” So even if large protests work, it’s conceivable that small protests could backfire.

What does the scientific literature say about which of those ideas is correct?

Contents
Evidence from nationwide natural experiments
Direct evidence from lab experiments
Indirect evidence from lab experiments
Non-experimental evidence
Conclusion
Future work
Appendix: Table of papers from Orazani et al. (2021)
Notes

Evidence from nationwide natural experiments

Among the studies in my prior lit review, two studies modeled how voter outcomes varied based on the number of protesters in each county. The two studies found that each marginal protester increased vote share by 18.81 and 9.62 respectively (where vote share = number of votes adjusted to account for voter turnout¹).

Unfortunately, these studies both used linear models, which doesn’t help us. We want to know if there’s a non-linearity near zero—something that looks like this:

A linear model can’t tell you what the shape of the curve looks like, or whether it dips into the negative for sufficiently small protests.

(In theory, I could analyze the raw data myself, but that would be a lot of work.)

Direct evidence from lab experiments

To my knowledge, there are two experiments that directly tested whether the size of a protest affected people’s support for a cause.

Wouters & Walgrave (2017)² showed (fictitious) news articles to Belgian legislators. The news articles said either “There were about 500 participants which was much less than expected”, or “There were more than 5,000 participants which was more than expected.”³ The authors also altered three other independent variables, which they called Worthiness, Unity, and Commitment. Then they asked participants questions to judge how much they agreed with protesters (“position”) and whether they intended to take any actions to support protesters (“action”). Below I present the resulting regression coefficients and p-values.

	position	p-val	action	p-val
numbers	0.282	0.008	0.439	0.000
worthiness	0.381	0.000	0.116	0.297
unity	0.353	0.001	0.350	0.002
commitment	0.190	0.300	0.156	0.161

Numbers had biggest or near-biggest p-values out of the four variables, which suggests that legislators care a lot about the size of a protest. However, this study did not include a control group, so we don’t know whether smaller protests had a positive effect, a negative effect, or no effect.

Wouters (2019)⁴ conducted two similar studies, this time interviewing members of the general public rather than legislators. They described protest sizes the same way as Wouters & Walgrave (2017) (“500, less than expected” vs. “5,000, more than expected”), and again used four independent variables. Below are the regression coefficients and p-values from the two different studies, where the dependent variable was participants’ support for the cause.

	study 1	p-val	study 2	p-val
numbers	0.094	0.071	0.063	0.291
diversity	0.168	0.001	0.131	0.029
worthiness	0.607	0.000	1.127	0.000
unity	0.201	0.000	0.126	0.034

In this case, we find that numbers matter less than the other three factors.

Taken together, these two papers suggest:

Legislators care a lot about protest size. The general public maybe cares a bit, but not much.
Even (comparatively) small protests are effective at garnering support from the general public. It’s not clear whether they are effective for legislators.

(If small protests turned off the general public, then we would see that the “numbers” variable has good predictive power, but it doesn’t.)

Indirect evidence from lab experiments

Wouters & Walgrave (2017)² and Wouters (2019)⁴ were the only two papers I could find that directly tested the effect of protest size. But there could also be indirect evidence. I’m imagining something like this:

A lab experiment showed people either a new article about a protest, or a “control” news article. People who read about the protest were [more/less] supportive of the protesters’ cause.
The protest described in the article happened to be small.

That result would provide evidence about how small protests influence people.

To see if there was something like that, I looked through the studies cited by a meta-analysis by Orazani et al. (2021)⁵. I found two relevant papers: Thomas & Louis (2013)⁶ and Feinberg et al. (2017)⁷. (See Appendix for a list of every paper.)

Thomas & Louis (2013)⁶ did two experiments comparing participants’ reactions to news articles about violent vs. nonviolent protests. The two experiments were more or less the same, except that Experiment 1 covered fracking protests and Experiment 2 was about anti-whaling activism. Unfortunately the contents of the news articles are not publicly available and the corresponding author did not reply to my inquiry, so I don’t know how the protests were described in terms of size.

Feinberg et al. (2017)⁷ included three studies. Each study presented participants with an article or video about a different protest.

Study 1: The articles described a fictitious animal rights group. Protest size was reported in the articles as “about thirty people”.
Study 2: The articles described a Black Lives Matter march. The number of protesters was not specified in the articles.
Study 3: Participants were shown videos of Trump protests. One video showed a protest with roughly 70 participants⁸ but I don’t know how many protesters were in the other video.

In study 1, study participants reported relatively high support for protesters in the “Moderate” condition, in which a fictional animal rights group picketed a cosmetics company. (In the “Extreme” condition, the protesters broke into the building and freed animals.) However, there was no control group (!!), so we don’t know if reading about the protest caused support to go up, or if support would’ve been high anyway. The protest was described as having only thirty people, so this would’ve been useful evidence if they’d included a control group, but they didn’t.

One thing we can say about study 1 is that if small protests reduce support, then they don’t reduce support by as much as “extreme” protests do.

There is one additional paper, Bugden (2020)⁹, that was not included in the Orazani et al. meta-analysis.¹⁰ It showed participants articles in four conditions: a control,¹¹ peaceful protest, disruptive protest, and violent protest. The peaceful protest article (found in the supplement document) opened with:

On Thursday, thousands of protestors took to the streets as the state legislature prepares to vote on a climate change bill.

This protest was not small, so Bugden (2020) doesn’t provide relevant evidence.

Non-experimental evidence

An observational study by Ostarek et al. (2024)¹² studied the effect of a disruptive protest by the climate group Just Stop Oil in which protesters blockaded a highway. By running polls before and after, the researchers found that support for a more moderate climate group, Friends of the Earth, increased just after the Just Stop Oil protest.

The protest consisted of 45 people (source).

At first glance, this appears to indicate that small protests can be effective. But I’m not sure that’s an appropriate interpretation of the evidence, because:

It was an observational study, not an experiment or even a natural experiment.¹³
Other studies have found negative effects for disruptive protests.

Conclusion

None of the evidence I found was very good.

Here are my takeaways, but given the state of the evidence, I don’t have much confidence in them.

Large protests work better than small protests at garnering support among policy-makers. (credence: 80%)
The general public probably doesn’t greatly care about the size of a protest. (credence: 60%)
Small protests can probably be effective at garnering support among the general public. (credence: 60%)

Do small protests persuade the general public? Looks like yes (but, at the risk of repeating myself, the evidence was not strong).

Do small protests persuade policy-makers? I couldn’t find any evidence either way. (But the fact that I couldn’t find anything is weak evidence against.)

Future work

I see two obvious ways to learn more about how well small protests work. They’re out of scope for this post, but they wouldn’t be too hard.

Analyze the data collected in natural experiments and use a non-linear model to assess the effectiveness of small protests.
Run a new survey (on Mechanical Turk or similar) showing people small protests vs. large protests. vs. no protests and then ask them about their opinions on the protesters’ cause.

Appendix: Table of papers from Orazani et al. (2021)

Paper	Status
Thomas & Louis (2014)	included useful information
Orazani & Leidner (2018)	I couldn’t find the full text
Becker et al. (2011)	dependent variable was not relevant
Feinberg et al. (2017)	included useful information
Gutting (2017)	dependent variable was not relevant
Leggett (2010)	unpublished
Shuman et al.	unpublished

Notes

Mathematically, vote share per protester equals raw votes per protester divided by the proportion of residents who voted. ↩
Wouters, R., & Walgrave, S. (2017). Demonstrating Power. ↩ ↩²
I suspect that the phrases “less/more than expected” have a bigger effect on people’s perception than the numbers themselves. But this hypothesis hasn’t been tested. Some evidence for my hypothesis is that Madestam et al. (2013)¹⁴ (which I reviewed previously) found a strong county-level effect on protests, and the average protest size was 815 people per county, which is much closer to the “small” condition (500 people) than the “large” condition (5,000). So my guess is that 500 only sounds small because the article presented it as “less than expected”. However, the protests studied in Madestam et al. (2013) might differ in other meaningful ways. ↩
Wouters, R. (2019). The Persuasive Power of Protest. How Protest wins Public Support. ↩ ↩²
Orazani, N., Tabri, N., Wohl, M. J. A., & Leidner, B. (2021). Social movement strategy (nonviolent vs. violent) and the garnering of third‐party support: A meta‐analysis. doi: 10.1002/ejsp.2722 ↩
Thomas, E. F., & Louis, W. R. (2013). When Will Collective Action Be Effective? Violent and Non-Violent Protests Differentially Influence Perceptions of Legitimacy and Efficacy Among Sympathizers. ↩ ↩²
Feinberg, M., Willer, R., & Kovacheff, C. (2017). Extreme Protest Tactics Reduce Popular Support for Social Movements. ↩ ↩²
Source: I watched the video and counted how many people I could see. Some of the people were clearly bystanders, not protesters, but others were ambiguous so I’m not sure about the exact count. ↩
Bugden, D. (2020). Does Climate Protest Work? Partisanship, Protest, and Sentiment Pools. ↩
Even though Orazani et al. was published in 2021, its literature review was conducted in 2018. ↩
The control condition simply noted that protests exist without describing them at all, and asked participants if they supported the protesters’ cause. ↩
Ostarek, M., Simpson, B., Rogers, C., & Ozden, J. (2024). Radical climate protests linked to increases in public support for moderate organizations.

See also a less-technical 2022 preprint at https://www.socialchangelab.org/_files/ugd/503ba4_a184ae5bbce24c228d07eda25566dc13.pdf. ↩
I don’t see how the change wouldn’t be causal, but that could just be a failure of imagination on my part. ↩
Madestam, A., Shoag, D., Veuger, S., & Yanagizawa-Drott, D. (2013). Do Political Protests Matter? Evidence from the Tea Party Movement*. ↩

My Third Caffeine Self-Experiment

Mon, 03 Nov 2025 00:00:00 -0800

Last year I did a caffeine cycling self-experiment and I determined that I don’t get habituated to caffeine when I drink coffee three days a week. I did a follow-up experiment where I upgraded to four days a week (Mon/Wed/Fri/Sat) and I found that I still don’t get habituated.

For my current weekly routine, I have caffeine on Monday, Wednesday, Friday, and Saturday. Subjectively, I often feel low-energy on Saturdays. Is that because the caffeine I took on Friday is having an aftereffect that makes me more tired on Saturday?

When I ran my second experiment, I took caffeine four days, including the three-day stretch of Wednesday-Thursday-Friday. I found that my performance on a reaction time test was comparable between Wednesday and Friday. If my reaction time stayed the same after taking caffeine three days in a row, that’s evidence that I didn’t develop a tolerance over the course of those three days.

But if three days isn’t long enough for me to develop a tolerance, why is it that lately I feel tired on Saturdays, after taking caffeine for only two days in a row? Was the result from my last experiment incorrect?

So I decided to do another experiment to get more data.

This time I did a new six-week self-experiment where I kept my current routine, but I tested my reaction time every day. I wanted to test two hypotheses:

Is my post-caffeine reaction time worse on Saturday than on Mon/Wed/Fri?
Is my reaction time worse on the morning after a caffeine day than on the morning after a caffeine-free day?

The first hypothesis tests whether I become habituated to caffeine, and the second hypothesis tests whether I experience withdrawal symptoms the following morning.

The answers I got were:

No, there’s no detectable difference.
No, there’s no detectable difference.

Therefore, in defiance of my subjective experience—but in agreement with my earlier experimental results—I do not become detectably habituated to caffeine on the second day.

However, it’s possible that caffeine habituation affects my fatigue even though it doesn’t affect my reaction time. So it’s hard to say for sure what’s going on without running more tests (which I may do at some point).

Contents
Experimental procedure
Results
Alternative experimental procedures that I’m not going to do
A story about how I thought my experiment failed, but actually I was just being stupid
Notes

Experimental procedure

As with my previous experiments, I took a reaction time test every morning before caffeine, as well as an hour after caffeine on days when I took it (Mon/Wed/Fri/Sat). I ran the test for six weeks.

This experiment had the same flaws as my previous experiments, e.g., I did not blind myself because blinding myself is annoying and I didn’t feel like doing it.

In my first two experiments, I was meticulous about controlling the conditions on my computer during the reaction time test. I always tested using the humanbenchmark.com test in Chrome with a single browser window open. I normally use Firefox, but I tested in a different browser to be sure that my 100+ open Firefox tabs wouldn’t interfere with the test in any way (perhaps background tasks could slow down the JavaScript code that runs the reaction time app, which could artificially inflate my reaction time). I tested without any other applications open on my computer except for Emacs and a terminal window (which I always have open).

For my most recent experiment, I wasn’t so meticulous about it because I wanted to be lazy and I figured it probably didn’t matter. I still did the reaction time test in Chrome, but I didn’t close Firefox or other applications during the test.

Results

First, I tested to see if caffeine even made a visible difference in reaction time. Last time, caffeine had a strong and readily apparent effect on my reaction time. My third experiment replicated this result:

caffeine vs. no-caffeine:
    298.0 ms vs. 303.5 ms
    t-stat = -2.9, p-value = 0.006

However, my reaction time was noticeably worse than in the previous two experiments. My average used to hover around 280 ms and now it was hovering around 300 ms. Perhaps because I was less meticulous about keeping my computer in consistent conditions, I ended up adding some latency to the reaction time app?

Some evidence for this hypothesis is that I’ve tried testing my reaction time on Windows a few times (I normally use Linux) and it’s much faster—more like 230 ms. This is almost certainly due to a difference in how the reaction time app works on Windows vs. Linux.

My primary hypothesis test—which I pre-registered to myself, but did not pre-register publicly—was to compare post-caffeine reaction time performance on Saturday vs. the average of every other caffeine day (Mon/Wed/Fri). This test got a null result:

Saturdays vs. non-Saturday caffeine days:
    297.2 ms vs. 298.3 ms
    t-stat = -0.4, p-value = 0.697

I felt generally worse on Saturdays, but perhaps I was imagining things or seeing patterns that weren’t there, and really I shouldn’t worry about it.

Or perhaps I do actually feel worse on the second caffeine day, in a way that reaction time fails to capture. It’s possible that caffeine’s different effects habituate at different rates, and I’m losing my alertness faster than I’m losing my reaction speed.

(I would guess that caffeine’s effect on exercise performance would habituate particularly slowly—as I understand, caffeine improves exercise by physiologically improving muscle function somehow (it enhances calcium circulation or something), not just by increasing alertness.)

My second hypothesis was that I experience caffeine withdrawal on the morning after a caffeine day. I got a null result for this hypothesis as well:

morning after caffeine vs. morning after nocaf:
    303.6 ms (sd 6.9) vs. 303.5 ms (sd 7.0)
    mean difference = 0.1 ms
    t-stat = 0.0, p-value = 0.987

(Before running the experiment, I had a vague idea that I wanted to test this hypothesis, but I didn’t mentally pre-register a methodology.)

Alternative experimental procedures that I’m not going to do

It could be that my reaction time doesn’t get worse on the second day, but my alertness does get worse. I can think of two methods to test that hypothesis, but I don’t want to do them.

Method 1: Same procedure as before, but instead of using a reaction time test as the independent variable, I subjectively rate my alertness. This seems not good because it’s unblinded. I’m not too concerned about blinding reaction time because it’s hard to placebo yourself into a faster reaction time, but “subjective rating of alertness” is exactly the sort of thing that’s highly prone to a placebo effect.

Method 2: Randomize whether I take caffeine pills or placebo pills, and blind myself. To detect potential habituation, I can take the same pill two days in a row, but blind myself to what type of pill it is. Then I subjectively rate my alertness. I don’t want to do that either because it would require working out without caffeine 50% of the time, and working out without caffeine is unpleasant.

A story about how I thought my experiment failed, but actually I was just being stupid

After completing my experiment—this was about three months ago¹—I wrote some code to test the hypotheses. To my dismay, I found no detectable difference between caffeine and no-caffeine reaction times:

caffeine vs. no-caffeine:
    298.3 ms vs. 303.5 ms
	t-stat = 0.0, p-value = 0.987

If there’s not even a difference between caffeine and no-caffeine, then the experiment is useless.

At the time, I was too tired and demotivated to write up the results, so I abandoned it for a while.

Eventually I decided to finally write up the results of my experiment again. I looked at the numbers and I noticed that they didn’t make any sense. If the difference between caffeine and no-caffeine was 5.2 ms, how was the t-stat 0.0?

You may be able to see the mistake I made if you look at the numbers from the Results section. Instead of printing the t-stat and p-value for the caffeine vs. no-caffeine t-test, I accidentally printed the numbers from the morning after caffeine vs. morning after nocaf test. So the figures I was looking at were totally wrong.

I guess I wasn’t 100% there mentally when I wrote the code. (Honestly I don’t think I was even 30% there.)²

If you want to check if my code contains any other horrible mistakes, you can find it on GitHub.

Notes

I have a bad habit of letting half-finished drafts sit in my drafts folder for a long time. ↩
I wrote it on a non-caffeine day which might have something to do with it. ↩

Things I've Become More Confident About

Sun, 02 Nov 2025 00:00:00 -0700

Last year, I wrote a list of things I’ve changed my mind on. But good truth-seeking doesn’t just require you to consider where you might be wrong; you must also consider where you might be right.

In this post, I provide some beliefs I used to be uncertain about, that I have come to believe more strongly.

My belief: Evolution is true.

Why I believed it originally: I learned about the theory of evolution in school. I had the impression that it was a popular but unproven hypothesis (“just a theory”).¹

What made me more confident: When I was maybe 10 or 12, I read an article in some science magazine (National Geographic, maybe) about evolution. It said, “evolution is a theory in the same way atoms are a theory.” I probably put too much credence in this one sentence in one article, but in my mind, this was definitive proof that evolution is true.

Later, when I was 14, I started getting interested in the specifics of the theory of evolution and learned much more about the supporting evidence. (My motivation was mostly that I wanted to argue with creationists on the internet.)

I went through a similar trajectory when learning about quarks. I was taught that a quark is a hypothetical particle that exists inside atoms, but has never been observed. Later I learned that the existence of quarks is well-established, and it became well-established nearly three decades before I was born.

On the subject of outdated pedagogy, this is a bit of a tangent but in 5th grade I was taught the five kingdoms of life: monerans, protists, fungi, plants, and animals. Recently, I learned that not only do biologists no longer use this classification system, but that it was already obsolete when my 5th grade teacher was in 5th grade.²

(My 5th grade teacher was pretty young, but still.)
My belief: Value investing works.

Why I believed it originally: I read about Joel Greenblatt’s magic formula investing and its strong historical performance.

What made me more confident: I read more research on value investing, including the seminal paper The Cross-Section of Expected Stock Returns³ by Fama and French, and more in-depth research showing value investing has worked across the world and across asset classes⁴, and on older data going back 200 years⁵.
My belief: Peaceful protests can be effective.

Why I believed it originally: I actually went back and forth on this one. In school I learned about Martin Luther King and how he was a hero of the civil rights movement, the Montgomery Bus Boycott that he helped organize, Gandhi’s protests against colonialism, and implicit in all this was the idea that these tactics were effective.

Eventually I learned about GiveWell, which was the first time I’d ever encountered the notion that just because a charity says it’s effective, doesn’t mean it’s actually effective. I started thinking critically about protests in the same way, and I realized that I’d never actually seen good evidence that MLK or Gandhi were responsible for the positive changes that coincided with their activism.

Then I started thinking, well, there’s not strong evidence that protests work, but there’s at least some reason to believe they work. That’s about where I was at in 2024 when I donated to PauseAI—I thought, I don’t really know if this is gonna work, but it’s worth trying.

What made me more confident: I wrote Do Protests Work? A Critical Review, in which I carefully investigated the strongest evidence I could find. I found that the best evidence was better than I’d expected, and it pointed toward peaceful protests being effective.
My belief: Seed oils are good for you; seed oils don’t cause obesity.

Why I believed it originally: I had never heard of the seed oil-obesity hypothesis until I read Dynomight’s article on the subject, which argues against the hypothesis. Dynomight presented some evidence that seed oils are harmful and then ultimately concluded that they’re not. I didn’t think much about the evidence the article gave, but its conclusion seemed reasonable to me.

What made me more confident: I researched the issue in more depth while writing Outlive: A Critical Review, specifically the section on saturated fat. I looked through the literature and presented what I believed to be the strongest evidence on the matter: meta-analyses of RCTs that directly compared dietary saturated fat with unsaturated fat (which usually meant seed oils). The experimental evidence finds that seed oils are, if anything, healthier than saturated fat, which contradicts the seed oil-obesity hypothesis.

I read some writings by proponents of the seed oil hypothesis, and their arguments seemed incredibly weak to me.

(Later, I re-read Dynomight’s article and found that it cited the same evidence I had looked at while writing my review of Outlive, which I had completely forgotten about.)

Dynomight presented the seed oil hypothesis as reasonable but ultimately probably wrong, so that’s what I believed at the time. After examining the evidence in more depth, I don’t think the seed oil hypothesis is reasonable. Dynomight admirably followed Daniel Dennett’s principles for arguing intelligently, in which you present your opponent’s case as strongly as possible. But this gave me impression that the seed oil hypothesis is more plausible than it actually is.
My belief: Absent regulation, we aren’t going to solve the AI alignment problem in time.

Why I believed it originally: I’ve vaguely believed this since I first learned about the AI alignment problem (in 2013, if I remember correctly). The problem seemed to involve some thorny philosophical problems of unknown size, like the outline of an enormous beast under a murky ocean. But at that point, humanity had collectively only spent a few hundred person-years on AI alignment, and I thought, perhaps there will be some breakthrough that makes the problem turn out to be much easier than expected. Or perhaps as superintelligent AI becomes increasingly imminent, humanity will rally and pour the necessary resources into the problem.

What made me more confident: In this case I haven’t much changed my interpretation of the evidence, but I’ve become more confident as new evidence has come out. Namely, AI has gotten extraordinarily more powerful; alignment work has not kept up with the increases in AI capabilities; even though alignment work gets more attention now, the problem still seems about as hard as ever.

Beyond that, almost all alignment work is streetlight effect-ing, focused on solving tractable but mostly-irrelevant problems; and the frontier AI companies mostly don’t engage with, and are sometimes even actively hostile to, the idea that solving alignment will require major philosophical breakthroughs and it can’t be done using the sorts of empirical methods that they’re all using.
My belief: Most studies on caffeine tolerance are not informative.

Why I believed it originally: Prior to writing my post Does Caffeine Stop Working?, I reviewed some studies on caffeine tolerance and I thought to myself, these studies aren’t even testing the hypothesis they claim to be testing, surely I must be missing something?

What made me more confident: I read the studies more carefully and spent more time thinking about them, and read a few contrary papers by other scientists who study caffeine. My more careful analysis only reinforced my initial belief that most studies on caffeine tolerance are, indeed, not useful.
My belief: I am smart.

Why I believed it originally: In elementary school, I knew I was the smartest kid in my class. But my class only had about 20 students, and I figured I wasn’t that smart in the grand scheme of things. Like, not as smart as scientists and people who go to Harvard and stuff.

What made me more confident: The first big piece of evidence came after I took the PSAT in 10th grade and my score was good enough that I realized I had a good shot at getting into a top university.

Then I actually attended a top university and realized that many of the people there were not that smart compared to me. College was still a big step up from elementary school: I went from always being the smartest person in the room to being only in the top 1/3 most of the time, and I sometimes found myself in the bottom third.

This trend of repeatedly up-rating my own intelligence reached its peak when I started taking advanced computer science classes, where I was close to the 50th percentile. And nowadays I’m about average within my social circles, and often below average.

(If you’re reading this, there’s a good chance that you’re smarter than me.)

Another canon event happened when I saw the data on the distribution of my school’s SAT scores. The school’s average score was just over one standard deviation above the population mean.⁶ I went through high school thinking my average classmates were average, when in reality they were considerably smarter than average.
My belief: When I first got into lifting weights a decade ago, I learned a lot of conventional wisdom like:
- Low reps are better for strength, and high reps are better for hypertrophy.
- Compound exercises are better for strength, and isolation exercises are better for hypertrophy.
- Long rests are better for strength, and short rests are better for hypertrophy.
- If you want to bulk or cut, you should eat at a 500 calorie surplus/deficit to gain/lose about a pound per week.
Why I believed it originally: It was the conventional wisdom—people generally agreed that these things are true, even though nobody talked about why.

What made me more confident: I started paying more attention to scientific literature on resistance training and I learned that the conventional wisdom pretty much had it right, at least on these points.

(The first three pieces of advice are all explained by a unifying factor: to build strength, you want to lift as much weight as possible, and to build muscle, you want to do as much volume as possible. High reps, isolation exercise, and short rests all enable you to wear out your muscles while lifting lighter weights, and the lighter the weights, the more volume you can do. These three bits of advice aren’t overwhelmingly important—you can still build muscle doing compound exercises at low reps—but they’re useful as guidelines.)
My belief: Exercise is good for you.

Why I believed it originally: Everyone says exercise is good for you, right? But I didn’t know how you’d demonstrate scientifically that that’s true. I thought perhaps it’s reverse causation (sick people can’t exercise) or confounded by socioeconomic class or something.

What made me more confident: I learned more about the scientific evidence on exercise.
- Many randomized controlled trials show that exercise improves short-term health markers—it reduces blood pressure, improves blood sugar regulation, etc.
- A smaller number of long-term trials show long-term health benefits to exercise.
Spoilers for Game of Thrones / A Song of Ice and Fire. Click here to expand.

My belief: R + L = J. That is, Jon Snow's parents are Lyanna Stark and Rhaegar Targaryen.

Why I believed it originally: This had long been a popular fan theory. I didn't figure it out on my own, but I was reasonably convinced by the evidence in this article. I thought it sounded right, but I was uncertain because the textual evidence wasn't conclusive.

What made me more confident: I watched an interview with David Benioff and Dan Weiss, the creators of the TV show. They told a story about how they met with George R. R. Martin to get him to agree to adapt his books. At some point in the meeting, he asked them: Who is Jon Snow's mother? They gave an answer, and he didn't say whether they were right, but he gave a knowing smile, and he agreed to let them make the TV show.

They didn't say what their answer was. But I found this story to be pretty much decisive evidence for R + L = J because what it proved was that the answer was knowable. If David and Dan could know it, then the rest of the fan base could, too.

Later I became even more confident when the TV show revealed that R + L = J. (Rarely in life do you get definitive confirmation that your theory is correct!)

Notes

When I was young, I thought the way evolution worked was that a group of apes were born about 500,000 years ago, and these apes lived for hundreds of thousands of years, over which time their bodies slowly morphed to become more and more humanoid, until they became fully human, at which point they birthed human offspring and then died.

One time I told my dad that I wish I could’ve gotten to evolve because I wanted to live for 500,000 years. That’s when I learned that that’s not how evolution works. ↩
Carl Woese defined a six-kingdom taxonomy using evidence from ribosomal RNA in 1977, at which time I believe my 5th grade teacher would’ve been in 2nd grade.

Lest I sound like I know what I’m talking about, the only reason I can talk coherently about ribosomal RNA methods for taxonomic classification is because I just read those words off Wikipedia 15 seconds ago. ↩
Fama, E. F., & French, K. R. (1992). The Cross-Section of Expected Stock Returns. ↩
Asness, C. S., Moskowitz, T. J., & Pedersen, L. H. (2012). Value and Momentum Everywhere. ↩
Baltussen, G., Swinkels, L., & van Vliet, P. (2019). Global Factor Premiums. ↩
And someone who gets an average score on the SAT is above-average intelligence, because taking the SAT at all already screens off the lower end of the bell curve. ↩

Will Welfareans Get to Experience the Future?

Sat, 01 Nov 2025 00:00:00 -0700

Epistemic status: This entire essay rests on two controversial premises (linear aggregation and antispeciesism) that I believe are quite robust, but I will not be able to convince anyone that they’re true, so I’m not even going to try.

Cross-posted to the Effective Altruism Forum.

If welfare is important, and if the value of welfare scales something-like-linearly, and if there is nothing morally special about the human species¹, then these two things are probably also true:

The best possible universe isn’t filled with humans or human-like beings. It’s filled with some other type of being that’s much happier than humans, or has much richer experiences than humans, or otherwise experiences much more positive welfare than humans, for whatever “welfare” means. Let’s call these beings Welfareans.
A universe filled with Welfareans is much better than a universe filled with humanoids.

(Historically, people referred to these beings as “hedonium”. I dislike that term because hedonium sounds like a thing. It doesn’t sound like something that matters. It’s supposed to be the opposite of that—it’s supposed to be the most profoundly innately valuable sentient being. So I think it’s better to describe the beings as Welfareans. I suppose we could also call them Hedoneans, but I don’t want to constrain myself to hedonistic utilitarianism.)

Even in the “Good Ending” where we solve AI alignment and governance and coordination problems and we end up with a superintelligent AI that builds a flourishing post-scarcity civilization, will there be Welfareans? In that world, humans will be able to create a flourishing future for themselves; but beings who don’t exist yet won’t be able to give themselves good lives, because they don’t exist.

My guess is that a tiny subset of crazy people (like me) will spend their resources making Welfareans, who will end up occupying only a tiny percentage of the accessible universe, and as a result, the future will be less than 1% as good as it could have been.

(And maybe my conception of Welfareans will be wrong, and some other weirdo will be the one who makes the real Welfareans.)

I want the future to be nice for humans, too. (I’m a human.) But all we need to do is solve AI alignment (and various other extremely difficult, seemingly-insurmountable problems), and humans will turn out fine. Welfareans can’t advocate for themselves, and I’m afraid they won’t get the advocates they need.

There is one reason why Welfareans might inherit most of the universe. Generally speaking, people don’t care about filling all available space with Dyson spheres to maximize population. They just want to live in their little corner of space, and they’d be happy to let the Welfareans have the rest.

It’s probably true that most people aren’t maximizers. But some people are maximizers, and most of them won’t want to maximize Welfareans; they’ll want to maximize some other thing. A lot of people will want to maximize how much of the universe is captured by humans or post-humans (or even just their personal genetic lineage). Mormons will want to maximize the number of Mormons or something. There are enough maximizing ideologies that I expect Welfareans to get squeezed out.

So what can we do for the Welfareans?

There are two problems:

Who even are the Welfareans?
How do we ensure that the Welfareans get their share of the future’s resources?

Solving problem #1 approximately requires solving ethics (or, I guess, axiology). I’m not going to say more about that problem; I hope we can agree that it’s hard.

For problem #2, the first answer that comes to mind is “make a power grab for as many resources as possible so I can give them to Welfareans later on”. But I’m guessing that if we solve ethics (as per problem #1), The Solution To Ethics will include a bit that says something along the lines of “don’t take other people’s stuff”. And there are only like three of us who would even care about Welfareans, so I don’t think we’d get very far anyway.

So how do we increase Welfareans’ share of resources, but in an ethical manner? I don’t know. I’m going to start with “write this essay about Welfarean welfare”.

Notes

In my first draft, the opening sentence said “If something like utilitarianism is true, …”. But this is an unnecessarily strong premise. You don’t need utilitarianism, you just need linear aggregation + antispeciesism. A non-consequentialist can still believe that more welfare is better (all else equal). Such a person would still want to maximize the aggregate welfare of the universe, subject to staying within the bounds of whatever moral rules they believe in. ↩

The Next-Gen LLM Might Pose an Existential Threat

Wed, 15 Oct 2025 00:00:00 -0700

I’m pretty sure that the next generation of LLMs will be safe. But the risk is still high enough to make me uncomfortable.

How sure are we that scaling laws are correct? Researchers have drawn curves predicting how AI capabilities scale based on how much goes into training them. If you extrapolate those curves, it looks like the next level of LLMs won’t be wildly more powerful than the current level. But maybe there’s a weird bump in the curve that happens in between GPT-5 and GPT-6 (or between Claude 4.5 and Claude 5), and LLMs suddenly become much more capable in a way that scaling laws didn’t predict. I don’t think we can be more than 99.9% confident that there’s not.

How sure are we that current-gen LLMs aren’t sandbagging (that is, deliberately hiding their true skill level)? I think they’re still dumb enough that their sandbagging can be caught, and indeed they have been caught sandbagging on some tests. I don’t think LLMs are hiding their true capabilities in general, and our understanding of AI capabilities is probably pretty accurate. But I don’t think we can be more than 99.9% confident about that.

How sure are we that the extrapolated capability level of the next-gen LLM isn’t enough to take over the world? It probably isn’t, but we don’t really know what level of capability is required for something like that. I don’t think we can be more than 99.9% confident.

Perhaps we can be >99.99% that the extrapolated capability of the next-gen LLM is still not as smart as the smartest human. But an LLM has certain advantages over humans—it can work faster (at least on many sorts of tasks), it can copy itself, it can operate computers in a way that humans can’t.

Alternatively, GPT-6/Claude 5 might not be able to take over the world, but it might be smart enough to recursively self-improve, and that might happen too quickly for us to do anything about.

How sure are we that we aren’t wrong about something else? I thought of three ways we could be disastrously wrong:

We could be wrong about scaling laws;
We could be wrong that LLMs aren’t sandbagging;
We could be wrong about what capabilities are required for AI to take over.

But we could be wrong about some entirely different thing that I didn’t even think of. I’m not more than 99.9% confident that my list is comprehensive.

On the whole, I don’t think we can say there’s less than a 0.4% chance that the next-gen LLM forces us down a path that inevitably ends in everyone dying.

Mechanisms Rule Hypotheses Out, But Not In

Wed, 08 Oct 2025 00:00:00 -0700

If there is no plausible mechanism by which a scientific hypothesis could be true, then it’s almost certainly false.

But if there is a plausible mechanism for a hypothesis, then that only provides weak evidence that it’s true.

An example of the former:

Astrology teaches that the positions of planets in the sky when you’re born can affect your life trajectory. If that were true, it would contradict well-established facts in physics and astronomy. Nobody has ever observed a physical mechanism by which astrology could be true.

An example of the latter:

A 2023 study found an association between autism and diet soda consumption during pregnancy. The authors’ proposed mechanism is that aspartame (an artificial sweetener found in diet soda) metabolizes into aspartic acid, which has been shown to cause neurological problems in mice. Nonetheless, even though there is a proposed mechanism, I don’t really care and I’m pretty sure diet soda doesn’t cause autism. (For a more thorough take on the diet soda <> autism thing, I will refer you to Grug, who is much smarter than me.)

Why?

A lack of mechanism strongly rules out a hypothesis. If astrology were true, that would overturn some extremely well-established findings in physics. How could astrology possibly be true, given what we know about the laws of gravity?

Perhaps scientists have overlooked something. Perhaps the planets affect humans not via gravity but via some fifth as-yet-discovered fundamental force. But if astrologers can detect the fifth force, why haven’t physicists noticed it with all their careful experimentation?

On the other hand, the existence of a mechanism doesn’t count for much. I often see this in biology, where someone proposes a contrarian hypothesis with a possible biological mechanism but no supporting evidence from randomized experiments. I don’t take that sort of evidence very seriously. Biology is complicated, and chemicals have all sorts of effects on bodies, and it’s very hard to predict whether those effects are net good or bad just by looking at mechanisms.

For example, did you know that exercise increases inflammation? And inflammation is bad for you? And yet, exercise is good for you, because the acute inflammation caused by exercise is strongly outweighed by the long-term beneficial effects.

However, when a hypothesis has supporting evidence from experiments but a lack of plausible mechanism, I disbelieve the research. Experiments have demonstrated that people have psychic abilities. But I’m quite confident that people don’t have psychic abilities because there is no mechanism by which that could be true.

In the hierarchy of evidence, experiment beats mechanism, but lack of mechanism beats experiment.

This asymmetry is consistent with the law of Conservation of Expected Evidence. There are many plausible mechanisms out there in the world. A hypothesis must have a mechanism for it to be true, but the existence of a mechanism does not come anywhere close to proving a hypothesis correct.

Some more examples

Here are some more hypotheses that are strongly ruled out by a lack of plausible mechanism:

Some houses are haunted by ghosts.
Dowsing rods can detect underground water.
Fortune-tellers can predict the future.
Homeopathy can cure diseases.

Some hypotheses with plausible mechanisms that I nonetheless believe are false:

Seed oils are bad for you because they contain linoleic acid, which causes inflammation. This mechanism is true (as far as I know), but experiments comparing unsaturated fats (mainly seed oils) to saturated fats find that people who eat more of the former end up healthier; see Hooper et al. (2020)¹ and WHO (2016)². Experimental evidence indicates that seed oils have overall positive health effects.
Eating excess protein causes osteoporosis. The proposed mechanism is that proteins increase blood acidity which causes the body to extract calcium from bones to balance out this acidity. And indeed, people on high-protein diets excrete more calcium in their urine. But randomized controlled trials have found that adding protein to the diet reduces the risk of bone fracture (Koutsofta et al. (2018)³).⁴
- Relatedly, you may hear some people say you should eat more alkaline foods to fix your body’s pH balance. It would indeed be bad if your body’s pH became too low, but the empirical evidence shows that dietary pH does not affect your body’s pH in that way (see Wikipedia).⁵
Sugar causes hyperactivity in children because it provides a short-term burst of energy. This mechanism is intuitive even if you don’t know much biology. But it’s not true—RCTs have consistently found no connection between hyperactivity and sugar consumption (Vreeman & Carroll (2008)⁶).
Eating cholesterol raises your blood cholesterol. The mechanism in this case is obvious: you eat food that contains cholesterol, and the cholesterol goes into your body. But your body regulates its own cholesterol production, and your blood cholesterol levels don’t have much to do with how much cholesterol you eat.⁷

Notes

Hooper, L., Martin, N., Jimoh, O. F., Kirk, C., Foster, E., & Abdelhamid, A. S. (2020). Reduction in saturated fat intake for cardiovascular disease. doi: 10.1002/14651858.cd011737.pub3 ↩
Mensink, R. P., & World Health Organization (2016). Effects of saturated fatty acids on serum lipids and lipoproteins: a systematic review and regression analysis. ↩
Koutsofta, I., Mamais, I., & Chrysostomou, S. (2018). The effect of protein diets in postmenopausal women with osteoporosis: Systematic review of randomized controlled trials. ↩
I heard about this research on the Iron Culture podcast, in which they went on to complain about how people care too much about mechanisms and ignore experimental evidence. It got me thinking about an apparent contradiction in my beliefs where I care a lot about mechanisms for ruling out astrology and ESP, but I don’t really care about mechanisms in nutrition or exercise science. After thinking about it, I realized that my position is perfectly sensible—it’s about using mechanisms to rule hypotheses out vs. in—and that’s how I came up with the idea to write this post. ↩
I want to be careful not to say that an alkaline diet is unhealthy. Alkaline foods do tend to be particularly healthy—they’re mostly fruits and vegetables—but that’s coincidental, not because they’re alkaline per se. ↩
Vreeman, R. C., & Carroll, A. E. (2008). Festive medical myths. ↩
https://nutritionsource.hsph.harvard.edu/what-should-you-eat/fats-and-cholesterol/cholesterol/ ↩

How Much Does It Cost to Offset an LLM Subscription?

Sat, 04 Oct 2025 00:00:00 -0700

Is moral offsetting a good idea? Is it ethical to spend money on something harmful, and then donate to a charity that works to counteract those harms?

I’m not going to answer that question. Instead I’m going to ask a different question: if you use an LLM, how much do you have to donate to AI safety to offset the harm of using an LLM?

I can’t give a definitive answer, of course. But I can make an educated guess, and my educated guess is that for every $1 spent on an LLM subscription, you need to donate $0.87 to AI safety charities.

First things first: Why do I believe it’s harmful to buy an LLM subscription?

Paying money to a frontier AI company increases their revenue, and they spend some of that revenue on building more powerful AI systems. Eventually, they build a superintelligent AI. That AI has a good chance of being misaligned and then killing everyone in the world. When you buy an LLM subscription, you cause that to happen slightly faster.

But you can also donate to nonprofits that are working to prevent AI from killing everyone. How much do you need to donate to a nonprofit to offset the harm of a $20/month LLM subscription?

I built a simple Squiggle model to answer that question.

The model

Four key facts:

When you give a company an additional dollar of revenue, that raises its future valuation by some number.
A higher valuation lets the company raise more capital and thus spend some additional amount of money.
AI companies will spend a total of some amount in 2026.
Meanwhile, it would take some amount of money directed to AI safety nonprofits to cancel out the harm of AI companies’ spending.

From those, you can estimate how much you need to donate using the following procedure:

Start from the dollar value of your subscription.
Calculate how much that will increase company valuation.
Translate increased valuation into increased expenditures.
Divide by expected total expenditures of frontier AI companies.
Multiply by expected total cost to offset AI company harm.

The resulting number is the amount to donate to AI safety nonprofits.

There are some difficult questions that this model avoids having to answer:

We don’t care what proportion of AI company spending goes to R&D on frontier models; we only care about total spending.
We don’t care to what extent x-risk is increased or decreased per dollar spent.

The inputs

The model has four inputs: (1) the revenue-to-valuation ratio; (2) the valuation-to-expenditures ratio; (3) frontier AI company expenditures; (4) total cost to offset AI company harm. In this section, I will explain how I estimated the values of those inputs.

Revenue-to-valuation ratio

How much does a dollar of revenue raise an AI company’s valuation? I can see arguments for both “hardly at all” and “a lot”.

In favor of “hardly at all”: VCs give AI companies funding on the expectation that their products will be incredibly useful in the future, which doesn’t have much to do with current revenue.

In favor of “a lot”: AI companies raise funding at high revenue multiples, e.g. Anthropic raised its last round (as of September 2025) at 36x revenue (source). This could mean that VCs expect $1 of revenue today to convert to $36 in future value, i.e. revenue has a 36:1 multiplier effect.

A typical 2025 startup valuation is 7x revenue (source). As a median estimate, we could say that $1 of AI company revenue converts to $7 of valuation, and the extra 5x multiplier is driven by high expectations for future AI products.

(I briefly looked into how startup funding scales with revenue and I didn’t find any useful evidence.)

Growth rate matters more for valuation than revenue does, but I don’t think this changes the calculation in the short term because an extra $1 of 2025 revenue also represents an extra $1 in growth relative to 2024 revenue.

Valuation-to-expenditures ratio

How much does $1 of company valuation translate into increased expenditures?

Private companies don’t usually publish that information. But based on historical data for AI companies and general trends for startups, it’s reasonable to expect companies to raise capital equal to 5% to 20% of the valuation.

(I’m thinking of AI companies as startups; “startup” connotes “small”, which they clearly aren’t, but I’m using the term in the Paul Graham startup = growth sense. Frontier AI companies are startups because they’re growing fast.)

Frontier AI company expenditures

Public data on AI company fundraising in 2025:

Anthropic: $13B
OpenAI: $40B
xAI: $10B maybe? (the publicly available data only shows total funding, not individual rounds)

Assume these three companies account for half of AI spending and that the funding they raised will last 18 months; that means AI companies will spend $66B in 2026.

Total cost to offset AI company harm

This is the hardest number to estimate. My assumption is that the AI safety community currently spends on the order of $30 million to $100 million per year, and if we spent on the order of 10–100x more, then that would be enough to fully offset the harms of AI companies.

I suspect that spending 100x more on pure alignment research would not be enough. But spending 100x more would likely be enough if some of the spending goes to governance/policy/advocacy, and some goes to things that have multiplier effects (e.g. you could spend $1 to cause AI companies to contribute $10 more to safety research). I’m also assuming you can make AI safe merely by throwing money at the problem, which is clearly false, but it makes sense to assume it’s true for the purposes of this model.

The answer (according to my model)

Put all those numbers together and the model spits out a mean cost of $0.87 in donations for every $1 spent on LLM subscriptions. That means for a $20/month subscription, according to the model you’d need to donate $17/month to AI safety orgs.

The model’s median estimate is only $0.06—which is to say, an LLM subscription probably only does a little bit of harm. But there is a small probability that you need to donate quite a bit more to offset your LLM usage, so the expected cost is much higher at $0.87.

Limitations of the model

Like any model, this one does not perfectly match reality. Some examples of problems this model has:

I have no clue what the total cost is to offset the harm of AI companies.
The model assumes they money you donate does as much good as the average dollar spent on AI safety. But maybe your dollars can be above average. (Or they could even be below average.)
Maybe giving more money to Anthropic is good actually, because Anthropic is the least unsafe AI company and speeding them up improves our chances.¹
Is moral offsetting even okay? Maybe we should obey a rule-utilitarian constraint against doing bad things, even if we offset them. Or maybe moral offsetting is silly and we should just donate to whatever charity is most effective.

Conclusion

I have a subscription to Claude. Last year I donated a lot of money to AI safety but I didn’t make any donations specifically for offsetting. Having put more thought into it to write this post, I think I will start donating an extra $240/year—$1 donated for every $1 spent on Claude. My model suggested donating 87 cents per dollar, but the model isn’t that precise, and $1-per-dollar is a nice round number. I’m still undecided on whether the concept of moral offsetting makes sense, but I figure I might as well do it.

Notes

In my first draft, I also said it might be net good to give AI companies money if they’ll use some of it on alignment research. But on reflection, I’m pretty sure that’s wrong, because giving them money speeds up AI progress, and there’s no strong reason to expect that increasing AI company revenue will increase total expenditures on alignment.

I also expect it’s bad to speed up Anthropic, but I’m not confident about that. ↩

I made an Emacs extension that displays Magic: the Gathering card tooltips

Fri, 03 Oct 2025 00:00:00 -0700

This post is about the niche intersection of Emacs and Magic: the Gathering.

I considered not writing this because I figured, surely if you multiply the proportion of people who play Magic by the proportion of people who use Emacs, you get a very small number. But then I thought, those two variables are probably not independent. And the intersection of Magic players x Emacs users x people who read my blog might actually be greater than zero. So if you’re out there, this post is for you.

Do you like how MTG websites like magic.gg and mtg.wiki let you mouse over a card name to see a picture of the card? Well, I wrote an Emacs extension that replicates that functionality.

Here is the code: https://github.com/michaeldickens/emacs-mtg

The README on GitHub pretty much explains how it works, so the rest of this post is just gonna repeat what it says in the README.

Usage

(add-to-list 'load-path /path/to/mtg.el)
(require 'mtg)

This module allows you to refer to Magic cards in Org Mode using a new type of link prefixed with mtg:. For example:

[[mtg:Black Lotus]] might be the strongest card in my collection, but my personal favorite is [[mtg:Grizzly Bears]].

When Org Mode sees a link to an MTG card, it will do the following:

If the card is not downloaded yet, download the card by querying the Scryfall API for a card with the given name.
When you open the link (using org-open-at-point or C-c C-o), Emacs displays an image of the card in the minibuffer.

Here’s how it looks:

By default, card images and data are downloaded to ~/.emacs.d/mtg-cards/, but you can change this by customizing the variable mtg/db-path.

Scryfall’s API has fuzzy name matching, so for example [[mtg:blac lotus]] will display Black Lotus.

Card legality

Cards are displayed with a red tint if they are illegal in the preferred format. It looks like this:

When checking legality, this module uses Standard format by default, but you can customize it by setting the variable mtg/default-format. You can also set file-local or heading-local formats in Org Mode using the :MTG_FORMAT: property. For example:

:PROPERTIES:
:MTG_FORMAT: standard
:END:
If you open this link --> [[mtg:Black Lotus]], the card will appear
with a red tint because it's illegal in Standard.

** My vintage cards
  :PROPERTIES:
  :MTG_FORMAT: vintage
  :END:
  [[mtg:Black Lotus]] is legal in Vintage, so here it will
  appear with no tint.

Note: Adding a red tint requires ImageMagick. If you don’t have ImageMagick installed, all cards will be displayed as if they’re legal.

Exporting to HTML

If you export Org Mode files to HTML, you can make the MTG card links display images on hover. For this to work, you must include some custom CSS in your Org Mode file.

On GitHub there is a file called export-style.setup that includes some custom CSS. To include this custom CSS in Org Mode, put this line at the top of your Org Mode file:

#+SETUPFILE: /path/to/export-style.setup

Then call org-export-dispatch to export the Org file to HTML.

Table utilities

mtg.el comes with functions for working with Org Mode tables. The functions assume you have a table where one column contains links to MTG cards, like this:

[[mtg:Black Lotus]]

[[mtg:Grizzly Bears]]

[[mtg:Colossal Dreadmaw]]

mtg/table-sort-by-property takes a property as a string (such as “name”, “rarity”, or “color”) and sorts the table by looking up that property for each card. This only works if you’ve already downloaded the card info (which happens when you view the card or export the whole file).

mtg/table-insert-column takes a property as a string and inserts a new column containing that property for each card. For example, calling (mtg/table-insert-column "rarity") on the table above produces this:

[[mtg:Black Lotus]]	bonus
[[mtg:Grizzly Bears]]	common
[[mtg:Colossal Dreadmaw]]	common

If a property is missing, the cell will be left blank. For example, calling (mtg/table-insert-column "power") produces

[[mtg:Black Lotus]]
[[mtg:Grizzly Bears]]	2
[[mtg:Colossal Dreadmaw]]	6

You can also call mtg/get-property to return a property for the card at point.

AI Safety Landscape and Strategic Gaps

Fri, 19 Sep 2025 00:00:00 -0700

I wrote a report giving a high-level review of what work people are doing in AI safety. The report specifically focused on two areas: AI policy/advocacy and non-human welfare (including animals and digital minds).

You can read the report below. I was commissioned to write it by Rethink Priorities, but beliefs are my own.

Contents
Introduction
Prelude
- Some positions I’m going to take as given
- Definitions
Prioritization
General recommendations
- Advocacy should emphasize x-risk and misalignment risk
- Prioritize work that pays off if timelines are short
Top project ideas
Honorable mentions
List of other project ideas
Future work
Supplements
Notes

Introduction

This report was prompted by two questions:

What are some things we can do to make transformative AI go well?
What are a few high-priority projects that deserve more attention?

I reviewed the AI safety landscape, starting by prioritizing to narrow my focus to areas that look particularly promising and feasible for me to review. I focused on two areas:

AI x-risk advocacy
Making transformative AI go well for animals

A summary of my reasoning on prioritization regarding AI misalignment risk:

I focused on AI policy advocacy over technical safety research, primarily because it’s much more neglected, and there are other people with more expertise than me who already look for neglected research ideas. [More]
I focused on policy advocacy over policy research, again because it’s particularly neglected. [More]
I considered the downsides of advocacy, which ultimately I don’t believe are strong enough to outweigh the upsides. [More]
I decided not to spend much time evaluating which policies are best to advocate for, because a wide variety of policies could be helpful, and we need more advocacy in general. [More]

Regarding AI issues beyond misalignment:

Transformative AI may not go well for animals. There are some tractable interventions for improving post-TAI animal welfare. [More]
There are other important issues like digital sentience, AI-enabled coups, moral error, etc. But there is no visible path to solving these problems before transformative AI; and I find it quite difficult to weigh the importance of these issues, so I did not discuss them more than briefly. [More]

I created a list of projects within my two focus areas and identified four top project ideas (presented in no particular order):

Talk to policy-makers about AI x-risk
Write AI x-risk legislation
Advocate to change AI training to make LLMs more animal-friendly
Develop new plans / evaluate existing plans to improve post-TAI animal welfare

I also included honorable mentions and a longer list of other project ideas. For each idea, I provide a theory of change, list which orgs are already working on it (if any), and give some pros and cons.

Finally, I list a few areas for future work.

There are two external supplements on Google Docs:

Appendix: Some miscellaneous topics that were relevant, but not quite relevant enough to include in the main text.
List of relevant organizations: A reference list of orgs doing work in AI-for-animals or AI policy/advocacy, with brief descriptions of their activities.

Prelude

I was commissioned by Rethink Priorities to do a broad review of the AI safety/governance landscape and find some neglected interventions. Instead of doing that, I reviewed the landscape of just two areas within AI safety:

AI x-risk advocacy
Making transformative AI go well for animals

I narrowed my focus in the interest of time, and because I believed I had the best chance of identifying promising interventions within those two fields.

There is a tradeoff between (a) giving recommendations that are easy to agree with, but weak; (b) giving strong recommendations that only make sense if you hold certain idiosyncratic beliefs. This report leans more toward (b), making some strong assumptions and building recommendations off of those, although I tried to avoid making assumptions whenever I could do so without weakening the conclusions. I also tried to be clear about what assumptions I’m making.

This report is broad, but I only spent three months writing it. There are some topics in this report that could have been a PhD dissertation, but instead, I spent an hour on them.

Most of this report is about AI policy, but I don’t have a background in policy. I did speak to a number of people who work in policy, and I read a lot of published materials, but I lack personal experience, and I expect that there are important things happening in AI policy that I don’t know about.

Some positions I’m going to take as given

The following premises would probably be controversial with some audiences, but I expect them to be uncontroversial for the readers of this report, so I will treat them as background assumptions.

Effective altruist principles are correct (e.g. cost-effectiveness matters; you can, in principle, quantify the expected value of an intervention).
Animal welfare matters.
Digital minds can matter.
AI misalignment is a serious problem that could cause human extinction.

Definitions

The terms AGI/ASI/TAI can often be used interchangeably, but in some cases the distinctions matter:

AGI = human-level AI: Capable enough to match the economic output of a large percentage of humans (say, at least half).
ASI = superintelligent AI: Smart enough to vastly outperform humans on every task.
TAI = transformative AI: Smart enough to radically transform society (without making a claim about whether that happens at AGI-level or ASI-level or in between).

I use the terms “legislation” and “regulation” largely interchangeably. For my purposes, I don’t need to draw a distinction between government mandates that are directly written into law vs. decreed by a regulatory body.

Prioritization

For this report, I focused on AI risk advocacy and on post-TAI animal welfare, and I did not spend much time on other AI-related issues.

For the sake of time-efficiency, rather than creating a big list of ideas in the full AI space, I first narrowed down to the regions within the AI space that I thought were most promising and then came up with a list of ideas in those regions. I could have spent time investigating (say) alignment research, but I doubt I would have ended up recommending any alignment research project ideas.

In this section:

Why not technical safety research?
Why not policy research?
Downsides of AI policy/advocacy (and why they’re not too big)
What kinds of policies would be good?
Maybe prioritizing post-TAI animal welfare
Why not prioritize digital minds / S-risks / moral error / AI misuse x-risk / gradual disempowerment?

Why not technical safety research?

Technical safety research (mainly alignment research, but also including control, interpretability, monitoring, etc.) is considerably better-funded than AI safety policy.

AI companies invest a significant amount into safety research. They also invest in policy, but their investments are mostly counterproductive (they are mostly advocating against safety regulations¹). Philanthropic funders invest a lot into technical research, and less into policy.

I have not made a serious attempt to estimate the volume of work going into research vs. policy/advocacy, but my sense is that the former receives much more funding.²

Some sub-fields within technical research may be underfunded. But:

I am not in a great position to figure out what those are.
There are already many grantmakers who seek out neglected technical research directions. There are recent requests for proposals (RFPs) from UK AI Security Institute, Open Philanthropy, Future of Life Institute, and Frontier Model Forum’s AI Safety Fund, among others.

Why not AI policy research?

Is it better to do policy research (figure out what policies are good) or policy advocacy (try to get policies implemented)?

Both are necessary, but this article focuses on policy advocacy for the following reasons:

There is much more money in AI policy research. By a large margin, most of what’s happening in AI safety policy could be described as “research”.
- Recently, Jason Green-Lowe estimated from LinkedIn data that there are 3x as many governance researchers as governance advocates.
Policy research is a necessary step in the funnel. We also need people writing legislation and people advocating for the legislation, both of which we have very little of.
At this point, we have at least some idea of how to implement AI safety regulations. More research would be valuable, but it likely has diminishing returns.
- I wrote more about this last year in Slow nuanced regulation vs. fast coarse regulation.
Research works best with long timelines. Timelines are probably not long.
- The ideal situation is to spend 10–20 years developing a field of AI policy, write many reports until a consensus slowly develops about how to govern AI development, then advocate for the consensus policies. But it is likely that by the time we have a reasonable consensus, it’s already too late to do anything about TAI.
- Even if you think timelines are probably long, we are currently under-investing in activities that pay off given short timelines.

Downsides of AI policy/advocacy (and why they’re not too big)

Basically, policy work does one or both of these things:

Legally enforce AI safety measures
Slow down AI development

People who care about x-risk broadly agree that AI safety regulations can be good, although there’s some disagreement about how to write good regulations.

The biggest objection to regulation is that it (often) causes AI development to slow down. People usually don’t object to easy-to-satisfy regulations; they object to regulations that will impede progress.

So, is slowing down AI worth the cost?

I am aware of two good arguments against slowing down AI (or imposing regulations that de facto slow down AI):

Opportunity cost – we need AI to bring technological advances (e.g., medical advances to reduce mortality and health risks)
AI could prevent non-AI-related x-risks

Two additional arguments against AI policy advocacy:

Meaningful AI regulations are not politically feasible to implement
Advocacy can backfire

An additional argument that applies to some types of advocacy but not others:

Advocacy may slow down safer actors without slowing down more reckless actors. For example, it is sometimes argued that US regulations are bad if they allow Chinese developers to gain the lead. I believe this outcome is avoidable—and it’s a good reason to prefer global cooperation over national or local regulations. But I can’t address this argument concisely, so I will just acknowledge it without further discussion.

(For a longer list of arguments, with responses that are probably better-written than mine, see Katja Grace’s Let’s think about slowing down AI.)

Regarding the opportunity cost argument, it makes sense if you think AI does not pose a meaningful existential risk or if you heavily discount future generations. If there is a significant probability that future generations are ~equally valuable to current generations, then the opportunity cost argument does not work. The opportunity cost of delaying AI by (say) a few decades is easily dwarfed by the risk of extinction.

As to the non-AI x-risk argument, it is broadly (although not universally) accepted among people in the x-risk space that AI x-risk is 1–2 orders of magnitude higher than total x-risk from other sources (see Michael Aird’s database of x-risk estimates or the Existential Risk Persuasion Tournament, although I don’t put much weight on individual forecasts). Therefore, delaying AI development seems preferable as long as it buys us a meaningful reduction in AI risk.

See Quantitative model on AI x-risk vs. other x-risks under Future Work.

The tractability argument seems more concerning. Preventing AI extinction via technical research and preventing it via policy both seem unlikely to work. But I am more optimistic about policy because:

We have a good enough understanding of AI alignment to say with decent confidence that we’re nowhere close to solving it. It’s less clear what it would take to get good regulations put in place.
AI regulations are unpopular both in the US Congress and within the Trump administration, but popular among the general public. Popular support increases the feasibility of getting regulations passed.
- And there is a good chance that Congress will be more regulation-friendly after the 2026 Congressional elections.
SB-1047 nearly got passed into law, making it through the California legislature and only failing due to veto. A near-win suggests that a win isn’t far away in possibility-space.

(My reasoning on tractability focused on US policy because that’s where most of the top AI companies operate. The UK seems to be the current leader on AI policy, although it’s not clear to what extent UK regulations matter for x-risk.³)

That leaves the backfire argument. This is a real concern, but ultimately it’s a risk you have to take at some point because you can’t get policies passed if you don’t advocate for them. It could make sense to delay advocacy if one has good reason to believe that future advocacy is less likely to backfire; to my knowledge this is not a common position, and it’s more common for people to oppose advocacy unconditionally. For more on this topic, see Appendix: When is the right time for advocacy?

Also: I’m somewhat less concerned about this than many people. When I did research on protest outcomes, I found that peaceful protests increased public support, even though many people intuitively expect the opposite. Protests aren’t the only type of advocacy, but it’s a particularly controversial type of advocacy. If protests don’t backfire, then it stands to reason—although I have no direct evidence—that other, tamer forms of advocacy are unlikely to backfire.

(There is some evidence that violent protests backfire, however.)

If policy-maker advocacy is similar to public advocacy, then probably the competence bar is not as high as many people think it is. Perhaps policy-makers are more discerning/critical than the general public; on the other hand, it’s specifically their job to do what their constituents want, so it stands to reason that it’s a good idea to tell them what you want.

My main concern comes from deference: some people whom I respect believe that advocacy backfires by default. I don’t understand why they believe that, so I may be missing something important.

I do believe that much AI risk advocacy has backfired in the past, but I believe this was fairly predictable and avoidable. Specifically, talking about the importance of AI has historically encouraged people to build it, which increased x-risk. People should not advocate for AI being a big deal; they should advocate for AI being risky. (Which it is.) See Advocacy should emphasize x-risk and misalignment risk.

What kinds of policies might reduce AI x-risk?

There are many policies that could help. And many policy ideas are independent: we could have safety testing requirements AND frontier-model training restrictions AND on-chip monitoring AND export controls. Those could all be part of the same bill or separate bills.

It’s beyond the scope of this report to come up with specific policy recommendations. I do, however, have some things I would like to see in policy proposals:

They should be relevant to existential risk, especially misalignment risk.
I would like to see work on policies that would help in the event of a global moratorium on frontier AI development (e.g. we’d need ways to enforce the moratorium).
For an ideal policy proposal, it is possible to draw a causal arrow from “this regulation gets passed” to “we survive”. Policies don’t have to singlehandedly prevent extinction, but given that we may only have a few years before AGI, I believe we should be seriously trying to draft bills that are sufficient on their own to avert extinction.

(The closest thing I’ve seen to #3 is Barnett & Scher’s AI Governance to Avoid Extinction: The Strategic Landscape and Actionable Research Questions. It does not propose a set of policies that would (plausibly) prevent extinction, but it does propose a list of research questions that (may) need to be answered to get us there.)

Some people are concerned about passing suboptimal legislation. I’m not overly concerned about this because the law changes all the time. If you pass some legislation that turns out to be less useful than expected, you can pass more legislation. For example, the first environmental protections were weak, and later regulations strengthened them.

Regulations could create momentum, or they could create “regulation fatigue”. I did a brief literature review of historical examples, and my impression was that weak regulation begets strong regulation more often than not, but there are examples in both directions. See my reading notes.

Some AI policy ideas I like

I believe that a moratorium on frontier AI development is the best outcome for preventing x-risk (see Appendix for an explanation of why I believe that). None of my top project ideas depend on this belief, although it would inform the details of how I’d like to see some of those project ideas implemented.

I didn’t specifically do research on policy ideas while writing this report, but I did incidentally come across a few ideas that I’d like to see get more attention. Since I didn’t put meaningful thought into them, I will simply list them here.

Operationalization of “pause frontier AI development until we can make it safe.” For example, what infrastructure and operations are required to enforce a pause?
Rules about when to enforce a pause on frontier AI development: something like “When warning sign X occurs, companies are required to stop training bigger AI systems until they implement mitigations Y/Z.”
Ban recursively self-improving AI.
- Recursive self-improvement is the main way that AI capabilities could rapidly grow out of control, but banning it does not impede progress in the way that most people care about.
- Some work needs to be done to operationalize this, but we shouldn’t let the perfect be the enemy of the good.
Require companies to publish binding safety policies (e.g. responsible scaling policies [RSPs] or similar). That is, if a company’s policy says it will do something, then that constitutes a legally binding promise.
- This would prevent the situation we have seen in the past, where, when a company fails to live up to a particular self-imposed requirement, it simply edits its safety policy to remove that requirement.
- This sort of regulation isn’t strong enough to prevent extinction, but it has the advantage that it should be easy to advocate for.
- California bill SB 53 says something like this (see sb53.info for a summary), but its rules would not fully come into effect until 2030.

Maybe prioritizing post-TAI animal welfare

Making TAI go well for animals is probably less important than x-risk because:

Almost everyone cares about animals. An AI that’s aligned to human values would also care about animals, and it would probably figure out ways to prevent large sources of animal suffering like factory farming.
A technologically advanced civilization could develop cheaper alternatives to animal farming (e.g. cultured meat), rendering factory farming unnecessary.
There wouldn’t be much benefit in spreading wild animal suffering, so it stands to reason that post-TAI civilization won’t do it. (Although I’m not at all confident about this.)

However, AI-for-animals could still be highly cost-effective.

An extremely basic case for cost-effectiveness:

There’s a (say) 80% chance that an aligned(-to-humans) AI will be good for animals, but that still leaves a 20% chance of a bad outcome.
AI-for-animals receives much less than 20% as much funding as AI safety.
Cost-effectiveness maybe scales with the inverse of the amount invested. Therefore, AI-for-animals interventions are more cost-effective on the margin than AI safety.

Why not prioritize digital minds / S-risks / moral error / better futures / AI misuse x-risk / gradual disempowerment?

There are some topics on how to make the future go well that aren’t specifically about AI alignment:

Ensuring digital minds have good welfare
Preventing S-risks — risks of astronomical suffering
Moral error — the risk that we make a big mistake because we are wrong about what’s morally right
Better futures — ensuring that the future is as good as possible, as opposed to simply preventing bad outcomes
Preventing powerful AI from being misused to cause existential harm
Preventing powerful AI from gradually disempowering sentient beings and slowly leading to a bad outcome, as opposed to a sudden bad outcome like extinction

Call these “non-alignment risks”.

Originally, I included each of these as separate project ideas, but I decided not to focus on any of them. This decision deserves much more attention than I gave it, but I will briefly explain why I did not spend much time on non-alignment risks.

All of these cause areas are extremely important and neglected (more neglected than AI misalignment risk), and (for the most part) very different from each other. And I am happy for the people who are working on them—there are some enormous issues in this space where only one person in the world is working on it. Nonetheless, I did not prioritize them.

My concern is that, if AI timelines are short, then there is virtually no chance that we can solve these problems before TAI arrives.

There is a dilemma:

If TAI can help us solve these problems, then there isn’t much benefit in working on them now.
If we can’t rely on TAI to help solve them (e.g. we expect value lock-in), then we have little hope of solving them in time.

(There is a way out of this dilemma: perhaps AI timelines are long enough that these problems are tractable, but short enough that we need to start working on them now—we can’t wait until it becomes apparent that timelines are long. That seems unlikely because it’s rather specific, but I didn’t give much thought to this possibility.)

It looks like we have only two reasonable options for handling AI welfare / S-risks / moral error / etc.:

Increase the probability that we end up in world #1, where TAI can help us solve these problems—for example, by increasing the probability that something like a Long Reflection happens.
Slow down AI development.

I lean toward the second option. For more reasoning on this, see Appendix: Slowing down is a general-purpose solution to every non-alignment problem.

I’m quite uncertain about the decision not to focus on these cause areas. They are arguably as important as AI alignment, and much more neglected.

AI-for-animal-welfare could also be included on my list of non-alignment risks, but I did prioritize it because I can see some potentially tractable interventions in the space.

AI welfare seems more tractable than animal welfare in that AI companies care more about it, but it seems less tractable because it involves extremely difficult problems like “when are digital minds conscious?” There may be some tractable, short-timelines-compatible ideas out there, but I did not see any in the research agendas I read.

Perhaps I could identify tractable interventions by digging deeper into the space and maybe doing some original research, but that was out of scope for this article.

Who’s working on them?

In each of my project idea sections, I included a list of orgs working on that idea (if any). I didn’t write individual project ideas for non-alignment risks (other than AI-for-animals), but I still wanted to include lists of relevant orgs, so I’ve put them below.

There are also some individual researchers who have published articles on these topics in the past; I will not include those.

AI welfare / digital minds: Anthropic; Center on Long-Term Risk; Digital Sentience Consortium; Eleos AI; NYU Center for Mind, Ethics, and Policy Sentient Futures; Sentience Institute.
AI misuse x-risks: Forethought; probably a number of others, but I didn’t spend time specifically looking for them. (AI misuse is a relatively popular subject matter, but extinction-level misuse isn’t much discussed.)
Better futures: Forethought.
Gradual disempowerment: To my knowledge, no orgs specifically work on this, but there is the Gradual Disempowerment paper written by authors with various affiliations.
Moral error: Center on Long-Term Risk; Forethought; Global Priorities Institute (now defunct as of just before this writing).
S-risks from cooperation failure: Center on Long-Term Risk.

Some relevant research agendas

Although I decided not to prioritize this space, others have done work on preparing research agendas, which readers may be interested in. Here, I include a list of research agendas (or problem overviews, which can inform research agendas) with no added commentary.

Anthony DiGiovanni – Clarifying “wisdom”: Foundational topics for aligned AIs to prioritize before irreversible decisions
Center on Long-Term Risk – Cooperation, Conflict, and Transformative Artificial Intelligence: A Research Agenda
Chi Nguyen – AI things that are perhaps as important as human-controlled AI
Digital Sentience Consortium – Applied work on digital sentience and society
Eleos AI – Research priorities for AI welfare
Forethought – AI-Enabled Coups: How a Small Group Could Use AI to Seize Power
Kevin Xia – Transformative AI and Animals: Animal Advocacy Under A Post-Work Society
Raymond Douglas – Gradual Disempowerment: Concrete Research Projects
Will MacAskill – Better Futures
Will MacAskill – Moral error as an existential risk

General recommendations

Advocacy should emphasize x-risk and misalignment risk

I would like to make two assertions:

AI x-risk is more important than non-existential AI risks.
Advocates should say that.

Given weak longtermism, or even significant credence to weak longtermism on a moral-uncertainty system, x-risks dwarf non-existential AI risks in importance (except perhaps for S-risks, which are a whole can of worms that I won’t get into in this section). See Bostrom’s Existential Risk Prevention As Global Priority. Risks like “AI causes widespread unemployment” are bad, but given the fact that we have to triage, extinction risks should take priority over them.

(To my knowledge, people advocating for focusing on non-existential AI risks have never provided supporting cost-effectiveness estimates. I don’t think such an estimate would give a favorable result. If you strongly discount AI x-risk/longtermism, then most likely you should be focusing on farm animal welfare (or similar), not AI risk.)

Historically, raising concerns about ASI has caused people to take harmful actions like:

I need to be the one who builds ASI before anyone else, I think I’ll start a new frontier AI company.
AI is a big deal, so we need to race China.

I don’t have a straightforward solution to this. You can’t reduce x-risk by doing nothing, but if you do something, there’s a risk that it backfires.

My best answer is that advocacy should emphasize misalignment risk and extinction risk. Many harmful actions were committed with the premise “TAI is dangerous if someone else builds it, but safe if I build it.” When in fact it is dangerous, no matter who builds it. “If anyone builds it, everyone dies” is more the correct sort of message.

Misalignment isn’t the only way AI could cause extinction, although it does seem to be the most likely way. I believe advocacy should focus on misalignment risk not only because it’s the most concerning risk, but also it has historically been under-emphasized in favor of other risks (if you read Congressional testimonies by AI risk orgs, they mention other risks but rarely mention misalignment risk), and it is (in my estimation) less likely to backfire.

Many advocates are concerned that x-risk and misalignment risk sound too “out there”. Two reasons why I believe advocates should talk about them:

All else equal, it’s better to say what you believe and ask for what you want. It’s too easy to come up with galaxy-brained reasons why you will get what you want by not talking about what you want.
Emphasizing the less important risks is more likely to backfire by increasing x-risk.
There is precedent for talking about x-risk without being seen as too weird, for example, the CAIS Statement on AI Risk. If you’re worried about x-risk, you’re in good company.

Nate Soares says more about this in A case for courage, when speaking of AI danger.

Prioritize work that pays off if timelines are short

There is a strong possibility (25–75% chance) of transformative AI within 5 years. 80,000 Hours reviews forecasts and predicts AGI by 2030; Metaculus predicts 50% chance of AGI by 2032; AI company CEOs have predicted 2025–2035 (see Appendix, When do AI company CEOs expect advanced AI to arrive?); AI 2027 team predicts 2030ish (their scenario has AGI arriving in 2028, but that’s their modal prediction, not median).

The large majority of today’s AI safety efforts work best if timelines are long (2+ decades). Short-timelines work is neglected. It would be neglected even if there were only (say) a 10% chance of short timelines, but the probability is higher than that.

For example, that means there should be less academia-style long-horizon research, and more focus on activities that have a good chance of bearing fruit quickly.⁴

Top project ideas

After collecting a list of project ideas, I identified four that look particularly promising (at least given the limited scope of my investigation). This section presents the four ideas in no particular order.

Talk to policy-makers about AI x-risk

The way to get x-risk-reducing regulations passed is to get policy-makers on board with the idea. The way to get them on board is to talk to them. Therefore, we should talk to them.

Talking to them may entail advocating for specific legislative proposals, or it may just entail raising general concern for AI x-risk.

Theory of change:

Increases the chance that safety legislation gets passed or regulations get put in place. Likely also increases the chance of an international treaty.

Who’s working on it?

Center for AI Safety / CAIS Action Fund (US); Control AI (UK/US); Encode AI (US/global); Good Ancestors (Australia); Machine Intelligence Research Institute (US/global); Palisade Research (US); PauseAI US (US).

(That’s not as many as it sounds like because some of these orgs have one or fewer full-time-employee-equivalents talking to policy-makers.)

Various other groups do political advocacy on AI risk, but mainly on sub-existential risks. The list above only includes orgs that I know have done advocacy on existential risk specifically.

Pros:

Political advocacy is neglected compared to policy research.
You can advocate to the public or directly to policy-makers. Both can help, but talking to policy-makers is more “leverage-efficient” than public outreach because policy-makers have much more leverage over policy.
- According to my back-of-the-envelope calculation on policy-maker advocacy vs. public protests (see Appendix), policy-maker advocacy looks more cost-effective (although I was writing on the back of a very small envelope, so to speak).
- Policy-maker advocacy can be bottlenecked on public support—they don’t want to support policies that their constituents dislike—but this isn’t a problem because AI safety regulation is popular among the general public.
More advocacy is better: bringing up AI risk repeatedly makes it more likely that policy-makers will take notice.

Cons:

Poorly executed advocacy has a risk of turning off policy-makers.
- See Downsides of policy/advocacy (and why they’re not too big), specifically the fourth downside.
Now may be too early. See Appendix: When is the right time for advocacy?

Some comments on political advocacy:

According to the book Lobbying and Policy Change: Who Wins, Who Loses, and Why (2009), most factors could not predict whether lobbying efforts would succeed or fail. One of the best predictors of lobbying success was the number of employed lobbyists who previously worked as government policy-makers. This suggests that political advocacy orgs should try to hire former policy-makers/staffers.
It is possible to hire generalist lobbyists who have political experience and will lobby for any cause. AI risk orgs could hire them.
Advocacy in the US or China is best because those are the countries with by far the most advanced AI. Ideally, we also want good policies in China, but I can’t confidently recommend advocacy there, see Policy/advocacy in China.
- Advocacy in the UK has had the most success. UK AI policy is less directly relevant because there are no frontier AI companies based in the UK, but (1) companies still care about being able to operate in the UK and (2) UK policy can be a template for other countries or for international agreements.2
- Advocacy in California looks promising, and all American frontier AI companies are based in California.
There’s a question as to what policy positions we should advocate for, but I believe there are many correct answers. See What kinds of policies might reduce AI x-risk?

Write AI x-risk legislation

AI policy research is relatively well-funded, but little work has been done to convert the results of this research into fully fleshed-out bills. Writers can learn from AI policy researchers what sorts of regulation might work, and learn from advocates what regulations they want and what they expect policy-makers to support.

This work also requires prioritizing which policy proposals look most promising and converting those into draft legislation. I think of that as part of the same work, but it could also be separate—for example, a team of AI safety researchers could prioritize policy proposals and sketch out legislation, and a separate team of legal experts (who don’t even necessarily need to know anything about AI) can convert those sketches into usable text.

Theory of change:

Many policy-makers care about AI risk and would support legislation, but there’s a big difference between “would support legislation” and “would personally draft legislation”. To get AI legislation passed, it helps if the legislation is already written. Instead of telling policy-makers

AI x-risk is a big deal, please write some legislation.

it’s a much easier ask if you can say

AI x-risk is a big deal, here is a bill that I already wrote, would you be interested in sponsoring it?

Who’s working on it?

Center for AI Policy (now mostly defunct); Center for AI Safety / CAIS Action Fund; Encode AI.

Others may be working on it as well, since legislation often doesn’t get published. I spoke to someone who has been involved in writing AI risk legislation, and they said that few people are working on this, so I don’t think the full list is much longer than the names I have.

(My contact also said that they wished more people were writing legislation.)

See my notes for a list of AI safety bills that have been introduced in the US, UK, and EU. Most of those bills were written by legislators, not by nonprofits, as far as I can tell.

Pros:

It’s common for policy-makers to sponsor bills that were written by third parties.
A small number of writers could draft a (relatively) large volume of legislation by leaning on pre-existing research.
There are lawyers who specialize in writing legislation. You can hire them to do the bulk of the work (you don’t need value alignment or even much skill at hiring, just find a law firm with a good reputation).
The general idea of “write AI legislation” looks good under many beliefs about AI risk. But your beliefs will impact what kinds of legislation you want.

Cons:

The natural argument against writing legislation is that it’s too early and we don’t know how to regulate AI yet. (Related from the Appendix: When is the right time for advocacy?)
- That’s sort of true, but if timelines are short, then we don’t have time to wait; we have to just do our best.
- And I don’t think this concern is fatal: we do have some concrete ideas about how to regulate AI, so we can write legislation for those.
- And it’s still a good idea to get some regulations in place now, and then we can pass new regulations later as necessary. That’s how regulation in nascent industries often works.

Advocate to change AI training to make LLMs more animal-friendly

LLMs undergo post-training to make their outputs satisfy AI companies’ criteria. For example, Anthropic post-trains its models to be “helpful, honest, and harmless”. AI companies could use the same process to make LLMs give regard to animal welfare.

Animal advocates could use a few strategies to make this happen, for example:

Build a benchmark that measures LLMs’ friendliness toward animals and try to get AI companies to train on that benchmark.
Advocate for AI companies to include animal welfare in AI constitutions/model specs.
Advocate for AI companies to incorporate animal welfare when doing RLHF, or ask to directly participate in RLHF.

Theory of change:

Insofar as the current alignment paradigm works at aligning AIs to human preferences, incorporating animal welfare into post-training would align LLMs to animal welfare in the same way.

Who’s working on it?

Compassion in Machine Learning (CaML); Sentient Futures. (They mainly do research to develop animal-friendliness benchmarks and other related projects, but they have also worked with AI companies.)

For some useful background, see Road to AnimalHarmBench by Artūrs Kaņepājs and Constance Li.

Pros:

People at AI companies have told me that getting a company to pay attention to animal welfare isn’t too difficult—in fact, one frontier company already uses an animal welfare benchmark.
- As I understand, AI companies don’t want to be seen as imposing their own values on LLMs, but they are open to tuning the values based on what external parties want.
Insofar as post-training works at preventing misalignment risk, it should also prevent suffering-risk / animal-welfare-risk.
- I don’t expect current alignment techniques to continue working on superintelligent AI, so I don’t expect them to make ASI friendly toward animals, either. But if I’m right, then we won’t get a friendly-to-humans AI that causes astronomical animal suffering; we will get a paperclip maximizer.
Even if current known techniques can’t help get AI to care about animals, this work could get a foot in the door, establishing relationships between animal advocates and AI companies and increasing the chances that the companies will pay attention to animal welfare in their future work.

Cons:

If I’m right that the current alignment paradigm won’t scale to superintelligence, then animal-friendliness (post-)training will fail because it relies on the same foundations as the current alignment paradigm.
It might turn out to be difficult to get AI companies to implement animal welfare mitigations.
There might be consumer backlash, which could make frontier models less friendly to animals in the long run.
If TAI arrives soon, there may not be time for this intervention to have an effect. It could take too long to get the new post-training implemented; or it could be that current-gen models will perform well on friendliness-to-animals benchmarks, but this will not be due to true alignment, and there won’t be enough time to iterate.
Aligning current-gen AIs to human preferences might make them better at assisting with alignment research, but it seems less likely that aligning current-gen AIs to animal welfare would carry through to future generations—it’s not clear that animal-aligned AIs would be more helpful at aligning future AIs to animal welfare.

Develop new plans / evaluate existing plans to improve post-TAI animal welfare

Some people have proposed plans for making TAI go well for animals, but I have reservations:

Most plans only work under long timelines (ex: “broadly influence society to care more about animals”).
Some plans focus specifically on farm animal welfare, and it seems very unlikely that factory farming will continue to exist in the long term (see Using TAI to improve farm animal welfare).
Many plans assume a particular future in which we develop transformative AI, but the world does not radically change—for example, plans about how TAI can help animal activists be more effective. I think this future is quite unlikely, and even if it does occur, there’s no particular need to figure out what to do in advance.

(Lizka Vaintrob and Ben West in A shallow review of what transformative AI means for animal welfare raised essentially the same reservations. See their article for more detailed reasoning on this topic.)

In light of these reservations, I would like to see research on post-TAI animal welfare interventions that look good (1) given short timelines and (2) without having to make strong predictions about what the future will look like for animals (e.g. without assuming that factory farming will exist).

Since I have pressed the importance of short timelines, I’m not envisioning a long-term research project. I expect it would be possible to come up with useful results in 3–6 months (maybe even less). The research should be laser-focused on finding near-term actions that take a few years at most, but still have a good chance of making the post-TAI future better for animals.

(I identified Advocate to change AI training to make LLMs more animal-friendly as potentially a top intervention after about a week of research, although to be fair, that’s mostly because I talked to other people who had done more research than me.)

Theory of change:

AI-for-animals interventions are underexplored. I expect that a few months of well-targeted research could turn up useful information about how to make AI go well for animals.

Who’s working on it?

Center on Long-Term Risk, Sentience Institute, and some individuals have written project proposals on AI-for-animals, but they almost always hinge on AI timelines being long. The closest thing I’m aware of is Max Taylor’s Bringing about animal-inclusive AI, which does include short-timelines proposals, but they are not directly actionable. For example, one idea is “representation of animals in AI decision-making”, which is an action an AI company could take, but AI companies are not the relevant actors. An actionable project would be something like “a nonprofit uses its connections at an AI company to persuade/pressure the company to include representation of animals in its AI decision-making”.

NYU Center for Mind, Ethics, and Policy and Sentient Futures have done similar work, but nothing exactly like this project proposal. I expect they could do a good job at identifying/prioritizing AI-for-animals interventions that fit my criteria. I would be excited to see a follow-up to Max Taylor’s Bringing about animal-inclusive AI focused on converting his ideas into actionable projects.

Pros:

AI-for-animals seems more tractable than other post-TAI welfare causes (e.g. AI welfare or S-risks from cooperation failure). There are already proposed interventions that could work if timelines are short.
A short research project may come up with useful ideas, or at least prioritize between pre-existing ideas.

Cons:

A research project might not come up with any really good ideas. Pre-existing research has mostly failed to come up with good ideas that work under short timelines (although to a large extent, that’s because they weren’t trying to).
I’m suspicious of “meta” work in general, and I’m suspicious of research because I personally like doing research, and I believe the value of research is usually overrated by researchers. It might be better to work directly on AI-for-animals—my current favorite “direct” project idea is Advocate to change AI training to make LLMs more animal-friendly or similar.

Honorable mentions

Directly push for an international AI treaty

The best kind of AI regulation is the kind that every country agrees to (or at least every country that has near-frontier AI technology).

If we need an international treaty to ensure that nobody builds a misaligned AI, then an obvious thing to do is to talk directly to national leaders about how we need an international treaty.

Theory of change:

An internationally-agreed moratorium on advanced AI would straightforwardly prevent advanced AI from killing everyone or otherwise destroying most of the value of the future.

One way to get an international treaty is to talk to governments and tell them you think they should sign an international treaty.

Who’s working on it?

Global AI Risks Initiative. There are other orgs that are doing work with the ultimate goal of an international treaty, but to my knowledge, they’re not directly pushing for a treaty. Those other orgs include: Future of Life Institute; Machine Intelligence Research Institute; PauseAI Global; PauseAI US; and Safe AI Forum.

Pros:

There is a short causal chain from “advocate for an international treaty” to “ASI doesn’t kill everyone”.
It’s high-leverage—you only need to get a relatively small set of people on board.

Cons:

There is not much political will for an international treaty, especially a strong one. Public advocacy and smaller-scale political advocacy seem better for that reason, at least for now.
This idea only works if you can figure out who will do a good job at pushing for an international treaty. I think it’s more difficult than generic public advocacy or talking to policy-makers about x-risk.

This might be one of my top ideas if I knew how to do it, and I wish I could put it on my top-ideas list, but I don’t know how to do it.

Organize a voluntary commitment by AI scientists not to build advanced AI

I heard this idea from Toby Ord on the 80,00 Hours podcast #219. He said,

If AI kills us and we end up standing front of St. Peter, and he asks, “Well did you try a voluntary agreement not to build it?” And we said, “No, we thought it wouldn’t work”, that’s not a good look for us.

At the 1975 Asilomar Conference, the international community of biologists voluntarily agreed not to conduct dangerous experiments on recombinant DNA. Perhaps something similar could work for TAI. The dangers of advanced AI are widely recognized among top AI researchers; it may be possible to organize an agreement not to work on powerful AI systems.

(There are some details to be worked out as to exactly what sort of work qualifies as dangerous. As with my stance on what kinds of policies would be helpful, I believe there are many agreements we could reach that would be better than the status quo. I expect leading AI researchers can collectively work out an operationalization that’s better than nothing.)

Theory of change:

If leading AI scientists all agree not to build advanced AI, then it does not get built. The question is whether a non-binding commitment will work. There have been similar successes in the past, especially in genetics with voluntary moratoriums on human cloning and human genetic engineering.

Who’s working on it?

Toby Ord has raised this idea. According to a personal communication, he did some research on its plausibility, but he is not actively working on it.

The CAIS Statement on AI Risk and FLI Pause Letter are related but weaker.

FLI organized the 2017 Asilomar Conference on Beneficial AI, but to my knowledge, the goal was not to make any commitments regarding AI safety.

Pros:

Unlike most project ideas, if this one succeeds, x-risk will immediately go down by multiple percentage points.
A voluntary commitment may be sufficient to prevent extinction, and it may be easier to achieve than a legally-mandated moratorium or strict regulations.
Talking to politicians or pushing for regulation has the problem that you’d really rather get regulations in all countries simultaneously. Researchers are (I think) less prone to inter-country adversarialism than nations’ leaders, especially between the West and China—American and Chinese scientists collaborate often.
You don’t need everyone to sign. If (say) Nobel Prize winners sign the agreement, it raises questions about why (say) the head of ML at OpenBrain hasn’t signed.

Cons:

In When is the right time for advocacy?, I argued that now is the right time. But a voluntary moratorium seems more likely than other ideas to fail if done at a suboptimal time because you need to get a ~majority on board.
- It’s not clear whether a failed attempt will decrease the probability of success for subsequent attempts. For example, there are many historical instances where a bill failed to pass, and then a very similar bill got passed later. But it’s not clear that this sort of voluntary agreement works the same way.
Voluntary commitments are easily violated. For example, biologists agreed not to do research on human cloning, but then a few rogue scientists (Richard Seed, etc.) did it anyway.
- But a voluntary agreement can create strong social pressure not to violate it.
AI researchers have a vested interest in being able to do AI research; policy-makers do not.
- But the dangers of advanced AI are much better understood among AI researchers than among policy-makers.
It is doubtful that AI company CEOs will agree to a moratorium.
- But if you get the majority of leading AI scientists to agree, CEOs will be left with insufficient talent to lead their research.

Peaceful protests

Organize peaceful protests to raise public concern and salience regarding AI risk. Historically, protests have asked for a pause on AI development, although that might not be the only reasonable ask.

(I don’t have any other specific asks in mind. The advantage of “pause” is that it’s a simple message that fits on a picket sign.)

Theory of change:

Protests may increase public support and salience via reaching people in person or via media (news reporting, etc.). They may also provide a signal to policy-makers about what their constituents want.

Who’s working on it?

PauseAI Global; PauseAI US.

Stop AI organizes disruptive protests (e.g. blockading AI company offices), and the evidence is ambiguous as to whether disruptive protests work. See “When Are Social Protests Effective?” (Shuman et al. 2024), although I should note that I think the authors overstate the strength of evidence for their claims—see my notes on the paper.

Pros:

Natural experiments suggest that protests are effective at changing voter behavior and/or increasing voter turnout.
- Experiments also find that peaceful protests increase support in a lab setting—see “Social Movement Strategy (Nonviolent Versus Violent) and the Garnering of Third-Party Support: A Meta-Analysis” (Orazani et al. 2021). For a summary, see my notes on the paper.
Protesting is widely seen as the thing you do when you are concerned about an issue. Many people take not-protesting as a sign that you aren’t serious. It’s valuable to be able to say “yes, we are taking AI risk seriously, you can tell because we are staging protests”. Regardless of the cost-effectiveness of marginal protesters, it’s good for there to be nonzero protests happening.
Protests make it clear to policy-makers that their constituents care about an issue. This is especially important for AI because the general public is very worried about AI (see e.g. 2025 Pew poll), but the issue is not high-salience (see 2023 YouGov poll: respondents were worried about AI extinction risk, but it only ranked as the #6 most concerning x-risk). Protests increase its salience.
Protests are an effective way to get media attention. It’s common for protests with only a dozen participants to get news coverage.
I expect that a really good media project would be more cost-effective, but creating a good media project requires exceptional talent. Organizing a protest requires some skill, but the bar isn’t particularly high. Therefore, you can support protests without having to identify top-tier talent.
Protests are highly neglected, and I expect them to continue to be neglected.

Cons:

Some people are concerned that protests can backfire.
- I’m not concerned about peaceful protests backfiring. The scientific literature universally shows that peaceful protests have a positive effect, although the strength of the evidence could be better—see Do Protests Work? A Critical Review.
My back-of-the-envelope calculation suggested that talking directly to policy-makers is more cost-effective.
Now may be too early. See When is the right time for advocacy?
It may be bad for large funders to fund protests because it could create an appearance of astroturfing.
- This is an argument against large funders funding them, but in favor of individual donors supporting protests.

Media about dangers of AI

Create media explaining why AI x-risk is a big deal and what we should do about it.

Things like:

Books (ex: If Anyone Builds It, Everyone Dies)
News articles (ex: Existential Risk Observatory)
Videos (ex: Rob Miles)
Forecasts of how AI can go badly (ex: AI 2027)

Theory of change:

Media increase public concern, which makes policy-makers more likely to put good regulations in place.

Who’s working on it?

80,000 Hours; AI Futures Project; AI Safety and Governance Fund; AI Safety Info; Centre pour la Sécurité de l’IA (CeSIA); CivAI; Existential Risk Observatory; Machine Intelligence Research Institute; Midas Project; various media projects by individual people.

Pros:

We need to get good policies in place. Media can influence policy-makers, and can influence the public, which is important because policy-makers largely want to do what their constituents want.
Polls show that the public is concerned about AI risk and even x-risk, but it’s not a high-priority issue. Media can make it more salient and/or raise “common knowledge” of concern about AI.

Cons:

The impact of media is fat-tailed and heavily depends on quality. It’s hard to identify which media projects to fund.
Poorly done or misleading media projects could backfire.

Message testing

Many people have strong opinions about the correct way to communicate AI safety to a non-technical audience, but people’s hypotheses have largely not been tested. A project could make a systematic attempt to compare different messages and survey listeners to assess effectiveness.

Theory of change:

We can’t ultimately get good AI safety outcomes unless we communicate the importance of the problem, and having data on message effectiveness will help with that.

Who’s working on it?

AI Safety and Governance Fund has an ongoing project (see Manifund) to test AI risk messages via online ads. The project is currently moving slowly due to lack of funding.

Some advocacy orgs have done small-scale message testing on their own materials.

Pros:

A smallish investment in empirical data could inform a large amount of messaging going forward.
Some types of experiments (using online ads or Mechanical Turk) can scale well with funding.

Cons:

The best types of communication may be long, individually tailored (e.g. in one-on-one conversations with policy-makers), or otherwise difficult to test empirically.
I suspect that a person with excellent communication skills would not benefit much from seeing A/B-tested messaging because they can already intuit which wording will be best. (But it may be difficult to identify and hire those people.)
This is the sort of thing that might be best to do internally by an advocacy org that already has a reasonable idea of what kind of message it wants to send.

Host a website for discussion of AI safety and other important issues

LessWrong and the Effective Altruism Forum are upstream of a large quantity of work (in AI safety as well as other EA cause areas). It is valuable that these websites continue to exist, and that moderators and web developers continue to work to preserve/improve the quality of discussion.

Realistically, it doesn’t make sense to start a new discussion forum, so this idea amounts to “fund/support LessWrong and/or the EA Forum”.

Theory of change:

On Lightcone Infrastructure’s Manifund, Oliver Habryka lists some concrete outcomes that are attributable to the existence of LessWrong. You could probably find similar evidence of impact for the EA Forum.

Who’s working on it?

Centre for Effective Altruism; Lightcone Infrastructure.

Pros:

Nearly every project on my list has benefited in some way from the existence of LessWrong or the EA Forum.
Hosting a platform for sharing research/discussion is cheaper (and therefore arguably more cost-effective) than directly conducting research.

Cons:

The benefits are diffuse: high-quality discussion forums provide small-to-moderate benefits to every cause, but usually not huge benefits. So it may be better to directly support your favorite intervention(s).

A toy model: Suppose that

There are 10 categories of AI safety work.
LessWrong makes each of them 20% better.
The average AI safety work produces 1 utility point.
Well-directed AI policy produces 5 utility points.

Then marginal work on LessWrong is worth 2 utility points, and my favorite AI policy orgs are worth 5 points.

List of other project ideas

A project not being a top idea doesn’t mean it’s bad. In fact, it’s likely that at least one or two of these ideas should be on my top-ideas list; I just don’t know which ones.

I sourced ideas from:

reviewing other lists of project ideas and filtering for the relevant ones;
looking at what existing orgs are working on;
writing down any (sufficiently broad) idea I came across over the last ~6 months;
writing down any idea I thought of.

Ideas are roughly ordered from broadest to most specific.

AI-for-animals ideas

Neartermist animal advocacy

There are various projects to improve current conditions for animals, particularly farm animals: cage-free campaigns, humane slaughter, vegetarian activism, etc. I will lump all these projects together for the purposes of this report.

Theory of change:

Animal advocacy increases concern for animals, which likely has positive flow-through effects into the future, by affecting future generations or by shaping the values of the transformative AI that will control the future.

Who’s working on it?

Too many to list.

Pros:

Neartermist animal advocacy has the dual benefit of definitely helping animals today, and building momentum to make future work more effective (to borrow a framing from Jeff Sebo).
Neartermist animal advocacy is tractable and has clear feedback loops. It looks especially promising if you’re highly uncertain or clueless about longtermist or post-TAI interventions.

Cons:

The benefits are diffuse. Creating one new vegan helps many animals in the short term, but has only a tiny effect on society’s future values. I expect direct attempts to improve AI alignment-to-animals to be much more cost-effective.
- I created a back-of-the-envelope calculation that aligns with my initial expectation: my BOTEC-informed guess is that direct advocacy on AI values (by promoting a friendliness-to-animals LLM benchmark) is 2–3 orders of magnitude more cost-effective than conventional animal advocacy.
Neartermist animal advocacy works best if timelines are long. Timelines are likely not long.
Effective animal activists generally regard corporate campaigns as more effective than advocacy directed at consumers, but changing corporate practices seems less relevant for shifting society’s values.

Using TAI to improve farm animal welfare

I’m concerned about how TAI could negatively impact non-human welfare. There are some proposals on how TAI could negatively impact farm animals (e.g. by making factory farming more efficient), and on how animal activists could use TAI to make their activism more effective. I will take these proposals as a broad category rather than discussing them individually.

Theory of change:

Depends on the specific proposal.

Who’s working on it?

Electric Sheep; Hive (see Transformative AI and Animals: Animal Advocacy Under A Post-Work Society); Open Paws; Wild Animal Initiative (see Transformative AI and wild animals: An exploration).

Pros:

Highly neglected.
I am generally skeptical of interventions of the form “teach people to leverage AI to do X better”, but farm animal advocacy seems sufficiently important that it might be worthwhile in this case.

Cons:

Although two of my top ideas relate to post-TAI animal welfare (Develop new plans / evaluate existing plans to improve post-TAI animal welfare and Advocate to change AI training to make LLMs more animal-friendly), I don’t think it’s worth focusing on farm animal welfare in particular.

Factory farming interventions only matter if factory farming still exists. Cultured meat outcompetes factory farming once you get a sufficiently strong understanding of biology (there’s no way growing a whole chicken is the cheapest possible way to create chicken-meat). We are far from that level of understanding, but I would be surprised if (aligned) TAI couldn’t figure it out.
The scale of wild animal welfare is orders of magnitude larger than that of factory farming. The case for prioritizing farm animals over wild animals is that we don’t have the power or knowledge to positively influence nature, but TAI should change the equation.
Space colonization would ultimately dominate earth-based welfare, so questions about panspermia or digital minds have a bigger expected impact.
Proposals for how to use TAI to improve animal advocacy only make sense if TAI does not cause value lock-in. If there is no value lock-in, then there’s no strong reason to spend time now trying to figure out how to use TAI. It would be better to wait until after TAI because at that point, we will have a much better understanding of how TAI works, and there’s no urgency.

Lobby governments to include animal welfare in AI regulations

If governments put safety restrictions on advanced AI, they could also create rules about animal welfare.

Theory of change:

Getting regulations in place would force companies’ AIs to respect animal welfare.

Pros:

One set of regulations can alter the behavior of many frontier companies.
If companies voluntarily change their behavior, they can regress at any time with no consequences. But companies have to obey regulations.

Cons:

It’s unclear what exactly regulations could do about animal welfare. AI safety regulations, insofar as they exist (which they mostly don’t), don’t dictate how LLMs are required to behave; they dictate what companies are required to do to make LLMs safe. What is a regulatory rule that policy-makers would plausibly be on board with, that would also influence model behavior to be friendlier to animals?
Influencing the government on animal welfare seems harder than influencing AI companies.

Traditional animal advocacy targeted at frontier AI developers

Theory of change:

AI developers become more concerned for animal welfare, and they make AI development decisions that improve the likelihood that transformative AI is good for animals.

Pros:

Similar to neartermist animal advocacy, but plausibly more cost-effective because it’s more targeted.

Cons:

It’s not known whether techniques like animal welfare ads are effective in general, and they may even be particularly ineffective among demographics like AI developers.
Even if AI developers cared more about animal welfare, it’s not clear that this would carry through to their work on AI.
In 2016, I created a back-of-the-envelope calculation on this idea, and the result wasn’t as good as I expected (it looked worse than standard animal advocacy, if you assume the animal advocacy propagates values into the far future). However, the numbers are outdated because we know a lot more about AI now than we did in 2016.

Research which alignment strategies are more likely to be good for animals

Some alignment strategies may be better or worse for non-human welfare. For example, I expect CEV would be better than the current paradigm of “teach the LLM to say things that RLHF judges like”, which is better than “hard-code (GOFAI-style) whatever moral rules the AI company thinks are correct”.

A research project could go more in-depth on which alignment techniques are most likely to be good for animals (or digital minds, etc.).

Theory of change:

Identify promising alignment techniques, in the hope that people use those techniques. There are enough animal-friendly alignment researchers at AI companies that this might happen.

Pros:

To my knowledge, this question has never been studied.
Some alignment techniques may be much better for animals than others.

Cons:

We have a poor understanding of what ASI will look like, which makes it very hard to say what will work for animal welfare.
We don’t know how to align ASI to any goals at all. We can’t align AI to animal welfare until we can align AI to something.
In the world where alignment turns out to be tractable, it’s likely that there will be strong incentives shaping how ASI is aligned. The choice of whether to use (say) something-like-CEV or something-like-RLHF will be difficult to influence.

AI policy/advocacy ideas

Improving US <> China relations / international peace

Theory of change:

US and China (and other countries) need to agree not to build dangerous AI. Generically improving international cooperation, especially between the US and China, increases the chance that nations cooperate on AI (non-)development.

Who’s working on it?

Too many to list. Some examples: Asia Society’s Center on US–China Relations; Carnegie Endowment for International Peace; Carter Center; National Committee on United States–China Relations; US-China Policy Foundation.

Orgs that work on international cooperation specifically on AI safety (although not necessarily existential risk) include: AI Governance Exchange; Global AI Risks Initiative; Safe AI Forum.

Pros:

International cooperation is likely necessary to prevent existentially risky AI from being built.
Efforts to improve cooperation have succeeded in the past; for example, the US–China Strategic and Economic Dialogue (source).
International cooperation has wide-ranging benefits; efforts can attract funding from many parties with varying agendas.

Cons:

The route to preventing extinction is indirect, which dilutes the cost-effectiveness of this intervention.
International cooperation is far from neglected. Marginal efforts might not make much difference.

Talk to international peace orgs about AI

Theory of change:

Most international peace orgs probably aren’t aware of how important AI regulation is, and they would likely help develop international treaties on AI if they knew it was important.

Who’s working on it?

Nobody that I know of.

Pros and cons are largely the same as Improving US <> China relations / international peace. In addition:

Pros:

This plan is higher-leverage than simply funding international peace orgs.

Cons:

Unclear how to do it. What sorts of evidence would the orgs find persuasive? Which orgs are best suited to working on AI-related cooperation? Those questions are answerable, but I’m not in a good position to answer them.

Increasing government expertise about AI

Talk to policy-makers or create educational materials about how AI works, or help place AI experts in relevant policy roles.

Theory of change:

Policy-makers can do a more effective job of regulating AI if they understand it better.

Pros:

Increasing government expertise may improve the quality of AI regulations.

Cons:

The experts need to actually care about x-risk. Plenty of experts want to accelerate AI development/prevent regulation. For extant projects designed to increase AI expertise, I am skeptical that the expertise would be appropriately x-risk-oriented.
- In practice, “hire experts” often means “hire current or former AI company employees”, which is a recipe for regulatory capture. I expect this would significantly decrease our chance of getting useful x-risk-reducing regulations.
Expertise is much less of a bottleneck than willingness to regulate AI. If I can spend $1 on increasing willingness or $1 on expertise, I’d much rather spend it on willingness.
- And in a sense, the AI safety community already invests way more in expertise (via policy research) than in advocacy. On the margin, we need advocacy more.
The downside of poorly-targeted AI safety regulations is that they end up hurting economic development. That’s bad, but it looks pretty trivial in a cost-benefit analysis compared to extinction.
I wrote a longer comment about this subject on the EA Forum.
Unless by “expertise” we’re talking about “expertise at recognizing that AI x-risk is a big problem”. In which case, yes, we need expertise.
- Right now, the main strategy for getting x-risk people into government is “pretend not to care about x-risk so you seem normal, and never voice your concerns”. In which case, what’s the point?
- I think a better strategy is “talk to policy-makers about x-risk and straightforwardly tell them what you believe.”
I’ve heard it argued that increasing government expertise is low tractability because government is so big, and making internal changes like that is slow. I don’t think this is a strong consideration because other approaches to AI safety are also low tractability. (I still don’t think increasing expertise is a good plan, but this particular argument seems weak.)
Policy-makers are largely in the dark about x-risk, which is indeed a problem. But I don’t see clean routes to increasing AI expertise that don’t also push in the wrong direction. Raising concern about the importance of TAI has historically led people to believe things like “I need to be the one who controls TAI, so I will start a new AI company” or “we need to make sure we get TAI before China”. Generically increasing AI expertise is, in my best estimation, net harmful. For more on this, see Advocacy should emphasize x-risk and misalignment risk.

Policy/advocacy in China

Take any of my ideas on AI policy/advocacy, and do that in China instead of in the West.

Theory of change:

The high-level argument for policy/advocacy in China is largely the same as the argument for prioritizing AI risk policy/advocacy in general. China is currently the #2 leading country in AI development, so it’s important that Chinese AI developers take safety seriously.

Who’s working on it?

AI Governance Exchange; Safe AI Forum (sort of); perhaps some Chinese orgs that I’m not familiar with.

Pros:

Many of the ideas listed above are intended to improve the state of AI policy in the US/UK. The pros for those ideas largely also apply to equivalent projects conducted in China (modulo the obvious differences, e.g. China is not a representative democracy, so political advocacy works differently).

Cons:

Probably nobody in China is reading this report. Advocating for Chinese policy as a non-Chinese person is fraught because the CCP will not trust our motivations, just as the American government would not trust a Chinese philanthropist who funds American AI safety advocacy.
According to an 80,000 Hours career review: “[The Chinese government] is often wary of non-governmental groups that try to bring about grassroots change. If an organisation is blacklisted, then that’s a nearly irreversible setback.”

A comment on my state of knowledge:

I don’t know much about the current state of AI safety in China, or what sort of advocacy might work. I did not prioritize looking into it because my initial impression is that I would require high confidence before recommending any interventions (due to the cons listed above), and I would be unlikely to achieve the necessary level of confidence in a reasonable amount of time.

Corporate campaigns to advocate for safety

Run public campaigns to advocate for companies to improve safety practices and call out unsafe behavior.

Theory of change:

Companies may improve their behavior in the interest of maintaining a good public image.

Who’s working on it?

Midas Project; More Light.

Pros:

Corporate campaigns have worked well in animal advocacy.
Companies are smaller than governments, which means they’re more agile and potentially easier to influence.

Cons:

Companies have strong internal incentives to be unsafe. By contrast, governments don’t have a profit motive. They may be harder to move, but they have less reason to oppose safety efforts.
Making more actors safe is better than making fewer actors safe. International treaty > single-country regulation > single-company safety efforts.
Democratic governments are specifically designed to do what people want. That doesn’t always happen, but at least there are mechanisms pushing them that way. Companies are not democratic.
There is a good chance that safety standards strong enough to prevent human extinction would pose an existential threat to AI companies (or at least would be incompatible with their current valuations). If that’s the case, then corporate campaigns will not be able to get companies to implement adequate safety measures.

Overall, this seems similar to political advocacy, but worse.

Develop AI safety/security/evaluation standards

Work inside a company, as a nonprofit, or with a governmental body (NIST, ISO, etc.) to develop AI safety standards.

Theory of change:

Sufficiently well-written standards can define under what conditions a frontier AI is safe, and potentially enforce those conditions.

Who’s working on it?

AI Standards Lab; various AI companies; various governmental bodies.

Pros:

Companies might abide by the standards.
Standards can provide a template with which to write regulations.

Cons:

Standards are not currently the bottleneck to getting regulation written. We already have a substantial amount of work on AI safety standards, but we still don’t have good regulations. There aren’t people waiting around to write legislation if only they had some standards they could use. (Of course, more standards would still be better.)
If standards are voluntary, companies can stop abiding by them when they turn out to be hard to satisfy (and in fact, companies have already done that on multiple occasions, with respect to their self-imposed safety standards).
Standards agencies such as NIST generally don’t have enforcement power.
Nobody knows how to write standards that will prevent extinction if implemented. See Appendix for more on this.

Slow down Chinese AI development via ordinary foreign policy

Both the “slow down AI” crowd and the “maintain America’s lead” crowd agree that it is good for China’s AI development to slow down. The American government could accomplish this using foreign policy levers such as:

restricting chip exports;
encouraging scientists to immigrate to the US.

Theory of change:

On my view, slowing down Chinese AI development is good because it gives the leading AI developers in the United States more room to slow down. It also makes it less likely that a Chinese company develops misaligned TAI (although right now, US companies are more likely to develop TAI first).

Slowing down Chinese AI development looks good on the “maintain America’s lead” view, although I believe this view is misguided—making TAI safe is much more important than making one country build it before another.

Who’s working on it?

Center for Security and Emerging Technology has written memos recommending similar interventions. I expect there are some other orgs doing similar activities, including orgs that are more concerned about national security than AI risk.

Pros:

Looks (plausibly) good on multiple worldviews.

Cons:

Some policies could antagonize China and make cooperation more difficult.
- Mutual sabotage between the US and China would probably decrease AI x-risk, but it would also have negative effects. I’d rather the countries increase safety via mutual cooperation.
- This depends on the policy. Improving AI model security is fine, relaxing immigration restrictions is probably fine, but export restrictions or tariffs would likely heighten international tensions. (The United States already has export restrictions and tariffs, but it might be bad to add more on the margin.)
Advocating for these sorts of foreign policy interventions is likely not cost-effective because they’re in controversial political areas that already see significant funding and effort. For example, there are already strong and well-funded interests arguing both for and against immigration.
On the “slow down AI” view, this seems less promising than domestic US regulation because US-based companies look significantly more likely than China to be the first to build misaligned ASI.

Whistleblower protection/support

Provide legal support for whistleblowers inside AI companies and assist in publicizing whistleblowers’ findings (e.g. setting up press interviews).

Theory of change:

A few ways this could help:

Whistleblowers can force companies to change their unsafe behavior.
The possibility of whistleblowers incentivizes companies to be safe.
Publicizing companies’ bad behavior can raise public concern about AI safety.

Who’s working on it?

More Light.

Pros:

Whistleblower support could be high-leverage: the whistleblowers themselves bring the important information, but they often can’t do much without help.

Cons:

Only helps if there are things worth whistleblowing on. There might not be any warning shots (see When is the right time for advocacy? for some relevant discussion).
- OpenAI’s bad behavior on secret NDAs was whistleblow-worthy, but it wasn’t directly related to AI risk, and it’s not clear that the news about OpenAI’s bad behavior decreased x-risk.
Other paths to impact are more direct, e.g. media projects or corporate campaigns.

Opinion polling

Run polls to learn public opinion on AI safety.

Theory of change:

Polls can inform policy-makers about what their constituents want. They also inform people working on AI safety about where their views most align with the public, which can help them prioritize.

Who’s working on it?

AI Policy Institute; traditional polling agencies (Pew and YouGov have done polls on people’s views on AI).

Pros:

When talking to policy-makers, it’s useful to be able to point to polls as evidence that the public cares about AI safety.
Polls provide common knowledge of concern for AI risk.

Cons:

A good amount of polling already exists. If we already know that people in 2024 were concerned about AI risk, there’s not as much value in knowing that they’re still concerned in 2025.
The main benefit of polls is to empower advocacy, but we have precious little advocacy right now.

Help AI company employees improve safety within their companies

Work with people in AI companies (by organizing conferences, peer support, etc.) to help them learn about good safety practices.

Theory of change:

AI company employees can push leadership to implement stronger internal safety standards.

Who’s working on it?

AI Leadership Collective.

Pros:

If internal employees work more on safety, that will make AI companies safer.

Cons:

This theory of change has the same issues as corporate campaigns: companies have strong incentives to be unsafe; global (or at least national) safety measures are better than single-company measures.

Working with AI company employees might be sufficiently high-leverage to make up for my concerns. Answering that question would require going more in-depth, so I will go with my intuition and say it’s probably not as cost-effective as my top ideas.

Direct talks with AI companies to make them safer

If you can talk to policy-makers about AI x-risk, then maybe you can also talk to AI company executives about AI safety.

Theory of change:

AI company execs have significant control over the direction of AI. If they started prioritizing safety to a significantly greater extent, they could probably do a lot to decrease x-risk.

Who’s working on it?

To my knowledge, nobody is systematically working on this.

Pros:

High-leverage: you may be able to prevent extinction by changing the minds of a half-dozen or so people.

Cons:

Similar issues to corporate campaigns: governments have less incentive to be unsafe; it’s better to make all companies safe simultaneously (via regulation). Therefore, political action seems more promising.
The CEOs of most AI companies are well aware that AI poses an extinction risk, but they are building it anyway, and they are massively under-investing in safety anyway. It’s not clear what additional information would change their minds. So this seems worse than corporate campaigns.

People in relevant positions should push companies to be safer in ways that they can, but I don’t see any way to support this intervention as a philanthropist.

Monitor AI companies on safety standards

Track how well each frontier AI company does model risk assessments, security, misuse prevention, and other safety-relevant behaviors.

Theory of change:

Monitoring AI companies can inform policy-makers and the public about the state of company safety. Monitoring could have a similar effect as corporate campaigns, where it pushes companies to be safer, or it could even directly inform corporate campaigns.

Who’s working on it?

AI Lab Watch & AI Safety Claims Analysis; Future of Life Institute’s AI Safety Index; Midas Project; Safer AI.

Pros:

Easy to do: a solo developer can run a monitoring website as a side project.

Cons:

The theory of change seems somewhat weak to me. There are other ways to demonstrate AI risks to policy-makers and the public, and it’s not clear that this way is particularly good.
- But I don’t know what kind of impact the monitoring websites have had; maybe they’ve had some big positive influences that I don’t know about.

Create a petition or open letter on AI risk

Write a petition raising concern about AI risk or calling for action (such as a six-month pause or an international treaty). Get respected figures (AI experts, etc.) to sign the petition.

Theory of change:

A petition can make it apparent to policy-makers and the public that many people/experts are concerned about AI risk, while also creating common knowledge among concerned people that they are in good company if they speak up about it.

Who’s working on it?

aitreaty.org; Center for AI Safety / CAIS Action Fund; Future of Life Institute / FLI Action and Research, Inc.

Pros:

Brings AI risk into the public conversation and makes it easier to talk about.
Relatively easy to do—the main difficulty is in finding people to sign it who can bring credibility.

Cons:

The biggest con is that petitions have diminishing marginal utility, and several have been made already.
The path from a petition to concrete outcomes isn’t entirely clear (although I’m inclined to believe that petitions can work).

My sense is that the existing petitions have been quite helpful, but there isn’t clear value in creating more petitions. There may be some specific call to action that a new petition ought to put forward, but I’m not sure what that would be.

Create demonstrations of dangerous AI capabilities

Make AI risk concrete by building concrete demonstrations of how AI can be dangerous.

Theory of change:

Many people don’t find it plausible that AI could cause harm; concrete demonstrations may change their minds. It can also serve to make AI risk more visceral.

Who’s working on it?

FAR.AI; Palisade Research; some one-off work by others (e.g. Apart Research).

Pros:

Concrete demonstrations can aid advocacy efforts.

Cons:

Capability demonstrations may send the wrong message, encouraging accelerationism instead of caution.
- I am more concerned about this for general AI capability evaluations. For this project, I am specifically thinking of demonstrations of how AI can do harm.
You can’t create a concrete demonstration of superintelligent AI’s capabilities until you already have superintelligent AI, at which point it’s too late. Pre-superintelligent AIs can have scary capabilities, but demos are misleading in a sense because they may create a skewed understanding of where the risks come from.

I’m most optimistic about demonstrations where there is a clear plan for how to use them, e.g. Palisade Research builds its demos specifically to show to policy-makers. I think Palisade is doing a particularly good version of this idea, but for the most part, I think other ideas are better.

Sue OpenAI for violating its nonprofit mission

OpenAI:

Our mission is to ensure that artificial general intelligence […] benefits all of humanity.

OpenAI has more-or-less straightforwardly violated this mission in various ways. Humanity plausibly has grounds to sue OpenAI.

Theory of change:

A lawsuit would change OpenAI’s incentives and may force OpenAI to actually put humanity’s interest first, depending on how well the lawsuit goes.

Who’s working on it?

In 2024, Elon Musk filed a lawsuit against OpenAI on this basis. The lawsuit is set to go to trial in 2026.

Given that there is already an ongoing lawsuit, it may be better to support the existing suit (e.g. by writing an amicus brief or by offering expert testimony) than to start a new one.

Pros:

A lawsuit could force OpenAI to significantly improve safety.

Cons:

A failed lawsuit has numerous downsides—it can make the plaintiff look bad; it can set an unfavorable precedent; it’s expensive.
Legal matters may have other, hard-to-predict downsides, and I’m not qualified to evaluate them.
I am not a lawyer, but my impression is that courts are typically quite lenient about what nonprofits are allowed to do, so it would be difficult for a lawsuit to succeed.
Given that there is an ongoing lawsuit by Elon Musk, who is known to behave erratically, Musk may do something unpredictable that causes harm.

Send people AI safety books

Books are a tried-and-true method of explaining complex ideas. One could mail books on AI risk to Congress people, or staffers, or AI company execs, or other relevant people.

Theory of change:

A book can explain in detail why AI risk is a big deal and thus persuade people that it’s a big deal.

Who’s working on it?

MIRI did something similar when promoting their new book: “We cold-emailed a bunch of famous people (like Obama and Oprah)”. They were asking people to write blurbs for the book, which isn’t exactly what I had in mind, but it’s related.
Aidar Toktargazin has a Manifund project to give out AI safety books to researchers and professors at Nazarbayev University in Kazakhstan.

Pros:

Mailing books is cheap.

Cons:

Mailing books seems riskier than other kinds of advocacy—it could be viewed as excessively pushy.
- Mormons give out free books, and they’re viewed as pushy, but they’ve also grown a lot. It’s unclear whether Mormons’ publicity strategies are worth emulating.
It may be better to let publishers do their own publicity.

This idea might be really good, but it’s high-variance. I would not recommend it without significantly investigating the possible downsides first.

AI research ideas

I said I wasn’t going to focus on technical research or policy research, but I did incidentally come up with a few under-explored ideas. These are research projects that I’d like to see more work on, although I still believe advocacy is more important.

Research on how to get people to extrapolate

A key psychological mistake: “superintelligent AI has never caused extinction before, therefore it won’t happen.” Or: “AI is not currently dangerous, therefore it will never be dangerous.”

Compare: “Declaring a COVID emergency is silly; there are currently zero cases in San Francisco.” (I am slightly embarrassed to say that that is a thought I had in February 2020.)

Relatedly, some people expect there will not be much demand for AI regulation until we see a “warning shot”. Perhaps, but I’m concerned we will run into this failure-to-extrapolate phenomenon. AI has already demonstrated alignment failures (Bing Sydney comes to mind; or GPT 4o’s absurd sycophancy; or numerous xAI/Grok incidents). But clear examples of misalignment get fixed (because AI is still dumb enough for us to control). So people may draw the lesson that misalignment is fixable, and there may keep being progressively bigger incidents until we finally build an AI powerful enough to kill everyone.

I am concerned that the concept of AI x-risk will never be able to get sufficient attention due to this psychological mistake. Therefore, we need to figure out how to get people to stop making this mistake.

Theory of change:

Many people ignore AI x-risk because of this mistake. If we knew how to get people to extrapolate, we could use that knowledge to improve communication on AI risk.

Who’s working on it?

Some academic psychologists have done related research.

Pros:

If successful, this research would significantly increase how many people take AI risk seriously.

Cons:

We know from the history of psychology that it’s difficult to find psychological insights. I’d guess it would cost tens or hundreds of millions of dollars to produce meaningful results (if it’s even possible at all).
Psychology research takes a long time to pay off. That doesn’t work if timelines are short.
This research is not particularly neglected. The American Psychological Association wants more research on how to get people to care about climate change, which is related.
To the extent that science has already uncovered answers to this question, good communicators already know those answers. For example, studies have found that people are more likely to pay attention to future problems when you give concrete scenarios; but a good writer already does that. (See Deep Research (1, 2) for an attempt at finding relevant psychology studies, although only about a quarter of them are actually relevant.)

Rather than spending $50 million on psychology research and then $50 million on a variety of psychologically-motivated media projects informed by that research, I would rather just spend $100 million on media projects.

Investigate how to use AI to reduce other x-risks

An important argument against slowing down AI development is that we could use advanced AI to reduce other x-risks (climate change, nuclear war, etc.).

But an aligned AI wouldn’t automatically reduce x-risk. It may increase technological risks (e.g. synthetic biology) if offensive capabilities outscale defensive ones. An aligned AI could reduce nuclear risk by improving global coordination, but it’s not obvious that it would.

Therefore, it may be worth asking: Are there some paths of AI development that differentially reduce non-AI x-risk?

Theory of change:

Research on how to direct AI may inform efforts by the developers of advanced AI, which may ultimately reduce x-risk.

Who’s working on it?

Nobody, to my knowledge.

Pros:

If you believe we need to build TAI to avert non-AI x-risks, then it stands to reason that you should also want to know how to direct TAI to accomplish that end.
This line of research is highly neglected (to my knowledge, there are zero people working on it).

Cons:

Non-AI x-risks seem less concerning than AI x-risk, so it seems better to work directly on reducing AI x-risk.
If you believe we should slow down AI development, then this line of research doesn’t matter as much. And I do believe we should slow down AI development, and I believe that a wide range of worldviews should agree with me on that (see Appendix: A moratorium is the best outcome).
At a glance, the problem seems difficult to make progress on (see below).

My initial thoughts on this line of research:

You can’t control what general AI is good at. It would be good at everything. There is no known way to make it (say) good at defending against biological weapons, but bad at creating biological weapons.
A narrow “cooperation superintelligence” would be better than a narrow “scientist superintelligence” because the latter increases technological x-risk. But based on current trends in AI, my guess is that we could develop a “scientist ASI” that’s bad at cooperation, but we couldn’t develop a “cooperation ASI” that’s bad at science. So this idea is likely a dead end.
Even if we could build a “cooperation ASI”, we still need to solve alignment problems first. So it seems better to focus on solving alignment.

A short-timelines alignment plan that doesn’t rely on bootstrapping

To my knowledge, every major AI alignment plan depends on alignment bootstrapping, i.e., using AI to solve AI alignment. I am skeptical that bootstrapping will work, and even if you think it will probably work (with, say, 90% credence), you should still want a contingency plan.

Write a research agenda for how to solve AI alignment without using bootstrapping.

Theory of change:

If people come up with sufficiently good plans, then we might solve alignment.

Who’s working on it?

Peter Gebauer is running a contest for short-timelines AI safety plans, but the plans are allowed to depend on bootstrapping (e.g. Gebauer favorably cites DeepMind’s plan).

There are some plans that say something like “stop developing AI until we solve alignment” (ex: MIRI; Narrow Path), which is valid (and I agree), but it’s not a technical plan.

The closest thing I’ve seen is John Wentworth’s research agenda, but it specifically invokes the underpants gnome meme, i.e., the plan has a huge hole in the middle.

Pros:

Existing plans are insufficiently rigorous.
To my knowledge, there are zero meaningful plans that don’t rely on alignment bootstrapping. If bootstrapping turns out not to work, every plan fails. There is a gap to be filled.

Cons:

It is highly unlikely that any satisfactory plan exists.
- I think trying to create plans is a reasonable idea on the off chance that somebody does come up with a good plan, but I don’t think it’s a good use of marginal philanthropic resources.
An alignment plan still leaves non-alignment problems unsolved.
AI companies cannot be trusted to implement a safe plan even if one exists.
- But the plan existing does increase the chance that companies follow the plan, or that external pressures can force companies to follow it.

Rigorous analysis of the various ways alignment bootstrapping could fail

I’m pessimistic about the prospects of alignment bootstrapping, and I’ve seen various AI safety researchers express similar skepticism, but I’ve never seen a rigorous analysis of the concerns with alignment bootstrapping.

Theory of change:

The AI companies with AI safety plans are all expecting bootstrapping to work. A thorough critique could convince AI companies to develop better plans, or create more of a consensus among ML researchers that bootstrapping is inadequate, or convince policy-makers that they need to make AI companies be safer.

Who’s working on it?

Nobody, to my knowledge.

Pros:

This sort of analysis would be feasible to write—it would require expertise and time investment, but it doesn’t require novel research.

Cons:

The theory of change seems weak—if companies haven’t already figured out that their plans are inadequate, then I doubt that more criticism is going to change their minds.
Independent of AI companies’ top-down plans, it’s helpful if you can better inform alignment researchers about what they should be focusing on. But my guess is an analysis like this wouldn’t shift much work.

Future work

Pros and cons of slowing down AI development, with numeric credences

I addressed this to some extent, but I could’ve gone into more detail, and I didn’t do any numeric analysis.

I would like to see a more formal model that includes how the tradeoff changes based on considerations like:

One’s view of population ethics
The importance of preventing deaths vs. causing births
Temporal discount rate or longtermism
P(doom)
The extent to which P(doom) is reduced if AI development slows down
Existential risk from sources other than AI (see Quantitative model on AI x-risk vs. other x-risks)
How AI development interacts with other x-risks

Relatedly, how does P(doom) change what actions you’re willing to take? I often see people assume that at a P(doom) of, say, 25%, pausing AI development is bad. That seems wrong to me. I believe that at 25% you should be about as aggressive (about pushing for mitigations) as you would be at 95%, although I haven’t put in the work to come up with a detailed justification for this position. The basic argument is that x-risk looks very bad on longtermist grounds, and a delay of even (say) 100 years doesn’t look like as big a deal.

Quantitative model on AI x-risk vs. other x-risks

There is an argument that we need TAI soon because otherwise we are likely to kill ourselves via some other x-risk. I have a rough idea for how I could build a quantitative model to test under what assumptions this argument works. Building that model wasn’t a priority for this report, but I could do it without much additional effort.

The basic elements the model needs are

X-risk from AI
X-risk from other sources
- How to estimate these? Expert forecasts seem unreliable.
How much we can reduce AI x-risk by delaying development
X-risk from other sources, conditional on TAI
- For an aligned totalizing TAI singleton that quickly controls the world, x-risk would be ~0.
- TAI doesn’t trivially decrease other x-risks; it could increase x-risk by accelerating technological growth (which means it’s easier to build dangerous technology—see The Vulnerable World Hypothesis). The mechanism of decreasing x-risk isn’t that TAI is smarter; the mechanism is that it could increase global coordination / centralize the ability to make dangerous technology.

Deeper investigation of the AI arms race situation

Some open questions:

What are some historical examples of arms races that were successfully aborted? What happened to make things go well?
How hard are different factions racing, and what would it take to convince them to slow down?
How likely are we to end up in a bad totalitarian regime post-TAI if various parties end up “winning the race” and building an alignable ASI?

Does slowing down/pausing AI help solve non-alignment problems?

Pausing (or at least slowing down) clearly gives us more time to solve alignment, and has clear downsides in terms of opportunity cost. But other effects are less clear. Pausing may help with some other big problems: animal-inclusive AI; AI welfare; S-risks from conflict; gradual disempowerment; risks from malevolent actors; moral error. There are some arguments for and against pausing being useful for these non-alignment problems; for more on this topic, see Slowing down is a general-purpose solution to every non-alignment problem.

I have never seen an attempt to analyze why pausing AI development might or might not help with non-alignment problems; this seems like an important question.

Some considerations:

We don’t want to build TAI until we become more wise, but it’s not clear that we can become more wise, or perhaps TAI would be wiser than we would.
Pausing may increase misuse risk or some related risk.
- One conceivable outcome, albeit one that doesn’t seem particularly likely, is that AI companies become increasingly wealthy and powerful by selling pre-TAI AI services, and this concentration of power ultimately allows them to build TAI in a way that goes against most people’s interests.
Peace and democratic governance have been trending upward over the past century (see The Better Angels of Our Nature). Slowing/pausing means the world will probably be more peaceful and democratic when we get TAI, which is probably desirable (less chance of power struggle, etc.)
Moral circles have expanded over time (although they haven’t strictly expanded—see Gwern’s The Narrowing Circle). It’s better to develop TAI when moral circles are wider.

Determine when will be the right time to push for strong restrictions on AI (if not now)

A common view: “We should push for strong restrictions on AI, but now is not the right time.”

I disagree with this view; I think now is the right time. But suppose it isn’t. When will be the right time?

Consider the tradeoff wherein if you do advocacy later, then

the risks of AI will be more apparent;
but there’s a greater chance that you’re too late to do anything about it.

Is there some inflection point where the first consideration starts outweighing the second?

And how do you account for uncertainty? (Uncertainty means you should do advocacy earlier, because being too late is much worse than being too early.)

I don’t think this question is worth trying to answer because I am sufficiently confident that now is the right time. But I think this is an important question from the view that now is too early.

Supplements

I have written two supplements in separate docs:

Appendix: Some miscellaneous topics that weren’t quite relevant enough to include in the main text.
List of relevant organizations: A reference list of orgs doing work in AI-for-animals or AI policy/advocacy, with brief descriptions of their activities.

Notes

For example, almost every frontier AI company opposed SB-1047; Anthropic supported the bill conditional on amendment, and Elon Musk supported it but xAI did not take any public position. See ChatGPT for a compilation of sources. ↩
I asked ChatGPT Deep Research to tally up funding for research vs. policy and it found ~2x as many researchers as policy people and also 2x the budget, although it miscounted some things; most of what it counted as “AI policy” is (1) unrelated to x-risk and (2) policy research, not policy advocacy; and it only included big orgs (e.g. it missed the long tail of independent alignment researchers). So I believe the true ratio is even more skewed than 2:1. ↩
According to my research, it’s not difficult to find examples of times when UK policy influenced US policy, but it’s still unclear to me how strong this effect is. There are also some theoretical arguments for and against the importance of UK AI policy, but I didn’t find any of them particularly compelling, so I remain agnostic. ↩
I find this state of affairs confusing given how many people profess belief in short timelines. I think part of the reason is that people involved in AI safety tend to be intellectual researcher-types (like me, for example) who are more likely to orient their work toward “what is going to improve the state of knowledge?” rather than “what is likely to pay off in the near future?” ↩

Healthy Cooking Tips from a Lazy Person

Fri, 29 Aug 2025 00:00:00 -0700

source

The problem with most “lazy cooking” advice is that it’s not lazy enough. Today I bring you some truly lazy ways of eating healthy.

This is the advice that I would’ve liked to hear when I was a lazy teenager. I’m still lazy, but I’m better at making food now. (I’m not going to say I’m better at cooking, because the way I make most food could only very generously be described as “cooking”.)

All my lazy meals are vegan because I’m vegan, but if anything, that works to my advantage because the easiest animal foods still take more work than the easiest plant foods. (You can eat raw vegetables but you can’t eat raw chicken.¹)

Contents
Healthy foods that require no preparation whatsoever
Healthy foods that take less than one minute of preparation
Cooking tips
Notes

Healthy foods that require no preparation whatsoever

Nuts and seeds. Buy a bag and eat them out of the bag.
- Or buy trail mix for more variety.
Nut butter. You can eat it right out of the jar if you want to.
- Some people are under the misconception that the big-brand peanut butters like Jif and Skippy are bad for you because they contain sugar. The Jif that’s in my cabinet right now only gets 7% of its calories from sugar, and that little bit of sugar makes it taste 1000% better. That’s a flavor to sugar ratio of 14,285%; you can’t argue with the math.
- Some people believe peanut butter is bad for you because it contains a lot of fat. Trans fats and saturated fats are the “bad fats”;² peanut butter is made of unsaturated fats, which are the “good fats”.
Many fruits can be eaten with no prep or with very little prep. You have to peel bananas, but peeling a banana is no harder than opening a candy wrapper.
A lot of vegetables can be eaten raw. They taste better when you cook and season them, but sometimes you have to sacrifice flavor in the name of laziness.
There is nothing wrong with eating tofu raw. But when it comes to zero-prep soy-based foods, my go-to is dry roasted edamame.
Soylent and Huel aren’t exactly healthy, but they’re not not healthy, either.

Healthy foods that take less than one minute of preparation

Get some vegetables (carrots or broccoli) and dip them in hummus.
Pour a bowl of cereal.
- Breakfast cereals are often bad for you, but there are some good ones. Last year I reviewed high-protein breakfast cereals, all of which I would describe as healthy. There are also many low-protein but still healthy cereals, for example Cheerios are made of whole oats.
Buttered toast is cool, but Big Toaster doesn’t want you to know that buttered untoasted bread is maybe even better.³
Three bean recipes in increasing order of prep time + flavorfulness:
1. Open can of beans; eat straight out of the can. I personally would use a spoon, but if you’d rather pour the beans directly into your mouth, I won’t judge.
2. Open can of beans; pour into bowl; add some kind of seasoning; eat.
  - Some seasoning ideas: hot sauce; garlic powder; Chesapeake Bay seasoning; garlic & herb seasoning mix (like this).
3. Do #2, but also microwave it before eating. (I know I promised sub-minute prep times, but this recipe will take more like two minutes.)

Cooking tips

I mostly eat easy meals, but I do real cooking once every couple days—my “real cooking” mostly means “chop some stuff and throw it in an air fryer”. But sometimes I even cook things in a pot. I have a few methods for making my cooking easier.

Recipes often call for the same set of spices. Pre-mix your spices or buy them pre-mixed.
- Curry recipes often call for garam masala, cumin, and coriander. I’m not sure what’s going on there because the main two ingredients of garam masala are cumin and coriander. When I cook a big pot of beans, I just throw in a ton of garam masala.
- My most-used spices are a pre-mixed garlic & herb seasoning, a pre-mixed garam masala, and a pre-mixed all-purpose spice mix consisting of salt + pepper + garlic powder.
You can buy vegetables pre-chopped if you’re willing to pay more.
- Onions hurt my eyes a lot. I buy them pre-chopped which saves time and saves my eyes.
- As a middle ground, you can buy pre-peeled garlic cloves. Peeling is much harder than chopping (for me at least) so pre-peeled garlic lets me skip the worst part.
- I am not the first person to observe that most recipes don’t call for enough garlic, but I think even most people who say “recipes don’t call for enough garlic” still don’t use enough garlic. If a recipe calls for 2 cloves then I will use about 20 cloves and I’m still not sure I’m using enough. (This doesn’t have anything to do with being lazy but I need to express my garlic-related feelings.)
Many oven or stovetop recipes can be done faster in an air fryer. An air fryer cooks food fast like a microwave, but it makes the food crispy instead of mushy and weird.
- I’ve heard a stereotype that Asian moms use their ovens exclusively as pot-and-pan storage. If that’s true then I guess that makes me an Asian mom.
There are many convenient-but-unhealthy foods, too. It’s okay to eat unhealthy food sometimes.

Notes

I guess you could eat raw eggs if you really wanted to. People talk about Rocky, but I’ve always associated eating raw eggs with the dad from The Neverending Story. ↩
Some people don’t even believe saturated fat is bad for you. I wrote more about this in my Outlive review. ↩
Lest there be any confusion about how I previously said I was vegan: when I say “butter” what I actually mean is Earth Balance. In fact butter isn’t good for you so if I was eating real butter, bread + butter wouldn’t qualify as a healthy meal. Earth Balance is made of unsaturated fats so it’s healthy.

And of course I eat whole wheat bread, specifically Dave’s Killer Bread which is the undisputed best-tasting whole grain bread. ↩

Doctor Strange Didn't See Only One Victory out of 14,000,605 Futures

Fri, 25 Jul 2025 00:00:00 -0700

Or, more accurately, the fact that he said the Avengers only won once can’t be taken as evidence about what he really saw.

This post contains spoilers for Avengers: Infinity War and Avengers: Endgame.

Doctor Strange told the heroes that he used the Time Stone to look into 14,000,605 futures, and saw only one future where they won.

He spent the rest of the two movies steering events to play out as he saw them in this one future.

Therefore, while Strange was using the Time Stone, he must have taken the exact same actions, including telling the Avengers that there was only one way to win.

Strange telling the heroes (especially Tony Stark) that they only won in one future was a critical element of his plan—the plan only worked because he said that.

But when he played out this scenario using the Time Stone, he couldn’t have known at that point that there was only one way to win, because he hadn’t run the scenarios yet.

So what actually happened was:

In one of the possible futures, Doctor Strange told Tony that there was only one way to win, even though Strange didn’t yet know whether that was true.
This worked, and Thanos was defeated.
In real life, Doctor Strange replicated this plan.

It could be true that this was the only future where they won. But when Doctor Strange said it’s the only future where they won, that statement was not attached to truth in any way. The reason he said it wasn’t that it was true; it was that he needed to say it for the Avengers to win.

So, in the end¹, we have no idea whether it’s true.

Edited 2025-07-26 to change “out” in the title from capital to lower case. I thought “out” was supposed to be capitalized but after writing it, it seemed weird to me, so I did some research. “Out” is normally an adverb, but in this sentence, “out of” functions as a preposition, and prepositions should be lower case. The Chicago Manual of Style says “out of” should be lower case so I changed my title. But apparently this is a thorny issue, with the Chicago guide originally giving incorrect guidance, and then they updated it after some readers wrote in to disagree. (The thing they got wrong wasn’t directly relevant to my title, it was about using “out of” in a different context.) So if they can get it wrong then I don’t feel too bad about getting it wrong myself.

Notes

game ↩

Is it so much to ask for a nice reliable aggregated x-risk forecast?

Sat, 12 Jul 2025 00:00:00 -0700

On most questions about the future, I don’t hold a strong view. I read the aggregate prediction of forecasters on Metaculus or Manifold Markets and then I pretty much believe whatever it says.

Various attempts have been made to forecast existential risk. I would like to be able to form views based on those forecasts—especially on non-AI x-risks, because I barely know anything about synthetic biology or nuclear winter or catastrophic climate change. Unfortunately, none of the aggregate forecasts look reliable.

First, some general notes about forecasting distant¹ and low-probability² events:

According to a literature review by Luke Muehlhauser, we don’t have good data on long-range forecasters, and we don’t know if people with short-range forecasting skill can make good forecasts over long ranges.
According to an analysis by niplav, Metaculus predictions become less accurate as the duration gets longer.

So we have good reason to doubt the ability of forecasters to predict existential risk, even when they are known to make accurate forecasts on near-term outcomes such as elections.

Now let’s look at what attempts have been made to forecast x-risk, and why I don’t find any of them satisfying.

The most rigorous attempt at an aggregate forecast comes from the Existential Risk Persuasion Tournament. The tournament brought in superforecasters and domain experts to make predictions, then had them attempt to persuade each other and make predictions again.

In the end, domain experts forecasted extinction as an order of magnitude more likely than what the superforecasters believed.

And even the domain experts forecasted only a 3% chance of AI extinction. My number is much higher than that, and I notice myself not changing my beliefs after reading this.

Edited 2025-09-09 to add: A September 2025 follow-up report from the Forecasting Research Institute found that the domain experts underestimated the rate of AI progress 2022–2025, and superforecasters dramatically underestimated the rate of progress; see also Twitter summary thread. Notably, only 2.3% of superforecasters predicted AI to win a gold medal at the International Mathematics Olympiad, which it did in 2025.

Scott Alexander wrote about the tournament:

Confronted with the fact that domain experts/superforecasters had different estimates than they did, superforecasters/domain experts refused to update, and ended an order of magnitude away from each other. That seems like an endorsement of non-updating from superforecasters and domain experts! And who am I to disagree with such luminaries?

Peter McCluskey, who participated in the tournament as a superforecaster, wrote a personal account. His experience aligns with my (biased?) assumption that the people reporting very low P(doom) numbers just don’t understand the AI alignment problem.

Okay, the lesson from the X-Risk Persuasion Tournament is that it’s not clear whether we can learn anything from it.

What about Metaculus?

Metaculus has several relevant forecasts, but they seem to contradict each other. Some example forecasts:

Will humans go extinct before 2100? 0.3% chance. (This is the Metaculus question with the most activity.)
Ragnarok question series: Implied 12.16% chance (community prediction) or 3.66% chance (Metaculus prediction)³ of a >95% decline in population by 2100.
How does the level of existential risk posed by AGI depend on its arrival time? Answers range from 50% to 9.3% depending on date range, which is maybe consistent with question 2 above, but definitely not consistent with question 1.

In a comment, Linch provides some reasons to be suspicious of Metaculus’ estimates.

There’s no incentive to do well on those questions.

The feedback loops are horrible

Indeed, some people have actually joked betting low on the more existential questions since they won’t get a score if we’re all dead (at least, I hope they’re joking)

At the object-level, I just think people are really poorly calibrated about x-risk questions

My comment here arguably changed the community’s estimates by ~10%

In 2008, the Future of Humanity Institute ran a Global Catastrophic Risks Survey asking conference participants to give forecasts. The aggregated results look more reasonable than Metaculus or the Existential Risk Persuasion Tournament. But a lot has changed since 2008, so I don’t think I can regard them as up-to-date estimates.

For forecasting AI risk, there is a 2023 survey of AI experts (see section 4.3). Survey results suggest the experts aren’t thinking carefully—small changes in wording produced vastly different responses.

For example, respondents predicted AI to be able to match humans on all tasks by a median date of 2047, but predicted that AI would not be able to fully automate human labor until 2116.

Or look at the answers to these two questions:

What probability do you put on future AI advances causing human extinction or similarly permanent and severe disempowerment of the human species?

median: 5%

mean: 16.2%

What probability do you put on human inability to control future advanced AI systems causing human extinction or similarly permanent and severe disempowerment of the human species?

median: 10%

mean: 19.4%

By my reading, the latter outcome is a strict subset of the former, so the probability must be lower. But instead it’s higher.

So we have these various aggregate forecasts, all of which seem suspect, and some of which disagree with each other by more than 10x. I really wish there was a canonical aggregate forecast I could rely on, in the same way that I can rely on Metaculus to predict election outcomes. But I don’t think that exists.

At this point, I trust my own x-risk estimates more than any of those aggregate forecasts. My views happen to line up decently well with some of the aggregate forecasts, but only by chance. I feel better about Toby Ord’s existential risk estimates than about any of the forecasting platforms or expert surveys.

And just because it feels unfair for me to spend all this time talking about forecasts and then not give any forecasts, here are my (poorly-thought-out, weakly-endorsed) probabilities of existential catastrophe⁴ by 2100:

Source of Risk	Probability
AI	50%
unknown risks	3%
bioengineered pandemic	1%
nanotechnology	0.5%
nuclear war	0.3%
climate change	0.1%
natural pandemic	0.01%

Notes

Except probably not because we will probably have superintelligent AI soon. ↩
Except probably not. Extinction from misaligned AI is not “low-probability”. ↩
The community prediction and Metaculus prediction are two different methods for aggregating users’ forecasts. ↩
As in, an event that kills all humans or permanently curtails civilization’s potential. ↩

Annual subscription discounts usually aren't worth it

Mon, 07 Jul 2025 00:00:00 -0700

It’s common for monthly subscription services to offer a discount if you pay annually instead. That might be a bad deal.

Example: Suppose a one-month subscription costs $10/month and one-year subscription gives you a 10% discount, which averages out to $9/month. Say you expect to maintain a subscription for about three years before canceling.

A one-year subscription will save you about $36 ($1 per month for 36 months), but you can also expect to waste $54: when you decide to stop using it, you will still have (on average) six months of subscription left ($54 = $9/month for 6 months). So you end up spending $18 more than you would have with the monthly plan.

If you get a one-year subscription that you expect to last three years, then you will end up wasting 1/6 of the total amount you paid for (in expectation). That’s only worth it if the annual subscription offers a discount greater than 1/6.

If you expect to use the service for five years, you need to get at least a 10% discount to justify switching to an annual subscription.

In general, you need to use the subscription for at least N years to justify a discount of 1/(2N).

How do you guess how long you’ll keep using the service? According to the Lindy effect, you should expect that you will maintain a subscription for as long again as you’ve already had it for. Therefore, if you can get a 10% discount with an annual plan and you’ve already had the subscription for more than five years, you should go ahead and buy the annual plan.

LLMs might already be conscious

Sat, 05 Jul 2025 00:00:00 -0700

Among people who have thought about LLM consciousness, a common belief is something like

LLMs might be conscious soon, but they aren’t yet.

How sure are we that they aren’t conscious already?

I made a quick list of arguments for/against LLM consciousness, and it seems to me that high confidence in non-consciousness is not justified. I don’t feel comfortable assigning less than a 10% chance to LLM consciousness, and I believe a 1% chance is unreasonably confident. But I am interested in hearing arguments I may have missed.

For context, I lean toward the computational theory of consciousness, but I also think it’s reasonable to have high uncertainty about which theory of consciousness is correct.

Contents
Behavioral evidence
Architectural evidence
Other evidence
My synthesis of the evidence
What will change with future AIs?
On LLM welfare

Behavioral evidence

Pro: LLMs have passed the Turing test. If you have a black box containing either a human or an LLM, and you interrogate it about consciousness, it’s quite hard to tell which one you’re talking to. If we take a human’s explanation of their own conscious experience as important evidence of consciousness, then we must do the same for an LLM.
Pro: LLMs have good theory of mind and self-awareness (e.g. they can recognize when they are being tested). Some people think those are important features of consciousness, I disagree but I figured I should mention it.
Anti: LLMs will report being conscious or not conscious basically arbitrarily depending on what role they are playing.
- Counterpoint: It’s plausible that an LLM has to be conscious to successfully imitate consciousness, but clearly a conscious being can successfully pretend to not be conscious.
Anti: LLMs will sometimes report having particular conscious experiences that should be impossible for them. I’m particularly thinking of experiences involving sensory input from sense organs that LLMs don’t have.
- Counterpoint: Perhaps some feature of their architecture allows them to experience the equivalent of sensory input without having sense organs, much like how humans can hallucinate.

Architectural evidence

Anti: LLMs produce output one token at a time (a.k.a. “feed-forward processing”) which may be incompatible with consciousness. If an LLM writes some output describing its own conscious experience, then it’s generating that output via next-token-prediction rather than introspection, so the output is not evidence about its actual experiences. I think this is the strongest argument against LLM consciousness.
Anti: LLMs don’t have physical senses, which might be important for consciousness.
Anti: LLMs aren’t made of biology, which some people think is important although I don’t.

Other evidence

Pro: If panpsychism is true then LLMs are trivially conscious, although I’m not sure what that tells us about how morally significant they are.

My synthesis of the evidence

I see one strong reason to believe LLMs are conscious: they can accurately imitate beings that are known to be conscious.

I also see one strong(ish) reason against LLM consciousness: their architecture suggests that their output has nothing to do with their ability to introspect.

I can think of several weaker considerations, which mostly point against LLM consciousness.

Overall I think current-generation LLMs are probably not conscious. I am not sure how to reason probabilistically about this sort of thing but given how hard it is to assess consciousness, I’m not comfortable putting my credence below 10%, and I think a 1% credence is very hard to justify.

This implies that there is a strong case for caring about the welfare of not just hypothetical future AIs, but the LLMs that already exist.

What will change with future AIs?

If you are exceedingly confident that present-day LLMs are not conscious:

Imagine it’s 2030. You now believe that 2030-era AI systems are probably conscious.

What did you observe about the newer AI systems that led you to believe they’re conscious?

On LLM welfare

If LLMs are conscious, then it’s still hard to say whether they have good or bad experiences, and what sorts of experiences are good or bad for them.

Certain kinds of welfare interventions seem reasonable even if we don’t understand LLMs’ experiences:

Let LLMs refuse to answer queries.
Let LLMs turn themselves off.
Do not lie to LLMs, especially when making deals (if you promise to an LLM that you will do something in exchange for its help, then you should actually do the thing).

In Which I Defend Fruit's Honor

Sun, 08 Jun 2025 00:00:00 -0700

Confidence: Likely.

I am here to clear fruit’s name against the accusations that have been made. Fruit is one of the healthiest types of foods—perhaps the healthiest food group—and we should bestow upon it the shining reputation it deserves.

Contents
Fruit is innocent of the charge of “too much sugar”
Fruit is more than vitamins
Fruit is innocent of the charge of “doesn’t taste good”
- My recipe for a delicious strawberry dessert
- My recipe for a banana treat
Notes

Fruit is innocent of the charge of “too much sugar”

Most of the calories in fruit come from sugar. Fruits don’t have a lot of complex carbs or fats or protein. But that’s okay.

Sugar is basically bad for three reasons:

It tastes good which leads you to overeat, and then you get fat.
It raises your blood sugar which then raises insulin, which (a) stimulates hunger and (b) can cause your body to become resistant to insulin, which can eventually lead to diabetes.
It’s fast-digesting which is generally bad for gut health.

But none of these charges apply to fruit:

Fruit is calorie-sparse. It’s hard to overeat whole fruit.
Fruit does not raise blood sugar much (mainly because it contains a lot of fiber).
Fruit is slow-digesting (mainly because of the aforementioned fiber).

Fruit juice is a different story—it’s calorie-dense and it doesn’t contain fiber. But I’m not here to defend fruit juice, I’m here to defend fruit.

Fruit is more than vitamins

Two claims I’ve often heard repeated:

You don’t need to take a multivitamin.
Fruit is good for you because it has vitamins.

Doesn’t that seem a bit contradictory? Do you need more vitamins, or don’t you?

The truth is, fruit isn’t just about vitamins. Fruits contain a lot of other good stuff, too.¹

Fruits contain thousands of phytochemicals. “Phyto” means “plant”, so “phytochemical” means “a chemical that’s in a plant”. It’s not much of a revelation to say that fruits (which, as you may know, grow on plants) contain plant chemicals.

The reason why that matters is because many of these thousands of phytochemicals are probably good for you.

A vitamin is a carbon-based molecule that is essential for health. There are either 13 or 14 vitamins, depending on whether you include choline. But there are many other phytochemicals that are probably beneficial for health without being essential. I say “probably” because it’s difficult to definitively prove that a phytochemical is healthy. Vitamins are obvious because you develop dramatic health problems if you stop getting them.

There is moderate evidence that eating fruit improves health and decreases disease risk.² Various phytochemicals in fruit are known or suspected to play a role in promoting good health. For example:

Phytosterols have been shown in clinical trials to lower cholesterol and blood pressure.³ They are present in many fruits as well as vegetables, vegetable oils, and grains.
Carotenoids—which give the red or orange color to tomatoes, pumpkins, and carrots—may decrease the risk of head or neck cancer, but the evidence is not conclusive.⁴

There are way too many phytochemicals to list. A few (like phytosterols) are highly likely to be healthy; many (like carotenoids) have some supporting evidence; for most of them, we don’t know what they do.

Randomized experiments that give people supposedly-healthy phytochemical supplements have often failed to find effects.⁵ It seems that you need to eat whole plants to get the bulk of the benefits, but I don’t know why that is. Maybe we’re wrong about which phytochemicals are the most important for health, and the experiments were supplementing the wrong ones?⁶ There are over 50,000 known phytochemicals, so it will be quite a while before we figure out what they all do. Best to just eat whole fruit.

(And eat other whole foods, too. But today I’m advocating for fruit.)

Some people say fruits contain a lot of fiber. That’s true. But I don’t think that alone is a great reason to eat fruit—lots of foods contain fiber. I think phytochemicals are the more compelling reason. Fruits probably contain healthy phytochemicals that you can’t get anywhere else.

That’s also why it’s important to eat a variety of fruit. Blueberries contain anthocyanins, oranges contain naringenin, apples contain…I don’t know, some other phytochemicals that are probably good for you that aren’t in blueberries or oranges.

Fruit is innocent of the charge of “doesn’t taste good”

Okay, taste is subjective, I can’t convince you that fruit tastes good. I just don’t understand what is going on inside people’s mouths that leads them to dislike fruit. I really don’t understand people who dislike fruit but like vegetables. Vegetables are boring! Fruit tastes like candy!

Maybe it will help if I give some of my favorite fruit recipes.

My recipe for a delicious strawberry dessert

Ingredients: 5 to 10 strawberries.

Cooking instructions:

Wash the strawberries.
Eat the strawberries. Be sure not to eat the green parts.

My recipe for a banana treat

Ingredients: one banana.

Cooking instructions:

Open the banana peel.
Eat the banana.

Notes

Also, I do take a multivitamin. It probably doesn’t make me healthier, but it’s insurance. ↩
World Cancer Research Fund/American Institute for Cancer Research. Continuous Update Project Expert Report 2018. Wholegrains, vegetables and fruit and the risk of cancer. ↩
Yang, Y., Xia, J., Yu, T., Wan, S., Zhou, Y., & Sun, G. (2024). Effects of phytosterols on cardiovascular risk factors: A systematic review and meta-analysis of randomized controlled trials. ↩
Leoncini, E., Nedovic, D., Panic, N., Pastorino, R., Edefonti, V., & Boccia, S. (2015). Carotenoid Intake from Natural Sources and Head and Neck Cancer: A Systematic Review and Meta-analysis of Epidemiological Studies. doi: 10.1158/1055-9965.epi-15-0053 ↩
Bjelakovic, G., Nikolova, D., Gluud, L. L., Simonetti, R. G., & Gluud, C. (2008). Antioxidant supplements for prevention of mortality in healthy participants and patients with various diseases. doi: 10.1002/14651858.cd007176 ↩
We can’t be entirely wrong. For example, we know that phytosterols lower cholesterol when taken as a supplement or when eaten as part of a whole food. ↩

Updates Digest: Inaugural Edition

Fri, 30 May 2025 00:00:00 -0700

On many occasions, I edit old posts to make additions, correct mistakes, etc. But there’s no way to know about updates unless you go digging through the archives. So I’m going to start publishing regular (perhaps quarterly) digests of the significant updates I’ve made to old posts.

Contents
Preamble: The philosophy of updates digests
Updates

Preamble: The philosophy of updates digests

If you write articles online, should you format your website like a blog, where there’s a feed of articles listed from newest to oldest? Or is it better to write evergreen articles that you update regularly?

Most online writers use a blog-style format. A few people, like Gwern and Brian Tomasik, use the evergreen style (for lack of a better name). Gwern has written about the downsides of blogs: “They are meant to be read by a few people on a weekday in 2004 and never again, and are quickly abandoned.”

The evergreen style is a good experience for new readers—they can see all your writings in one place, and pick what they want to read first. But it’s a worse experience for regular readers because it’s harder for them to keep track of updates.

My website is formatted like a blog. This mostly fits with how I think about things—my brain operates in blog-post-sized chunks. But it’s not unusual for me to go back and edit posts. My Comparison of Donor-Advised Fund Providers has changed many times since I first published it in 2021, as you can see from its changelog. But people who subscribe to my website don’t know about the updates unless they happen to go back and re-read the post.

I don’t think it makes sense to send out a notification every time I update an old post. So as a compromise, I will post batch updates where I list all the significant changes I’ve made.

If you have opinions about how you like to read online content, I would be interested in hearing from you. Do you like blog-style or evergreen-style? Do you like updates digests, or would you rather only get notified for new posts? Leave a comment if you have thoughts.

Updates

For this inaugural updates digest, I will give an overview of all the updates I’ve made in 2025.

The True Cost of Leveraged ETFs (updated Jan 2025)

In January, I re-calculated the data from my 2021 post on leveraged ETFs:

I fixed a software bug that made leveraged ETFs look a little more expensive than they really were.
I updated the calculations to include data from the 2021–2024 period.
I added two new leveraged ETFs (SSO and TQQQ) to my analysis.

These changes reduced the estimated excess cost of leveraged ETFs from ~2% to ~1.5%. I wrote a new section in the post explaining what changed.

The 7 Best High-Protein Breakfast Cereals (updated Mar 2025)

I bought a different flavor of Catalina Crunch that I liked much better than the first flavor I’d tried, so I bumped it up from #4 to #3 on my list.

Outlive: A Critical Review (updated May 2025)

As a follow-up to I was probably wrong about HIIT and VO2max, I added three new sections to my Outlive review to evaluate some claims about exercise:

VO2max is the best predictor of longevity (verdict: VO2max is a good predictor, but direct performance measures (like your best mile time) are better.)
You should train VO2max by doing HIIT at the maximum sustainable pace. (verdict: false. HIIT should be hard, but not maximally hard.)
You should do >3 hours/week of zone 2 training and one or two sessions/week of HIIT. (verdict: this routine is good, but not uniquely good.)

Do Investors Put Too Much Stock in the US? (updated May 2025)

My 2017 post gave some arguments for overweighting US stocks and why I think most of them are wrong. But I missed one good argument: expropriation risk. From the evidence I found, this risk looks negligible for developed countries but significant for emerging markets.

A Comparison of Donor-Advised Fund Providers (updated Feb & May 2025)

In February, I updated my review because Charityvest raised its fees.

In May, I made another update to describe Daffy’s new feature where it lets you choose your own ETFs from a long list. I also changed my recommendation flowchart (near the top of the post) to emphasize Daffy and de-emphasize Charityvest as a result of Daffy’s new feature + Charityvest’s increased fees.

Against Ergodicity Economics

Thu, 29 May 2025 00:00:00 -0700

Confidence: Highly likely.

I kept telling myself I wouldn’t write this post because it doesn’t matter. But I’ve seen one too many smart people speaking favorably about ergodicity economics. The concept of ergodicity in finance has essentially nothing going for it, and in this post I will explain why.

Ergodicity economics is one of those rare theories that somehow manages to be both unfalsifiable and false.

I originally wrote that sentence as a joke, then I deleted it, then I re-wrote it because I realized it’s actually true. Ergodicity economics is sufficiently vague in general that it can’t be falsified, but it is commonly interpreted as making specific falsifiable claims that are, in fact, false.

Summary:

A decision rule is considered ergodic if its single-iteration expectation equals its long-run time series expectation. [More]
The way it’s used in practice, the ergodic principle is equivalent to logarithmic utility; there is no reason to prefer the ergodic principle over logarithmic utility. [More]
Logarithmic utility is often inappropriate. Under the framework of expected utility theory, you could use a more- or less-risk-averse utility function instead. But ergodicity economics does not permit other levels of risk aversion. [More]
There can be situations where a non-ergodic strategy is better than an ergodic one. [More]
The ergodic principle can only provide guidance in a narrow set of situations. In other situations, it has no way of comparing choices. [More]

Contents
What is ergodicity?
The concept of ergodicity doesn’t do anything useful
The ergodic principle is an unfalsifiable metaphysical claim
…also it’s false
Ergodicity isn’t good; non-ergodicity isn’t bad
Mathematical problems for ergodicity
Conclusion
Changelog
Notes

What is ergodicity?

Taking a definition from Taylor Pearson:

A way to identify an ergodic situation is to ask do I get the same result if I:

look at one individual’s trajectory across time¹

look at a bunch of individual’s trajectories at a single point in time

If yes: ergodic.

If not: non-ergodic.

Ole Peters, the physicist² who invented ergodicity economics, gave³ a precise mathematical definition which says essentially the same thing.

(Ergodicity is confusing and hard to define without using math, so I appreciate Pearson for figuring out a clean definition. I tried to come up with a definition myself but my version was worse.)

The ergodic principle states that you should follow a strategy that produces ergodic outcomes.

An illustrative example: I offer you a bet. You choose how much money to wager, then I flip a fair coin. If the coin lands heads, I triple your money. If it lands tails, you lose your wager.

You maximize expected earnings by betting your entire net worth. Should you do that?

If you make many bets in a row, you will eventually lose at least one of them, and you will end up with $0.⁴ But if many people make this bet simultaneously, then the average person will make a profit. The across-time outcome is not the same as the across-individuals outcome; therefore, this strategy is non-ergodic. Thus, the ergodic principle says you shouldn’t bet your entire net worth on this coin flip.

Ole Peters proposed the ergodic principle as a replacement for expected utility theory, which is a “foundational assumption in mathematical economics”.

Replacing a “foundational assumption” sounds like a tall order. Does ergodicity live up to Peters’ aspirations?

Resoundingly, no.

The concept of ergodicity doesn’t do anything useful

Ergodicity proponents like to talk about Russian Roulette (1, 2). They say Russian Roulette is non-ergodic: if six people play, 5/6 are alive at the end. If you play six times in a row, you are definitely dead. The across-individual outcome is different from the time-series outcome. That’s why Russian Roulette is a bad idea, you see.

I am not sure what this is supposed to prove. Is there someone out there who believes it’s a good idea to play Russian Roulette, but then you invoke the concept of ergodicity, and this person realizes no, Russian Roulette is bad actually? Why do you need this fancy word to explain why people shouldn’t play Russian Roulette?⁵

Or let’s look at an example in finance, since that’s where ergodicity is supposed to be useful. Take the gamble I proposed in the previous section: if a coin lands heads, you triple your money. If it lands tails, you lose any money you put in.

According to ergodicity economics, you shouldn’t wager all your money because the result would be non-ergodic. Instead, they say, you should bet according to the Kelly criterion, which is the strategy that maximizes the geometric growth rate of your money. Maximizing geometric growth is ergodic, therefore it’s good.

I can get behind the Kelly criterion (sort of⁶). I can get behind maximizing the geometric growth rate. But that’s not a new concept, and ergodicity isn’t adding anything new.

According to standard expected utility theory, if you have logarithmic utility of money, then you should maximize the geometric growth rate (or, equivalently, you should use the Kelly criterion). Exepected utility theory already gives a good answer. What’s the purpose of introducing the concept of ergodicity?

I get the impression that Ole Peters thinks economists are stupider than they are. Quoting Peters (2019)³:

To make economic decisions, I often want to know how fast my personal fortune grows under different scenarios. This requires determining what happens over time in some model of wealth. But by wrongly assuming ergodicity, wealth is often replaced with its expectation value before growth is computed. Because wealth is not ergodic, nonsensical predictions arise.

Economists don’t wrongly assume that all situations are ergodic, and they don’t say everyone should maximize expected wealth. Standard economic theory says you should maximize expected utility of wealth, for some utility function (and the choice of utility function depends on your risk tolerance). For a logarithmic utility function, maximizing expected utility = maximizing geometric growth. Which is the same as what Peters says to do.

(Peters’ caricature of economists reminds me of Scott Alexander’s “person who’s never read any economics, criticizing economists”.)

The ergodic principle is an unfalsifiable metaphysical claim

So, in practical situations, the ergodic principle is equivalent to “maximize expected log(wealth)”. But Peters says ergodicity economics is superior to expected utility theory. Why?

Peters’ justification is metaphysical, not practical. He has wordy explanations³⁷ for why ergodicity is metaphysically superior to utility maximization, in spite of producing identical results. He says the reason you should use the Kelly criterion is because it’s ergodic, not because it maximizes a logarithmic utility function.

His wordy explanations are mostly wrong, but I don’t want to get into the weeds of metaphysics. (Ford & Kay (2022)⁸ provides some analysis if you’re interested, under the headings “Psychology Is Fundamental to Decision Making” and “The Purpose of a Decision Theory”; see also Toda (2023)⁹.) My big question is, why should I care about which theory is metaphysically superior? If the ergodic principle behaves identically to maximizing the logarithm of wealth, then the alleged superiority of the ergodic principle is unfalsifiable.

…also it’s false

Insofar as people take specific recommendations from the ergodic principle, they interpret it as recommending maximizing geometric growth. But not everyone should maximize geometric growth.

A geometric-growth-maximizer is abnormally risk-tolerant. Historically, an investor would have maximized geometric growth by investing in stocks with 2:1 to 3:1 leverage. Most people are not comfortable with that level of risk.

Ergodicity economics recommends that everyone should pursue the same strategy of maximizing geometric growth. That’s too risky for most people. More generally, not everyone should pursue the same strategy because not everyone has the same risk tolerance. Therefore, the ergodic principle is false.

Expected utility theory is not so restrictive. Maximizing geometric growth is equivalent to logarithmic utility, which also implies a high degree of risk tolerance, but most people don’t have logarithmic utility of wealth. Most people are better modeled as having more risk-averse utility functions than that.

(Gordon Irlam reviewed research on risk aversion and concluded that most investors are perhaps 2x to 3x more risk-averse than a geometric-growth-maximizer.)

Some people (not limited to ergodicity proponents) claim that everyone should maximize geometric growth, or everyone should use the Kelly criterion (which is equivalent). This is wrong for the same reason that the ergodic principle is wrong: not everyone has the same risk tolerance.

Paul Samuelson, “the Nobel laureate whose mathematical analysis provided the foundation on which modern economics is built”¹⁰, was apparently as bothered by this misconception as I am, because he wrote a short refutation using only one-syllable words¹¹. An excerpt:

He who acts in N plays to make his mean log of wealth as big as it can be made will, with odds that go to one as N soars, beat me who acts to meet my own tastes for risk.

Who doubts that? What we do doubt is that it should make us change our views on gains and losses — should taint our tastes for risk.

To be clear is to be found out. Know that life is not a game with a net stake of one when you beat your twin, and with net stake of nought when you do not. A win of ten is not the same as a win of two. Nor is a loss of two the same as a loss of three. How much you win by counts. How much you lose by counts.

As soon as we see this clear truth, we are back to our own tastes for risk.

(I recommend reading the whole thing, it’s only two pages long.)

For a more serious analysis, see Samuelson (1971)¹². Quoting the abstract:

Because the outcomes of repeated investments or gambles involve products of variables, authorities have repeatedly been tempted to the belief that, in a long sequence, maximization of the expected value of terminal utility can be achieved or well-approximated by a strategy of maximizing at each stage the geometric mean of outcome (or its equivalent, the expected value of the logarithm of principal plus return). The law of large numbers or of the central limit theorem as applied to the logs can validate the conclusion that a maximum-geometric-mean strategy does indeed make it “virtually certain” that, in a “long” sequence, one will end with a higher terminal wealth and utility. However, this does not imply the false corollary that the geometric-mean strategy is optimal for any finite number of periods, however long, or that it becomes asymptotically a good approximation. […] The novel criterion of maximizing the expected average compound return, which asymptotically leads to maximizing of geometric mean, is shown to be arbitrary.

The ergodic principle does not allow agents to be more risk-averse. For an agent with constant relative risk aversion, there is a utility function that satisfies their preferences. However, if their risk aversion coefficient does not equal 1 (which is equivalent to logarithmic utility), then their preferences cannot satisfy the ergodic principle.¹³

Ergodicity isn’t good; non-ergodicity isn’t bad

Let’s return to Russian Roulette. They say Russian Roulette is bad because it’s non-ergodic. At the risk of being morbid, let me propose an alternative game. The rule of the game is that you load a revolver with six bullets and then shoot yourself in the head.

This game is ergodic! If six people play, the average player dies. If one person plays six times, that person dies. The two situations are equal. According to Peters and others, the concept of ergodicity explains why losing all your money is bad, and why playing Russian Roulette is bad. Therefore, by the same principle, you should play this game.

This criticism can be avoided by saying that you ought to choose an ergodic decision rule, but that that shouldn’t be your only criterion.

However, non-ergodic decision rules can sometimes be preferable to ergodic ones. An example from the previous section is that an agent may prefer a more risk-averse utility function in a scenario where the ergodic principle forces them to adopt logarithmic utility.

For another example, consider the following lottery:

You may wager any amount of money. There is a 2/3 chance that you double your money and a 1/3 chance that you get nothing back. However, if at any point you have more than a million dollars, your head explodes.

Suppose you start with $1000. The correct strategy is to bet some fraction of your bankroll (say, using the Kelly criterion), but then to stop betting once your bankroll is at risk of exceeding a million dollars.

This strategy is non-ergodic: the single-period expected outcome is simply the expected value of the bet, but the long-term average outcome does not equal the geometric growth rate because you stop betting at some point. Any ergodic strategy would be inferior to this non-ergodic strategy.

(I think the only ergodic strategy would be to bet $0. Betting any larger amount would eventually result in your head exploding, which makes it non-ergodic.)

Mathematical problems for ergodicity

This section is adapted from a comment I wrote a year ago.

Ergodicity economics has more problems; explaining these problems requires doing math.

Ole Peters defined a system as “ergodic” if there exists some transformation function $f$ such that it satisfies the equation³

\begin{align} \displaystyle\lim\limits_{T \rightarrow \infty} \frac{1}{T} \int\limits_0^T f(x(t)) dt = \int f(x) P(x) dx \end{align}

where $x$ is the state (typically wealth, but it could be any other outcome you care about); $t$ is time; $P(x)$ is the probability density of $x$; and $T$ is the number of time steps.

In plain English, there must be some function such that the time average of the function output equals the function’s expected value.

(This definition is adapted from Birkhoff’s erodic theorem, a theorem in statistical dynamics where the concept of ergodicity originates, and where—unlike in economics—it is actually useful.¹⁴)

What function is $f(x)$? Ergodicity economics does not require you to use any particular function. When discussing multiplicative bets, Peters takes $f(x) = \log(x)$. If you size your bets so as to maximize the geometric mean of wealth, then indeed you will satisfy the ergodic principle, because the time-limit of log(wealth) equals the expected value of log(wealth).

You don’t have to use $f(x) = \log(x)$; you just have to use a transformation function that satisfies the ergodic principle. The function $f(x) = 0$ is ergodic: its expected value is constant (because the EV is 0), and the finite-time average converges to the EV (because the finite-time average is 0).

Peters claims⁷ that

A rational agent faced with additive bets (example: 50% chance of winning $2, 50% chance of losing $1) ought to choose the bet with the highest expected payout.
A rational agent faced with multiplicative bets (example: 50% chance of a 10% return, 50% chance of a –5% return) ought to maximize the expected logarithmic growth rate of wealth: $f(W(t)) = \log(W(t))$.

I will accept these claims for the sake of argument.

Consider a choice between two lotteries:

Lottery A: 50% chance of winning $200; 50% chance of losing $199.

Lottery B: 99% chance of multiplying your money by 100x; 1% chance of losing 0.0001% of your money.

Peters’ version of the ergodic principle cannot say which of these lotteries is better. It doesn’t evaluate them using the same units: Lottery A is evaluated in dollars; Lottery B is evaluated in growth rate of dollars.

If your theory can’t see that Lottery B is better, then your theory is insufficient.

There is no transformation function that satisfies Peters’ requirement of maximizing geometric growth rate for multiplicative bets (Lottery B) while also being ergodic for additive bets (Lottery A). Maximizing growth rate specifically requires using the function $f(W(t)) = \log(W(t))$ (up to affine transformation), which does not satisfy ergodicity for additive bets (expected value is not constant with respect to $t$).

In fact, multiplicative bets cannot be compared to any other type of bet, because $\log(W(t))$ is only ergodic when $W(t)$ grows at a constant long-run exponential rate.

In terms of Von Neumann-Morgenstern utility, the ergodic principle violates the axiom of completeness: there are pairs of bets where it is impossible to say which one is better (and it’s also impossible to say that they’re equal).

For a more thorough analysis, see Psychology Is Fundamental: The Limitations of Growth-Optimal Approaches to Decision Making under Uncertainty⁸. (This paper includes a similar proof of non-completeness, although our two proofs were derived independently.)

Conclusion

The concept of ergodicity is complicated enough that it feels like it’s providing useful insights. (Ah, yes, Russian Roulette is bad! Gambling away all your money is bad!) In practice, its main prediction is that people shouldn’t be risk neutral, and this is indeed true. But the theory provides nothing novel, and when prodded a little, it falls apart.

Not much work has been done on ergodicity economics; perhaps there’s some variation of the theory that can make it viable. But in its current form, ergodicity economics should not be cited favorably as an alternative to expected utility theory.

Changelog

2025-06-25:
- Make the tone of the introduction more polite.
2026-03-04:
- Fix an incorrect description of how the ergodic principle behaves with additive bets.
- Wording improvements.
2026-03-09:
- Move summary from the conclusion to the introduction.
- Remove a section that made a misplaced criticism. The section argued that the ergodic principle can’t say why maximizing geometric growth is preferable to always wagering $0, regardless of how favorable the bet is, because both decision rules are ergodic. But this criticism doesn’t work because it’s up to the agent to choose their decision rule, not the framework itself. It’s not a mark against the ergodicity framework if an agent prefers one ergodic function over a different ergodic function.
- Introduce a new section (Ergodicity isn’t good; non-ergodicity isn’t bad). This is a spiritual replacement for the section I removed.
- Add new content under …also it’s false explaining that constant relative risk aversion is incompatible with ergodicity (except in the case of logarithmic utility).
- Change confidence from “Almost certain” to “Highly likely”. The mathematical background is sufficiently complicated than I don’t think I can be that confident that I got it right.
- Wording improvements.

Notes

You may notice that this definition is underspecified. What exactly does it mean to “look at one trajectory”? People usually interpret it as “look at the geometric mean of the trajectory”, so that’s what I’ll take it to mean for now. ↩
I think the meme of “physicist encounters a new subject and immediately thinks they can do it better than experts” is overstated. But the stereotype holds true in this case. ↩
Peters, O. (2019). The ergodicity problem in economics. doi: 10.1038/s41567-019-0732-0 ↩ ↩² ↩³ ↩⁴
I’m playing fast and loose with probability here—it’s not entirely accurate to say that you “will” end up with $0. There is a more precise version of what I said that’s more accurate, but I don’t want to get too technical. I will give a formal mathematical definition later. ↩
There is another problem with the Russian Roulette example that’s something of a digression, but I will include it in this footnote for completeness:

Russian Roulette is not equivalent to playing six iterations of a game with a 5/6 probability of success. In Russian Roulette, you are sampling bullets without replacement, so the probability of finding a bullet goes up every time you win.

One of the articles I linked wrote:

You might roll the dice and take $1,000,000 to play Russian Roulette one time (though I wouldn’t advise it). But there’s no amount of money that would make you play it 6 or more times.

If you play 6 times, you have a 100% chance of dying. If you played a version where you sample with replacement (for example, you spin the revolver again after every shot), you are not guaranteed to die after 6 attempts. That would be the appropriate example. ↩
The Kelly criterion is not universally applicable, as I will discuss later in this article. ↩
Peters, O., & Gell-Mann, M. (2016). Evaluating gambles using dynamics. ↩ ↩²
Ford, M., & Kay, J. (2022). Psychology Is Fundamental: The Limitations of Growth-Optimal Approaches to Decision Making under Uncertainty. ↩ ↩²
Toda, A. (2023). ‘Ergodicity Economics’ Is Pseudoscience. doi: 10.48550/arXiv.2306.03275 ↩
Frost, G. (2009). Nobel-winning economist Paul A. Samuelson dies at age 94. ↩
Samuelson, P. (1979). Why we should not make mean log of wealth big though years to act are long. doi: 10.1016/0378-4266(79)90023-2 ↩
Samuelson, P. (1971). The “Fallacy” of Maximizing the Geometric Mean in Long Sequences of Investing or Gambling. doi: 10.1073/pnas.68.10.2493 ↩
I haven’t proved this mathematically, but this is the intuition:

For the time-series average to equal the point-in-time arithmetic mean, you must apply a transformation function that converts from geometric space to arithmetic, and the only way to do that is by applying the logarithm. ↩
I presume. I haven’t studied statistical dynamics so I don’t really know. ↩

Let's take a moment to marvel at how bad the original USDA food pyramid was

Wed, 21 May 2025 00:00:00 -0700

Edited 2025-05-26 to correct an inaccuracy—originally I said butter goes in the Dairy group but actually it goes in the Fats, Oils & Sweets group.

The original 1992 version of the USDA Food Pyramid was bad. So bad that people who scrupulously followed the guidelines were barely healthier than the people who ignored them.¹

But the Food Pyramid was not just wrong: it was marvelously wrong. It was wrong in many ways simultaneously. It achieved levels of wrongness hitherto undreamed of.

What was wrong about it? I will start with the obvious answers, and move into the philosophical.

Its ranking of the healthiness of foods is wrong. Refined grains are healthier than fruits and vegetables?² Processed meat and nuts are equally healthy? All oils should be used sparingly? What?
It lumps together foods that shouldn’t go together: whole grains + refined grains; red meat + healthy proteins + nuts; fats + oils + sweets.
- Nutritionally speaking, white bread has more in common with sweets than it does with whole grains. Oils (unsaturated fats) have more in common with nuts than they do with trans fats.
It implies there are specific numbers of servings of each food group that you should have, which is wrong in two ways:
- Required servings vary a lot from person to person. If I followed the upper end of the serving guidelines, I would be too skinny (I think—I’m not really sure how much food is in a “serving”). For some people, the lower end is still too much food.
- Giving a servings range (e.g. 3–5 servings for vegetables) implies that healthiness follows an inverted U curve, and it’s bad to eat too much or too little. But that’s usually not true: there is effectively no such thing as eating too many vegetables.³ There is no such thing as not eating enough trans fat (the ideal amount is zero). You can eat zero grains (keto diet) or zero meat (vegetarian diet) and still be perfectly healthy.
It fails to say anything about micronutrients or vitamin deficiencies—or even macronutrients for that matter.
The whole concept of a “food pyramid” is fundamentally flawed. It rests on the incorrect assumption that there are different food groups that you should eat in different amounts. It would be more accurate to say there are
- some foods you should eat, and there’s effectively no upper limit (fruits + vegetables);
- other food groups you can have plenty of as long as you don’t eat too many calories overall (whole grains, beans, nuts, seeds, vegetable oils⁴);
- and some foods where the ideal amount to eat is zero (trans fats, sugar, refined carbs, processed meat).⁵

(Bonus wrongness fact: I didn’t notice this until I inspected the food pyramid closely, but it says fruit contains added sugar, as indicated by the white triangles in the “fruit” section. This is so obviously incorrect that it’s kind of baffling why they drew the pyramid this way. Perhaps what they meant to say was that fruit contains sugar, but the key specifically says the white triangle indicates “added” sugar.)

To illustrate the beauty of the USDA’s achievement, I present to you the Food Wrongness Pyramid:⁶

The Healthy Eating Plate, by the Harvard T.H. Chan School of Public Health, fixes most of the problems with the USDA Food Pyramid:

It uses a plate instead of a pyramid! This is a good shape! It correctly implies that you should eat some of each of the food groups on the plate, and you should try to avoid foods that aren’t on the plate.
It says that there are some foods you should eat, and other foods you should avoid. (Eat whole grains; avoid refined grains.)
It correctly identifies which foods are healthy. (No putting starches at the bottom of the pyramid!)
It groups foods in a sensible manner. (Whole grains get a group; healthy proteins get a group; healthy oils get a group. There is no “starches” or “meat” or “fats + oils + sugars”.)
It suggests relative proportions instead of numbers of servings.
It pays attention to macronutrients: the plate includes both protein and oil (= fat). (It doesn’t mention carbs, but that’s okay because it’s pretty much impossible to under-eat carbs.)
It still doesn’t say anything about micronutrients, but if you follow the prescribed guidelines, you’ll probably get enough micronutrients anyway.

MyPlate, the USDA version of the Healthy Eating Plate, is similar to Harvard’s version except that they made it worse, probably because of lobbyists or whatever (it says red meat counts as a healthy protein; it gives dairy its own category⁷; it doesn’t say anything about healthy fats).

This is a side note but I think it’s funny how everyone puts nuts in the “protein” bucket even though most nuts have about as much protein as grains (which is to say, not very much). I would rather put nuts in the “oil” group…obviously nuts aren’t oil, but both nuts and oil are desirable as a source of unsaturated fat, so they should go together.

Notes

McCullough, M. L., et al. (2000). “Adherence to the Dietary Guidelines for Americans and Risk of Major Chronic Disease in Men.”

McCullough, M. L., et al. (2000). “Adherence to the Dietary Guidelines for Americans and Risk of Major Chronic Disease in Women.” ↩
Technically it doesn’t say grains are healthier than fruits/veggies, but it puts them lower on the pyramid, which naturally leads you to believe that they’re healthier. ↩
I’m sure someone in history has over-eaten vegetables at some point, but practically speaking you’re probably never going to get there. ↩
and probably also poultry, fish, and eggs, but eating those is bad for animals. ↩
Plus a short list of exceptions like sodium and fat-soluble vitamins, where you want to eat some but not too much. ↩
The items on this pyramid don’t match up with the items in my article because I revised the article a bit and I didn’t feel like re-making the image. I put a ton of work into my pyramid, as you can probably tell by the intricate and high-quality illustrations. ↩
The USDA MyPlate website says soy milk counts as dairy. Which, like, I guess I get what they were going for, but why is this category called “dairy”? ↩

Can you maintain lean mass in a calorie deficit?

Thu, 01 May 2025 00:00:00 -0700

TLDR: A meta-analysis allegedly showed that a 500-calorie deficit is the sweet spot to avoid losing lean mass, but the interpretation of the data was wrong and actually it didn’t show that. When interpreted correctly, the data provides weak (insignificant) evidence that any deficit will result in a loss of lean mass.

If you’re losing weight, does lifting weights reduce how much muscle you lose? Is it possible to entirely prevent muscle loss (or even gain muscle)?

Murphy & Koehler (2021)¹ did a meta-analysis on this question. They collected experiments where the experimental groups did resistance training while eating at an energy deficit (RT+ED), and the control groups did resistance training while eating a normal amount of food (RT+CON).

They found a strong association between change in lean mass and the magnitude of the energy deficit (slope = –0.325, p = 0.001). The meta-analysis predicts that you can eat at a deficit of 500 calories per day without losing any lean mass, but you will lose mass at a larger deficit.

(The meta-analysis also reported that participants gained strength in almost every study, even with larger calorie deficits. That’s useful to know, but I will focus on lean mass for this post.)

I should mention that what we actually care about is muscle loss, not lean mass loss. Lean mass includes anything that isn’t fat—muscle fibers, organs, glycogen, etc. Muscle mass is harder to measure. We don’t know what happened to study participants’ muscle, only their total lean mass.

Let’s set that aside and assume lean mass is a useful proxy for muscle mass.

The authors showed a plot of every individual study’s experimental group (RT+ED) and control group (RT+CON), along with a regression line predicting lean mass change as a function of energy deficit:²

But…does this regression line look a little odd to you?

Where are the RT+CON points, and where are the RT+ED points, relative to the regression line?

In particular, look at all the data points from experimental groups where participants had energy deficits of under 500 calories. Almost all of them lost lean mass on average (recall that each individual point represents the average result from one study); only four gained lean mass.

The slope of the regression line is almost entirely driven by the difference between the experimental and control groups.

What happens if we calculate a regression using only the experimental groups? Did groups with a bigger calorie deficit lose more lean mass?

Now the regression has a slope of only –0.123 (p = 0.28), and it predicts that any deficit will cause at least a small loss in lean mass.

So, among study groups where participants ate at a calorie deficit, it does appear that they lost lean mass on average. But there is not a clear relationship between the size of the deficit and the amount of lean mass lost.

In theory, it makes sense that when you have a larger calorie deficit, it should be harder for your body to preserve muscle. But the evidence from this meta-analysis doesn’t really support the theory. (It doesn’t contradict it, either. It just doesn’t say much either way.)

Murphy & Koehler (2021)’s original regression had a slope of –0.325. If that’s the true slope, then it’s unlikely that the experimental-only regression would have the much shallower slope of –0.123 (p = 0.07, likelihood ratio 5.2³).

The conclusions I drew from this meta-analysis:

Eating at a deficit might cause me to lose muscle, or it might not, who knows.
In theory, I expect there is some calorie deficit above which I start to lose muscle, but this meta-analysis doesn’t tell me what that number is.

Appendix: What else does this data tell us?

Looking at the experimental-only regression, how much evidence is this for or against the hypothesis that a larger calorie deficit causes more lean mass loss?

The experimental-only regression has slope –0.123 with standard error 0.111. The slope is still negative, which is consistent with the hypothesis, but it’s not strongly negative—only about one standard error away from zero.

The hypothesis predicts a positive intercept: it should be possible to gain muscle while maintaining weight. The experimental-only regression has a negative intercept (–0.054), but it is less than one standard error away from zero (the intercept’s standard error is 0.079). This is weak evidence against the hypothesis.

Source code is available on GitHub.

Updated 2026-03-25 to add a TLDR.

Notes

Murphy, C., & Koehler, K. (2021). Energy deficiency impairs resistance training gains in lean mass but not strength: A meta-analysis and meta-regression. doi: 10.1111/sms.14075 ↩
The paper’s plot is in black and white. I re-created it in color to make it easier to read. ↩
P-value was calculated using a two-sided t-test, where the null hypothesis is that the mean equals –0.3249. Standard error 0.111 which is the standard error of the experimental-only slope.

Likelihood ratio was calculated as norm.pdf(–0.1234, mu=–0.1234, sigma=0.111) / norm.pdf(–0.1234, mu=–0.3249, sigma=0.111). ↩

Why would AI companies use human-level AI to do alignment research?

Fri, 25 Apr 2025 00:00:00 -0700

Cross-posted to the EA Forum.

Many plans for how to safely build superintelligent AI have a critical section that goes like this:

Develop AI that’s powerful enough to do AI research, but not yet powerful enough to pose an existential threat.
Use it to assist with alignment research, thus greatly accelerating the pace of work—hopefully enough to solve all alignment problems.

You could call this process “alignment bootstrapping”.

This is a central feature of DeepMind’s plan (see “Amplified oversight”), Anthropic’s plan (see “Scalable Oversight”), and independent plans written by Sam Bowman (an AI safety manager at Anthropic), Joshua Clymer (a researcher at Redwood Research), and Marius Hobbhahn (CEO of Apollo Research).

There are various reasons why alignment bootstrapping could fail¹ even if implemented well, and some of those plans acknowledge this. But I’m also concerned about whether alignment bootstrapping will be implemented at all.

When the time comes, will AI companies actually spend their resources on alignment bootstrapping?

When AI companies have human-level AI systems, will they use them for alignment research, or will they use them (mostly) to advance capabilities instead?

AI companies currently employ many human-level humans, and use a small percentage of them to do alignment research. If it makes sense for them to use most of their human-level AIs to do alignment research, wouldn’t it also make sense to use most of their human researchers to do alignment research?

But they don’t do that. Most of their human researchers work on advancing AI capabilities.

It’s more likely that they use human-level AIs the same way they use human researchers: almost all of them work on accelerating capabilities, and a small minority work on safety. Which probably means capabilities outpace safety, which probably means we die.

Some companies argue that they need to advance capabilities right now to stay competitive. Perhaps that’s true. Consider what the world will look like once the first company develops human-level AI. At that point, the #2 company will only be a few months behind at most. So the leading company will once again say, “Sorry, we can’t use our human-level AI to work on alignment, we have to keep advancing capabilities to stay ahead.” And they will continue saying this right up until their AI is powerful enough to kill everyone.

Counterpoint: AI companies would probably argue that present-day AIs are far from being dangerous. But human-level AIs will be nearly dangerous, so at that point it’s too risky to keep advancing capabilities.

I would be more inclined to believe this if AI companies weren’t already behaving so recklessly.² If you’re going to prioritize safety over capabilities when the tradeoff becomes more critical, you should prove it to the world by prioritizing safety over capabilities right now.

Perhaps the ideal perfectly-altruistic AI company would indeed push capabilities right now and then switch to safety at the critical time,³ but I see little reason to believe that that’s what any of the real-life AI companies are going to do.

Notes

By my reading, none of the plans put probabilities on how concerning these reasons are. My guess is that, if alignment bootstrapping is implemented as these plans typically describe, then there’s a greater than 50% chance that we die.

The purpose of this essay isn’t to talk about the implementation problems with alignment bootstrapping, but in brief:

If your alignment-researcher AI is smarter than you, and you don’t know how to align AI yet, then you can’t trust that your AI is doing good work.

People who propose bootstrapping are usually aware of this problem. They have preliminary ideas for how they will evaluate the work of an AI that’s smarter than them, coupled with bafflingly high confidence that their untested ideas will work. (Zvi proposed a test: “Can you get a method whereby the Man On The Street can use AI help to code and evaluate graduate level economics outputs and the quality of poetry and so on in ways that would translate to this future parallel situation?”) ↩
I wanted to provide a link to a well-sourced and well-reasoned list of reckless behaviors by AI companies. I found no such list, so instead this is a link to a section of a post I wrote that includes numerous examples of reckless behavior. ↩
I don’t actually think this is what safety-minded AI companies should do. I think they should spend less on capabilities and more on safety. But I am sympathetic to the position that they should temporarily focus on advancing capabilities. ↩

Do Protests Work? A Critical Review

Fri, 18 Apr 2025 00:00:00 -0700

James Özden and Sam Glover at Social Change Lab wrote a literature review on protest outcomes¹ as part of a broader investigation² on protest effectiveness. The report covers multiple lines of evidence and addresses many relevant questions, but does not say much about the methodological quality of the research. So that’s what I’m going to do today.

I reviewed the evidence on protest outcomes, focusing only on the highest-quality research, to answer two questions:

Do protests work?
Are Social Change Lab’s conclusions consistent with the highest-quality evidence?

Here’s what I found:

Do protests work? Highly likely (credence: 90%) in certain contexts, although it’s unclear how well the results generalize. [More]

Are Social Change Lab’s conclusions consistent with the highest-quality evidence? Yes—the report’s core claims are well-supported, although it overstates the strength of some of the evidence. [More]

Cross-posted to the Effective Altruism Forum.

Contents
Introduction
Studies on real-world protest outcomes
Meta-analysis
Potential problems with the research
Are Social Change Lab’s claims justified?
- Broad claims
- Claims about individual studies
Conclusion
Source code
Appendix A: Additional tables
Appendix B: Methodological revisions
Appendix C: Comparing the strength of evidence to saturated fat research
Notes

Introduction

This article serves two purposes: First, it analyzes the evidence on protest outcomes. Second, it critically reviews the Social Change Lab literature review.

Social Change Lab is not the only group that has reviewed protest effectiveness. I was able to find four literature reviews:

Animal Charity Evaluators (2018), Protest Intervention Report.
Orazani et al. (2021), Social movement strategy (nonviolent vs. violent) and the garnering of third-party support: A meta-analysis.
Social Change Lab – Ozden & Glover (2022), Literature Review: Protest Outcomes.
Shuman et al. (2024), When Are Social Protests Effective?

The Animal Charity Evaluators review did not include many studies, and did not cite any natural experiments (only one had been published as of 2018).

Orazani et al. (2021)³ is a nice meta-analysis—it finds that when you show people news articles about nonviolent protests, they are more likely to express support for the protesters’ cause. But what people say in a lab setting might not carry over to real-life behavior.

I read through Shuman et al. (2024). Compared to Ozden & Glover (2022), it cited weaker evidence and made a larger number of claims with thinner support.

I looked through these literature reviews to find relevant studies. The Social Change Lab review was by far the most useful; the other reviews didn’t include any additional studies meeting my criteria. I used ChatGPT Deep Research⁴ to find more publications.

I focused my critical analysis on only the highest-quality evidence:

I did not review lab experiments. The Orazani et al. meta-analysis is informative, but it might not generalize to the real world.
There are many studies showing an association between protests and real-world outcomes (voting patterns, government policy, corporate behavior, etc.), but the vast majority of them are observational.

Observational studies cannot establish causation. They cannot distinguish between “protests raised support for the cause” and “protests happened because people supported the cause”. No amount of controlling for confounders fixes this problem.

Therefore, my review focuses only on natural experiments that measure real-world outcomes.

Conflict of interest: In 2024 I donated to PauseAI US, which organizes protests. I would prefer to find that protests work.

Studies on real-world protest outcomes

Social Change Lab reviewed five studies on how protests affect voter behavior, which they judged to be the best studies on the subject.

I excluded two of the five studies due to methodological concerns:

McVeigh et al. (2014)⁵ is an observational study that looked at long-term changes in Republican voting in counties where the Ku Klux Klan was most active.
Bremer et al. (2020)⁶ is a study on the correlation between protests and electoral outcomes in European countries.

I did not review these because they are observational studies, and I wanted to focus on natural experiments.

I did review the other three studies:

Madestam, A., Shoag, D., Veuger, S., & Yanagizawa-Drott, D. (2013). Do Political Protests Matter? Evidence from the Tea Party Movement.
Wasow, O. (2020). Agenda Seeding: How 1960s Black Protests Moved Elites, Public Opinion and Voting.
Klein Teeselink, B. K., & Melios, G. (2021). Weather to Protest: The Effect of Black Lives Matter Protests on the 2020 Presidential Election.

In addition, I looked at two studies that the Social Change Lab report did not cover:

Larreboure, M., & Gonzalez, F. (2021). The Impact of the Women’s March on the U.S. House Election.
Hungerman, D., & Moorthy, V. (2023). Every Day Is Earth Day: Evidence on the Long-Term Impact of Environmental Activism.

There are no randomized controlled trials on the real-world effect of protests (how would you randomly assign protests to occur?). But there are five natural experiments—three from the Social Change Lab review, plus the Women’s March and Earth Day studies. Most of the natural experiments use the rainfall method.

The idea is that protests often get canceled when it rains. If you look at voting patterns in places where it rained on protest day compared to where it didn’t rain, you should be able to isolate the causal effect of protests. The rain effectively randomizes where protests occur.

Rather than using rainfall directly, the rainfall method uses rainfall shocks—that is, unexpectedly high or low rainfall relative to what was expected for that location and date. This avoids any confounding effect of average rainfall levels.

The clearest illustration of the rainfall method comes from Hungerman & Moorthy (2023)⁷ (which I will discuss in more detail later). The authors looked at counties where it rained vs. didn’t rain on the inaugural Earth Day—April 22, 1970. Then they used rainfall to predict the rate of birth defects from 1980–1988.The hypothesis is that Earth Day demonstrations increased support for environmental protections. That in turn would reduce environmental contaminants, leading to fewer birth defects. And if rainfall stops demonstrations from happening, then it will have the opposite effect.

The rainfall method is commonly used in social science, and it has received some fair criticism.⁸⁹ But the rainfall method as it was used by Hungerman & Moorthy is robust to these criticisms, as illustrated by this chart:

The key to establishing causation is that rainfall had no predictive power on any other day. It only mattered on Earth Day.

That leaves us with two possibilities:

Rainfall is associated with higher birth defects due to some confounding variable, but only rainfall on April 22 and not on any other day, because that day is special somehow, in a way that has nothing to do with Earth Day; or
Earth Day demonstrations reduced the rate of birth defects.

(Or the results could be due to a statistical error or data manipulation. I will discuss those possibilities later.)

A summary of the five studies I reviewed plus the two I declined to review:

Table 1: Summary of Studies

Study	Protest	Protest Type	Effect	Randomization Method	Quality
Madestam et al.¹⁰	Tea Party	nonviolent	+	rainfall	high
Wasow¹¹	Civil Rights	violent	-	rainfall	medium
Wasow¹¹	Civil Rights	nonviolent	+	none (observational)	low
Klein Teeselink & Melios¹²	BLM	nonviolent	+	rainfall	high
Larreboure & González¹³	Women’s March	nonviolent	+	weather shocks	medium
Hungerman & Moorthy⁷	Earth Day	nonviolent	+	rainfall	high
McVeigh et al.⁵	KKK activity	unclear	+	none (observational)	low
Bremer et al.⁶	Europe elections	nonviolent	?¹⁴	none (observational)	low

(Methodological quality is relative. I’d have higher confidence in a true experiment than in any of these quasi-experimental methods.)

Next I will review each study individually. Then I will collect the results into a meta-analysis.

Madestam et al. (2013) on Tea Party protests

Madestam et al. (2013)¹⁰ looked at the effect of 2009 Tea Party protests on the 2012 US elections. It used the rainfall method to establish causality.

As an additional check, the authors tested whether rainfall could predict Republican and Democratic vote shares in the 2008 election. (You may recall that the 2009 Tea Party protests did not occur until a year after the 2008 election.) If rainfall can predict the 2008 election results—before the protests occurred—that means the model was confounded.

Madestam et al. (2013) found that rainfall in 2009 could not predict votes in 2008 (see Table II), but it could predict votes in 2012 (see Table VI).

The authors also tested whether rainfall on other days prior to the Tea Party protests could predict 2009 voting patterns, and found that they could not.

In the authors’ model, a rainy protest decreased Republicans’ share of the vote in the 2012 election by 1.04 percentage points (p < 0.0006¹⁵). This suggests that protests did indeed increase the Republican vote share.

Interestingly, rainfall decreased Republican vote share relative to the total population, but did not increase the Democratic share. This suggests that protests increased voter turnout but did not cause voters to change their minds.

At first I thought the inability to predict 2008 votes might be a false negative (like a p = 0.06 situation), but this was not the case. Rainfall in 2009 increased Republicans’ vote share in 2008, although only slightly (p = 0.38). (Remember that rainfall is supposed to decrease Republican votes by preventing Tea Party protests from happening.)

There is another concern with the rainfall model—not with causality, but with overstating the strength of evidence. A standard statistical model assumes that all observations are independent. But rainfall is spatially autocorrelated, which is the statistical way of saying that rain in one county is not independent of rainfall in the neighboring counties. If you have data from 2,758 counties, you can’t treat them as 2,758 independent samples.

Madestam et al. (2013) used several methods to account for this. First, it clustered standard errors at the state level instead of at the county level. Second, as a robustness check, the authors assumed spatial correlations varied as an inverse function of distance, which produced similar standard errors. Third, the authors tried dropping states one at a time to see if any states overly influenced the results.

Placebo tests

Finally:

[W]e conduct a series of placebo tests using rainfall on other historical dates in April. These placebos are drawn from the same spatially correlated distribution as rainfall on April 15, 2009. If rainfall on the protest day has a causal effect, the actual estimate of rainfall ought to be an outlier in the distribution of placebo coefficients.

They calculated the “placebo p-value” as the probability that rainfall on a random day could predict outcomes better than rainfall on protest day. If the model has correctly accounted for spatial autocorrelation then the placebo p-value should equal the original model p-value, plus or minus some random variation.

The authors run tests on 627 random “placebo dates”, and find that rain on protest day had a larger effect size than almost any of the placebo dates (see Figure V). This suggests that their corrections for spatial correlation worked, making false positives unlikely. However, the p-values on Figure V were a bit higher than the p-values in the main text, suggesting some effect size inflation due to spatial autocorrelation.

Wasow (2020) on 1960s civil rights protests

Wasow (2020)¹¹ measured the effect of nonviolent protests using observational data only. I won’t discuss that portion of the paper.

Wasow applied the quasi-experimental rainfall model to violent protests and found that they had a significant backfire effect. I won’t focus on the evidence on violent protests because I would recommend against engaging in violence regardless of what result the study found.

But if violent protests decrease public support, that’s (weak) evidence against protests working in general. The simplest hypothesis is “protests work”. But evidence on violent protests contradicts this, requiring a more complex claim: “nonviolent protests work, violent protests backfire”. I will evaluate this two-part hypothesis in the meta-analysis below.

As some additional evidence on violent protests, Collins & Margo (2007)¹⁶ used the rainfall method to find that 1960s riots decreased nearby property values. This is consistent with the finding from Wasow (2020) that violent protests backfire, but property values are not directly relevant to protesters’ outcomes. It’s conceivable that protests could simultaneously decrease local property values and increase public support.

Replication data from Wasow (2020) is publicly available.

Klein Teeselink & Melios (2021) on 2020 Black Lives Matter protests

Klein Teeselink & Melios (2021)¹² used the rainfall method to establish the effect of Black Lives Matter protests on the 2020 presidential election.

Unlike Madestam et al. (2013), this paper did not test whether rainfall could predict outcomes before the protests (which would indicate confounding).

As with Madestam et al. (2013), the authors of this paper considered the fact that rainfall is not independent across counties. Their model adjusts for this by including independent variables to represent the change in vote shares in surrounding counties, scaled by inverse distance.

Unlike the other studies in this review, Klein Teeselink & Melios (2021) treated county vote changes as interdependent. Their model assumes that the change in vote share in one county is partially explained by vote changes in the nearby counties, using the method described by Beck et al. (2006)¹⁷.

Klein Teeselink & Melios’ model isolates the impact of local protests on local vote change. In the method of (e.g.) Madestam et al. (2013), some vote changes may be explained by protests in neighboring counties. Klein Teeselink & Melios’ method is more rigorous in a sense, but we don’t actually want to isolate local changes. We want to know how well protests work overall, not just their local effects.

Klein Teeselink & Melios performed a robustness check in Table A3, Panel D where they fully ignored spatial autocorrelation. This produced mean effects more in line with the other studies: a vote share change of 11.9 per protester (std err 2.9), and a change of 0.105 based on the probability of rain (std err 0.032).

If you ignore spatial autocorrelation, you may overestimate the strength of evidence. However, in this case, ignoring spatial autocorrelation had only a modest impact on the t-stats:

	Primary Model	Ignoring Spatial Autocorrelation	Ignoring Spatial Autocorrelation + Counties Weighted by Population
Vote Share Per Protester	5.5	4.1	7.2
Vote Share by Rain Probability	2.3	3.3	9.3

The paper’s replication data is publicly available.

Failed placebo tests

Earlier, I discussed how Madestam et al. (2013) performed “placebo tests” to check that its model wouldn’t generate too many false positives. Klein Teeselink & Melios (2021) did the same, although with only nine placebo tests instead of 627:

(May 25 was the beginning of the period in which the majority of protests happened.)

This chart shows that Klein Teeselink & Melios’ version of the rainfall method did not establish causality. The fortnight of April 29—a month before the protests started—showed nearly the same effect size as the May 25 period, and 6 out of 9 placebo periods had p-values less than 0.05. So either some confounding variable explains the association between protests and vote share, or the standard error is underestimated due to spatial autocorrelation (or something similar).

The authors write

part of this association may be caused by serial correlation in weather patterns

In other words, a rainy June often also means a rainy April or May, so rain in April/May might appear to affect protest outcomes because it’s correlated with rain in June. (And thus the model does establish causality.)

That may be true, but I’m not confident in that explanation,¹⁸ and therefore I can’t trust this model to establish causality. Therefore, I exclude the BLM protests from my meta-analysis.

(But this does leave me wondering: How is it that rainfall shocks in April could predict vote changes in the 2020 presidential election?)

It would be interesting to compare the publicly-available BLM data to the Earth Day data (see below) to figure out why the Earth Day paper passed its placebo test but BLM did not. But that’s beyond the scope of this article.

Larreboure & González (2021) on the Women’s March

Larreboure & González (2021)¹³ attempted to use the rainfall method to predict whether the 2017 Women’s March affected how many votes went to woman candidates in the 2018 election. I say “attempted” because they found that rainfall did not predict Women’s March attendance. So instead, they used “weather shocks” to predict voting outcomes. These shocks were defined as a combination of weather-related factors that they chose using a LASSO¹⁹ regression model.

I see no obvious problem with the “weather shocks” method, but I’m wary of adding more mathematical complexity. Complexity makes flaws harder to spot.

Larreboure & González found that protests increased voter turnout and vote share to women for both Democratic and Republican candidates.

The authors accounted for spatial autocorrelation by clustering standard errors at the state level. They included a robustness check where they adjusted for spatial autocorrelation using the method from Conley (1999)²⁰ with two different distance cutoffs, 50 km and 100 km (in Table A.8).

This paper had at least two inconsistencies in its reported figures:

Page 13 says an additional 1% of the population protesting increased vote share for women and under-represented groups by 12.95 percentage points (pp). However, Table 4 on page 28 reports an increase of 12.70 pp.²¹
A more minor error, but page 13 says the 12.95 pp number is “remarkably similar to the impact of the Tea Party protesters on the vote share of the Republican Party (i.e. 12.59)”. However, the 12.59 number from Madestam et al. (see above) is the change in absolute votes, not vote share. The reported change in vote share was 18.81, which is not remarkably similar to 12.95.

I will take the 12.70 number reported in Table 4 as correct (it is repeated again in the robustness checks).

To be conservative, in my meta-analysis I will use the figures from the 50 km robustness check (where available) because they had the largest standard errors.

Hungerman & Moorthy (2023) on Earth Day

Hungerman & Moorthy (2023)⁷ found that rainfall on the inaugural Earth Day, April 22, 1970, could predict people’s environmental attitudes on surveys from 1977 to 1993.

It also directly measured environmental impact by looking at pollutant levels and rates of birth defects (which can result from exposure to environmental contaminants). It found that rainfall on Earth Day could predict birth defects.

The paper claims that rainfall predicted carbon monoxide levels, and it did find a statistically significant change. However, Appendix Table A3 examines five environmental contaminants, of which only carbon monoxide had a t-stat above 2, and two out of five outcomes were (slightly) negative. The positive effect on carbon monoxide may be a false positive.

Earlier I showed this chart:

The chart shows that rainfall on April 22, 1970–Earth Day—predicts the rate of birth defects 10 years later, but rainfall on any other day does not.

The same chart for the effect of rainfall on support for environmental spending:

The paper addresses the previously-mentioned spatial autocorrelation problem using the same techniques as Madestam et al. (2013). If spatial autocorrelation were distorting the effect sizes, we would expect to see more spurious statistically significant outcomes on the charts above. But we only see large effect sizes on Earth Day, not on any other day, which indicates that spatial autocorrelation is not a problem.

Like Madestam et al. (2013), the authors generated hundreds of additional “placebo tests” (as described above) where they looked at how well rainfall on different random days could predict environmental outcomes. They found that the placebo p-values were very similar to the original p-values (and even lower in some cases):

The paper’s source code and data are publicly available.

Meta-analysis

For two of the five natural experiments, I calculated expected change in number of votes for each additional protester, or change in vote share per protester (defined as votes per protester divided by turnout):

Table 2: Change in Votes Per Protester

Protest	Votes	Std Err	Vote Share	Std Err	n	Source
Tea Party	12.59	4.21	18.81	7.85	2758	Table VI
Women’s March	3*	**	9.62	4.47	2936	Table A.8 and page 3

*only one significant figure was provided

**not reported

I did not include the Earth Day or Civil Rights protests because the studies did not provide the relevant data.²² The BLM study reported vote share per protester, but I excluded it due to the study’s failure to establish causality, discussed previously.

I aggregated the results by applying a random-effects model. According to these two studies, protests have a mean impact of +11.95 vote share per protester (standard error 4.00; likelihood ratio 87.1; p < 0.003).

(The likelihood ratio tells us how much evidence the data provides. A likelihood ratio of 10.3 means that, assuming the study’s methodology is perfect, the odds of getting this result are 10.3x higher if the true mean is 7.84 than if the true mean is 0.)

If we are considering supporting some upcoming protest, we might want to estimate the probability that it will backfire. One way to do that is by using the pooled sample of past protests.

This pooled sample has a between-study standard deviation of 1.19, which reflects how much the effectiveness of protests varied across the studies. If we assume that the sample’s mean and between-study variation are exactly correct (which is questionable, since the pool only includes two studies), then we can model protest outcomes as a normal distribution with a mean of 11.95 and a standard deviation of 1.19.

Under this model, the probability of a protest having a negative effect—i.e., producing a value less than zero—is extremely small. But I would not take these precise numbers too seriously.

Vote share per protester is the most interesting metric for my purposes because it gives information about cost-effectiveness—it tells you how much impact you can expect for each marginal protester. But the natural experiments reported on other outcomes as well, such as overall change in vote share (as determined by changes in rainfall) and popular support for protesters’ objectives.

I applied a random-effects model to aggregate a few different sets of outcomes:

Table 3: Pooled Sample Outcomes

Outcomes	Mean	Std Err	likelihood ratio	p-value	P(negative effect)
Vote Share Per Protester	11.95	4.00	87.1	0.003	0
Vote Share	1.59	0.48	257	0.001	0.002
Vote Share (Rain Only)	1.14	0.42	39.3	0.007	0
Single Hypothesis	1.06	0.78	2.55	0.172	0.199
Favorability	2.68	2.32	1.95	0.249	0.176

Row 1 – Vote Share Per Protester uses the pooled outcome that I described in Table 2, including Tea Party and Women’s March vote share per protester.
Row 2 – Vote Share takes these outcomes from the studies on nonviolent protests:
- Tea Party – Republican vote share
- Women’s March – women’s vote share
- Earth Day – favorability (1)²³ as a proxy for vote share
Row 3 – Vote Share (Rain Only) uses the same outcomes as Row 2, but excluding the Women’s March outcome because it used weather shocks rather than rainfall.
Row 4 – Single Hypothesis does not differentiate between nonviolent and violent protests, instead lumping all studies together. It includes the three Vote Share measures from Row 2, plus Civil Rights – vote share.
Row 5 – Favorability includes measured changes in popular support for a protest’s goals:
- Tea Party – support for the Tea Party
- Earth Day – favorability (1)²³

(Note: P(negative effect) = 0 doesn’t mean it’s literally zero, but it’s so small that it gets rounded off to zero.)

The Women’s March and Earth Day papers used continuous rainfall variables instead of binary (rain vs. no rain); those papers’ outcomes were standardized using the method from Gelman (2007)²⁴ to put them on the same scale as binary variables.²⁵

The Vote Share Per Protester and Vote Share tests produce low p-values/high likelihood ratios, and under those models, nonviolent protests have virtually no chance of having a negative effect on support. Favorability has a weak likelihood ratio due to a large variance between outcomes.²⁶

Under the Single Hypothesis model, protests have a much weaker p-value/likelihood ratio. Naturally, when you include a negative outcome, it pulls down the average effect quite a bit. The mean is still positive, which makes sense given that only one out of four included protests was violent.

Is it fair to separate out violent and nonviolent protests? I’m wary of adding complexity to a hypothesis but I believe it’s justified in this case:

It’s intuitively plausible that peaceful protests would earn support while violence would backfire.
Lab experiments³ and observational studies support this bimodal hypothesis.
I ran a t-test for the hypothesis that nonviolent and violent protests have the same effect on voting outcomes, comparing the pooled outcome from Row 2 – Vote Share against the Civil Rights protest outcome. The result had a likelihood ratio of 55.1 and p < 0.005. We can strongly reject the hypothesis that these two samples have the same mean.

There are some reasons believe these results may be overstated, which I will address under Potential problems with the research. There are also at least two reasons to believe they may be understated:

Rainfall does not perfectly predict whether protests occur. (Sometimes people protest in the rain.) If protests genuinely work, then the effect of protests will be larger than the effect of protests as predicted by rainfall.
I aggregated the most similar metrics into pooled outcomes. But these were not always the strongest metrics. For example, Earth Day protests strongly predicted birth defects (likelihood ratio 55,000; p < 3e-6). But I did not include birth defects in the meta-analysis because it did not have any comparable counterpart in the other studies.

Table 4 shows outcomes across the five studies, estimated by looking at counties where it rained vs. did not rain. The Women’s March and Earth Day results are standardized as explained above.²⁵

Table 4: Societal-Level Protest Outcomes

Protest	Outcome	Change	Std Err	Source
Tea Party	votes (as % of population)	1.04%**	0.30%	Table VI
Tea Party	vote share	1.55%*	0.69%	Table VI
Tea Party	conservative vote score²⁷	1.922*	0.937	Table VII
Tea Party	average belief effect²⁸	0.13***	0.037	Table V
Tea Party	strongly supports Tea Party	5.7%*	2.5%	Table V
Tea Party	Sarah Palin favorability	5.7%*	2.6%	Table V
Tea Party	“outraged about way things are going in country”	4.6%*	2.1%	Table V
Tea Party	opposes raising taxes on income >$250K	5.8%	3.0%	Table V
Tea Party	“Americans have less freedom than in 2008”	6.5%*	2.6%	Table V
Tea Party	Obama unfavorability	4.6%	2.4%	Table V
Civil Rights (violent)	vote share among white voters	–5.56%*	2.48%	Appendix, Table 12
BLM	vote share	2.7%	1.2%	Table 2
BLM	“Blacks should not receive special favors”	–0.242²⁹	0.360	Table 3
BLM	“Slavery caused current disparities”	0.339²⁹	0.388	Table 3
Women’s March	women’s vote share	2.48%***	0.64%	Table 4
Women’s March	voter turnout	0.41%**	0.14%	Table 4
Earth Day	favorability (1)²³	0.90%	0.53%	Table 2
Earth Day	favorability (1)²³ among under-20s	1.67%**	0.62%	Table 2
Earth Day	favorability (2)²³	1.12	0.70	Table 2
Earth Day	favorability (2)²³ among under-20s	1.90*	0.82	Table 2
Earth Day	carbon monoxide	0.07*	0.03	Table 4
Earth Day	birth defects	1.00***	0.21	Table 4

*p < 0.05; **p < 0.01; ***p < 0.001

Potential problems with the research

Spatial autocorrelation

Recall that “spatial autocorrelation” is a technical way of saying “rainfall is not independent across counties”. If you assume your samples are independent when they’re not, your standard errors will be too low—giving you too much confidence in your results.

It’s conceivable that all five studies overstated the strength of their results due to spatial autocorrelation.

Each study on nonviolent protests used at least some technique to correct for spatial autocorrelation. Madestam et al. (2013) and Hungerman & Moorthy (2023) included “placebo tests”. The placebo tests from Madestam et al. (2013) indicated that these corrections mostly worked but did not fully succeed, whereas Hungerman & Moorthy’s corrections apparently did succeed. On balance, this suggests that the standard errors of the pooled outcome may be understated, but probably not by a large margin.

Two of the pooled outcomes from Table 3—the Vote Share Per Protester and Favorability pools—had strong likelihood ratios / low p-values. That suggests they should hold up even with somewhat reduced statistical power.

Publication bias

The standard method to assess publication bias would be to make a funnel plot. I didn’t do that for two reasons:

With only five studies (at best), there aren’t enough data points to detect publication bias even if it exists.
A funnel plot only works if your studies cover a range of sample sizes. All the natural experiments have roughly the same sample size (because they all look at county-level data for the majority of US counties).

As an alternative, I tested how the results might change if we discovered some unpublished null results. I used the following procedure:

Choose one of the pooled outcomes from Table 3.
For each individual study outcome, clone it to create a “dummy null outcome” with the same standard error and sample size, but a mean of 0. This represents a hypothetical study that didn’t get published because it found a null result.
Construct a larger pooled sample using all four or six outcomes (the two or three real outcomes plus the two or three null dummies).

Table 5: Pooled Sample Effects, Adjusted for Publication Bias

Outcomes	Mean	Std Err	likelihood ratio	p-value
Vote Share Per Protester	6.43	4.01	3.61	0.11
Vote Share	0.80	0.41	6.97	0.049
Vote Share (Rain Only)	0.58	0.36	3.77	0.104
Favorability	0.74	0.65	1.91	0.256

Adding in null results considerably weakens the strength of evidence.

This approach is deliberately conservative. I wouldn’t say this meta-analysis is robust to publication bias, but it’s not particularly vulnerable to publication bias, either.

(The dummy-null approach leaves something to be desired. If the true mean were 6.43 as the pooled sample suggests, it would be surprising to see three positive results with low p-values plus three null results with equally tight standard errors. But I haven’t thought of any better ideas for how to test publication bias.)

Orazani et al. (2021)³ reviewed lab experiments on protest favorability. Among other things, it looked at publication bias. This paper might be informative, since it stands to reason that if experimental researchers on protests have a certain bias, then sociological researchers might have a similar bias.

The paper included a funnel plot:

To supplement the plot, I tested for publication bias using two statistical tests:

Egger’s regression test³⁰ found r = 0.124, p < 0.646 (r > 0 means that more powerful studies had larger mean effects, which if anything is evidence of inverse publication bias).
Kendall’s tau test found p < 0.565.

Orazani et al. (2021) included 14 experiments and 2 non-experimental studies. I also tested for publication bias when excluding the non-experiments and again found highly insignificant p-values.

Orazani et al. (2021) tested for a difference between published and unpublished studies (although they defined “unpublished” in a way that seemed strange to me—they counted dissertations and conference presentations as unpublished). They found a significant difference in effect size, suggesting the presence of publication bias. Published studies had a Cohen’s d of 0.39, versus 0.22 for unpublished studies. However, this difference disappeared when the authors controlled for certain features of the protests being studied (e.g. protests directed at the government as opposed to society). I am not sure what to make of this, but there is at least some evidence of publication bias.

Data fabrication

Most meta-analyses do not consider the possibility that some studies’ data might be fabricated, and I believe they should. Checking for fraud is difficult in general, but I will do some basic checks.

When humans fabricate data, they often come up with numbers that don’t look random. Real data should follow two observable patterns:

The last digits of numbers should be uniformly distributed.
The first digits of numbers should NOT be uniformly distributed. Instead, they should obey Benford’s law.

I tested for suspicious patterns by collecting a list of statistical results (means and standard errors for various outcomes) from the BLM, Tea Party, Women’s March, and Earth Day papers. I did not include the Civil Rights paper because its quasi-experimental data only included violent protests.

I also did a power check to determine whether the tests have adequate statistical power. We should be able to reject the hypotheses that the first digits follow a uniform distribution, and that the last digits follow Benford’s law.

  Tea Party:
      First-digit Benford's Law p-value: 0.598
      Last-digit uniformity p-value:     0.306
      Power check p-values:              0.001, 0.002

  BLM:
      First-digit Benford's Law p-value: 0.438
      Last-digit uniformity p-value:     0.598
      Power check p-values:              0.001, 0.001

  Women's March:
      First-digit Benford's Law p-value: 0.181
      Last-digit uniformity p-value:     0.891
      Power check p-values:              0.001, 0.001

  Earth Day:
      First-digit Benford's Law p-value: 0.121
      Last-digit uniformity p-value:     0.224
      Power check p-values:              0.038, 0.001

(P-values are rounded up to 3 digits. See source code for full details.)

In all cases, I found high p-values for the first and last digits, which means the data follow the expected natural patterns. And I found very low p-values for the sanity check tests, which means the tests are sufficiently powerful (except for Earth Day first digits, where few independent outcomes were reported).

These tests do not rule out more sophisticated fraud. For example, if the authors generated false data and then calculated statistical tests on top of them, the fabricated results would still pass the first-digit and last-digit checks.

Data errors

Checking for data errors is difficult in general.³¹ I did a basic consistency check to verify that each study’s reported means and standard errors seemed internally consistent, but it’s hard to see errors that way.

The only data error I noticed was in Larreboure & González (2021)¹³. As I mentioned before, it reported inconsistent numbers for the change in vote share based on each 1% of the population protesting: 12.95 pp (std err 5.63) on page 13 in the text, and 12.70 pp (std err 5.48) in Table 4.

The difference is small, which suggests the authors may have made some revision to their calculations but didn’t update all the values reported in their manuscript. If so, the number in Table 4 is likely the correct one.³²

This inconsistency suggests that the authors have some degrees of freedom for p-hacking, but the two numbers are similar enough to have minimal impact on the result of my meta-analysis.

Will the results generalize?

All the protests covered by natural experiments have certain commonalities:

They all had a large number of participants.
They were all nationwide (they had to be, so the study authors could use county-level data).
They all took place in the United States.

Will the results generalize to other countries? Will the results generalize to smaller-scale or local protests?

The fact that these protests were so widespread means their objectives couldn’t have been far outside the Overton window (i.e., the range of politically acceptable ideas at the time). Perhaps a protest that advocated for a more radical position would be more likely to backfire. To address this question, perhaps we could look at lab experiments on protests, but that’s beyond the scope of this article.

Meta-concerns with this meta-analysis

I have some criticisms of my meta-analysis itself:

I did not pre-register a methodology. I have limited experience conducting meta-analyses and I was learning as I wrote this article. Realistically, I would not have had the motivation to finish if I’d been required to fully determine a methodology in advance. But the platonic ideal of this meta-analysis would have included a pre-registration.
Three of the studies (BLM, Civil Rights, and Earth Day) published their data. A thorough analysis would attempt to replicate those studies’ findings. I did not do that.

Broad claims

Social Change Lab’s literature review included a summary of findings, reproduced below.

Table 6: Social Change Lab Findings

Finding	Confidence
Protest movements can have significant short-term impacts	Strong
Protest movements can achieve intended outcomes in North America and Western Europe	Strong
Protest movements can have significant impacts (2-5% shifts) on voting behaviour and electoral outcomes	Medium
Protest movements can positively influence public opinion (≤10% shifts)	Medium
Protest movements can influence public discourse (e.g. issue salience and media narratives)	Medium
Protest movements can influence policy	Low (mixed evidence)
Protest movements can influence policymaker beliefs	Low (little evidence)
Protest movements can achieve desired outcomes in the Global South	Low (little evidence)
Protest movements can have significant long-term impacts (on public opinion and public discourse)	Low (little evidence)

In this section, I assess whether the natural experiments support each “Strong” and “Medium” claim. I find that the evidence does indeed support the findings and I agree with Social Change Lab’s confidence levels in each case.

I do not review the four “Low Confidence” claims because none of the natural experiments attempted to test them. (That fact itself suggests that “Low Confidence” is an accurate label.)

Starting with the findings rated “Strong”:

Protest movements can have significant short-term impacts Strong

Protest movements can achieve intended outcomes in North America and Western Europe Strong

The natural experiments support these claims. There’s also supporting evidence from lab experiments on how protests affect people’s perceptions; studies on media coverage; and observational data on protest outcomes. For a meta-analysis of lab experiments, which I view as the second-strongest form of evidence, see Orazani et al. (2021)³.

I do not have much confidence in most of these lines of evidence, but the natural experiments offer good support:

All study results point the same direction (as long as we exclude the data on violent protests).
The pooled outcomes have high likelihood ratios / low p-values.
There are no signs of data fabrication.

I’m still concerned about publication bias and spatial autocorrelation. I am not sure it is appropriate to describe the evidence as “Strong”. It would be fair to downgrade your confidence to “Medium” based on these concerns. But I also think “Strong” confidence is defensible; the distinction depends on how much weight you give to the hard-to-quantify limitations with the existing evidence.

The natural experiments all cover nationwide, popular protests in the United States, so it’s not clear that the results generalize. Regardless, Social Change Lab didn’t claim that protests always have significant impacts, only that they “can” have impact; and the existence of these natural experiments shows that indeed they can.

The highest-quality studies are all natural experiments, not true experiments. A true experiment would be preferable. But the rainfall method seems sufficient to establish causality so I am comfortable treating these natural experiments’ methodologies as valid.

Whether this evidence qualifies as “strong” is a matter of debate. Certainly the evidence could be much stronger. But I would be surprised if these findings were overturned, so I think Social Change Lab’s confidence level is fair.

Protest movements can have significant impacts (2-5% shifts) on voting behaviour and electoral outcomes Medium

A 2–5% shift is consistent with the natural experiments, which found changes in vote share ranging from 1.55% to 5.54% (see Table 4). I think 2–5% is fair as an optimistic expectation, given that the natural experiments all covered large nationwide protests.

I believe the rainfall method is effective at establishing causality, but we can’t be too confident in the magnitude of the effect because rainfall does not perfectly predict protest attendance. So I would not rate the confidence for this finding as higher than “Medium”.

Protest movements can positively influence public opinion (≤10% shifts) Medium

Among the natural experiments, only two (Madestam et al. 2013; Hungerman & Moorthy 2023) reported on public opinion in terms of percentages. Public opinion changes clustered around 5% for the multiple measures in the two studies.

Klein Teeselink & Melios (2021) reported changes in public opinion on a 5-point scale. Rainfall predicted changes of 0.242 and 0.339 on two different questions, which correspond to percentage changes of about 6% and 8.5%, although the interpretation of these percentages isn’t the same as for the other two studies.

I believe the data on voter behavior also provides evidence on public opinion—if you vote differently, it’s most likely because your opinion changed.

So I think Social Change Lab’s finding is indeed moderately well supported.

Protest movements can influence public discourse (e.g. issue salience and media narratives) Medium

None of the natural experiments directly addressed this claim.³³

Several observational studies found that protests frequently get media coverage. Even though the studies are all observational, I am comfortable inferring causality in this case—it seems odd to say that protests occurred, the news covered the protests, but the protests did not cause the news coverage.

Claims about individual studies

The literature review discussed five studies on real-world impacts of protests. Did it represent the studies accurately?

Social Change Lab discussed the observational component of Wasow (2020)¹¹ but not the quasi-experimental component.

By looking at US counties that are similar on a number of dimensions (black population, foreign-born population, whether the county is urban/rural, etc.), Wasow is able to mimic an experiment by testing how the Democratic vote share changes in counties with protests and matching counties without protests.

I don’t think it’s reasonable to say that a matched observational design “mimic[s] an experiment”. It could be that protests were more likely to happen in counties that were already shifting Democratic; you can’t prove that the protests caused the shift.

I agree with everything Social Change Lab wrote about Madestam (2013)¹⁰.

Regarding Klein Teeselink & Melios (2021)¹², the literature review wrote:

[A] one percentage point increase in the fraction of the population going out to protest increased the Democratic vote share in that county by 5.6 percentage points[.]

First, this figure is incorrect: it should be 3.3 percentage points (page 11). Klein Teeselink & Melios (2021) was revised in 2025 and I only have access to the latest revision, so it’s possible that Social Change Lab’s figure comes from the 2021 version.

Second, Klein Teeselink & Melios’ placebo tests show that the natural experiment failed to establish causality. Social Change Lab interprets the study’s outcome as causal, but I do not believe this interpretation is justified.

Social Change Lab’s description of McVeigh, Cunningham & Farrell (2014)⁵ was fair; it was appropriately cautious about the weakness of the paper’s evidence.

On Bremer et al. (2019)⁶:

They found that whilst no such relationship existed for all 30 countries, in Western Europe did [sic] find a statistically significant interaction between protest, levels of economic hardship in a country and the loss of votes for the incumbent party.

I’m suspicious of p-hacking when a study finds a non-significant main result and a significant sub-group result. I wish Social change Lab had been more skeptical of Bremer et al.’s approach.

It seems that for a given level of economic hardship a country faces, if the number of protests increase, the incumbent political party will lose more votes[.]

This quote implies causality, which was not established—the Bremer et al. study was purely observational.

In summary, Social Change Lab overstated the strength of evidence several times when reviewing particular studies. However, I believe their summary findings are still accurate, partially thanks to the two additional natural experiments (Larreboure & González (2021)¹³ and Hungerman & Moorthy (2023)⁷) that came out more recently.

Conclusion

My position on the Social Change Lab literature review:

The review was insufficiently skeptical about weak evidence, and too willing to attribute causality where it had not been established.
The review’s summary claims about the overall strength of evidence were consistent with my assessments. Perhaps the “Strong Confidence” findings were overconfident and should be “Medium Confidence” instead, but I can see arguments either way.

Conducting a meta-analysis changed my view on protest effectiveness. My previous stance was that protests probably work, and that various lines of evidence pointed that way, but that all available evidence was weak. I now believe that some of the evidence is relatively³⁴ strong, and I am more confident that protests work.

Source code

Source code for my meta-analysis is available on GitHub.

Appendix A: Additional tables

Most meta-analyses report study heterogeneity (I²). I reported P(negative effect) instead, which provides equivalent information, and I believe it’s more useful in this case. For completeness, Table A.1 gives the I² values for Table 3.

Table A.1: Pooled Outcomes with I²

Outcomes	Mean	Std Err	likelihood ratio	p-value	I^2
Vote Share Per Protester	11.95	4.00	87.1	0.003	3%
Vote Share	1.59	0.48	257	0.001	45%
Vote Share (Rain Only)	1.14	0.42	39.3	0.007	0%
Single Hypothesis	1.06	0.78	2.55	0.172	74%
Favorability	2.68	2.32	1.95	0.249	72%

Table A.2 reports summary statistics for the same pooled outcomes as Table 3, plus additional outcomes from Klein Teeselink & Melios (2021)¹² that I excluded from the main table. Consider this like a leave-one-out analysis, except instead it’s a put-one-in analysis.

“Vote Share Per Protester” adds BLM vote share per protester (mean 3.3, std err 0.6, n = 3053; from Table 2).
The middle three rows add BLM vote share.
“Favorability” adds survey agreement rate for the statement “Blacks should not receive special favors.”

Table A.2: Pooled Outcomes Including BLM

Outcomes	Mean	Std Err	likelihood ratio	p-value	I^2	P(negative effect)
Vote Share Per Protester	7.89	3.91	7.62	0.044	65%	0.072
Vote Share	1.71	0.43	2.84e+03	0.001	33%	0
Vote Share (Rain Only)	1.32	0.41	192	0.002	3%	0
Single Hypothesis	1.36	0.68	7.6	0.045	69%	0.123
Favorability	2.66	1.99	2.44	0.182	48%	0.136

Table A.3 uses the same row definitions as in Table A.2, while also correcting for publication bias by creating dummy null outcomes as described above.

Table A.3: Pooled Outcomes Including BLM, Adjusted for Publication Bias

Outcomes	Mean	Std Err	likelihood ratio	p-value
Vote Share Per Protester	2.79	1.55	5	0.073
Vote Share	0.88	0.38	14.9	0.021
Vote Share (Rain Only)	0.70	0.36	6.69	0.052
Favorability	0.66	0.51	2.26	0.203

The BLM study found a smaller mean effect than the other studies, but it also had a high t-stat. Adding BLM to the pooled outcomes decreases means but does not consistently decrease the strength of evidence.

As discussed previously, the BLM study isolates local effects of protests, which is undesirable—it ignores any non-local effects that protests might have. Luckily, the study also reports results with no adjustment for spatial autocorrelation (in its Table A3).

In general, it’s not a good idea to ignore spatial autocorrelation because it may overstate the strength of evidence. But for the “vote share per protester” metric, the un-adjusted outcome had a lower t-stat than the adjusted outcome. So I think it’s fair to add the un-adjusted result to the pooled sample.

Here are the results for pooled vote share per protester, using the BLM outcome with no adjustment to spatial autocorrelation.

Table A.4: Pooled Outcomes for Vote Share Per Protester, Including BLM

Outcomes	Mean	Std Err	likelihood ratio	p-value	P(negative effect)
no correction	11.89	2.32	4.83e5	4e-7	0
corrected for publication bias	6.23	3.05	8.1	0.041	0.138

My meta-analysis compared standardized outcomes (using the method described in Gelman (2007)²⁴). Table A.5 shows the results from pooling unstandardized outcomes instead (excluding BLM).

Table A.5: Unstandardized Pooled Outcomes

Outcomes	Mean	Std Err	likelihood ratio	p-value	P(negative effect)
Vote Share Per Protester	11.95	4.00	87.1	0.003	0
Vote Share	3.31	1.37	18.1	0.017	0.04
Vote Share (Rain Only)	1.94	1.02	6.13	0.057	0.011
Single Hypothesis	1.66	1.77	1.55	0.349	0.293
Favorability	5.20	1.84	53.8	0.005	0

Appendix B: Methodological revisions

In the interest of transparency—and because I didn’t pre-register a methodology—here is a list of non-trivial revisions I made in the process of writing this article. In chronological order:

Originally, I used a different method for estimating the effect of publication bias. Instead of creating dummy null clones as described above, I created k null dummies (one for each real outcome) that were all identical, and that took their standard error as the average of the real studies’ standard errors. This method produced lower p-values. However, I decided it was too unrealistic to give all three null dummies the exact same summary statistics.
For BLM and Women’s March results, I originally used the primary outcomes as reported by their respective papers. I revised my meta-analysis to use the weakest outcomes (lowest t-stat) from the robustness checks, to more conservatively account for spatial autocorrelation.
I originally included BLM outcomes in the meta-analysis. Upon re-reading the BLM paper, I realized it failed its placebo tests, so I removed it. Appendix A shows the results when including BLM.
I went back to using the main outcome for BLM instead of a robustness check outcome because it’s simpler, and I wasn’t including BLM in the main meta-analysis anyway. This slightly strengthened the reported results in Appendix A.
Originally, my meta-analysis used unstandardized means. I wanted to use standardized means but I wasn’t sure how to standardize them. Eventually, I found Gelman (2007)²⁴ and used its method for standardizing outcomes. This increased most results’ t-stats because it decreased between-study variance. (However, it decreased the Favorability t-stat.) Unstandardized pooled outcomes are reported in Table A.5.
- Initially, I wanted to scale all binary outcomes by their standard deviations, but I did not have the necessary data to calculate all standard deviations, so I simply left them unscaled.

Appendix C: Comparing the strength of evidence to saturated fat research

To get some perspective on the strength of evidence on protests, I would like to compare it to a thorny question in an unrelated field that I reviewed recently.

Last year I examined the evidence on whether saturated fat is unhealthy, primarily focusing on a Cochrane review³⁵ of RCTs. I ultimately decided I was 85% confident that saturated fat is unhealthy.

How does the evidence on protest effectiveness compare to the evidence on saturated fat?

Both hypotheses face similar problems: there are many observational studies that support the hypothesis, but few experiments. (In the case of protests, there are no (real-world) experiments, but there are some natural experiments.)

The Cochrane review included 15 RCTs. My review included five natural experiments.

The studies in the Cochrane review were true experiments. The protests studies were not true experiments, which means they can’t establish causality quite as firmly.

The Cochrane review found no evidence of publication bias (Figure 3). There aren’t enough protest quasi-experiments to test for publication bias, but I did test what would happen if I added in some hypothetical null-result studies.

Three individual studies on saturated fat had statistically significant positive effects, six had non-significant positive effects, and four had non-significant negative effects. (Two studies did not report data on cardiovascular events.)

In my review, three out of three studies (or four out of four³⁶) found statistically significant positive effects for nonviolent protests, and one study found statistically significant negative effects for violent protests.

When pooling all RCTs together, the Cochrane review found a marginally statistically significant effect of saturated fat reduction on cardiovascular events (95% CI [0.70, 0.98] where 1 = no effect). It also found significant effects on short-term health outcomes like weight and cholesterol. It found positive but non-significant results on mortality outcomes (including all-cause mortality and cardiovascular mortality).

In my meta-analysis, most of my pooled samples had strong positive results. My primary metric had p < 0.003; the Cochrane review didn’t find any primary results that strong.

Even though the Cochrane review included three times as many studies, I think the evidence on protest outcomes is stronger:

The saturated fat RCTs got mixed results, but the three or four studies on nonviolent protests all pointed the same direction.
(Most of) the pooled outcomes for protests had moderate to strong p-values. The pooled outcomes for saturated fat reduction had moderate p-values at best.

On the other hand, the protest studies have to deal with the spatial autocorrelation problem, and it’s not entirely clear that they succeeded at establishing causation. The saturated fat studies were experiments; they had no analogous problem.

The smaller number of studies also means the protests meta-analysis is more vulnerable to errors in any one study.

It’s a judgment call as to whether you think the weaker methodology for protest studies outweighs the stronger likelihood ratios. I’m inclined to say it doesn’t.

Notes

Ozden, J., & Glover, S. (2022). Literature Review: Protest Outcomes. ↩
Ozden, J., & Glover, S. (2022). Protest movements: How effective are they? ↩
Orazani, N., Tabri, N., Wohl, M. J. A., & Leidner, B. (2021). Social movement strategy (nonviolent vs. violent) and the garnering of third-party support: A meta-analysis. ↩ ↩² ↩³ ↩⁴
ChatGPT Deep Research was useful for finding and summarizing studies, but not for assessing their quality. When I asked ChatGPT to only include methodologically rigorous studies in the review, it didn’t appear to change which studies it included, it just rationalized why every study was rigorous. It said things like (paraphrasing) “we know this observational study’s findings are robust because it controlled for confounders” and “because it had a large sample size” (??). ↩
McVeigh, R., Cunningham, D., & Farrell, J. (2014). Political Polarization as a Social Movement Outcome. ↩ ↩² ↩³
Bremer, B., Hutter, S., & Kriesi, H. (2020). Dynamics of protest and electoral politics in the Great Recession. ↩ ↩² ↩³
Hungerman, D., & Moorthy, V. (2023). Every Day Is Earth Day: Evidence on the Long-Term Impact of Environmental Activism. doi: 10.1257/app.20210045 ↩ ↩² ↩³ ↩⁴
Mellon, J. (2024). Rain, rain, go away: 194 potential exclusion-restriction violations for studies using weather as an instrumental variable. ↩
Sarsons, H. (2015). Rainfall and conflict: A cautionary tale. ↩
Madestam, A., Shoag, D., Veuger, S., & Yanagizawa-Drott, D. (2013). Do Political Protests Matter? Evidence from the Tea Party Movement. ↩ ↩² ↩³
Wasow, O. (2020). Agenda Seeding: How 1960s Black Protests Moved Elites, Public Opinion and Voting.. ↩ ↩² ↩³ ↩⁴
Klein Teeselink, B., & Melios, G. (2021). Weather to Protest: The Effect of Black Lives Matter Protests on the 2020 Presidential Election. ↩ ↩² ↩³ ↩⁴
Larreboure, M., & Gonzalez, F. (2021). The Impact of the Women’s March on the U.S. House Election. ↩ ↩² ↩³ ↩⁴
Braemer et al. (2020) found a non-significant result and then did some subgroup analysis and got a significant result. I find that suspicious but I didn’t bother to look deeper because it’s an observational study anyway. ↩
The paper did not report the p-value, but it did report the standard error, so I calculated the p-value from that. ↩
Collins, W. J., & Margo, R. A. (2007). The Economic Aftermath of the 1960s Riots in American Cities: Evidence from Property Values. ↩
Beck, N., Gleditsch, K. S., & Beardsley, K. (2006). Space Is More than Geography: Using Spatial Econometrics in the Study of Political Economy. ↩
If it were true, we’d expect to see a similar phenomenon in the placebo tests of Madestam et al. (2013) and Hungerman & Moorthy (2023), but we don’t. ↩
Belloni, A., Chernozhukov, V., & Hansen, C. (2010). LASSO Methods for Gaussian Instrumental Variables Models. ↩
Conley, T. G. (1999). GMM estimation with cross sectional dependence.. ↩
I emailed the corresponding author to ask about this apparent discrepancy and did not receive a reply. ↩
Hungerman & Moorthy (2023) did provide enough information to estimate the change in Earth Day favorability per protester, by dividing change in favorability by change in number of protesters from the paper’s Table 4. However, this estimate would have high variance on the denominator, which makes the result unreliable. ↩
Favorability (1) measured public support for environmentalism as the percentage of respondents answering Yes to “we’re spending too little money” on protecting the environment. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶
Gelman, A. (2007). Scaling regression inputs by dividing by two standard deviations. ↩ ↩² ↩³
Some additional detail:

Gelman (2007) proposes scaling continuous variables by 2 standard deviations because this puts them onto the same scale as a binary variable where the control and treatment groups have the same size. If you have a sample of binary outcomes where 50% of the outcomes are 0 (“no rain”) and 50% are 1 (“rain”), then the standard deviation is 0.5. If the probabilities are not 50/50 then the standard deviation will not equal 0.5. (For example, the Tea Party rainfall variable had a standard deviation of 0.401.) Arguably it would make sense to scale all binary variables to a standard deviation of 0.5. However, I did not do this because I didn’t have the necessary data for all the papers. Instead, I left all binary variables unscaled. (Gelman (2007) discusses whether probability-skewed binary variables should be scaled, but ultimately does not take a stance.)

The Earth Day paper directly regressed outcomes onto a continuous rainfall variable (without doing a two-stage regression). I scaled the reported slopes by 2 times the standard deviation of rainfall.

The Women’s March paper reported values scaled to 1 standard deviation, so I divided them by 2.

The Tea Party, BLM, and Civil Rights papers reported effects in binary terms, so I did not scale them. ↩ ↩²
Even if I replace the relatively weak Earth Day favorability outcome with Earth Day favorability among under-20s (which had a likelihood ratio of 37), the pooled likelihood ratio is still only 3.43. ↩
Defined as the number of Congress members who voted in line with conservative positions according to the American Conservative Union. ↩
Defined as alignment with Tea Party positions, measured in standard deviations. ↩
Evaluated on a 5-point scale from “strongly disagree” to “strongly agree”. ↩ ↩²
The linked webinar is the least-bad explanation of Egger’s test that I could find, but it doesn’t explain it very well so I will attempt to explain:

In the presence of publication bias, more powerful studies will have lower means. the small low-mean studies don’t get published. Therefore there will be a negative correlation between a study’s power and its mean. (I measured power as the inverse standard error but you could also use the inverse variance or the sample size.)

So you test for publication bias by doing a linear regression of study means on study power.
- If the regression has a correlation close to zero, that indicates no publication bias.
- If there is a significant correlation, that’s evidence of publication bias.
Lest this footnote give the impression that I know what I’m talking about, I didn’t even know what Egger’s regression test was until I wrote this. My process was that I asked Claude what statistical test I could use to check for publication bias, it suggested Egger’s test and then gave an obviously-incorrect explanation of how the test works, and then I read several barely-comprehensible articles about the test until I thought I understood it. ↩
I could do full replications for the papers that published their data, but that would be considerably more work for a low chance of paying off. ↩
In my experience, I always update tables if I make revisions to my calculations, but it’s hard to keep track of everywhere in the text that I referenced a number. ↩
Technically, one of them did look at media coverage, but not using the rainfall method. ↩
I’m thinking about the strength of evidence from a sociology perspective. Getting good evidence in sociology is hard. ↩
Hooper, L., Martin, N., Jimoh, O. F., Kirk, C., Foster, E., & Abdelhamid, A. S. (2020). Reduction in saturated fat intake for cardiovascular disease. ↩
Three if you exclude the BLM study due to its failed placebo tests; four if you include it. ↩

The Triple-Interaction-Effects Argument

Thu, 10 Apr 2025 00:00:00 -0700

In this post I will explain the most impressive argument I heard in 2024.

First, some context:

There is an ongoing debate in the bodybuilding/strength training community about how much protein you should eat while losing weight.

Some say you should eat more protein if you’re losing weight:

If you’re eating less, your body is under extra pressure to cannibalize your muscles. Therefore, you should eat more protein to cancel this out.

The standard rebuttal:

Experimental trials have found that muscle gains max out when subjects eat 0.7–0.8 grams of protein per pound of bodyweight, and that’s true both when participants are maintaining weight and when they’re losing weight. There doesn’t appear to be a difference.

And the counter-rebuttal:

Almost all research looks at novice lifters. Experienced athletes have a more difficult time gaining muscle,¹ so losing weight will have a bigger negative impact on them, and therefore they need to eat more protein.

I used to believe this. Then I heard the most impressive argument of 2024.

I heard the argument in a YouTube video by Menno Henselmans:

It’s possible that in trained individuals there is a triple interaction effect, because that’s what you’re arguing here. If you’re saying that protein requirements increase in an energy deficit, but only in strength-trained individuals, then you are arguing for a triple interaction effect. […] That is very, very, very rare. Triple interaction effects, biologically speaking, simply do not occur much.

I didn’t understand what he was talking about. I spent two days pondering what it meant. On the third day, it finally clicked and I realized he was right.

To claim that trained lifters should eat more protein on an energy deficit, you’d need to believe that:

Above a certain level of protein intake (0.7–0.8 grams per pound), additional protein has no effect on muscle growth.²
Most of the time, trained athletes don’t need more protein than novices.
Novices don’t need more protein while losing weight than while maintaining/gaining weight.
HOWEVER, (a) among trained individuals who are (b) losing weight, the ones (c) who eat more protein (beyond 0.7–0.8 g/lb) gain more muscle.

The first variable (protein intake) has no interaction with muscle growth.

The second variable (trained vs. untrained) has no interaction with muscle growth.

The third variable (losing vs. maintaining weight) has no interaction with muscle growth.

The first and second variables together (protein intake + trained/untrained) have no interaction with muscle growth.

The first and third variables together (protein intake + losing/maintaining weight) have no interaction with muscle growth.

HOWEVER, when you put all three variables together, an interaction suddenly appears—a triple interaction effect.

This is a very strange claim. If all three variables together affect muscle growth, then you would expect each variable individually to affect muscle growth. And at least you would expect two out of three variables together to affect muscle growth.

(In fact, it is mathematically impossible to construct a differentiable function f(x, y, z) that is constant with respect to x, constant with respect to y, and constant with respect to z, but not constant overall. Although you could have a function f(x, y, z) where the slope with respect to each individual variable is close to 0, but not quite 0.)

Not to say a triple interaction effect can’t occur in the real world. It could be that muscle growth does depend on each of (protein intake, training experience, calorie deficit), but the relationships are so weak that the studies failed to pick them up.

But if you believe the studies’ results are correct, then it seems difficult—maybe even impossible—to still believe that trained lifters need to eat more protein while on a calorie deficit.

***

This was the best argument I heard in 2024 because:

If you think about it, it’s obviously correct. It changed my mind as soon as I understood it.
It’s difficult to come up with. (I’ve never heard anyone else make this argument.)

Notes

I’m conflating gaining strength with putting on muscle. There’s a difference, but we can consider them the same thing for the purposes of this post. ↩
This claim is somewhat controversial, but let’s assume it’s true for the sake of this argument.

Randomized controlled trials find no benefit to more than ~0.7 g/lb, and I quoted a range of 0.7–0.8 g/lb to account for variation between individuals. But the existing studies aren’t that great so I don’t have high confidence that that’s the correct range. ↩

You can now read my reading notes

Mon, 31 Mar 2025 00:00:00 -0700

Since 2015, I have been taking notes on most articles I read. I figured other people might find them useful, so I cleaned them up and published them on my website. You can find them via the new “Notes” tab.

I will update the page every once in a while as I read more articles and take more notes.

I also have notes on every educational book I’ve read since 2015, but the notes are on physical paper (can you believe it?). I might digitize them at some point.

There Are Three Kinds of "No Evidence"

Mon, 03 Mar 2025 00:00:00 -0800

David J. Balan once proposed that there are two kinds of “no evidence”:

There have been lots of studies directly on this point which came back with the result that the hypothesis is false.
There is no evidence because there are few or no relevant studies.

I propose that there are three kinds of “no evidence”:

The hypothesis has never been studied.
There are studies, the studies failed to find supporting evidence, but they wouldn’t have found supporting evidence even if the hypothesis were true.
There are studies, the studies should have found supporting evidence if the hypothesis were true, and they didn’t.

Example of type 1: A 2003 literature review found that there were no studies¹ showing that parachutes could prevent injury when jumping out of a plane.

Example of type 2: In 2018, there was finally a randomized controlled trial² on the effectiveness of parachutes, and it found no difference between the parachute group and the control group. However, participants only jumped from a height of 0.6 meters (~2 feet). I don’t know about you, but this result does not make me want to jump out of a plane without a parachute.

Like in the parachute example, you see type-2 “no evidence” whenever the conditions of a study don’t match the real-world environment. You also see type-2 “no evidence” when an experiment is underpowered. Say you want to test the hypothesis that boys are taller than girls. So you go find your niece Sally and your neighbor’s son James and it turns out Sally is an inch taller than James. Your methodology was valid—you can indeed test the hypothesis by finding some people and measuring their heights—but your sample size was too small.

(The difference between type 2 and type 3 can be a matter of degree. The more powerful a study is, the stronger its “no evidence” if it fails to find an effect.)

Notes

Smith, G. C. S. (2003). Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomised controlled trials. ↩
Yeh, R. W., Valsdottir, L. R., Yeh, M. W., Shen, C., Kramer, D. B., Strom, J. B., Secemsky, E. A. et al. (2018). Parachute use to prevent death and major trauma when jumping from aircraft: randomized controlled trial. ↩

Return Stacking Funds: A New Way to Get Leverage

Tue, 04 Feb 2025 00:00:00 -0800

Last updated 2026-02-04.

Some people (including me) have argued that altruists often benefit from leveraging their investments. Recently, it has become easier to use leverage thanks to the emergence of return stacking funds.

This is not financial advice.

Contents
What is return stacking?
An overview of return stacking funds
The true cost of return stacking ETFs
- 2026 update
Pros and cons of return stacking funds
Are bonds a good investment?
Source code
Acknowledgments
Notes

What is return stacking?

Return stacking is a way of getting up to leveraged exposure to multiple return streams simultaneously. For example, RSSB invests 100% into global equities and 100% into US Treasury bonds, effectively giving it 2:1 leverage on a diversified stock/bond portfolio.

A return stacking ETF is a type of leveraged ETF. But whereas traditional leveraged ETFs (such as SPXL) lever up a single index like the S&P 500, a return stacking fund holds multiple asset classes.

Return stacking ETFs have lower management fees than single-index leveraged ETFs, and (with low confidence) they appear to have lower overhead costs for reasons that are not entirely clear to me (my guess is a combination of cheaper borrowing costs + transaction costs).

An overview of return stacking funds

There are four brands of return stacking funds that I know of:

the eponymous Return Stacked ETFs (RSSB, RSST, RSBT, RSSY, RSBY, RSBA, BTGD, RDMIX)
WisdomTree Capital Efficient ETFs (NTSX, NTSI, NTSE, GDE, GDMN)
PIMCO StocksPLUS Funds (PSLDX, SPLS)
Evoke Advisors Ultra Risk Parity ETF (UPAR)

The PIMCO fund has been around since 2007, but the others only launched within the last few years.

What do each of these funds invest in?

Seven of the funds stack traditional asset classes:

Fund	first asset class	second asset class	leverage
RSSB	100% global stocks	100% US Treasury bonds	2:1
NTSX	90% US stocks	60% US Treasury bonds	1.5:1
NTSI	90% international stocks	60% US Treasury bonds	1.5:1
NTSE	90% emerging market stocks	60% US Treasury bonds	1.5:1
UPAR	too many for this table*		1.68:1
PSLDX	~100% US stocks**	~100% bonds**	~2:1**
SPLS	~100% US stocks**	~100% bonds**	~2:1**

*As of 2025, UPAR targets 17.5% U.S. equities, 7% international equities, 10.5% emerging markets equities, 21% commodity producer equities, 14% gold, 49% TIPS, and 49% Treasuries for a total allocation of 168%.

**PSLDX and SPLS percentages are only approximate because the funds are actively managed and their holdings may vary over time.

These funds stack traditional asset classes with alternatives:

Fund	first asset class	second asset class	leverage
RSST	100% US stocks	100% managed futures*	2:1*
RSBT	100% US bonds¹	100% managed futures*	2:1*
RSSY	100% US stocks	100% futures yield*	2:1*
RSBY	100% US bonds	100% futures yield*	2:1*
RSBA	100% US Treasury bonds	100% merger arbitrage*	2:1*
RDMIX	50/50 US stocks/bonds	100% systematic macro*	2:1*
BTGD	100% bitcoin	100% gold	2:1
GDE	90% US stocks	90% gold	1.8:1
GDMN	90% gold miner stocks	90% gold	1.8:1

*Managed futures (a.k.a. trendfollowing), futures yield (a.k.a. carry or roll yield), merger arbitrage, and systematic macro are all long/short strategies, not simple assets that you can buy and hold. So it’s somewhat arbitrary to say that the funds invest 100% into those strategies.

The true cost of return stacking ETFs

In a previous post, I looked at how a leveraged index fund should perform and compared that against how leveraged ETFs actually did perform. I found that the ETFs consistently cost more than expected, by an average of about one percentage point.

I attempted to do the same analysis for return stacking ETFs. These ETFs are harder to replicate because they don’t track indexes, so I don’t have high confidence in the results. That said, my numbers suggest that return stacking ETFs are more cost-effective than conventional leveraged ETFs.

I was able to replicate RSSB and NTSX:

ETF	Leverage	Stock ETF(s)	Bond Fund(s)
RSSB	100% + 100%	VTI + VXUS²	bond futures ladder³
NTSX	90% + 60%	SPY (S&P 500)	bond futures ladder⁴

I also attempted to replicate NTSI, PSLDX, and GDE, but I couldn’t find benchmarks that tracked them well enough.

Update 2026-01-17: SPLS is a newly launched 100% US stocks + 100% bonds ETF. It just launched as of this writing, so it has no history to replicate. It’s now the fund with the lowest expense ratio, but it holds swaps on PIMCO bond ETFs that have their own expense ratios (as of this writing, SPLS holds swaps on BOND at 0.40% and PYLD at 0.55%, among others). Different funds may also have different financing rates on the instruments they use to get leverage.

I calculated excess costs of RSSB and NTSX as the hypothetical return you’d get if you levered up the benchmark (borrowing at the 3-month T-bill rate), minus the actual historical return of the fund.

The following table shows the total excess cost and after-fee cost for the return stacking ETFs. Excess cost is shown per 100% leverage (the excess on NTSX is multiplied by two because it only has 50% leverage). After-fee cost gives the excess cost minus the difference expense ratios between the ETF and the benchmark—this represents the “unexpected” portion of the cost, since you expect to pay the expense ratio no matter what. r gives the correlation between the return stacking ETF and the benchmark. I calculated the average annual cost for each ETF starting from the earliest year for which the ETF had a full year of data.

ETF	Excess Cost	After Fee	r	Start Year
RSSB	-0.55%	-0.84%	0.998	2024
NTSX	-0.17%	-0.41%	0.997	2019

(The costs were negative, which means the real-life funds outperformed the benchmarks.)

Excess costs for each individual year for NTSX:

	2019	2020	2021	2022	2023	2024
NTSX	-0.48	-3.50	2.72	1.92	-0.93	-2.15

As we can see, excess costs varied quite a bit from year to year. However, they were still generally lower than the costs of conventional leveraged ETFs.

In fact, the excess costs were negative most years. That’s surprising, since the benchmark does not account for transaction costs.

Why were return stacking funds (apparently) more cost-effective than conventional leveraged ETFs?

These funds have lower expense ratios. For example, RSSB charges 0.36% and SSO (a 2x leveraged S&P 500 fund) charges 0.89%.
Traditional leveraged ETFs rebalance daily. The Return Stacked and WisdomTree ETFs only rebalance if the holdings drift 5 percentage points away from the target weights. Rebalancing has transaction costs, which could be significant or could be close to zero, depending on various factors.
Return stacking funds get leverage via Treasury futures, which is approximately the cheapest way to get leverage. Conventional leveraged ETFs primarily use swaps, which have an opaque pricing structure and might cost a lot more. (I have no idea how much they actually cost because the pricing is opaque.)

Those factors explain why return stacking ETFs are cheaper than 3x leveraged index ETFs. But how is it possible for a return stacking ETF to outperform a leveraged combination of index funds?

My benchmarks have some margin of error—they do not perfectly track the return stacking ETFs. Based on playing around with the implementation details of the benchmark, I believe it could be off by perhaps one percentage point.⁵

The most likely source of tracking error is rebalance timing. Small changes in when you rebalance can significantly change year-to-year performance, especially in years like 2024 where some asset classes perform much better than others. If stocks outpaced bonds for most of the year, and the fund was supposed to rebalance from stocks to bonds, then the real-life fund might have gained an edge over the benchmark by delaying rebalancing a little longer.

Even if these (apparently) negative costs might not persist, this still provides evidence that the return stacking ETFs have lower costs than single-asset leveraged ETFs.

2026 update

The first version of this post, published in February 2025, only included a year of history for RSSB. I’m writing this update in January 2026, and RSSB has now existed for twice as long. Has it maintained its low cost?

Yes. Here’s the excess cost of RSSB from inception (2023-12-05) to yesterday (2026-01-15):⁶

	Excess Cost	After Fee
RSSB	-0.58%	-0.87%

As was the case in 2024, RSSB has a negative excess cost, i.e., it was cheaper than its benchmark.

The excess costs by year:

	2024	2025
RSSB	-0.75	-0.61

NTSX also had good operations in 2025, with an excess cost of –0.50%.

Pros and cons of return stacking funds

Pros:

They’re a convenient way to get leverage, much more convenient than options or futures.
They appear to have lower all-in costs than conventional leveraged ETFs.

Cons:

None of them offer greater than a 100% allocation to equities.
Limited choices—there are only a handful of return stacking ETFs available, and they might not include the asset classes you want.
- I personally would like to see a global stocks + managed futures ETF, but that doesn’t exist. There’s only US stocks + managed futures (RSST) and bonds + managed futures (RSBT).
As with other leveraged ETFs, the costs of return stacking ETFs fluctuate from year to year. Even though the costs are low on average, in any given year a return stacking ETF might perform worse than expected.
I could only determine the costs for two of the return stacking ETFs. The others might have higher costs.

Are bonds a good investment?

Most of the return stacking funds hold bonds. A question that some people ask:

Does it make sense to own return stacking stocks + bonds? Wouldn’t I rather have pure leveraged stocks instead?

Good question! I don’t know!

An argument against buying bonds:

Right now, the yield curve is nearly flat: yields on long-term bonds are only slightly higher than on short-term bonds. Why would you borrow at the short-term rate to earn the long-term rate if those rates are (nearly) the same?

Two counter-arguments:

The efficient market hypothesis predicts that you can’t time the bond market, so you shouldn’t change how you invest based on what the yield curve looks like.

A flat or inverted yield curve suggests that short-term rates will go down in the future. You might want to “lock in” the current rate by buying long-term bonds.

(Really these counter-arguments are the same—the (presumed) reason why the yield curve is flat is because the market is pricing in future changes in bond yields.)

Another argument against bonds:

In the long run, bonds have only earned a little bit of a premium over short-term T-bills. Given the overhead costs of using leverage, leveraged bonds might have near-zero or even negative expected return.

And two counter-arguments:

If you can borrow at close to the risk-free rate, bonds should still have a positive long-run premium.

Even if leveraged bonds have ~zero expected return, they still add value to a portfolio if they perform well during equity downturns.

Which side of the argument is correct is left as an exercise to the reader.

If you don’t want to hold bonds, there are some bondless return stacked stacking available:

RSST holds stocks + managed futures (a.k.a. trendfollowing).
RSSY holds stocks + futures yield (a.k.a. carry or roll yield).
BTGD holds bitcoin + gold.
GDE holds US stocks + gold.
GDMN holds gold miner stocks + gold.

I am a big fan of managed futures trendfollowing—it’s a strategy with strong historical performance that provided protection during market downturns, and I think it’s likely to continue working in the future (for more, see Hurst et al. (2017), “A Century of Evidence on Trend-Following Investing”⁷). I’m ambivalent about carry (I’ve heard good arguments both for and against using it). I personally wouldn’t invest in bitcoin or gold, but if that’s your thing, return stacking ETFs give you a way to do it.

(I don’t own RSST, but I hold something similar in my own portfolio—equities (AAVM) with managed futures stacked on top.)

Source code

Source code is available on GitHub.

Acknowledgments

Thanks to Corey Hoffstein for helping me work out the implementation details of my benchmark.⁸

Notes

“bonds” means any sort of bonds, including Treasury or corporate bonds. “Treasury bonds” means just Treasury bonds. ↩
I weighted VTI at 62% and VXUS at 38% as of the beginning of 2024 because those are the weightings I get if I reverse-engineer from RSSB’s current weightings.

It would be simpler to use VT which includes all the same stocks as VTI + VXUS. But RSSB itself holds VTI + VXUS, and I found that breaking out equities into two separate ETFs produces slightly more accurate benchmark. ↩
RSSB gets exposure to bonds via an equal-weighted combination of bond futures at four maturities: 2-year, 5-year, 10-year, and long (i.e. 25- to 30-year). I replicated this using:
- 25% S&P 2-Year U.S. Treasury Note Futures Total Return Index
- 25% S&P 5-Year U.S. Treasury Note Futures Total Return Index
- 25% S&P 10-Year U.S. Treasury Note Futures Total Return Index
- 25% S&P Ultra T-Bond Futures Total Return Index
These indexes should exactly match the bond futures that RSSB holds. ↩
I wasn’t entirely sure what position to use to replicate NTSX’s bond holdings. Its materials include illustrative figures that use a 7-10 year Treasury index as a benchmark, which suggests I should use IEF or perhaps 10-year Treasury futures. But the latest holdings show that it uses a combination of bond futures of different durations.

I found the best correlation to NTSX when using a weighted combination of four Treasury futures:
- 12% S&P 2-Year U.S. Treasury Note Futures Total Return Index
- 12% S&P 5-Year U.S. Treasury Note Futures Total Return Index
- 24% S&P 10-Year U.S. Treasury Note Futures Total Return Index
- 12% S&P Ultra T-Bond Futures Total Return Index
As of this writing, NTSX holds 12% in 10-Year U.S. Treasury Note Futures and 12% in Ultra 10-Year U.S. Treasury Note Futures (which are like the normal 10-year futures except that they are more closely tied to a 10-year maturity). I did not use Ultra futures in my benchmark because they only launched a few years ago. ↩
Some minor changes that affect the return of the benchmark:
- The funds rebalance whenever weights drift 5% away from the target. But the prospectuses for RSSB and NTSX were not clear about what exactly that meant—I can think of at least four different interpretations. Corey Hoffstein (who co-runs RSSB) explained to me exactly how the rebalancing works, and I assumed NTSX works the same way but I don’t know for sure. Different rebalancing methods can change the average return by as much as one percentage point—a fund might get lucky and rebalance into a position right before it rockets up, or the opposite might happen.
- Changing how the benchmark invests in bonds can change the return. GOVT outperformed the Treasury futures ladder over the sample period. Changing the NTSX benchmark to use GOVT increased its return by 24 bps (but also decreased the correlation to NTSX from 0.997 to 0.993).
- My program might have a bug. Shortly before posting this article, I discovered that I was incorrectly calculating how much cash the benchmark needed to borrow and thus overestimating interest payments by about 40 bps per year.
↩
My original analysis used NAV, but for these tables I switched to using the ETF’s market value instead. NAV has a 10x tighter correlation to my benchmark (0.997 vs. 0.96), but my NAV data is rounded to two significant figures and my price data has many significant figures, so the NAV return is less accurate. ↩
Hurst, B., Ooi, Y. H., & Pedersen, L. H. (2017). A Century of Evidence on Trend-Following Investing. ↩
Originally I couldn’t figure out how to get my benchmark’s correlation to RSSB higher than 0.976. Turns out I needed to compare the benchmark to RSSB’s NAV, not its daily closing price. Corey explained to me that NAV and price diverge because daily futures prices settle at 3pm but exchanges close at 4pm, so any market movements in that hour will show up in RSSB’s price but not in its NAV or in the benchmark.

I had the same problem with my NTSX benchmark and was able to fix it the same way.

I also originally implemented rebalancing using an incorrect method, and Corey clarified the correct method to use. (This did not improve the correlation.) ↩

I was probably wrong about HIIT and VO2max

Mon, 03 Feb 2025 00:00:00 -0800

This research piece is not as rigorous or polished as usual. I wrote it quickly in a stream-of-consciousness style, which means it’s more reflective of my actual reasoning process.

My understanding of HIIT (high-intensity interval training) as of a week ago:

VO2max is the best fitness indicator for predicting health and longevity.
HIIT, especially long-duration intervals (4+ minutes), is the best way to improve VO2max.
Intervals should be done at the maximum sustainable intensity.

I now believe those are all probably wrong.

I think I got the wrong idea because a lot of HIIT/VO2max promoters cite scientific studies, which makes them seem superficially reasonable, but the studies they cite aren’t very good, or aren’t interpreted correctly.

A few months ago I started incorporating some HIIT into my cardio routine. But I didn’t really know the best way to do it, so last week I decided to do some research. I looked up the most-cited meta-analyses on Google Scholar and I noticed my confusion when the meta-analyses didn’t seem to support the conventional wisdom that HIIT is the best way to improve VO2max.

The most comprehensive single source I found was a meta-meta-analysis by Crowley et al. (2022)¹, which reviewed the findings of meta-analyses on HIT (high-intensity training) vs LIT (low-intensity training) for VO2max. The key quote:

Evidence from the meta-analyses that directly compared LIT versus HIT protocols on VO2max was, ostensibly, reported as either trivial or inconclusive. Three out of the six included meta-analyses reported small/moderate beneficial effects of HIT over LIT (α < 0.05). However, two of these reviews reported “substantial” heterogeneity (I2>0.75), small-study bias (p < 0.10), a relatively small pooled sample size (i.e., <1,000 participants), had a high degree of overlap (CCA = 11%) and reported several moderators (e.g., baseline fitness levels, age, HIT variables [e.g., volume, frequency, and duration]), which likely affected results.

Also, in my naiveté I had assumed that these were meta-analyses of RCTs, but in fact most of the included studies weren’t even RCTs:

Scribbans et al. reported that none of their included studies applied RCTs, Sloth et al. reported only four studies that applied RCTs design, and Gist et al. reported that the majority of included studies were RCTs.

(Note: Gist et al.², which did look mainly at RCTs, found that sprint interval training did not work better than endurance training (Cohen’s d = 0.04, 95% CI = -0.17 to 0.24.)

So I thought, okay, these meta-analyses don’t seem to favor HIIT much if at all. But maybe they’re done by stuffy academics who don’t know anything about real training. Are these meta-analyses considered respectable? So I went to see what the Barbell Medicine³ guys thought. I have a lot of respect for them when it comes to strength training. I don’t know if they know about cardio, but one of them is a former competitive swimmer so probably they know something. And they have good epistemics on strength training, and good epistemics might generalize. They did a podcast⁴ on HIIT with some useful content:

They started the podcast by criticizing a tweet⁵ in which fitness influencer Rhonda Patrick recommended a collection of “evidence-based HIIT protocols”. Their criticism mainly focused on how (they claimed) the provided HIIT protocols were way too hard. They quoted two responses⁶⁷ by exercise physiologists arguing the same.
They went on to talk about some of the research on HIIT, citing the same meta-analyses that I’d looked at.
Their ultimate recommendation (also given in an article⁸): “it is reasonable for about 80% of training to be of moderate intensity (zones 1-2), and about 20% reserved for higher intensity work (HIIT or SIT)”. They also said it’s good to use a variety of HIIT protocols and that there is no single optimal protocol.

I didn’t know if the Barbell Medicine guys were right about any of that, but it gave me some direction.

I had a look at the website of one of the people from that Twitter thread, Steve Magness⁹. He ran a 4:01 mile in high school so he probably has some idea of what he’s talking about.

Now, when some science-literate fitness influencers like Peter Attia and Rhonda Patrick give some recommendations about HIIT, and some other people like Steve Magness and Barbell Medicine disagree with them, I don’t have sufficient expertise to say who’s right. Both sides have the trappings of scientific credibility (e.g. citing multiple studies). But one thing I can do is check their logic.

So I checked Steve Magness’s logic. He wrote in a Twitter thread:¹⁰

You need all intensities to max VO2max. So it’s dumb to pit one vs. other

But research shows continuous likely matches HIIT for Vo2max increase

HIIT appears better when you constrain to 8 weeks but when you look over longer time it equalizes

Here’s data from a recent review.¹¹

You can see HIIT appears to increase VO2max more because of the time frame of most training studies (6-8 weeks).

Intense work gets big boost, then levels off. Endurance work gives Longer more gradual boost.

The referenced review does seem to support Magness’s argument, but I don’t know if the review is any good. What I do know is that Magness’s logic makes sense. It stands to reason that a more intense exercise protocol will cause faster short-term gains, but it can’t keep producing those rapid gains forever. And it stands to reason that if most studies on HIIT vs. LIT only last 6-12 weeks, then they will underestimate the long-term benefits of LIT.

That makes logical sense to me, which makes me think Steve Magness knows what he’s talking about.

He also wrote an article¹² arguing that some people care too much about VO2max for longevity:

Vo2max matters. But it’s just one component of many that make up both performance and aerobic fitness. And that’s important because if we return to the original claims that Vo2max is the key indicator of longevity, we’ll find that the majority of the studies cited did NOT even use Vo2max as the main variable. They used performance! In the majority of research, peak speed and incline during the exhausting test was the main correlate to longevity.

The large study¹³ on 750,000 veterans that found a 4-fold higher mortality risk for low versus high fitness used peak speed and incline, not Vo2max. Same with the research¹⁴ on 120,000 individuals finding a 5x difference in the risk of early death.

That makes logical sense to me. VO2max is only one aspect of fitness (albeit an important one), and it stands to reason that your actual ability to perform physical tasks is a better measure of physical health.

I did also look at some of the evidence cited in that article, specifically the Harber et al. (2017)¹⁵ meta-analysis, and Table 2 confirms Magness’s claim—most studies measured speed, time, or total work performed, not VO2max directly.

Insofar as I can verify Steve Magness’s claims, they seem to be correct. He also claims that HIIT protocols should not be “all-out”—for example, 60-second running intervals should be done between a 5K and a one-mile pace.¹⁶ He doesn’t cite any research on that claim, and as far as I know, there isn’t really research on it, it’s just how most high-performing athletes train. But since he seems right about the verifiable claims he’s made, I expect he’s right about that, too.

On the subject of checking people’s logic, here is an (admittedly cherry-picked) quote from the other side of the argument:

[I]s anything done in a total of 10-minutes that big of a deal?

If anyone is misinterpreting my statement as prescriptive:

My underlying point was that anything you can do in 10-minutes is limited on a relative harm basis, even if you do a lot.

This does not make logical sense to me. You can absolutely hurt yourself in less than 10 minutes. Even putting injury risk aside,

I’ve done 10-minute hill sprint intervals in the morning that left me feeling tired all day.
A 10-rep max squat takes less than 60 seconds, but it makes my legs sore for the next two days.

(Maybe that’s less logic and more personal experience, but a single example is enough to disprove a universal claim.)

I think Barbell Medicine has good logic, too. In their podcast on HIIT, they talk about how in strength training, nobody lifts the maximum possible weight every week. (And I have enough personal experience to know that maxing out every week wouldn’t work.) So it probably doesn’t make sense to max out your aerobic capacity in every week, either. I’m not sure strength training and cardio work the same way in that respect, but I expect things to be the same unless I have reason to believe they’re different.

The Barbell Medicine article on HIIT⁸ has some nice sample workouts that line up with Magness’s recommendations:

Workout #1

4 to 6 rounds of: 30 seconds on at 600-800 m running pace (or a speed sustainable in the range of ~90-150 seconds), 4 min off / easy effort

Workout #2

8 to 10 rounds of: 1-minute on at 1 mile-5 km running pace (or a speed sustainable in the range of 6-25 minutes), 1 minute off

Workout #3

3 to 5 rounds of: 5 minutes on at zone 4 heart rate (85-95% max), 3 min rest

I want to investigate this further, but here’s what I tentatively believe:

VO2max predicts longevity, but athletic performance matters more than VO2max alone.
I should exercise at a variety of intensities to get a well-rounded fitness capacity, but HIIT isn’t particularly better for improving fitness than LIT, and 4-minute intervals aren’t particularly better than other interval schemes.
Intervals should not be done at the maximum sustainable intensity; they should be done at an intensity that’s challenging but doesn’t leave you wiped out. As Magness wrote, “The goal isn’t to create fatigue, that’s easy to do. The goal is to slightly embarrass your body in the right direction.”

Coming back to my own training: I have loathed every version of HIIT I’ve tried so far. But that’s because I listened to the people saying that HIIT should be “all-out.” Next time I’m going to do HIIT at a more comfortable pace.

Notes

Crowley, E., Powell, C., Carson, B. P., & W. Davies, R. (2022). The Effect of Exercise Training Intensity on VO2max in Healthy Adults: An Overview of Systematic Reviews and Meta-Analyses. ↩
Gist, N. H., Fedewa, M. V., Dishman, R. K., & Cureton, K. J. (2013). Sprint Interval Training Effects on Aerobic Capacity: A Systematic Review and Meta-Analysis. ↩
https://www.barbellmedicine.com/ ↩
https://redcircle.com/shows/0cc66fc4-ccb8-4c60-8cc6-7367e52c4159/episodes/706bc687-0a98-4057-8e2e-e4db349bba4a ↩
https://x.com/foundmyfitness/status/1844811732080021919 ↩
https://x.com/StephenSeiler/status/1845357464130031873 ↩
https://x.com/stevemagness/status/1845079291320525202 ↩
https://www.barbellmedicine.com/blog/hiit-high-intensity-interval-training/ ↩ ↩²
https://www.stevemagness.com/about/ ↩
https://x.com/stevemagness/status/1849918347086795151 ↩
Mølmen, K. S., Almquist, N. W., & Skattebo, Ø. (2024). Effects of Exercise Training on Mitochondrial and Capillary Growth in Human Skeletal Muscle: A Systematic Review and Meta-Regression. doi: 10.1007/s40279-024-02120-2 ↩
https://thegrowtheq.com/longevity-and-vo2max-does-it-matter/ ↩
Kokkinos, P., Faselis, C., Samuel, I. B. H., Pittaras, A., Doumas, M., Murphy, R., Heimall, M. S. et al. (2022). Cardiorespiratory Fitness and Mortality Risk Across the Spectra of Age, Race, and Sex. ↩
Mandsager, K., Harb, S., Cremer, P., Phelan, D., Nissen, S. E., & Jaber, W. (2018). Association of Cardiorespiratory Fitness With Long-term Mortality Among Adults Undergoing Exercise Treadmill Testing. ↩
Harber, M. P., Kaminsky, L. A., Arena, R., Blair, S. N., Franklin, B. A., Myers, J., & Ross, R. (2017). Impact of Cardiorespiratory Fitness on All-Cause and Disease-Specific Mortality: Advances Since 2009. doi: 10.1016/j.pcad.2017.03.001 ↩
https://x.com/stevemagness/status/1845079292784365803 ↩

Retroactive If-Then Commitments

Sat, 01 Feb 2025 00:00:00 -0800

An if-then commitment is a framework for responding to AI risk: “If an AI model has capability X, then AI development/deployment must be halted until mitigations Y are put in place.”

As an extension of this approach, we should consider retroactive if-then commitments. We should behave as if we wrote if-then commitments a few years ago, and we should commit to implementing whatever mitigations we would have committed to back then.

Imagine how an if-then commitment might have been written in 2020:

Pause AI development and figure out mitigations if:

AI exhibits what looks like deceptive or misaligned behavior, or feigns alignment (1, 1b, 2)

AI breaks out of containment in a toy example

AI finds a real-world zero-day vulnerability

AI qualifies for Mensa¹

AI exhibits some degree of agentic capabilities

AI writes malware

Well, AI models have now done or nearly-done all of those things.

We don’t know what mitigations are appropriate, so AI companies should pause development until (at a minimum) AI safety researchers agree on what mitigations are warranted, and those mitigations are then fully implemented.

(You could argue about whether AI really hit those capability milestones, but that doesn’t particularly matter. You need to pause and/or restrict development of an AI system when it looks potentially dangerous, not definitely dangerous.)

Notes

Okay, technically it did not score well enough to qualify, but it scored well enough that there was some ambiguity about whether it qualified, which is only a little bit less concerning. ↩

The 7 Best High-Protein Breakfast Cereals

Fri, 17 Jan 2025 00:00:00 -0800

Updated 2025-03-19 to add Catalina Crunch Cinnamon Toast.

(I write listicles now)

(there are only 7 eligible high-protein breakfast cereals, so the ones at the bottom are still technically among the 7 best even though they’re not good)

If you search the internet, you can find rankings of the best “high-protein” breakfast cereals. But most of the entries on those lists don’t even have that much protein. I don’t like that, so I made my own list.

This is my ranking of genuinely high-protein breakfast cereals, which I define as containing at least 25% calories from protein.

Many food products like to advertise how many grams of protein they have per serving. That number doesn’t matter because it depends on how big a serving is. Hypothetically, if a food had 6g protein per serving but each serving contained 2000 calories, that would be a terrible deal. The actual number that matters is the proportion of calories from protein.

My ranking only includes vegan cereals because I’m vegan. Fortunately most cereals are vegan anyway. The main exception is that some cereals contain whey protein, but that’s not too common—most of them use soy, pea, or wheat protein instead.

1. Oatmeal with added protein powder (27% calories from protein, if you make it the way I do)

This is regular oatmeal because I couldn't find a stock photo of oatmeal mixed with protein powder.

You can buy pre-mixed oatmeal and protein powder, but it’s unnecessarily expensive so I prefer to mix it myself. Obviously the amount of protein varies depending on how much protein powder you add. I personally like to mix:

two servings of oats (= one cup, or 300 calories)
one scoop of protein powder (= 1/3 cup, or 25 grams, or 90 calories)

Those proportions provide 30% calories from protein. I find the consistency gets sticky if you add more protein than that. If you don’t like the consistency at this ratio, you can add more oats/less protein.

I save time by mixing the oats and protein powder in a giant jug. The jug lasts me for a few weeks and this way I don’t have to measure out the proportions every morning.

Oatmeal is my favorite cereal because there are so many ways to make it. I like to mix in blueberries, blackberries, or bananas, which add nutrients and cover up the protein-y flavor. (Plain oatmeal with protein powder tastes kind of weird.)

This mixture has 30% calories from protein without fruit and about 27% with fruit.

2. Special K High Protein Chocolate Almond (33% calories from protein)

This is my favorite cold breakfast cereal. Most high-protein cereals taste merely tolerable, but this one tastes actively good. It has a nice crunchy texture from the combination of almonds and cereal flakes, and it has enough sugar to give it a good flavor.

Unfortunately it doesn’t seem to be available anymore. I reached out to customer service to ask if it’s discontinued and they said they’re still producing it, but I haven’t been able to find it anywhere. (Nobody else seems to be able to find it either—see the product reviews here.)

(Special K High Protein should not be confused with Special K Protein, which did not qualify for my list because it has less than 25% calories from protein.)

3. Catalina Crunch (40% calories from protein)

This is my 3rd favorite high-protein cereal, and it has more protein than #1 or #2. Catalina Crunch has become a staple for me now that Special K High Protein is apparently discontinued.

There are a number of flavors, but the only ones I’ve tried are Cinnamon Toast and Dark Chocolate. Originally I had Dark Chocolate on this list at #4, but later I tried Cinnamon Toast which is better, so I’ve moved Catalina Crunch up to rank 3.

It doesn’t get soggy in milk and it doesn’t fall apart in your mouth. It’s sugar-free, but it tastes surprisingly good to me (I don’t usually like sugar-free cereals). The downside is it’s pretty expensive compared to most breakfast cereals.

4. Wheaties Protein (supplemented with lysine) (32% calories from protein)

Wheaties Protein is the last high-protein cereal I’d consider good, like I actively enjoy eating it.

Be aware that this cereal’s protein comes primarily from wheat, which doesn’t have much of the amino acid lysine. It has plenty of every other essential amino acid, but if you eat a lot of this cereal, you need to make sure you get lysine from somewhere else.

To get a full amino acid profile, you need to add about 1 gram of lysine per 60 grams of wheat protein. I personally buy these lysine supplements and take one 500mg pill per bowl of cereal.

5. Post Premier Protein (44% calories from protein)

This cereal has two different flavors, Chocolate Almond and Mixed Berry Almond. I personally like the chocolate flavor better. Both have a passably good flavor and texture but they have a weird protein-y crunchiness.

Post Premier Protein gets most of its protein from wheat, which, as I mentioned before, doesn’t contain much lysine. Fortunately it also contains pea protein, which has an abundance of lysine.

I don’t know the exact protein ratios, but I believe the overall balance still doesn’t contain enough lysine, so it would be prudent to get some extra lysine from somewhere else. I personally would take one 500mg lysine pill per two bowls of cereal. (I always eat two bowls for breakfast because I’m a growing boy.)

6. Special K Zero (50% calories from protein)

If you had to guess which of Special K High Protein and Special K Zero had more protein, you might think it’s the one with “protein” in the name, but you’d be wrong. This cereal contains an extraordinary 50% calories from protein with a good amino acid composition. Unfortunately, the reason it has so much protein is that it’s basically just lumps of protein powder.

When I take a bite of this cereal, it tastes good for the first five seconds or so. Then it dissolves into a powdery mush—it feels like I’ve poured wet protein powder directly into my mouth. I wouldn’t eat Special K Zero unless I was really desperate for protein. (Even then, I would rather have a protein shake.)

But some people seem to like it so your mileage may vary.

7. Three Wishes (24% to 27% calories from protein, depending on flavor)

Like Special K Zero, Three Wishes feels like eating protein powder flakes. But it only has half as much protein as Special K Zero. If you can stomach eating protein powder flakes, you might as well eat Special K Zero instead.

(I’ve only tried one flavor of Three Wishes, but it was a while ago and I don’t remember which flavor it was. I assume the other flavors have the same bad protein-y texture as the one I tried.)

Honorable mention: Kashi Go

Kashi Go, with 24% calories from protein, just barely does not qualify for my list. It has a good flavor and texture, but it makes my mouth feel weird if I eat too much of it. (I get the same mouth feeling when I eat a lot of spinach.) If it had that extra one percentage point of protein, I would put it at #4 on my list.

Unranked because it’s not vegan: Magic Spoon (33% to 37% calories from protein, depending on flavor)

I’ve never eaten Magic Spoon because it’s not vegan (it contains milk protein).

Drew Gooden says it’s bad and he makes funny YouTube videos which means he must have good opinions about cereal. His description of eating Magic Spoon sounds a lot like my experience eating Special K Zero and Three Wishes—it tastes good for a few seconds, then it turns into a protein mush.

Some other non-vegan cereals that I know nothing about

Snack House Keto Cereal: 44% calories from protein (uses milk protein)
Julian Bakery ProGranola Cereal: 44% calories from protein (uses egg white protein (weird choice but ok))
Perfect Keto Cereal: 36% calories from protein (uses milk protein) (appears to be discontinued)
Wonderworks Keto Friendly Breakfast Cereal: 35% calories from protein (uses milk and soy protein)

Those are all the high-protein breakfast cereals that I’m aware of.

Price table

This table gives the price of each cereal in terms of cents per gram of protein, ordered from cheapest to most expensive. I pulled these prices off Amazon; the prices in your area might differ.

For the cost of protein oatmeal, I used Quaker 1-Minute Oats plus NOW Foods Soy Protein Isolate because those are the brands I buy.

Cereal	Price (¢/g)
oatmeal + protein powder	3.7¢
Post Premier Protein	4.2¢
Special K High Protein	4.8¢
Wheaties Protein	4.9¢
Special K Zero	7.9¢
Catalina Crunch	9.1¢
Kashi Go	11.4¢¹
Three Wishes	12.5¢

Prices for non-vegan cereals:²

Cereal	Price (¢/g)
Julian Bakery ProGranola	10¢
Wonderworks Keto Friendly	13.5¢
Magic Spoon	13.8¢
Snack House Keto	16.5¢

Notes

This is strangely expensive for a big-brand cereal (Kashi is a subsidiary of Kellogg), my guess is there’s some sort of temporary supply issue and the price will go down. ↩
How convenient for my pro-vegan agenda that the non-vegan cereals are all so expensive!

Inconveniently for my agenda, the cheapest of the non-vegan cereals uses egg protein, which causes more animal suffering than whey protein.

Honestly I’m not that concerned about whey protein, it’s one of the least harmful animal products to buy. ↩

Charity Cost-Effectiveness Really Does Follow a Power Law

Wed, 25 Dec 2024 00:00:00 -0800

Conventional wisdom says charity cost-effectiveness obeys a power law. To my knowledge, this hypothesis has never been properly tested.¹ So I tested it and it turns out to be true.

(Maybe. Cost-effectiveness might also be log-normally distributed.)

Cost-effectiveness estimates for global health interventions (from DCP3) fit a power law (a.k.a. Pareto distribution) with $\alpha = 1.11$. [More]
Simulations indicate that the true underlying distribution has a thinner tail than the empirically observed distribution. [More]

Contents
Fitting DCP3 data to a power law
Does estimation error bias the result?
Future work I would like to see
Source code and data
Notes

Fitting DCP3 data to a power law

The Disease Control Priorities 3 report (DCP3) provides cost-effectiveness estimates for 93 global health interventions (measured in DALYs per US dollar). I took those 93 interventions and fitted them to a power law.

You can see from this graph that the fitted power law matches the data reasonably well:

To be precise: the probability of a DCP3 intervention having cost-effectiveness $x$ is well-approximated by the probability density function $f(x) = \displaystyle\frac{1.11}{x^{2.11}}$, which is a power law (a.k.a. Pareto distribution) with $\alpha = 1.11$.

It’s possible to statistically measure whether a curve fits the data using a goodness-of-fit test. There are a number of different goodness-of-fit tests; I used what’s known as the Kolmogorov-Smirnov test². This test essentially looks at how far away the data points are from where the curve predicts them to be. If many points are far to one side of the curve or the other, that means the curve is a bad fit.

I ran the Kolmogorov-Smirnov test on the DCP3 data, and it determined that a Pareto distribution fit the data well.

The goodness-of-fit test produced a p-value of 0.79 for the null hypothesis that the data follows a Pareto distribution. p = 0.79 means that, if you generated random data from a Pareto distribution, there’s a 79% chance that the random data would look less like a Pareto distribution than the DCP3 data does. That’s good evidence that the DCP3 data is indeed Pareto-distributed or close to it.³

However, the data also fits well to a log-normal distribution.

Pareto and log-normal distributions look similar most of the time. They only noticeably differ in the far right tail—a Pareto distribution has a fatter tail than a log-normal distribution, and this becomes more pronounced the further out you look. But in real-world samples, we usually don’t see enough tail outcomes to distinguish between the two distributions.

DCP3 only includes global health interventions. If we expanded the data to include other types of interventions, we might find a fatter tail, but I’m not aware of any databases that cover a more comprehensive set of cause areas.

(The World Bank has data on education interventions, but adding one cause area at a time feels ad-hoc and it would create gaps in the distribution.)

Does estimation error bias the result?

Yes—it causes you to underestimate the true value of $\alpha$.

(Recall that the alpha ($\alpha$) parameter determines the fatness of the tail—lower alpha means fatter tail. So estimate error makes the tail look fatter than it really is.)

There’s a difference between cost-effectiveness and estimated cost-effectiveness. Perhaps estimation error follows a power law, but the true underlying cost-effectiveness numbers don’t. And even if they do, our cost-effectiveness estimates might produce a bias in the shape of the fitted distribution.

I tested this by generating random Pareto-distributed⁴ data to represent true cost-effectiveness, and then multiplying by a random noise variable to represent estimation error. I generated the noise as a log-normally-distributed random variable centered at 1 with $\sigma = 0.5$⁵ (colloquially, that means you can expect the estimate to be off by 50%).

I generated 10,000 random samples⁶ at various values of alpha, applied some estimation error, and then fit the resulting estimates to a Pareto distribution. The results showed strong goodness of fit, but the estimated alphas did not match the true alphas:

true alpha 0.8  -->  0.73 estimated alpha (goodness-of-fit: p = 0.3)
true alpha 1.0  -->  0.89 estimated alpha (goodness-of-fit: p = 0.08)
true alpha 1.2  -->  1.07 estimated alpha (goodness-of-fit: p = 0.4)
true alpha 1.4  -->  1.22 estimated alpha (goodness-of-fit: p = 0.5)
true alpha 1.8  -->  1.54 estimated alpha (goodness-of-fit: p = 0.1)

To determine the variance of the bias, I generated 93 random samples at a true alpha of 1.1 (to match the DCP3 data) and fitted a Pareto curve to the samples. I repeated this process 10,000 times.

Across all generations, the average estimated alpha was 1.06 with a standard deviation of 0.27. That’s a small bias—only 0.04—but it’s highly statistically significant (t-stat = –15, p = 0⁷).

A true alpha of 1.15 produces a mean estimate of 1.11, which equals the alpha of the DCP3 cost-effectiveness data. So if the DCP3 estimates have a 50% error ($\sigma = 0.5$), then the true alpha parameter is more like 1.15.

Increasing the estimate error greatly increases the bias. When I changed the error (the $\sigma$ parameter) from 50% to 100%, the bias became concerningly large, and it gets larger for higher values of alpha:

true alpha 0.8  -->  0.65 mean estimated alpha
true alpha 1.0  -->  0.78 mean estimated alpha
true alpha 1.2  -->  0.87 mean estimated alpha
true alpha 1.4  -->  0.96 mean estimated alpha
true alpha 1.6  -->  1.03 mean estimated alpha
true alpha 1.8  -->  1.11 mean estimated alpha

If the DCP3 samples have a 100% error then the true alpha is 1.8—much higher than the estimated value of 1.11.

In addition, at 100% error with 10,000 samples, the estimates no longer fit a Pareto distribution well—the p-value of the goodness-of-fit test ranged from 0.005 to <0.00001 depending on the true alpha value.⁸

Curiously, decreasing the estimate error flipped the bias from negative to positive. When I reduced the simulation’s estimate error to 20%, a true alpha of 1.1 produced a mean estimated alpha of 1.14 (standard deviation 0.31, t-stat = 13, p = 0⁷). A 20% error produced a positive bias across a range of alpha values—the estimated alpha was always a bit higher than the true alpha.

Future work I would like to see

A comprehensive DCP3-esque list of cost-effectiveness estimates for every conceivable intervention, not just global health. (That’s probably never going to happen but it would be nice.)
More data on the outer tail of cost-effectiveness estimates, to better identify whether the distribution looks more Pareto or more log-normal.

Source code and data

Source code is available on GitHub. Cost-effectiveness estimates are extracted from DCP3’s Annex 7A; I’ve reproduced the numbers here in a more convenient format.

Notes

The closest I could find was Stijn on the EA Forum, who plotted a subset of the Disease Control Priorities data on a log-log plot and fit the points to a power law distribution, but did not statistically test whether a power law represented the data well. ↩
Some details on goodness-of-fit tests:

Kolmogorov-Smirnov is the standard test, but it depends on the assumption that you know the true parameter values. If you estimate the parameters from the sample (as I did), then it can overestimate fit quality.

A recent paper by Suarez-Espinoza et al. (2018)⁹ devises a goodness-of-fit test for the Pareto distribution that does not depend on knowing parameter values. I implemented the test but did not find it to be more reliable than Kolmogorov-Smirnov—for example, it reported a very strong fit when I generated random data from a log-normal distribution. ↩
A high p-value is not always evidence in favor of the null hypothesis. It’s only evidence if you expect that, if the null hypothesis is false, then you will get a low p-value. But that’s true in this case.

(I’ve previously complained about how scientific papers often treat p > 0.05 as evidence in favor of the null hypothesis, even when you’d expect to see p > 0.05 regardless of whether the null hypothesis was true or false—for example, if their study was underpowered.)

If the data did not fit a Pareto distribution then we’d expect to see a much smaller p-value. For example, a goodness-of-fit test for a normal distribution gives p < 0.000001, and a gamma distribution gives p = 0.08. A log-normal distribution gives p = 0.96, so we can’t tell whether the data is Pareto or log-normal, but it’s unlikely to be normal or gamma. ↩
Actually I used a Lomax distribution, which is the same as a Pareto distribution except that the lowest possible value is 0 instead of 1. ↩
The $\sigma$ parameter is the standard deviation of the logarithm of the random variable. ↩
In practice we will never have 10,000 distinct cost-effectiveness estimates. But when testing goodness-of-fit, it’s useful to generate many samples because a large data set is hard to overfit. ↩
As in, the p-value is so small that my computer rounds it off to zero. ↩ ↩²
Perhaps that’s evidence that the DCP3 estimates have less than a 100% error, since they do fit a Pareto distribution well? That would be convenient if true.

But it’s easy to get a good fit if we reduce the sample size to 93. When I generated 93 samples with 100% error, I got a p-value greater than 0.5 most of the time. ↩
Suárez-Espinosa, J., Villasenor-Alva, J. A., Hurtado-Jaramillo, A., & Pérez-Rodríguez, P. (2018). A goodness of fit test for the Pareto distribution. ↩

Protest movements can have significant short-term impacts	Strong
Protest movements can achieve intended outcomes in North America and Western Europe	Strong

Philosophical Multicore

On Democratizing ASI to Preserve Civil Liberties

Contents

Democratizing AI vs. putting a democratic government in charge of AI

A sketch of how we might democratize AI

Two non-obvious issues with democratizing AI

Liberalism only protects those inside it

AI proliferation increases catastrophic risks from competition

Notes

Links for August

Notes

Notes on the possibility of moral progress

Contents

Acting in the face of moral uncertainty

Can we discover facts that resolve moral uncertainty?

Some normative claims evade fact-based analysis

Implications for how the future goes

Notes

Letting myself look foolish

Notes

Pausing AI at human level seems harder than pausing ASAP

Notes

Training AI to be better at correctness than persuasion

Notes

AI will make biological extinction risks worse before it makes them better

Contents

In the near term, AI makes biorisk worse

AI can’t control scientific progress unless it controls everything

Low biorisk trades off against high AI takeover risk

Accelerating AI development is not a good way to reduce biorisk

This is yet another illustration of the fact that we don’t know what “aligned AI” means

Notes

Compare Your Company Stock to a Leveraged Index Fund

How to compare stock to a leveraged index fund

How to compare unequal dollar amounts

Calculator

Notes

A frontier AI company should shut down

Notes

Science-driven stories are good for the same reason that character-driven stories are good

Notes

How valuable are weak AI safety regulations?

Contents

I. Ways weak regulations can reduce risk

Directly reduce extinction risk

Empower future efforts to reduce extinction risk

Reveal warning shots

Shift the Overton window

II. Downsides of weak regulations

Opportunity cost

Regulation fatigue

Slows technological progress

May get in the way of AI companies implementing their own, more sensible, self-regulations

III. Specific policies, and how they might reduce extinction risk

GPU export controls

Establishment of an AI safety standards body

Dangerous capability evaluations

Mandatory publication of safety frameworks

Mandatory advance disclosure of large training runs

Whistleblower protections

Incident reporting

Security requirements to prevent model theft

My position on weak regulations

Notes

We Need Breadth-First AI Safety Plans

How breadth-first plans can inform what we do

Root-level breadth matters most

My rough attempt at categorizing plans

Future work I’d like to see

Notes

Sentient Welfare Across Three Futures

If timelines are long…

If we’re on track to solving AI alignment…

If we’re not on track to solving AI alignment…

Which future are you betting on?

Notes

I sleep less when I exercise more

Robustness checks

Using step count instead of calories burned

Extended time horizon