The ethical principles that most people hold—and hold most strongly—go completely out the window when it comes to war.

Continue reading

AI companies want to bootstrap weakly-superhuman AI to align superintelligent AI. I don’t expect them to succeed. I could give various arguments for why alignment bootstrapping is hard and why AI companies are ignoring the hard parts of the problem; but you don’t need to understand any details to know that it’s a bad plan.

When AI companies say they will bootstrap alignment, they are admitting defeat on solving the alignment problem, and saying that instead they will rely on AI to solve it for them. So they’re facing a problem of unknown difficulty, but where the difficulty is high enough that they don’t think they can solve it. And to remediate this, they will use a novel technique never before used in history—i.e., counting on slightly-superhuman AI to do the bulk of the work.

If they mess up and this plan doesn’t work, then superintelligent AI kills everyone.

And they think this is an acceptable plan, and it is acceptable for them to build up to human-level AI or beyond on the basis of this plan.

What?

Continue reading

I’ve been playing a lot of MTG Arena lately, but I refuse to spend any money on it, which means I can’t craft many rare cards. When I look up meta decklists, they always include a lot of rares and mythic rares. I don’t want to spend all my rare wildcards on one deck!

That’s sort of what the Pauper format is for. Pauper decks are only allowed to use common cards, which makes them cheap. But that format isn’t quite what I’m looking for, for four reasons:

1. There is no Pauper ladder on Arena, there are only tournaments.¹ I don’t want to play a tournament, I just want to be able to hop on and play a few games.
2. I have some rare wildcards that I can use to craft rare cards; I don’t have to limit myself to commons only. I just don’t have very many rare wildcards, so I want to spend them judiciously.
3. All the strongest Pauper decks are aggro decks. What if I don’t want to play aggro?
4. There are thousands of MTG cards that aren’t playable on Arena, so I can’t build most Pauper decks anyway.

There’s the Artisan format which is Arena-specific (so it fixes problem 4), but it still has the other three problems.

What I really want is to build a Standard deck using only 4–8 wildcards to craft the most important rares, and then if I decide I like the deck enough, I can craft some more. Which means I want to know which rares I really need, and which ones I can replace with common or uncommon substitutes.

Continue reading

Inspired by this post by Tomás Bjartur, which is an allegory; but I’m not writing an allegory, I’m writing about the rules of Quidditch.

The rules of Quidditch have a big problem. The game ends when a seeker catches the snitch, and the snitch is worth 150 points. So most of the players on the field don’t matter; in almost all games, the only thing that matters is who catches the snitch.

This also makes it a bad spectator sport because you can’t see the snitch, so nobody knows what the hell is going on.

I propose some rule changes:

Continue reading

Sometimes I’m talking about lifting weights and someone asks me, “What’s your goal weight?” I don’t understand why I would have a goal weight.

Say I want to bench press 300 pounds. What happens when I reach 300? I just give up on the bench press now? That would be silly. If I can keep getting stronger, I should.

What happens if I fall short of my goal? Say I haven’t been able to bench more than 285.¹ Should I start eating 5000 calories a day to put on as much muscle as possible? No, I’m not going to do that, I don’t want to get fat. Realistically, if I fall short of my goal, the answer to the question of what I should change is “nothing”.

The point of a goal is to make tradeoffs between objectives. But when you set goals, you have less information about your costs than when you’re trying to implement them. At implementation time, you have new information that might change how you prioritize things, which may result in failing to achieve a goal; and that’s perfectly fine.

Sometimes a goal turns out to be easier than you thought; that doesn’t mean you should give up after you achieve it.

Sometimes a goal turns out to be harder than you thought; that doesn’t mean you should sacrifice everything else for it.

Continue reading

A curiosity stopper is an answer to a question that gets you to stop asking questions, but doesn’t resolve the mystery.

There are some curiosity stoppers that I’ve heard many times:

• Why doesn’t cell phone radiation cause cancer? Because it’s non-ionizing radiation.
• Why are antioxidants good for you? Because they eliminate free radicals.
• Why do bicycles stay upright? Because of gyroscopic forces.
• Why do solids hold together? Because of intermolecular forces of attraction.

For the first three, those answers confused me because I didn’t know what those words meant. I guess I know what an ion is (it’s an atom with an electrical charge) but why do I care whether radiation is ionizing? And what makes radiation ionizing or non-ionizing?

What’s a free radical? Why is it bad?

What’s a gyroscopic force? (What even is a gyroscope? It’s some sort of top, right?) How on earth does a bicycle generate a gyroscopic force?

The fourth curiosity stopper—”intermolecular forces of attraction”—is even more of a non-answer. Of course solids hold together because a force holds them together. That’s what a force is. But what is the force, and where does it come from?

Another genre of curiosity stopper is the out-of-context number:

• “The Dow is down 600 points today.” (How much is that?)
• “My proposed policy will create two million jobs.” (What percentage is that? What are the odds that I, personally, get a new job?)
• “This product has 7 grams of protein per serving!” (How big is a serving? How much would I need to eat to meet my daily protein requirement?)

Answers (sort of)

Continue reading

Last year I gave my reasoning on cause prioritization and did shallow reviews of some relevant orgs. I’m doing it again this year.

Continue reading

Vizzini: Inconceivable!

Inigo: You keep using that word. I do not think it means what you think it means.

What did Inigo mean by this?

(Don’t laugh, this is serious.)

Continue reading

Introduction

Even if we solve the AI alignment problem, we still face non-alignment problems, which are all the other existential problems¹ that AI may bring.

People have written research agendas on various imposing problems that we are nowhere close to solving, and that we may need to solve before developing ASI. An incomplete list of topics: misuse; animal-inclusive AI; AI welfare; S-risks from conflict; gradual disempowerment; risks from malevolent actors; moral error.

The standard answer to these problems, the one that most research agendas take for granted, is “do research”. Specifically, do research in the conventional way where you create a research agenda, explore some research questions, and fund other people to work on those questions.

If transformative AI arrives within the next decade, then we won’t solve non-alignment problems by doing research on how to solve them.

Continue reading

Previously, I reviewed the five strongest studies on protest outcomes and concluded that peaceful protests probably work (credence: 90%).

But what about disruptive or violent protests?

Peaceful protests use nonviolent, non-disruptive tactics such as picketing and marches.

Disruptive protests use nonviolent, in-your-face tactics such as civil disobedience, sit-ins, and blocking roads.

Violent protests use violence.

There isn’t much evidence on the other two categories of protest. My best guesses are:

• Violent protests probably don’t work. (credence: 80%)
• Violent protests may reduce support for a cause, but it’s unclear. (credence: 40%)
• For disruptive protests, it’s hard to say whether they have a positive or negative impact on balance. I’m about evenly split on whether a randomly-chosen disruptive protest is net helpful, neutral, or harmful.
• A typical disruptive protest doesn’t work as well a typical peaceful protest. (credence: 80%)
• Peaceful protests are a better idea than disruptive protests. (credence: 90%)

Continue reading