Recently, I was browsing IMDb’s list of top-rated TV shows:

According to IMDb ratings, Planet Earth II is the second-best TV show of all time, with 9.5 stars out of 10. But if you look at the ratings of each individual episode, they range from 6.8 to 7.9¹:

In general, the rating of a TV show usually differs from the average rating of that show’s episodes. What does the list of top TV shows look like if we sort by average episode rating instead of show rating? Perhaps voters have different motivations when they’re rating shows than when they’re rating individual episodes, and it could be interesting to see how the ratings differ.

So I downloaded the IMDb public database to find out².

Continue reading

Experiments are a critical part of science—perhaps even the central feature. But middle school and high school science experiments don’t teach students how experiments are supposed to work.

The way I did science experiments in school went something like this:

1. Learn about some natural phenomenon.
2. Teacher explains an experiment intended to test the natural phenomenon or at least vaguely relate to it.
3. We run the experiment, with the ostensible goal of observing the natural phenomenon.
4. We get results totally different from what the laws of nature predict.
5. Whatever, let’s move on to the next subject.

For example, I remember in physics class we learned about how acceleration due to gravity changes when an object rolls down an incline based on the steepness of the incline. Then we did an “experiment” to test this by rolling marbles down inclines and measuring how far they got in a fixed amount of time. The results we got were inconsistent with the laws of mechanics, but nobody questioned ths. We all assumed that our experiment was not sufficiently well-controlled to produce reliable results (which was accurate).

This is the antithesis of how experiments are supposed to work. The point of running an experiment is to learn something about the world. Experiments should be well controlled so you can be confident that you are learning something.

Running a good experiment is not easy. Experiments can easily fail to produce good results, so they must be designed carefully. Designing good experiments is a skill. And the way experiments are done in school does nothing to teach this skill.

If you know in advance that you have bad methodology and you’re going to throw away the results of your experiment, what’s the point? Experiments as they are done in school don’t teach about natural laws (because you ignore whatever results you get), and they don’t teach how to design good experiments (because no effort is made to produce consistent results).

I can imagine an effective science class that focused on teaching students how to design experiments. You could perhaps start by providing students a simple natural law, such as an object’s acceleration on an inclined plane, then challenge them to produce an experiment that replicates the results. If they don’t produce consistent results, push them to figure out why, and refine their experimental conditions until they can get reliable measurements.

But the point of an experiment isn’t (usually) to reproduce known results—it’s to figure out something unknown. A good experiment should be able to falsify a hypothesis; you shouldn’t just keep changing your experiment until you get the expected results. (The process I described in the previous paragraph is basically P-hacking.) I don’t know how you would teach people to get from “design an experiment that can consistently replicate a known natural law” to “design an experiment that can tell you something you don’t already know, and be confident that it’s correct.” But I’ve only been thinking about it for a few minutes. We are collectively wasting tens of millions of hours per year having students run experiments while learning nothing about how to run experiments, and I’m sure we can do better.

Let me throw out a slightly more sophisticated idea for how to teach experiments. Give students a natural phenomenon to investigate; it should be something they probably don’t already know (so they don’t know what result to expect), but that isn’t too hard to test. Divide the students into groups and have them design and implement experiments to figure out the phenomenon. Then challenge them to peer review each other’s experiments and look for flaws. Refine the experiments until most of the class agrees on the correct methodology and can replicate each other’s results.

This also provides a natural way to teach students statistics. If you need to develop good experimental methodologies, you need to have a way of knowing how reliable your results are and how many trials to run. Some students will try to understand how to do this, and as they begin to think more deeply about it, they will inevitably ask the same questions that inferential statistics is meant to answer. This is the perfect time to equip them with some statistics knowledge that they can use to improve their understanding of science.

I’m tempted to get overzealous about how significant it would be if we consistently ran science classes this way. I would like to say that it would solve the replication crisis, bring an end to shoddy news reporting, and revolutionize politics. Probably none of that would happen, and maybe this whole thing isn’t even a good idea. I’m just theorizing, I haven’t tested any of these ideas experimentally.

Altruists often would like to get good predictions on questions that don’t necessarily have great market significance. For example:

• Will a replication of a study of cash transfers show similar results?
• How much money will GiveWell move in the next five years?
• If cultured meat were price-competitive, what percent of consumers would prefer to buy it over conventional meat?

If a donor would like to give money to help make better predictions, how can they do that?

You can’t just pay people to make predictions, because there’s no incentive for their predictions to actually be accurate and well-calibrated. One step better would be to pay out only if their predictions are correct, but that still incentivizes people who may be uninformed to make predictions because there’s no downside to being wrong.

Another idea is to offer to make large bets, so that your counterparty can make a lot of money for being right, but they also want to avoid being wrong. That would incentivize people to actually do research and figure out how to make money off of betting against you. This idea, however, doesn’t necessarily give you great probability estimates because you still have to pick a probability at which to offer a bet. For example, if you offer to make a large bet at 50% odds and someone takes you up on it, then that could mean they believe the true probability is 60% or 99%, and you don’t have any great way of knowing which.

You could get around this by offering lots of bets at varying odds on the same question. That would technically work, but it’s probably a lot more expensive than necessary. A slightly cheaper method would be to determine the “true” probability estimate by binary search: offer to bet either side at 50%; if someone takes the “yes” side, offer again at 75%; if they then take the “no” side, offer at 62.5%; continue until you have reached satisfactory precision. This is still pretty expensive.

In theory, if you create a prediction market, people will be willing to bet lots of money whenever they think they can outperform the market. You might be able to start up an accurate prediction market by seeding it with your own predictions; then savvy newcomers will come and bet with you; then even savvier investors will come and bet with them; and the predictions will get more and more accurate. I’m not sure that’s how it would work out in practice. And anyway, the biggest problem with this approach is that (in the US and the UK) prediction markets are heavily restricted because they’re considered similar to gambling. I’m not well-informed about the theory or practice of prediction markets, so there might be clever ways of incentivizing good predictions that I don’t know about.

Anthony Aguirre (co-founder of Metaculus, a website for making predictions), proposed paying people based on their track record: people with a history of making good predictions get paid to make more predictions. This incentivizes people to establish and maintain a track record of making good predictions, even though they don’t get paid directly for accurate predictions per se.

Aguirre has said that Metaculus may implement this incentive structure at some point in the future. I would be interested to see how it plays out and whether it turns out to be a useful engine for generating good predictions.

One practical option, which goes back to the first idea I mentioned, is to pay a group of good forecasters like the Good Judgment Project (GJP). In theory, they don’t have a strong incentive to make good predictions, but they did win IARPA’s 2013 forecasting contest, so in practice it seems to work. I haven’t looked into how exactly to get predictions from GJP, but it might be a reasonable way of converting money into knowledge.

Based on my limited research, it looks like donors may be able to incentivize donations reasonably effectively with a consulting service like GJP, or perhaps by doing something involving predictions markets, although I’m not sure what. I still have some big open questions:

1. What is the best way to get good predictions?
2. How much does a good prediction cost? How does the cost vary with the type of prediction? With the accuracy and precision?
3. How accurate can predictions be? What about relatively long-term predictions?
4. Assuming it’s possible to get good predictions, what are the best types of questions to ask, given the tradeoff between importance and predict-ability?
5. Is it possible to get good predictions from prediction markets, given the current state of regulations?

Discuss on the Effective Altruism Forum.

Disclaimer: I am not an investment advisor and nothing in this essay serves as investment advice.

Introduction

Robin Hanson: If More Now, Less Later

The rate of return on investment historically has been higher than the growth rate–or, as they say, r > g. If you save your money to donate later, you can earn enough interest on it that you eventually have the funds to donate a greater amount. Because r > g, you should invest your money for as long as you can before donating¹–or so the argument goes.

Traditionally, we’d apply a discount rate of g to future donations, because that’s the rate at which people get richer and therefore the rate at which money becomes less valuable for them. But this ignores some important factors that affect how much we should discount future donations, and we can create a much more detailed estimate. This essay will explore that in detail. Exactly what factors determine the investment rate of return and the discount rate on poverty alleviation? Can we gain any information about which is likely greater?

Continue reading

Some people have observed that small and large donors follow different giving patterns. Small donors who give out of their salary—that is, most people—tend to donate money more or less as soon as they earn it (usually within a year). Large donors—e.g., extremely wealthy people and foundations—tend to slowly distribute their money and hold on to most of it¹. For example, large foundations typically donate little more than the legally required 5% of assets each year. Why do they behave differently?

I don’t believe this difference is surprising, and actually it’s not really even a difference.

Continue reading

This is a collection of writings on where people are donating. It only includes writings that I am aware exist (obviously) and that are written by effectiveness-minded people.

My descriptions are paraphrased from the linked writings as much as possible. The writing in this post includes combinations of my own and the linked writers’ words. My summaries often do not do the original writers justice, so I recommend reading all of the linked articles if you are interested.

Continue reading

Summary: The stock market can be modeled as Omega in Newcomb’s problem. On average, an asset will only outperform if the market predicts that you won’t buy it. So you cannot say “if I had bought that, I would have made a lot of money”, just as in Newcomb’s problem you can’t say “if I had taken both boxes, I would have gotten more money than if I only took one”.

Continue reading

Summary: All expected value distributions must either (1) have a mean of infinity, or (2) have such thin tails that you cannot ever reasonably expect to see extreme values. When we’re estimating the utility distribution of an intervention, both of these options are bad.

Continue reading

I have removed Disqus and replaced it with built-in static comments. Disqus comments are disabled, but still visible on any old posts¹. New posts going forward will only use the new static comment system.

I had been wanting to switch off Disqus for a while. It has a few disadvantages:

1. I have no control over comments except for the moderation tools Disqus provides.
2. I have no control over how comments are displayed.
3. I don’t know what Disqus is doing or might do with commenters’ personal information.

The new comment system does exactly what I want it to do and nothing more.

Edited to add: If anyone’s interested, I’m using the Jekyll Static Comments plugin by Matt Palmer, with a few personal modifications.

Notes

Summary: Large donors may create better incentives for both charities and small donors if, rather than providing fixed funding to a charity, they offer to match all donations to that charity over a relatively long time horizon.

Continue reading