There Are Three Kinds of "No Evidence"
David J. Balan once proposed that there are two kinds of “no evidence”:
- There have been lots of studies directly on this point which came back with the result that the hypothesis is false.
- There is no evidence because there are few or no relevant studies.
I propose that there are three kinds of “no evidence”:
- The hypothesis has never been studied.
- There are studies, the studies failed to find supporting evidence, but they wouldn’t have found supporting evidence even if the hypothesis were true.
- There are studies, the studies should have found supporting evidence if the hypothesis were true, and they didn’t.
Example of type 1: A 2003 literature review found that there were no studies1 showing that parachutes could prevent injury when jumping out of a plane.
Example of type 2: In 2018, there was finally a randomized controlled trial2 on the effectiveness of parachutes, and it found no difference between the parachute group and the control group. However, participants only jumped from a height of 0.6 meters (~2 feet). I don’t know about you, but this result does not make me want to jump out of a plane without a parachute.
Like in the parachute example, you see type-2 “no evidence” whenever the conditions of a study don’t match the real-world environment. You also see type-2 “no evidence” when an experiment is underpowered. Say you want to test the hypothesis that boys are taller than girls. So you go find your niece Sally and your neighbor’s son James and it turns out Sally is an inch taller than James. Your methodology was valid—you can indeed test the hypothesis by finding some people and measuring their heights—but your sample size was too small.
(The difference between type 2 and type 3 can be a matter of degree. The more powerful a study is, the stronger its “no evidence” if it fails to find an effect.)
Notes
-
Smith, G. C. S. (2003). Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomised controlled trials. ↩
-
Yeh, R. W., Valsdottir, L. R., Yeh, M. W., Shen, C., Kramer, D. B., Strom, J. B., Secemsky, E. A. et al. (2018). Parachute use to prevent death and major trauma when jumping from aircraft: randomized controlled trial. ↩