It has often been said that scholars don’t read their own citations. Out of curiosity, I decided to go through one of my longer essays to see how many of my citations I read.

(I actually did this exercise a while ago, around the time I published the original essay. Today I was going through my personal journal and found my notes on the exercise, and I thought it might be worth sharing publicly.)

The results:

• My essay cites a total of 37 academic papers.
• For 9 citations, I read the entire thing top to bottom and took notes to help me remember.
• For another 6 citations, I skimmed them but didn’t read carefully or take notes.
• For the remaining 22, I only read the abstracts.

I also replicated the results of 7 of the 37 citations. I didn’t do full replications, but I did reproduce at least one result from each paper. 6 out of those 7 were what you might call “trusting” replications: I got data from the study authors and used that to replicate their findings, but I’m still trusting that they constructed their data set correctly. For one of the 7 replications, I used an independent data set.

(There’s another level of reading that’s less rigorous than a replication but more rigorous than a full read-through: a “close reading” with the intentional goal of identifying mathematical or logical errors. I did not do that for any of my citations. Replicating a paper is (arguably) harder than doing a close reading, but I preferred to do outright replications because I find it more fun.)

Why did I approach the citations this way? Here’s why:

• A minority of papers were directly relevant to my thesis, so I read those in full. Whether I read or skimmed a paper largely depended on how well it was written. I can barely force myself to read papers with bad prose, so I just skim those.
• For most citations, I just wanted to know some specific fact, and I searched the literature until I found one that included the fact I wanted. For example: I wondered, “do mutual funds have any net exposure to the value factor?” I found a couple of papers that purportedly answered this question—Blitz (2017) and Lettau et al. (2018)—skimmed them to find the relevant tables, and then cited them, trusting that they were methodologically sound.
• Another common cause of me not reading citations is that I’d read an interesting fact in an article or book, decide to put the fact in my essay, and then cite the same paper that that article or book cited. (Pretty much the classic “don’t cite Wikipedia, just read Wikipedia and then copy its citations” technique.)

As a heuristic, if multiple studies find similar results, that gives me more confidence in their conclusions without having to read them. It’s not perfect, but it’s a reasonable way to save time. For example, I haven’t read Fama & French (1992) beyond the abstract, but I know it’s been replicated lots of times.

Sure, in an ideal world, I should carefully analyze every paper I cite to check for mistakes. But given time constraints, it makes sense for me not to read most of the papers I cite. Still, I can see why bad papers often get widely cited—nobody reads them! I wish I had more time and energy to read more stuff. (I also wish most academic papers weren’t written so poorly. If they were better written, reading them might actually be fun.)