Inspired by Applied Divinity Studies’ Unemployment Part 2

Many people, such as Cal Newport, say that you can only do about four hours of deep work per day. I am a lot worse at deep work than that.

When I worked full-time as a software developer, I tried pretty hard to avoid distractions and stay focused on work. At the end of each day, I made a quick estimate of how much I got done that day. I rated myself on a 5-point productivity scale. A fully productive day, where I spent the bulk of the day doing meaningful work, earned the full 5 points. My estimates were by no means objective, but according to my own perception, I scored 5 points on a total of 91 out of 602 work days (that’s 15%). A 5-point day usually meant I spent around four hours doing deep work, and most of the rest of the day doing important shallow work.

When I failed to do sufficient deep work, it usually wasn’t because I spent too much time on meetings and emails (although that did happen occasionally). Most of the time, I was simply too dumb to get enough useful work done.

Most of my work required having insights. If I’m trying to debug a problem, figure out how to extend some bit of code, or plan out a new feature, I need to have a series of small ideas about how to make progress. For example, if I’m looking for the source of a bug, I might have the idea, “The problem might be in this particular function. I’ll step through the function and look for any anomalies.” On my less productive days, I would simply fail to generate any such ideas, preventing me from making any progress.

On my worst days (which happened about 15% of the time), I couldn’t manage even simple tasks. This was more about focus than about insight. It would usually go like this:

Me: Here’s the task that we need to do. As you can see, I’ve opened the relevant file and I’m currently looking at it. We need to do X to this file. Brain: I don’t want to do that. Me: Come on, you know exactly what to do, you should just do it. Brain: No. Can we do something fun instead? Me: We have to do this. Brain: Well I’m not gonna.

Since I refused to get distracted and my brain refused to do the actual work, I’d end up staring at the screen doing nothing for about an hour. Eventually I’d give up on that and go read Hacker News or think about investing—things that feel work-adjacent but don’t qualify as genuine work.¹ After doing that for about 15 to 30 minutes, I’d feel guilty enough to go back to staring at my code while doing nothing.

What predicts productivity?

I tracked a bunch of variables to see if anything could predict whether I’d have a good or bad day. I looked at sleep, how much I ate, the glycemic index of my lunch², whether I went to the gym, and whether I had melatonin the previous night; none showed a statistically significant effect. The only thing that clearly mattered was whether I had caffeine. On a five-point scale, caffeine boosted by average productivity by about 0.7 points.

(I’m highly confident that I perform worse when I don’t get enough sleep. But my sleep schedule was consistent enough that I didn’t have a sufficiently large sample of sleep-deprived work days to show a visible effect.)

(I didn’t blind myself, so any positive results could be explained by a placebo effect. But I only got one positive result, and I wasn’t expecting to get it before I analyzed the data—subjectively, I felt about the same on caffeine days and non-caffeine days.)

About a year ago, when I started working full-time on independent research, the frequency of fully-productive days went up from 15% to 24% (53 out of 221). But median productivity declined, probably because I stopped trying to force myself to work when I didn’t want to.

My productivity tended to come in waves. I’d think of some interesting and digestible sub-problem, make a bunch of progress on it for a few days or weeks (depending on the size of the problem), figure out all the obvious stuff, and then lose steam and stop being productive for a while.

During waves of low productivity, the issue wasn’t that I didn’t know what to work on. I had plenty of ideas about important research areas. The problem was that I didn’t feel motivated to work on them. Usually, I’d only feel motivated to work on a problem shortly after I came up with it. Once the problem became stale, I’d lose interest. But sometimes I’d get a sudden second wind on an old stale problem. It would be nice if I knew what caused those second winds, or if I could find a way to reliably trigger them. But so far I haven’t figured out how to do that.

Notes