In many contexts, progress largely comes not from incremental progress, but from sudden and unpredictable insights. This is true at many different levels of scope—from one person’s current project, to one person’s life’s work, to the aggregate output of an entire field. But we know almost nothing about what causes these insights or how to increase their frequency.

Cross-posted to LessWrong.

Incremental progress vs. sudden insights

To simplify, progress can come in one of two ways:

1. Incremental improvements through spending a long time doing hard work.
2. Long periods of no progress, interspersed with sudden flashes of insight.

Realistically, the truth falls somewhere between these two extremes. Some activities, like theorem-proving, look more like the second case; other activities, like transcribing paper records onto a computer, look more like the first. When Andrew Wiles proved Fermat’s Last Theorem, he had to go through the grind of writing a 200-page proof, but he also had to have sparks of insight to figure out how to bridge the missing gaps in the proof.

The axis of incremental improvements vs. rare insights is mostly independent of the axis of easy vs. hard. A task can be sudden and easy, or incremental and hard. For example:¹

	incremental work	sudden insights
easy	algebra homework	geometry homework
hard	building machine learning models	proving novel theorems

Insofar as progress comes from “doing the work”, we know how to make progress. But insofar as it comes from rare insights, we don’t know.

Some meditations on the nature of insights

Why did it take so long to invent X?

Feynman on finding the right psychological conditions

Physicist Richard Feynman talks about this in Take the World from Another Point of View:

I worked out the theory of helium, once, and suddenly saw everything. I’d been struggling, struggling for two years, and suddenly saw everything at one time. […] And then you wonder, what’s the psychological condition? Well I know at that particular time, I simply looked up and I said wait a minute, it can’t be quite that difficult. It must be very easy. I’ll stand back, I’ll treat it very lightly, I’ll just tap it, and there it was! So how many times since then, I’m walking on the beach and I say, now look, it can’t be that complicated. And I’ll tap it, tap it, nothing happens.

Feynman tried to figure out what conditions lead to insights, but he “never found any correlations with anything.”

P vs. NP

A pessimistic take would be that there’s basically no way to increase the probability of insights. Recognizing insights as obvious in retrospect is easy, but coming up with them is hard, and this is a fundamental mathematical fact about reality because P != NP (probably). As Scott Aaronson writes:

If P=NP, then the world would be a profoundly different place than we usually assume it to be. There would be no special value in “creative leaps,” no fundamental gap between solving a problem and recognizing the solution once it’s found. Everyone who could appreciate a symphony would be Mozart; everyone who could follow a step-by-step argument would be Gauss; everyone who could recognize a good investment strategy would be Warren Buffett. It’s possible to put the point in Darwinian terms: if this is the sort of universe we inhabited, why wouldn’t we already have evolved to take advantage of it?

I’m not quite so pessimistic. I agree with Scott Aaronson’s basic argument that solving problems is much harder than recognizing good solutions, but there might still be ways we could make it easier to solve problems.

johnswentworth on problems we don’t understand

The concept of sudden-insight problems relates to johnswentworth’s concept of problems we don’t understand. Problems we don’t understand almost always require sudden insights, but problems that require sudden insights might be problems we understand (for example, proving theorems). johnswentworth proposes some types of learning that could help:

• Learn the gears of a system, so you can later tackle problems involving the system which are unlike any you’ve seen before. Ex.: physiology classes for doctors.
• Learn how to think about a system at a high level, e.g. enough to do Fermi estimates or identify key bottlenecks relevant to some design problem. Ex.: intro-level fluid mechanics.
• Uncover unknown unknowns, like pitfalls which you wouldn’t have thought to check for, tools you wouldn’t have known existed, or problems you didn’t know were tractable/intractable. Ex.: intro-level statistics, or any course covering NP-completeness.

I would expect these types of learning to increase the rate of insights.

Learning how to increase the frequency of insights

Insights happen less frequently under bad conditions: when you’re sleep-deprived, or malnourished, or stressed out, or distracted by other problems. Some actions can increase the probability of insights—for example, by studying the field and getting a good understanding of similar problems. But even under ideal conditions, insights are rare.

Interestingly, most of the things that increase the frequency of insights, such as sleep and caffeine, also increase the speed at which you can do incremental work. It’s possible that these things speed up thinking, but don’t increase the probability that any particular thought is the “right” one.

I can come up with one exception: you can (probably?) increase the frequency of insights on a problem if you understand a wide variety of problems and concepts. I don’t believe this does much to speed up incremental work, but it does make sudden insights more likely. Perhaps this happens because sudden insights often come from connecting two seemingly-unrelated ideas. I’ve heard some people recommend studying two disparate fields because you can use your knowledge of one field to bring a unique perspective to the other one.

Overall, though, it seems to me that we as a society basically have no idea how to increase insights’ frequency beyond a basic low level.

Instead of directly asking how to produce insights, we can ask how to learn how to produce insights. If we wanted to learn more about what conditions produce insights, how might we do that? Could we formally study the conditions under which geniuses come up with genius ideas?

If someone gave me a pile of money and asked me to figure out what conditions best promote insights, what would I do? I might start by recruiting a bunch of mathematicians and scientists to regularly report on their conditions along a bunch of axes: how long they slept, their stress level, etc. (I’d probably want to figure out some axes worth studying that we don’t already know much about, since we know that conditions (like sleep quality) do affect cognitive capacity.) Also have them report whenever they make some sort of breakthrough. If we collect enough high-quality data, we should be able to figure out what conditions work best, and disambiguate between factors that help provide insights and factors that “merely” increase cognitive capacity.

I’m mostly just speculating here—I’m not sure the best way to study how to have insights. But it does seem like an important thing to know, and right now we understand very little about it.

Notes