Albert Einstein in the early 1900s. Aretha Franklin in the 1960s. Steve Jobs in the 2000s. There are certain spans of time when scientists, artists, and inventors have phenomenal periods of productivity.
This is also true of most people—at least on a smaller scale. Aren’t there periods when you feel like you’re effortlessly flourishing at work, while other times you feel incompetent and uninspired? You might recognize these periods of concentrated success among your friends, peers, and competitors too.
The Northwestern University economist Dashun Wang calls these special bursts of creativity “hot streaks”—a term usually reserved for sports. “Ninety percent of people have a hot streak in their career,” Wang told me. “Most people have just one. Some people have two. It’d be nice to have more.”
In the past few years, Wang has peeled back the mystery of why these special creativity clusters happen and how individuals and companies can multiply and extend them. Three years ago, he co-wrote a paper with researchers at Northwestern, the University of Miami, Penn State, and Central European University, in Budapest, that used large data sets to trace the career outputs of more than 20,000 artists, film directors, and scientists. The researchers found that almost all of them had clusters of highly successful work, as determined by higher-than-average art-auction prices, IMDb film ratings, or scientific-journal citations. “Bursts of high-impact works [are] remarkably universal across diverse domains,” he and his co-authors wrote. Just about everybody has a period in their life when they produce at their best, even if, unlike Aretha, they aren’t pumping out some of the greatest work of the 20th century.
So where do hot streaks come from? And how can each of us plan for one, or two—or 100? Wang spent several years trying to answer that question. His search uncovered mostly dead ends. “The more we tried and the more failed attempts we had, the idea of hot streaks seemed very random,” he said.
The conventional wisdom is that hot streaks happen in our middle age. One famous analysis of scientists and inventors found that their ability to produce Nobel Prize–winning insights and landmark technological contributions peaks between the ages of 35 and 40. Another analysis of “age-genius curves” for jazz musicians found that musical productivity rises steadily until about the age of 40 and then declines sharply.
Wang’s analysis—which used a broader measure of productivity for a much larger group of people—didn’t find anything special about the productivity of middle-aged people. Instead, hot streaks were equally likely to happen among young, mid-career, and late-career artists and scientists. Other theories fell flat too. Maybe, he thought, getting hot is a numbers game, and hot streaks happen when you produce the most work. Or maybe extremely successful work periods are all about focusing on one specific type of art or scientific discipline—as the 10,000-hours-of-practice rule popularized in Malcolm Gladwell’s book Outliers suggests. Or maybe hot streaks are more about who else we’re working with, and we’re most successful when we cozy up to superstars in our domain. But no explanation fit the data set.
Until this year. This summer, Wang and his co-authors published their first grand theory of the origin of hot streaks. It’s a complicated idea that comes down to three words: Explore, then exploit.
In 1991, the Stanford Graduate School of Business professor James G. March published an influential paper, “Exploration and Exploitation in Organizational Learning,” which broke down work into two big categories: exploring new ideas and exploiting old certainties. Say you’re a car manufacturer. Every year, you must decide between investing in future innovations, such as self-driving software, and finding ways to squeeze new revenue out of existing technologies and materials. Too much fanciful R&D spending, and this year’s profit plummets. Too much emphasis on tweaking existing product lines, and you get squashed by some fresh upstart in a decade.
Individuals face the same choice. Every week, I can write about pretty much whatever I want. I love exploring new ideas and emerging industries. But if I write an article about a totally new subject, it’s possible that I’ll do a bad job, or that nobody will want to read about it. Meanwhile, whenever I write articles about the future of work, a lot of people read them. Should I allow my curiosity to wander into new fields that might be dead ends for my career, or should I double down on becoming a full-time work futurist? It’s the same tension: explore or exploit?
In Wang’s most recent analysis, he found that artists and scientists tend to experiment with diverse styles or topics before their hot streak begins. This period of exploration is followed by a period of creatively productive focus. “Our data shows that people ought to explore a bunch of things at work, deliberate about the best fit for their skills, and then exploit what they’ve learned,” Wang said. This precise sequence—exploration, followed by exploitation—was the single best predictor of the onset of a hot streak.
Wang pointed to Jackson Pollock, the artist known for splashing and dribbling paint on a canvas. When Pollock started painting, in the early 1930s, he experimented with a variety of styles, including abstract art and surrealism reminiscent of Marc Chagall’s work. Suddenly, in the mid-’40s, he honed a mystically messy “drip style,” in which he painted almost exclusively for about four years. In 1949, Life magazine made him a household name and asked if he was “the greatest living painter in the United States.” The next year, at the height of his fame, Pollock abruptly abandoned his drip method—and started experimenting again, until his death.
At least for artists, film directors, and scientists, neither exploration nor exploitation does much good on its own. “When exploitation occurs by itself,” Wang and his co-authors wrote, “the chance that such episodes coincide with a hot streak is significantly lower than expected, not higher, across all three domains.” Only when periods of trial and error are followed right away by periods of deliberate focus does the probability of a hot streak increase significantly.
The research suggests something fundamentally hopeful: that periods of failure can be periods of growth, but only if we understand when to shift our work from exploration to exploitation. If you look around you at this very moment, you will see people in your field who seem wayward and unfocused, and you might assume they’ll always be that way. You will also see people in your field who seem extremely focused and highly successful, and you might make the same assumption. But Wang’s paper asks us to consider the possibility that many of today’s wanderers are also tomorrow’s superstars, just a few months or years away from their own personal hot streak. Periods of exploration can be like winter farming; nothing is visibly growing, but a subterranean process is at work and will in time yield a bounty.
Several years ago, the journalist David Epstein wrote the book Range: Why Generalists Triumph in a Specialized World, which argued that early specialization was a poor strategy for succeeding in a world of complex problems that defy easy answers. Instead, Epstein said, people are better off exploring a variety of fields and approaches and braiding their knowledge to produce new solutions. Wang’s research seems to back up that claim. The central paradox of the explore-exploit sequence is that hot streaks are examples of specialization, but specialization itself doesn’t lead to hot streaks. Today’s best exploiters were yesterday’s best explorers.
The benefits of exploration don’t end in school. A 2014 study of youth employment found that people who switch jobs more frequently early in their career tend to have higher wages and incomes in their prime working years. Job-hopping might seem like the work of an uncertain dilettante, but it improves the odds that you’ll find a job that combines mastery, meaning, and a good amount of money. Once again, exploration pays long-term dividends.
Finally, Wang and his colleagues offer a broader criticism of America’s education and innovation systems. U.S. education pushes early specialization in youth sports, music, and other extracurricular activities that promise to produce child prodigies. But if extended periods of trial and error are crucial for doing our most creative work, Americans’ fixation on early specialization is narrowing our collective potential. Several studies of funding decisions by the National Institutes of Health and the National Science Foundation have found systemic bias against extremely original research ideas and barriers for researchers working in exploratory fields. Even in areas, such as scientific grants, that are theoretically devoted to the frontier of knowledge, Americans keep trying to do incrementally more with what we already know—rather than expanding the scope of our intelligence.
The point is not that exploration is good and exploitation is bad. It’s that all success—career success, corporate thriving, national flourishing—requires that we pay close attention to the interplay between scouting new ideas and pumping established wells. By and large, America seems to suffer from too much exploitation and too little exploration. “We’ve gotten very good at encouraging people to be more and more focused and at penalizing people who wander outside their lane,” Wang said. “I don’t think America is particularly good at rewarding novel thinking.” Indeed, we have a national scouting deficit, because our theories of success emphasize immediate productivity in a way that might obscure the benefits of a little bewilderment and curiosity.