"fallacy"

How Researchers Discovered the Basketball ‘Hot Hand’

It should come as no surprise that the U.S. men’s Olympic basketball team has some very talented shooters on it. But the best of them is probably Klay Thompson, the Golden State Warriors shooting guard known for suddenly catching fire — swish, swish, swish — and becoming seemingly unable to miss. The most spectacular manifestation of Thompson’s shooting abilities came two Januaries ago, in a game against the Sacramento Kings. That was when he dropped 37 points in a single quarter, smashing the previous record of 33, shared by George Gervin and Carmelo Anthony:

The majority of NBA players never score 37 points in a game. How can you explain a performance like Thompson’s? When something like this happens, most observers and players believe the player in question has a “hot hand” — that for whatever reason, they have entered a state that makes shooting and scoring easier for them than it normally is (or even easier, in the case of a born-to-score star like Thompson). But it isn’t just pro athletes who sometimes feel this way. Anyone who has played enough pickup basketball has, at one time or another, gone through a hot-hand period that felt too good to be true.

But does the hot hand really exist, or is it an illusion? Are players who hit a few shots in a row really more likely to hit their next one than they would have been otherwise? It’s a question that has launched a surprisingly complex and interesting academic battle stretching back decades — a neat case study in how human bias works, in more ways than one.

Today Thomas Gilovich is a well-known social psychologist and behavioral economist at Cornell University. But back when he was a lowly, first-year grad student at Stanford, he took a seminar on human judgment with Amos Tversky, one of the godfathers of behavioral economics — a field obsessed with the ways that human bias can affect perception and decision-making. (Tversky died in 1996.) The subject of randomness came up — or, specifically, the way humans have a tendency to see patterns in randomness where there aren’t any. For example, as a species we tend to underestimate how (for lack of a better word) clumpy random patterns can get. If you flip a coin over and over again, you’ll get a fair number of HHHHHH or TTTTTT sequences, and other runs of consecutive heads or tails. These leap out to us as patterns, or as proof that the coin isn’t fair, but really they’re just the natural result of randomness.

Gilovich played a lot of basketball and was curious whether this common misperception applied to the idea of hot hands: Were people seeing patterns where there was really just randomness? Initially, Tversky told him it would be impossible to get enough data to really test this, but after Tversky brought the idea back to Gilovich during the latter’s last year of grad school, the duo paired up with Robert Vallone, a statistician, to try to determine whether the hot hand really existed.

Klay Thompson #11 of the Golden State Warriors reacts after he made a three-point basket in the third quarter of their game against the Sacramento Kings at ORACLE Arena on January 23, 2015 in Oakland, California. Thompson scored 37 points in the third quarter to set a NBA record. Photo: Ezra Shaw/Getty Images

In 1985, the trio published a paper in Cognitive Psychology that turned the intuitive consensus belief about hot hands on its head. After presenting survey evidence showing, unsurprisingly, that a sample of college-student basketball fans believed in the hot hand, the researchers examined real-life NBA data from the Philadelphia 76ers, in search of patterns suggesting that players who had run up a string of consecutive successful shots were more likely to hit their next shot than they would be otherwise. They found effectively no evidence for any such pattern, and analyses of New Jersey Nets and New York Knicks shooting records, as well as NBA free-throw data, all failed to turn up anything either. Everything was just random. There was little sign that players got “hot,” despite many players’ own adamant insistence that they did. Finally, Gilovich and his co-authors invited a bunch of players from Cornell’s basketball teams to take a bunch of shots in the more controlled setting of a gym. Maybe the hot streak didn’t exist in an in-game situation, but would it show up there? Nope: still nothing. And the researchers also found no evidence that players’ own “sense of being ‘hot’” actually predicted whether they would hit their next shot.

While academics familiar with statistics and probability accepted the findings, other academics had trouble doing so. “It was very hard to publish it,” Gilovich said, “and the same kind of reaction you see in the world at large was true on the part of many people in the academic world.” To many people, the hot hand seemed so obviously true that there was no way they were going to be convinced otherwise by some dry number-crunching. Gilovich said he had a lot of trouble convincing basketball players — who perhaps hold tighter to the idea of the hot hand than anyone — that it didn’t really exist. But he did come up with a method: He asked them to think about the feeling of “heating up.” Didn’t they sometimes get a feeling “creeping up their spines” that they were worried their hot streak would end? Any honest player would respond yeah, of course. Then Gilovich asked: And could they tell when it would end? No, the basketball player would respond. To Gilovich, that was the closest he could get to a eureka moment of understanding. His whole point was that past success doesn’t predict future success, and that hot streaks can end, randomly, on any shot attempt — there’s no there there.

In the 30-plus years since Gilovich’s paper — during which the result has been replicated on a number of occasions, using both in-game data and gym sessions — the so-called “hot-hand fallacy” has become a favorite teachable moment for professors (and smart-asses) everywhere. That’s partly because, during this period, there has been an explosion in research, both from behavioral economists like Tversky and from other subfields of psychology as well, on the aforementioned tendency of human perception and judgment to be led astray by various sorts of false signals. The hot-hand fallacy is a remarkably useful example for explaining that just because something feels true, doesn’t mean it is. Our lying eyes often deceive us, the logic goes. “Our intuitions are based on some very impressive psychological processes that can boil down lots of data for us in a split second, and it’s an impressive intellectual accomplishment,” said Gilovich. “But the whole reason we have the so-called scientific method is to guard against our unaided intuitions, [since they] don’t give us an accurate view of the world.”

The hot-hand fallacy, then, could be seen as a tidy case of careful empirical research beating out less-rigorous, popular folk-wisdom — an outcome any fan of Nate Silver can appreciate. Except … it turns out there probably is a hot hand, after all. It just took some new technology and an accidental statistical discovery to figure it out.

***

Gilovich, perhaps unsurprisingly, doesn’t think much of most of the efforts that researchers have undertaken to prove the hot-hand effect really exists. But he does acknowledge that two recent pieces of research have swayed him at least a little. The first was a paper presented at the 2014 MIT Sloan Sports Analytics Conference by Andrew Bocskocsky, John Ezekowitz, and Carolyn Stein.

The basic ideas behind the paper had been percolating since the three authors were undergraduate classmates at Harvard. In 2009 or 2010, Stein recalled, she and Ezekowitz were talking about Gilovich’s paper — specifically, the insistence of NBA players interviewed by the researchers that the hot hand did exist. They realized that the paper, epochal as it was, may have had an important shortcoming: It didn’t account for how players reacted to their own perceived hotness, nor for how they reacted to the hotness of those they were guarding on defense. “If they believe in [the hot hand], and they behave accordingly, if you make four shots in a row, that fifth shot you try will probably be from further away, and because the defender thinks you’re hot, he’s going to be covering you tighter,” said Stein. That could lead to a different interpretation of player’s field-goal percentages staying the same after a string of makes. “If your shooting percentage is staying the same, it almost seems like you must be becoming a better shooter,” she said. “Your shooting percentage is staying constant, but you’re attempting more difficult shots.” If they could control for shot difficulty, Stein and her classmates realized, they could test the hot-hand effect in a much more robust manner. But at the time, “short of watching hours of NBA games” and hand-coding the difficulty of each shot, there was just no way to get that kind of data.

SportVU changed that. Introduced league-wide at the start of the 2013–2014 season, it is a sophisticated system, based on Israeli military technology, that tracks every NBA player’s every movement during every second of every game, and translates all that frenetic activity into hard data. Since its launch, SportVU has revolutionized the way NBA geeks and front offices understand how American professional basketball is played. In Cambridge, Stein and Ezekowitz realized they could use SportVU data to control for shot difficulty, so they teamed up with Bocskocsky — who Stein described as a talented programmer, well-equipped to translate the system’s complex output into something easily crunchable — and got to work.

Once the young researchers completed their analysis, out popped a hot hand — well, a warm one, at least. All else being equal, they found that an NBA player who hits four shots in a row is about 2 percent more likely to hit his next one than one who has hit just two of his last four shots. This translates to about one percentage point. So if a player normally shoots 45 percent from the field, the SportVU data suggests that, after four makes, even controlling for shot difficulty, he will be a 46-percent shooter on his next shot.

It’s not a huge number, but it’s statistically robust, said Stein. She allowed that the SportVU data, while impressive, isn’t perfect. For one thing, it captures players’ locations as X-Y coordinates, but not where their arms are at a given moment, which can matter a lot in terms of how much a given shot is bothered. Stein also said that if she revisited the hot-hand debate, which she might, she would probably employ statistical bells and whistles she didn’t know about as an undergrad (and would make one particularly important adjustment, which we’ll get to in a bit). For now, though, Stein said she considers the jury to be somewhat out on the question of the hot hand. She’s more confident in the findings about how players react to the prospect of someone heating up, she said: The SportVU data showed pretty clearly that players who are heating up do, in fact, take tougher shots, and that the defenders who are on them do, in fact, play them tighter.

Two other researchers, who are behind the other work Gilovich mentioned, are a lot more sure of what they have found. And their key insight wasn’t empirical at all — rather, it came from a bolt-of-lightning discovery about the fabric of statistics. In a draft paper that began circulating last summer, Joshua B. Miller, a researcher at Bocconi University in Milan, and Adam Sanjurjo, at the University of Alicante in Spain, overturned, in one stroke, a huge amount of the past work that had been done on the hot hand.

They did so by pointing out a fact that, at first, sounds really unlikely. Let’s say you flip a coin four times and write down the outcomes. If you go back and look at your results and choose one of the heads flips at random, what is the probability that the next flip will also have been a heads? If you said .5, that’s understandable, since it makes perfect intuitive sense — but it turns out to be wrong.

Perhaps the easiest way to explain what’s going on here is to steal a table from one of Miller and Sanjurjo’s papers, which simply lists all the ways a series of four coin flips can go.

Ignore the fancy symbols and language in the caption (which comes straight from the paper), unless you’re statistically inclined. What matters is what happens when you add up all the sequences which start with an H, and then check to see how frequently the next flip is an H as compared to a T:

What the table shows, convincingly, is that due to what had been an as-yet-undiscovered quirk of the math, there is a cold hand built into the very laws of probability. The probability of getting tails on any individual flip is, of course, always 50 percent. But when you have a finite number of coin flips — or shot attempts, or any other probability-based event — the sequences with consecutive identical outcomes can only be arranged in so many ways. As a result, a given flip of heads is more likely to be followed by tails than by another heads.

Gilovich, Tversky, and Vallone didn’t account for this, nor did the many subsequent papers which seemed to confirm their results. Miller and Sanjurjo realized this could be a big deal: if a cold hand is baked into the stats, then if players are maintaining a consistent shooting percentage after a series of makes, that actually means they’re shooting at a higher percentage. So they dug up the data from the original Gilovich paper and adjusted for the quirk they had found. Suddenly, the estimate of the hot hand effect in the 26 collegiate players studied by Gilovich and his coauthors went from effectively zero to “around 12 percentage points,” said Miller in an email, meaning that, on average, when a player hit 3 shots in a row in the Gilovich sample the next shot was 12 percentage points more likely to be a hit than when the same player missed 3 shots in a row. That’s huge: in the NBA, a 35-percent shooter of three-pointers is middle of the pack, while a 47-percent shooter is one of the most deadly sharpshooters in the league. Miller and Sanjurjo also applied their fix to a bunch of other past data dealing with the hot-hand question, and while the effect size has been smaller at times, wherever Miller and Sanjurjo have tweaked old datasets, they have found hot hands.

It varies, as one might expect, from player to player, and some individual players don’t have any hot hand at all (Dwight Howard can shoot three-pointers all day, and it’s unlikely he will get into a rhythm). In a sample of data Miller and Sanjurjo collected from the 8 semipro players they asked to shoot in a gym, the average was about 3.5 - 4 percentage points. Miller and Sanjurjo also expanded upon one of the more well-known replications of the original Gilovich result, a paper published in 2003 in which Jonathan J. Koehler and Caryn Conley looked at four years worth of NBA three-point shooting contest from the 1990s and found no hot hand. When Miller and Sanjurjo applied the fix, and expanded the dataset to include all the contests from 1986 on to give them a bigger sample size to work with, they found a hot-hand effect of 8 percentage points, on average, among the participants (of course, players in the three-point contest are there because they are good at shooting from outside, so this could be a higher hot-hand effect than one might see otherwise).

As soon as Miller and Sanjurjo realized the potential importance of their statistical discovery, they temporarily set aside their new gym data and got to work circulating drafts of their research which focused on their fix and its potentially profound implications for the hot-hand debate. And as word percolated about what they had found, it became clear that even important names in the field were impressed. In an email, Miller said that a big moment for them was when they went into the very well-known Columbia statistician and blogger Andrew Gelman’s office and successfully convinced him that they had uncovered something legit. “Hey—guess what?” enthused Gelman in the headline of his subsequent blog post last July. “There really is a hot hand!”

Miller and Sanjurjo are working on revisions for a big paper that will tie all this together: the error they discovered, what happened when they applied it to old hot-hand data sets, and the new data they collected from the semipros. They hope it’ll be published soon, and they believe that once it is, it will be very hard for anyone to deny that the hot-hand effect is real.

***

Gilovich isn’t ready to acknowledge that the hot-hand fallacy is itself a fallacy. He sparked a micro-controversy among those with an interest in this topic when, in October of last year, a New York Times Sunday Review article on the hot-hand debate by George Johnson included a quote in which Gilovich played down the ramifications of Miller and Sanjurjo’s finding. “The larger the sample of data for a given player, the less of an issue this is,” Gilovich said in an email he sent to Johnson. “Because our samples were fairly large, I don’t believe this changes the original conclusions about the hot hand.

Later that day, Gelman published a blog post in which he dinged Gilovich for the quote, and for not acknowledging the hot hand. “It’s really too bad to hear Gilovich write this,” Gelman wrote. “Isn’t it always the way, though: People don’t like to admit that they made a mistake, even an honest mistake.” Gelman pointed out that the Gilovich, Tversky, and Vallone paper was still about two-thirds right. In addition to its more famous findings, it showed, apparently correctly, that people tend to overrate the hot-hand effect, and to discern random patterns where there aren’t any. As Gelman said, “Not bad for a 30-year-old paper and no cause for embarrassment on Gilovich’s part.”

I decided to check in with Gilovich on this: I sent him a link to a draft paper by the two researchers, and emphasized that they not only pointed out the error he and everyone else committed, but also reran the numbers from the original study to show that Gilovich’s college shooters appeared to demonstrate a hot hand once the statistical adjustment was made. “Like all good papers, it stimulates thought and I now have an idea about a new study to run that I think will inform the debate,” he said in response. But he’s still not there yet, he said — he still has some questions, though he does view the paper as “the most interesting challenge” to the original research.

Miller and Sanjurjo, meanwhile, plan on releasing their latest draft this week. We’ll see if they manage to pick up more converts — at the moment, they certainly appear to be on something of a hot streak.

How Researchers Discovered the Basketball ‘Hot Hand’