Skip to content, or skip to search.

Skip to content, or skip to search.

The Spreadsheet Psychic

ShareThis

In this year’s Democratic primary, for example, the polls were all over the place. Before the Iowa caucuses on January 3, one poll had Clinton winning by nine, one had Clinton by two, and one had Obama by one. Obama won by seven. In the New Hampshire primary, five days later, one poll had Obama by thirteen and most others had him winning by eight or nine. Clinton won by three. Primaries are notoriously difficult to poll, because unlike in a general election, turnout is very unpredictable and people are much more likely to switch their choice at the last minute. As the primaries went on, however, Silver, who had been writing an anonymous diary for the liberal Website Daily Kos, made an observation about this year’s voters: While the polls were wobbling wildly state-to-state, the demographic groups supporting each candidate, and especially Clinton and Obama, were remarkably static. He wasn’t the only one who noticed this, of course—it was a major narrative theme of the campaign. One pundit summed it up by saying that Clinton had “the beer track”—blue-collar whites, Latinos, and seniors—while Barack had African-Americans and “the wine track”: young voters and educated whites.

Every other pundit, though, was doing what they’ve always done, i.e., following the polls. Silver decided to ignore the polls. Instead, he used this observation about demographics to create a model that took voting patterns from previous primaries and applied them to upcoming contests. No phone calls, no sample sizes, no guesswork. His crucial assumption, of course, was that each demographic group would vote in the same way, in the same percentages, as they had in other states in the past.

Like many of the so-called Moneyball breakthroughs in baseball, this was both a fairly intuitive conclusion and a radical break from conventional thinking. (In Moneyball, for example, the idea that players who get on base most often are the most valuable—which now seems kind of obvious—was a major breakthrough in strategy.) After all, political pundits love to talk about states as voting blocs—New Hampshire’s leaning this way, North Carolinians care about this, etc.—as though residency is the single most important factor in someone’s vote. Silver’s model more or less ignored residency. But his hunch about demographics proved correct: It’s how he called the Indiana and North Carolina results so accurately when the polls got them so wrong.

In baseball, a hit is a hit. With polls, a yes isn’t always a yes. Sometimes it’s more like “I don’t know, but I’ll say yes anyway to get you off the phone.”

The model didn’t always work throughout the primaries: Silver missed on Kentucky and South Dakota. But the model proved that the kind of creative swashbuckling that exemplifies Baseball Prospectus—the institutional obsession with questioning assumptions, even your own, even (or especially) to the point of heresy—could work when applied to politics as well. When I asked Joe Sheehan to sum up the Baseball Prospectus philosophy, he said simply, “Back up your argument. Because too many people are telling stories, as opposed to actually looking for the truth.”

Meanwhile, even as his primary model attracted attention, Silver was cooking up another idea. He figured there must be a better way to use the daily tracking polls to predict a candidate’s future, just as he’d once found a better way to use baseball stats to predict how many home runs a player might hit. His simple goal, as he explained on Daily Kos in late February, was to “assess state-by-state general-election polls in a probabilistic manner.” In other words, he wanted to find a way to use all those occasionally erratic, occasionally unreliable, occasionally misleading polls to tell him who would win the election in November, which at that point was over 250 days away.

It’s a tough business, being an oracle. Everyone cheers when you hit a bull’s-eye, but no one’s arrows fly true all the time. “Sometimes being more accurate means you’re getting things right 52 percent of the time instead of 50,” says Silver. “PECOTA is the most accurate projection system in baseball, but it’s the most accurate by half a percent.” That half-percent, though, makes all the difference. Silver’s work, in both baseball and politics, is about finding that slim advantage. “I hate the first 90 percent [of a solution],” he says. “What I want is that last 10 percent.”

As a kid, Silver was not a dork in a plastic bubble, as you might expect, gobbling stats and spouting figures. He grew up in East Lansing, Michigan, a typical baseball fan with a Tigers pennant on his bedroom wall. In person, talking baseball, he hardly comes off as a human computer; rather, he talks with the same bursts of enthusiasm familiar to any engaged fan in a sports bar. (And it’s been a rough year for Silver, fanwise: His home team, the Tigers, were an underperforming disaster, and his two adopted teams, the Cubs and the White Sox, were both quickly and tragically dispatched from the playoffs this year.)


Related:

Advertising
Current Issue
Subscribe to New York
Subscribe

Give a Gift

Advertising