Skip to content, or skip to search.

Skip to content, or skip to search.

The Spreadsheet Psychic

Nate Silver is a number-crunching prodigy who went from correctly forecasting baseball games to correctly forecasting presidential primaries—and perhaps the election itself. Here’s how he built a better crystal ball.

ShareThis

In a month when the Dow had its worst single-day plunge in over twenty years, when Lehman imploded, AIG faltered, and WaMu failed, when the word crisis became an everyday staple in newspaper headlines and the presidential race pulled close, then pulled apart, when the Chicago Cubs kicked off a playoff quest to win their first championship in 100 years (then got swept out in three straight games) and, for good measure, some scientists in an underground lab near the Swiss Alps fired up a Large Hadron Collider that some serious observers warned might create a black hole that would swallow up the Earth, it was comforting to sit down and have lunch in midtown with a man who can see the future. It’s not that Nate Silver is psychic, or even that he’s right all the time. He’s just proved very good, especially of late, at looking at what’s already happened and using that information to predict what will happen next.

Silver, who’s 30, thin, and lives in Chicago, had been flown to New York at the invitation of a hedge fund to give a talk. “They just said, ‘Why don’t you come in, talk about your models,’ ” he said with a shrug. “I’ll probably just take a lot of questions.” Silver doesn’t know all that much about high finance; these days, he’s spending most of his energy on his political Website, FiveThirtyEight (the total number of Electoral College votes), where he uses data analysis to track and interpret political polls and project the outcome of November’s election. The site earned some national recognition back in May, during the Democratic primaries, when almost every other commentator was celebrating Hillary Clinton’s resurgent momentum. Reading the polls, most pundits predicted she’d win Indiana by five points and noted she’d narrowed the gap with Obama in North Carolina to just eight.

Silver, who was writing anonymously as “Poblano” and receiving about 800 visits a day, disagreed with this consensus. He’d broken the numbers down demographically and come up with a much less encouraging outcome for Clinton: a two-point squeaker in Indiana, and a seventeen-point drubbing in North Carolina. On the night of the primaries, Clinton took Indiana by one and lost North Carolina by fifteen. The national pundits were doubly shocked: one, because the results were so divergent from the polls, and two, because some guy named after a chili pepper had predicted the outcome better than anyone else.

Silver’s site now gets about 600,000 visits daily. And as more and more people started wondering who he was, in May, Silver decided to unmask himself. To most people, the fact that Poblano turned out to be a guy named Nate Silver meant nothing. But to anyone who follows baseball seriously, this was like finding out that a guy anonymously running a high-fashion Website turned out to be Howard Cosell. At his day job, Silver works for Baseball Prospectus, a loosely organized think tank that, in the last ten years, has revolutionized the interpretation of baseball stats. Furthermore, Silver himself invented a system called PECOTA, an algorithm for predicting future performance by baseball players and teams. (It stands for “player empirical comparison and optimization test algorithm,” but is named, with a wink, after the mediocre Kansas City Royals infielder Bill Pecota.) Baseball Prospectus has a reputation in sports-media circles for being unfailingly rigorous, occasionally arrogant, and almost always correct.

This season, for example, the PECOTA system predicted that the Tampa Bay Rays would win 90 games. This seemed bold, even amusing, given that the Rays were arguably the worst team in baseball. In 2007, they’d lost 96 games. They’d finished last in all but one season of their ten-year existence. (In 2004, they finished fourth.) They had some young talent, sure, but most people, even those in the Rays’ front office, thought that if the team simply managed to win more games than it lost, that would represent a quantum leap.

PECOTA, however, saw it differently. PECOTA recognized that the past Rays weren’t a hopelessly bad team so much as a good team hampered by a few fixable problems—which, thanks to some key off-season changes, had been largely remedied. Silver argued on the Baseball Prospectus Website that the long-suffering team had finally “decided to transform themselves from a sort of hedge fund for undervalued assets into a real, functional baseball club.”

PECOTA, as it turns out, wasn’t exactly right. The Rays didn’t win 90 games this year. They won 97 games and are currently playing the Red Sox for the American League championship.

So, Nate Silver: What happens next?

Sports and politics offer several obvious parallels. Both involve a competition, essentially between two teams. Both involve reams of statistical data available for devotees to sort through—or, more commonly, for intermediary experts to sort through, analyze, and then interpret for you. In baseball, these stats track player performance—how many hits a player gets, and when, and against what kind of pitchers—while in politics, the data tracks voter preferences. Who do you like and why? What kind of choice are you likely to make on Election Day? These stats, on their face, seem pretty straightforward. If a hitter hits .300, he’s valuable. If Obama opens up a six-point national lead, he’s in good shape.


Related:

Advertising
Current Issue
Subscribe to New York
Subscribe

Give a Gift

Advertising