The Spreadsheet Psychic

Photo: Melissa Ann Pinney

In a month when the Dow had its worst single-day plunge in over twenty years, when Lehman imploded, AIG faltered, and WaMu failed, when the word crisis became an everyday staple in newspaper headlines and the presidential race pulled close, then pulled apart, when the Chicago Cubs kicked off a playoff quest to win their first championship in 100 years (then got swept out in three straight games) and, for good measure, some scientists in an underground lab near the Swiss Alps fired up a Large Hadron Collider that some serious observers warned might create a black hole that would swallow up the Earth, it was comforting to sit down and have lunch in midtown with a man who can see the future. It’s not that Nate Silver is psychic, or even that he’s right all the time. He’s just proved very good, especially of late, at looking at what’s already happened and using that information to predict what will happen next.

Silver, who’s 30, thin, and lives in Chicago, had been flown to New York at the invitation of a hedge fund to give a talk. “They just said, ‘Why don’t you come in, talk about your models,’ ” he said with a shrug. “I’ll probably just take a lot of questions.” Silver doesn’t know all that much about high finance; these days, he’s spending most of his energy on his political Website, FiveThirtyEight (the total number of Electoral College votes), where he uses data analysis to track and interpret political polls and project the outcome of November’s election. The site earned some national recognition back in May, during the Democratic primaries, when almost every other commentator was celebrating Hillary Clinton’s resurgent momentum. Reading the polls, most pundits predicted she’d win Indiana by five points and noted she’d narrowed the gap with Obama in North Carolina to just eight.

Silver, who was writing anonymously as “Poblano” and receiving about 800 visits a day, disagreed with this consensus. He’d broken the numbers down demographically and come up with a much less encouraging outcome for Clinton: a two-point squeaker in Indiana, and a seventeen-point drubbing in North Carolina. On the night of the primaries, Clinton took Indiana by one and lost North Carolina by fifteen. The national pundits were doubly shocked: one, because the results were so divergent from the polls, and two, because some guy named after a chili pepper had predicted the outcome better than anyone else.

Silver’s site now gets about 600,000 visits daily. And as more and more people started wondering who he was, in May, Silver decided to unmask himself. To most people, the fact that Poblano turned out to be a guy named Nate Silver meant nothing. But to anyone who follows baseball seriously, this was like finding out that a guy anonymously running a high-fashion Website turned out to be Howard Cosell. At his day job, Silver works for Baseball Prospectus, a loosely organized think tank that, in the last ten years, has revolutionized the interpretation of baseball stats. Furthermore, Silver himself invented a system called PECOTA, an algorithm for predicting future performance by baseball players and teams. (It stands for “player empirical comparison and optimization test algorithm,” but is named, with a wink, after the mediocre Kansas City Royals infielder Bill Pecota.) Baseball Prospectus has a reputation in sports-media circles for being unfailingly rigorous, occasionally arrogant, and almost always correct.

This season, for example, the PECOTA system predicted that the Tampa Bay Rays would win 90 games. This seemed bold, even amusing, given that the Rays were arguably the worst team in baseball. In 2007, they’d lost 96 games. They’d finished last in all but one season of their ten-year existence. (In 2004, they finished fourth.) They had some young talent, sure, but most people, even those in the Rays’ front office, thought that if the team simply managed to win more games than it lost, that would represent a quantum leap.

PECOTA, however, saw it differently. PECOTA recognized that the past Rays weren’t a hopelessly bad team so much as a good team hampered by a few fixable problems—which, thanks to some key off-season changes, had been largely remedied. Silver argued on the Baseball Prospectus Website that the long-suffering team had finally “decided to transform themselves from a sort of hedge fund for undervalued assets into a real, functional baseball club.”

PECOTA, as it turns out, wasn’t exactly right. The Rays didn’t win 90 games this year. They won 97 games and are currently playing the Red Sox for the American League championship.

So, Nate Silver: What happens next?

Sports and politics offer several obvious parallels. Both involve a competition, essentially between two teams. Both involve reams of statistical data available for devotees to sort through—or, more commonly, for intermediary experts to sort through, analyze, and then interpret for you. In baseball, these stats track player performance—how many hits a player gets, and when, and against what kind of pitchers—while in politics, the data tracks voter preferences. Who do you like and why? What kind of choice are you likely to make on Election Day? These stats, on their face, seem pretty straightforward. If a hitter hits .300, he’s valuable. If Obama opens up a six-point national lead, he’s in good shape.

And yet in both sports and politics, there’s an industry built around studying this data, making up stories about it, and then trying to sell those stories to you. For example: A-Rod, for all his greatness, can’t deliver in the clutch. Obama, for all his charisma, has struggled to connect with white working-class voters. The Mets are a bunch of chokers. This election, it’s all about the hockey moms.

As a result, in baseball and, now, politics, there exists a small subculture of counterexperts: People who argue against these conventional story lines using new interpretations of the raw data to make their case. In baseball, this counterculture has been growing for roughly 30 years and can be traced, improbably, to one man: Bill James, a cranky Midwesterner who started writing articles about baseball while working the night shift as a security guard at a pork-and-beans factory in Lawrence, Kansas. In 1977, he published a photocopied newsletter called Baseball Abstract, which found a cult following that later blossomed into a national audience. By the late eighties, he was hailed as the founder of “sabermetrics”—a new field dedicated to better analysis of baseball stats—and the father of a revolution. Once considered a Unabomber-style outcast, James now consults for the Red Sox.

And in the nineties, the study of sabermetrics exploded. For starters, the development of so-called fantasy baseball—a game in which fans draft a team of real players, then compete with each other based on the players’ on-field success—created a huge new market for performance projections. (If you want to win your fantasy league, you care a lot less about who hit 40 home runs last year than you do about who’ll hit 40 next year.) And the advent of the Internet allowed fans unprecedented access to stats, both raw and packaged by various experts. Then, in 2003, Michael Lewis wrote Moneyball, a best-selling book that valorized Billy Beane, the general manager of the Oakland A’s, for using some of these new insights to overcome the financial advantage of richer teams.

In the midst of all this, in 1996, Baseball Prospectus was born. Founded by five baseball fans who met each other online, the BP crew are like the bratty children of Bill James, adding a new level of analytical sophistication to his contrarian philosophy. “When he started, Bill James had to actually call up teams and ask for their information,” says Joe Sheehan, one of BP’s founders. “Now we’re able to download databases. We can do things with one-tenth the effort and a hundred times the available data.” Also, whereas James used stats to explain what a player had done, BP uses stats to predict what a player might do. As a result, BP has built a small but successful empire of smarty-pants, with a Website, syndicated columns, and most prominently, a preseason annual full of player projections.

As an avid fantasy player, I’ve spent long hours combing through the pages of the Baseball Prospectus book. It arrives each season huge, heavy, and intimidatingly dense. (“It’s longer than Moby-Dick and heavier than War and Peace,” jokes Steven Goldman, one of the editors.) Last year’s edition weighed in at 605 oversize pages and offered essays like “Expanding the Cannon: Quantifying the Impact of Outfield Throwing Arms.” The writing is lively and funny, nerd-nip for baseball obsessives. But the book, with its extensive charts and graphs, its talk of VORP (value over replacement player) and SNLVAR (support-neutral lineup adjusted value added above replacement), its SAC percent and EqH9 (oh, never mind), can also make you feel like you’re reading a Ph.D. dissertation in statistics or a book by Dr. Seuss—or both, at the same time.

In fact, the work of stat hounds in general, and of Baseball Prospectus in particular, is so obviously the product of high-wattage brainpower and creativity that you can’t help but occasionally wonder: What if someone applied all this energy to something that actually mattered, like, I don’t know, politics?

Last year, at the start of an unusually unpredictable election season, Nate Silver began to wonder the same thing.

As stats are to baseball, polls are to politics; i.e., the basic numeric measurement of how things have gone in the past and how they might go in the future. Ask any pollster, though, and he will tell you that polls aren’t meant to be used as predictive tools—they’re simply a rough measure of where the electorate stands at a given moment. As pollster John Zogby put it to me, “We take snapshots. And when you take many snapshots in a row, you get motion pictures.”

But unlike baseball stats, polls are a notoriously imprecise measurement. In baseball, at least, a hit is a hit. With polls, a yes isn’t always a yes. Sometimes it’s more like a “maybe,” or a “yes, until I change my mind,” or an “I don’t know, but I’ll say yes anyway to get you off the phone.” Poll results can vary dramatically based on what you’re asking, who you’re asking, how you’re asking, and how many people decide to answer you. Three different polls were conducted recently asking Americans how they felt about the federal $700 billion bailout. They all asked the question in slightly different ways and the results were essentially useless: One poll had people in favor of the bailout 57 to 30 percent, one had them against it 55 to 31, and one was basically split down the middle. In other words, polls are, at best, educated guesses. But if there’s one thing Nate Silver loves to make, it’s an educated guess.

In this year’s Democratic primary, for example, the polls were all over the place. Before the Iowa caucuses on January 3, one poll had Clinton winning by nine, one had Clinton by two, and one had Obama by one. Obama won by seven. In the New Hampshire primary, five days later, one poll had Obama by thirteen and most others had him winning by eight or nine. Clinton won by three. Primaries are notoriously difficult to poll, because unlike in a general election, turnout is very unpredictable and people are much more likely to switch their choice at the last minute. As the primaries went on, however, Silver, who had been writing an anonymous diary for the liberal Website Daily Kos, made an observation about this year’s voters: While the polls were wobbling wildly state-to-state, the demographic groups supporting each candidate, and especially Clinton and Obama, were remarkably static. He wasn’t the only one who noticed this, of course—it was a major narrative theme of the campaign. One pundit summed it up by saying that Clinton had “the beer track”—blue-collar whites, Latinos, and seniors—while Barack had African-Americans and “the wine track”: young voters and educated whites.

Every other pundit, though, was doing what they’ve always done, i.e., following the polls. Silver decided to ignore the polls. Instead, he used this observation about demographics to create a model that took voting patterns from previous primaries and applied them to upcoming contests. No phone calls, no sample sizes, no guesswork. His crucial assumption, of course, was that each demographic group would vote in the same way, in the same percentages, as they had in other states in the past.

Like many of the so-called Moneyball breakthroughs in baseball, this was both a fairly intuitive conclusion and a radical break from conventional thinking. (In Moneyball, for example, the idea that players who get on base most often are the most valuable—which now seems kind of obvious—was a major breakthrough in strategy.) After all, political pundits love to talk about states as voting blocs—New Hampshire’s leaning this way, North Carolinians care about this, etc.—as though residency is the single most important factor in someone’s vote. Silver’s model more or less ignored residency. But his hunch about demographics proved correct: It’s how he called the Indiana and North Carolina results so accurately when the polls got them so wrong.

In baseball, a hit is a hit. With polls, a yes isn’t always a yes. Sometimes it’s more like “I don’t know, but I’ll say yes anyway to get you off the phone.”

The model didn’t always work throughout the primaries: Silver missed on Kentucky and South Dakota. But the model proved that the kind of creative swashbuckling that exemplifies Baseball Prospectus—the institutional obsession with questioning assumptions, even your own, even (or especially) to the point of heresy—could work when applied to politics as well. When I asked Joe Sheehan to sum up the Baseball Prospectus philosophy, he said simply, “Back up your argument. Because too many people are telling stories, as opposed to actually looking for the truth.”

Meanwhile, even as his primary model attracted attention, Silver was cooking up another idea. He figured there must be a better way to use the daily tracking polls to predict a candidate’s future, just as he’d once found a better way to use baseball stats to predict how many home runs a player might hit. His simple goal, as he explained on Daily Kos in late February, was to “assess state-by-state general-election polls in a probabilistic manner.” In other words, he wanted to find a way to use all those occasionally erratic, occasionally unreliable, occasionally misleading polls to tell him who would win the election in November, which at that point was over 250 days away.

It’s a tough business, being an oracle. Everyone cheers when you hit a bull’s-eye, but no one’s arrows fly true all the time. “Sometimes being more accurate means you’re getting things right 52 percent of the time instead of 50,” says Silver. “PECOTA is the most accurate projection system in baseball, but it’s the most accurate by half a percent.” That half-percent, though, makes all the difference. Silver’s work, in both baseball and politics, is about finding that slim advantage. “I hate the first 90 percent [of a solution],” he says. “What I want is that last 10 percent.”

As a kid, Silver was not a dork in a plastic bubble, as you might expect, gobbling stats and spouting figures. He grew up in East Lansing, Michigan, a typical baseball fan with a Tigers pennant on his bedroom wall. In person, talking baseball, he hardly comes off as a human computer; rather, he talks with the same bursts of enthusiasm familiar to any engaged fan in a sports bar. (And it’s been a rough year for Silver, fanwise: His home team, the Tigers, were an underperforming disaster, and his two adopted teams, the Cubs and the White Sox, were both quickly and tragically dispatched from the playoffs this year.)

His approach to politics is similar—he’s an engaged fan. He unapologetically roots for Obama. One of his early posts as a contributor to Daily Kos, titled “I Got Dinner With Barack Obama,” recounts with gee-whiz wonder a chance sighting of Obama during Silver’s birthday outing at a Mexican restaurant. (“At first I was pissed off with my friend for not doing more to alert me,” he wrote, “but if I’d had more advance warning, I’d probably have done something stupid like scream ‘Fired up!,’ which would have been embarrassing in retrospect.”) But he doesn’t try to pummel you with numbers to prove his argument, like a typical hot-blooded partisan. Instead, on his site, he exhibits the cool confidence of someone who’s simply used to knowing his stuff better than anyone else in the room.

Not that he can’t pick, and win, a good fight. In a post on Daily Kos last December titled “Is a Bad Poll Better Than No Poll at All?” Silver singled out a few pollsters, particularly American Research Group, to show that their consistently off-base numbers will skew polling averages so severely that they harm one’s results. Of ARG he wrote, “They have a track record of rolling out some polls that are completely different from anybody else in the race, and when they do, they are almost always wrong.”

Dick Bennett, the pollster for ARG, responded by posting items on his Website such as “Nate Silver is Wrong Again,” and mocking FiveThirtyEight’s slogan (“Electoral Projections Done Right”) in a tone that echoed current political attack ads. (“So much for electoral projections done right.”) Then, in June, Silver posted an open letter to Bennett, which read, “It has been a long and hard-fought primary campaign. We’ve both had our share of successes, and made our share of mistakes. Granted, you made a few more than I did”—and in that last sentence, every word but “Granted” was a separate link to an ARG polling misfire. Then Silver challenged Bennett to a contest, in which each site would call the elections results, state-by-state, with a $1,000 bounty per state. Bennett never took him up on it, and this is what he has to say about Silver now: “What he does is different than what I do. There’s a market for that. There’s also a lady down the street who will read your palm.”

As a high-schooler, Silver was a state-champion debater, though he claims to be only a so-so public speaker. I asked if he ever thought of becoming a baseball G.M., like the 34-year-old boy wonder (and sabermetrics proponent) Theo Epstein, who took the Red Sox to two championships (and counting). Silver said, “The people who do that are very talented. They’re very smart, very polished. And I’m not much of a schmoozer. With Baseball Prospectus, you still have a voice and it’s influential. I prefer shaping public opinion, I suppose.”

After earning a degree in economics from the University of Chicago, Silver took a corporate job at a consulting firm but found it boring. He seems endlessly distractable; for example, in 2007, he started a Website, The Burrito Bracket, that rated Mexican restaurants in Wicker Park. (“Each week, I will be visiting two restaurants and having the same item of food [carne asada burritos, for example] at each one. The restaurant that provides the superior experience advances to the next round of the bracket.”) For a while, he was supplementing his income playing online poker, and even earned six figures one year, but eventually he quit. “For a while, there was a lot of money to be made, but you kind of eliminate one sucker at a time,” he says, “until finally you’re the sucker.” After he developed PECOTA and joined Baseball Prospectus, he turned his eye to political analyses, thus finding another field in which to identify suckers and eliminate them one at a time.

In concocting FiveThirtyEight, Silver decided the best way to read the polls was to put them all together, with the idea that averaging ten polls would give you a better result than trying to pick out the best one. Again, he wasn’t the first person to do this—other sites like RealClearPolitics and Pollster offer the same service. But, as Silver told me, “Sometimes the answer is in looking at other alternatives that exist in the market and saying, ‘They have the right idea, but they’re not doing it quite the right way.’ ”

Silver wanted to average the polls, but he wanted the polls that were more accurate to count for more, while the bad polls would be discounted. Other sites, like RealClearPolitics, treat every poll as equal. Silver knows that some polls are simply better than others. Yet it’s hard to know how accurate a general-election poll is before the actual election.

Photo: Jamie Squire/Getty Images

So he came up with a system that predicts a pollster’s future performance based on how good it’s been in the past. In finding his average, Silver weights each poll differently—ranking them according to his own statistic, PIE (pollster-introduced error)—based on a number of factors, including its track record and its methodology. One advantage of this system is that, during the primaries, the system actually got smarter. Because each time a poll performed well in a primary, its ranking improved.

For the general election, this gets trickier, since you have polls coming out every single day and you can’t know which ones are getting it right until Election Day. You can, however, weigh these new polls based on the pollster’s history, the poll’s sample size, and how recently the poll was conducted. You can also track trends over time and use these trend lines to forecast where things will end up on November 4. You can also, as Silver has done, analyze all the presidential polling data back to 1952, looking for information as to what is likely to happen next. (For example, how much the polls are likely to tighten in the last month of the race, which they traditionally do.) You can also run 10,000 computer simulations of the election every day based on your poll projections. (Think of this as sort of like that scene at the end of WarGames, where the computer blurs through every possible nuclear-war scenario.) As of October 8, the day after the town-hall debate, Silver’s simulations had Obama winning the election 90 percent of the time.

“Literally, the composition of the North Carolina electorate is different than it was six months ago.”

All of which is very seductive (and heartening to Obamaphiles), especially when you see it laid out on Silver’s site, with its pleasing graphs, compelling charts, and graphically vibrant electoral maps. But in essence, Silver’s whole undertaking is premised on breaking the first rule of reading polls: He’s assuming—in his complex, elegant, partially proprietary and yet-to-be-entirely-validated way—that today’s polls can predict tomorrow’s election. Rival statisticians, in particular Sam Wang at Princeton (who runs his own poll-aggregation blog), have criticized Silver, arguing that polls can’t and shouldn’t be used as a crystal ball. Other critics have argued that the idea of a projection model is inherently flawed because it can’t predict the unpredictable—for example, before the financial meltdown and McCain’s campaign-suspension stunt, the polls were much tighter and Silver’s electoral map had McCain on top.

Silver agrees, to a point, comparing daily polls, especially ones that come out months before the election, to “a 162-game baseball season, where one individual win or loss doesn’t really tell you that much about the ultimate outcome.” But for him, you have to use polls to predict the future—that’s the whole point. Unlike other electoral projection maps, Silver calls each state for one candidate or the other—there are no undecideds—because the goal is to approximate what the map will look like after Election Day. “I think the entire value of the exercise is in predicting the outcome in November,” Silver says, in a response to Wang. “What would happen in an election held today is a largely meaningless question.”

And Silver is right. The truth is that everyone reads polls this way. When you pick up the paper and see McCain up by three or Obama up by six, you assume that means that candidate is on his way to a win. Silver’s goal with FiveThirtyEight, then, is to simply do what everyone does, but do it better—to read the polls in such a way that those assumptions we all naturally make will actually turn out to be true.

For all the numbers and nuance, the adjustments and algorithms, there’s really only one stark, looming, unambiguous test for the political prognosticator. “Pretty soon, there’s going to be an Election Day,” says pollster J. Ann Selzer of Selzer & Co. “And you’re either going to be golden or a goat.” (Selzer, this season, has consistently been golden, calling the Iowa caucus flawlessly, which is partly why Selzer & Co. is Silver’s top-ranked pollster on FiveThirtyEight.)

Like everyone else calling this election, Silver’s day of reckoning will come on November 4. In the meantime, he’s become an increasingly confident commentator, growing into a national role as a calming anti-pundit among the white noise of partisan spin. FiveThirtyEight not only tracks numbers, but features field reports from the 50 states by Silver’s colleague, Sean Quinn, and the documentary photographer Brett Marty. And Silver posts to his blog several times daily, spotting and dissecting surprising trends or aberrations, such as a recent poll in Minnesota that handed McCain a sudden one-point lead. Normally, you’d expect a liberal-leaning commentator to read such a result and blame bias, or error, or voodoo. Silver, however, poked around and determined that McCain had been recently outspending Obama three-to-one in Minnesota, making it the only state in the country where he was out-advertising Obama. “So McCain may literally have bought his way into a competitive race,” Silver wrote. “So, yes, you can beat a state into submission if you really want to … But whether it’s been a good use of resources, we’ll have to see.”

I caught up with Silver on the phone recently, on a day when he’d just arrived back in Chicago from New York, having appeared the night before on The Colbert Report. We talked about Obama’s widening lead over McCain, and the remaining undecideds—a group, he says, that’s “mostly older rather than younger, more religious than not, and a lot of Independents, which is typical”—and how a lot of formerly solid red states now look like good bets to turn blue. “My pet theory is that these states along the Atlantic coast, like Virginia and North Carolina, are growing so fast that you have a lot of newly registered Democrats. That universe now contains people that it didn’t a month ago. Literally, the composition of the North Carolina electorate is different than it was six months ago. They update their registration figures on a weekly basis, and last week the Democrats registered about 16,000 new voters—which would represent one percent of their turnout from 2004.” And this year, at least, for all the surprises and Sturm und Drang, the electorate appears to be acting rationally. “The conventional punditry underestimates voters. The voters are pretty smart. They picked two very strong candidates. Even if McCain’s in trouble now, if it had been Fred Thompson, he might have conceded already.”

Even as he updates his projections and runs his 10,000 simulations a day, Silver wonders if maybe we don’t yet know what the narrative of this election will be. In September, he wrote a post on Obama’s extensive “ground game”—his efforts to set up outposts and register new voters, which have far outstripped McCain’s—and suggested, “Suppose that, because of their ground efforts, the Obama campaign is 5 percent more efficient at turning out its vote than the McCain campaign on Election Day … The implications of this would be enormous—a net of two to three points in each and every swing state—but we know zip, zilch, nada at this stage about their ultimate effect.”

This is the paradoxical spirit of the stat-heads: They can be arrogant, sure, and even bullying as they charge forward, brandishing their spreadsheets. But they are just as happy to prove themselves wrong as they are to debunk anyone else. This, I think, is at the heart of Silver’s appeal. (From a recent random Facebook status update: “I am an empiricist and I trust Nate Silver. Read it and chill.”) “Nate’s medium-term goal is to accomplish what we’ve accomplished at Prospectus—to change the conversation,” says Sheehan. “And Nate’s growth curve has been much sharper than ours ever was. He’s crammed about five years’ of BP’s growth into five months. And if you get good enough arguments out there, if you do your work well enough, then other people have to do their work better. Nate’s watched that at Prospectus. But FiveThirtyEight can do things for America that Baseball Prospectus never could.”

FiveThirtyEight is the product of a movement, but also of a moment. The political media is polarized. Cable anchors choke on their own spin. The red states and blue states act like the Jets versus the Sharks—they don’t trust us and we don’t trust them. So we all rail against the enemy in the echo chambers of comment boards, retreating to the bomb-shelter safety of partisan blogs.

It’s not that Silver is objective or impartial—he’s not. He’s still that young guy who almost yelled “Fired up!” across a crowded Mexican restaurant. But his ultimate goal is simple and nonpartisan: to build a better expert. Sure, he’ll be disappointed if Obama loses. But he also says, “If Obama does lose, I think it’s healthy to try and understand why, rather than just kicking and throwing things.” If he ever decides to run for office, that wouldn’t make for a bad slogan. Nate Silver: More understanding. Less kicking and throwing things.

The Spreadsheet Psychic