And yet in both sports and politics, there’s an industry built around studying this data, making up stories about it, and then trying to sell those stories to you. For example: A-Rod, for all his greatness, can’t deliver in the clutch. Obama, for all his charisma, has struggled to connect with white working-class voters. The Mets are a bunch of chokers. This election, it’s all about the hockey moms.
As a result, in baseball and, now, politics, there exists a small subculture of counterexperts: People who argue against these conventional story lines using new interpretations of the raw data to make their case. In baseball, this counterculture has been growing for roughly 30 years and can be traced, improbably, to one man: Bill James, a cranky Midwesterner who started writing articles about baseball while working the night shift as a security guard at a pork-and-beans factory in Lawrence, Kansas. In 1977, he published a photocopied newsletter called Baseball Abstract, which found a cult following that later blossomed into a national audience. By the late eighties, he was hailed as the founder of “sabermetrics”—a new field dedicated to better analysis of baseball stats—and the father of a revolution. Once considered a Unabomber-style outcast, James now consults for the Red Sox.
And in the nineties, the study of sabermetrics exploded. For starters, the development of so-called fantasy baseball—a game in which fans draft a team of real players, then compete with each other based on the players’ on-field success—created a huge new market for performance projections. (If you want to win your fantasy league, you care a lot less about who hit 40 home runs last year than you do about who’ll hit 40 next year.) And the advent of the Internet allowed fans unprecedented access to stats, both raw and packaged by various experts. Then, in 2003, Michael Lewis wrote Moneyball, a best-selling book that valorized Billy Beane, the general manager of the Oakland A’s, for using some of these new insights to overcome the financial advantage of richer teams.
In the midst of all this, in 1996, Baseball Prospectus was born. Founded by five baseball fans who met each other online, the BP crew are like the bratty children of Bill James, adding a new level of analytical sophistication to his contrarian philosophy. “When he started, Bill James had to actually call up teams and ask for their information,” says Joe Sheehan, one of BP’s founders. “Now we’re able to download databases. We can do things with one-tenth the effort and a hundred times the available data.” Also, whereas James used stats to explain what a player had done, BP uses stats to predict what a player might do. As a result, BP has built a small but successful empire of smarty-pants, with a Website, syndicated columns, and most prominently, a preseason annual full of player projections.
As an avid fantasy player, I’ve spent long hours combing through the pages of the Baseball Prospectus book. It arrives each season huge, heavy, and intimidatingly dense. (“It’s longer than Moby-Dick and heavier than War and Peace,” jokes Steven Goldman, one of the editors.) Last year’s edition weighed in at 605 oversize pages and offered essays like “Expanding the Cannon: Quantifying the Impact of Outfield Throwing Arms.” The writing is lively and funny, nerd-nip for baseball obsessives. But the book, with its extensive charts and graphs, its talk of VORP (value over replacement player) and SNLVAR (support-neutral lineup adjusted value added above replacement), its SAC percent and EqH9 (oh, never mind), can also make you feel like you’re reading a Ph.D. dissertation in statistics or a book by Dr. Seuss—or both, at the same time.
In fact, the work of stat hounds in general, and of Baseball Prospectus in particular, is so obviously the product of high-wattage brainpower and creativity that you can’t help but occasionally wonder: What if someone applied all this energy to something that actually mattered, like, I don’t know, politics?
Last year, at the start of an unusually unpredictable election season, Nate Silver began to wonder the same thing.
As stats are to baseball, polls are to politics; i.e., the basic numeric measurement of how things have gone in the past and how they might go in the future. Ask any pollster, though, and he will tell you that polls aren’t meant to be used as predictive tools—they’re simply a rough measure of where the electorate stands at a given moment. As pollster John Zogby put it to me, “We take snapshots. And when you take many snapshots in a row, you get motion pictures.”
But unlike baseball stats, polls are a notoriously imprecise measurement. In baseball, at least, a hit is a hit. With polls, a yes isn’t always a yes. Sometimes it’s more like a “maybe,” or a “yes, until I change my mind,” or an “I don’t know, but I’ll say yes anyway to get you off the phone.” Poll results can vary dramatically based on what you’re asking, who you’re asking, how you’re asking, and how many people decide to answer you. Three different polls were conducted recently asking Americans how they felt about the federal $700 billion bailout. They all asked the question in slightly different ways and the results were essentially useless: One poll had people in favor of the bailout 57 to 30 percent, one had them against it 55 to 31, and one was basically split down the middle. In other words, polls are, at best, educated guesses. But if there’s one thing Nate Silver loves to make, it’s an educated guess.