What a Chocolate Hoax Can Teach Us About Junk Science

By
Photo: Getty Images

You may have read (and vaguely believed) that dark chocolate is good for you thanks to its antioxidant content — after all, some diets even promote small amounts of chocolate as a component of a healthy eating plan. Last month, a study circulated suggesting that eating a dark-chocolate bar daily while on a low-carb diet could actually help you lose weight. The news was picked up by a handful of press outlets

The people behind that study included a journalist named John Bohannon, who, along with German documentary filmmakers, set out to exploit weaknesses in the scientific process to see if reporters would take the bait; now, on i09, they've admitted it was all a hoax.

As research on their film about how junk science can lead to lots of big headlines, they purposely commissioned a faulty scientific study, had a financial analyst crunch the results, set up a website for the fake Institute of Diet and Health, and listed the lead author as Johannes Bohannon, Ph.D. (Bohannon does have a Ph.D., but in molecular biology.)

They had a general practitioner run the real clinical trial on only 16 people: five men and 11 women. They split the participants into three groups: people following a low-carb diet, that diet plus 1.5 ounces of of 81 percent dark chocolate every day, and a control group who changed nothing. Here's what actually happened: Both diet groups lost five pounds, but the chocolate group lost 10 percent of the weight faster, had better cholesterol readings, and reported higher levels of well-being.

Yet Bohannon and his colleagues claimed the difference in weight loss was "statistically significant" — meaning there is a small chance of the finding being attributed to sheer luck. For instance, he points out that women's weight can fluctuate by five pounds simply because of their menstrual cycle. That's why researchers need to do studies on large groups of people and include a range of ages and genders. (He said most reputable journals flat-out reject trials with fewer than 30 subjects.) So he sought out non-reputable ones, submitting the study in April to 20 journals that are known to skip the peer-review process. The research was published less than two weeks later in the International Archives of Medicine for a fee of 600 euros. He then worked with a science PR friend to cook up a good press release — which didn't mention the number of subjects or exactly how much weight the two diet groups lost — and the news stories followed.

The German documentary is set to air next week, so now he's come forward and admits it was all a lie. Bohannon hoped that other reporters would question him on the details, but most didn't. "For far too long, the people who cover this beat have treated it like gossip, echoing whatever they find in press releases," he wrote on io9. "Hopefully our little experiment will make reporters and readers alike more skeptical."