Vegetables don’t want to be eaten and other lessons from Britain's organic food war

Vegetables don’t want to be eaten and other lessons from Britain's organic food war. By Trevor Butterworth
STATS.org, August 4, 2009

A major British study recently turned conventional wisdom on organic food on its head, triggering a war between science writers, reporters, activists and chefs. Was it a “myth” that organic produce was nutritionally superior to conventional food – or did an agency with an agenda cook up some flawed science to appease big agribusiness?

Bad Science, a book by Ben Goldacre, has become an unusual bestseller in the British Isles, powering its way into the higher reaches of nationally and locally compiled bestseller lists since its publication in the Fall of 2008. Goldacre is a doctor for Britain’s National Health Service (though he plays this down on the grounds that arguments from positions of expertise are often self-defeating with the public), and the book is a continuum of a column by the same name he writes for the left-leaning Guardian newspaper. The Royal Statistical Society awarded him first prize in their inaugural 2007 award for statistical excellence in journalism, and the British Medical Journal, in reviewing “Bad Science,” declared that Goldacre “is fighting what sometimes seems like a one man battle against a tide of pseudoscience and an army of quacks,” and that the country was lucky to have him.

The book’s popularity seems to speak to increasing consumer frustration with information promoted as “scientific,” whether in news stories, government pronouncements, or advertisements for pills and panaceas, and to the hopeful sign that people want to know – or want someone to examine on their behalf – the underlying principles that determine whether such research claims can be considered reliable or unreliable.

These principles came to the forefront in Britain last week – and the rest of the world – with the publication of a new study claiming that there was no reliable evidence that organically-produced food was better, nutritionally, than conventionally-produced food.
The study, “Nutritional quality of organic foods: a systematic review,” was funded by Britain’s Food Standards Agency (FSA), and conducted by researchers from the Nutrition and Public Health Intervention Research Unit at the Department of Epidemiology and Population Health, London School of Hygiene & Tropical Medicine; it was published in the peer-reviewed American Journal of Clinical Nutrition.

Senior reporter Karen McVeigh told readers of the Guardian in the opening paragraph of its news report on June 30 that the review’s “conclusions have been called into question by experts and organic food campaigners,” and more than half of the article focused on criticisms of the study, namely that the researchers had been “selective in the extreme,” used “questionable methodology, were contradicted by numerous other studies, and neglected to mention the risks of pesticides and fertilizers in conventional farming. As Peter Melchett, policy director at the Soil Association, the non-profit that advocates for and certifies organic farming in Britain, told the paper, “The review rejected almost all of the existing studies of comparisons between organic and non-organic nutritional differences.”

Given the way the story was reported, with the validity of the study immediately questioned in the opening paragraph (and with the paper being home to Goldacre’s column), readers could have been forgiven for concluding that the FSA had indeed funded some dodgy research, peer-review notwithstanding.

But at the Guardian’s sister, Sunday paper, The Observer, science editor, Robin McKie, defended the study. “[I]t is certainly not the work of dogmatic and intractably hostile opponents of the cause,” he wrote before weighing in on one of the key criticisms of the study, namely, that it did not take pesticide and fertilizer residues on conventional food into consideration.

“For a start, the idea that organic fruit and veg contain no harmful chemicals compared with non-organic produce is simply wrong, scientists argue. Certainly, there are pesticide residues in the latter but there is no evidence these are cumulatively harmful.

More to the point, organic crops - because they are untreated with chemicals - have correspondingly high levels of natural fungal toxins. Thus they balance out: artificial pesticide residues in non-organic crops, natural fungal toxins in organic.”

As Professor Ottoline Leyser, a molecular biologist at York University told McKie:

“People think that the more natural something is, the better it is for them. That is simply not the case. In fact, it is the opposite that is the true: the closer a plant is to its natural state, the more likely it is that it will poison you. Naturally, plants do not want to be eaten, so we have spent 10,000 years developing agriculture and breeding out harmful traits from crops. ‘Natural agriculture’ is a contradiction in terms.”

Over at the Times of London, science editor Mark Henderson took a similar position, as well as noting that

“Research that appears to support health claims for organic food also suffers from a quality problem. Many studies lack proper controls or fail to detail the organic regime and crop variety being evaluated or the analytical techniques used for assessment.

Studies that fail to meet these standards cannot provide useful evidence and are rightfully excluded from systematic reviews. It is no coincidence that the school had to throw out about two thirds of the available literature.”

The Times also noted that previous reviews by the French and Swedish food standards agencies had come to the same conclusion as this new study.

But as the Observer called into question the thrust of the Guardian’s initial news report, so the Times sister paper, The Sunday Times seemed to question the daily paper’s characterization of the study.

“We dig out the facts from the manure,” said the article’s sub head, but as reporter Chris Gourlay dug away, he seemed less convinced by the FSA’s evidence: The new study’s findings were “controversial” and the Food Standards Agency “claimed a comprehensive review,” but as Carlo Leifert, Professor of Ecological Agriculture at Newcastle University told Gourlay, the researchers “have ignored all the recent literature as well as new primary research which show the health advantages of organic.” He added that he intended “to rip their study apart in scientific journals.”

Other newspapers, such as the Daily Mail, warned that “studies have found” that children born to farmers in summer, when pesticide use was highest, were “less intelligent.” One columnist rued the focus on nutrition in the Daily Telegraph noting that “All food is nutritious; having no food is what kills. The wider benefits of organic foods are still worth pursuing. It is what food does not contain and the effects that it does not have that really matter.” The Telegraph’s gossip columnist warned that the pro-organic produce Prince Charles had reportedly taken a dim view of the FSA study and was girding for battle.

One notable pattern emerged in the coverage: If the reporter specialized in science, they thought the study well done and conclusive; if the reporter was a generalist, the study was flawed and controversial. So what did the scourge of bad science make of the review and the media coverage?

Goldacre began his column by noting that news coverage had given organic advocates a blanket right of reply to the study. This, he said, was “testament to the lobbying power of this £2bn [$3.38 billion dollars] industry, and the cultural values of people who work in the media.”

He pointed out one of the salient aspects of the study, namely, that it was only about the nutritional content of organic and conventional food, and not about any other kind of benefit. Critics of the study, however, only wanted to talk about other kinds of benefits to prove that the study was flawed; this was, he said, “gamesmanship.” And it was gamesmanship that worked to undermine the public’s understanding and ability to engage in a debate on the evidence by claiming that key evidence was ignored by the FSA.

“The accusation is one of ‘cherry-picking’, and it is hard to see how it can be valid in the kind of study conducted by the FSA, because in a ‘systematic review’, before you begin collecting papers, you specify how you will search for evidence, what databases you will use, what types of studies you will use, how you will grade the quality of the evidence (to see if it was a ‘fair test’), and so on.

What is it that the FSA ignored which so angered the Soil Association? As an example, from their press release, they are ‘disappointed that the FSA failed to include the results of a major European Union-funded study involving 31 research and university institutes and the publication, so far, of more than 100 scientific papers, at a cost of €18m [$25.9 million dollars], which ended in April this year’. They gave the link to qlif.org.

I followed this link and found the list of 120 papers. Almost all are irrelevant. The first 14 are on ‘consumer expectations and attitudes’, which are correctly not included in a systematic review of the evidence on food composition. Then there are 22 on ‘effects of production methods’: here you might expect to find more relevant research, but no.

The first paper (‘The effect of medium term feeding with organic, low input and conventional diet on selected immune parameters in rat’), while interesting, will plainly not be relevant to a systematic review on nutrient content. The same is true of the next paper, ‘Salmonella infection level in Danish indoor and outdoor pig production systems measured by antibodies in meat juice and fecal shedding on-farm and at slaughter’: it is not relevant.

Furthermore, the overwhelming majority of these are unpublished conference papers, and some of them are just a description of the fact that somebody made an oral presentation at a meeting. The systematic review correctly looked only at good-quality data published in peer-reviewed academic journals.”

This is a devastating indictment, not just of the Soil Association’s position, but the degree to which reporters did little more than act as stenographers to its criticisms of the FSA study. [Yes, we too followed the link to http://www.qlif.org/ and found that Goldacre was correct in his categorization of the research]. Readers of the Guardian may have been forgiven for wondering why they bothered to read the initial news story, given that the reporter’s focus on what was wrong with the study turned out to be more spin than science.

The uncomfortable question for the media – and the Guardian in particular – is to what degree would Goldacre’s rearguard defense of science be needed if the journalists who reported on the latest data did a better job of analysis before presenting it to the public?