A little bird sent me this request: “How about an analysis of the Roundup thing?”
I’d read about the multimillion-dollar lawsuits recently won against Monsanto, an agricultural biotechnology best known for its public relations challenges and questionable ethics. However, I wasn’t aware that Bayer had purchased the company last year and can now look forward to over 11,000 lawsuits! It certainly appeared as if the verdict was in about the dangers of Roundup, but jurors aren’t known for their ability to evaluate scientific evidence. While Monsanto has a financial incentive to try to influence studies, lawyers also do well in times of public hysteria (“mesothelioma” was the top-paying Internet ad at my last job). So let’s take a crack at getting beyond all of the perverse incentives at play and focus on the main question: Does Roundup cause cancer?
With controversial topics like these, it’s important to first look for a scientific consensus. In this case, the EPA, European Food Safety Authority, Food and Agriculture Organization, European Chemicals Agency, Health Canada, German Federal Institute for Risk Assessment and others have concluded that, at the levels people are exposed to glyphosate, the active ingredient in Roundup, it does not pose a risk of cancer. However, the consensus on glyphosate is not unanimous; there is one organization, the International Agency for Research on Cancer (IARC) which classified glyphosate as a “probable carcinogen.” Is this the only agency to escape Monsanto’s influence or is there another explanation?
It turns out that the IARC evaluates risk in a different way than the other agencies. It determines if the substance can cause cancer with exposure levels far more extreme than any that would be found in the real world. Practically everything is dangerous in high amounts, (including water) and the IARC, accordingly, has only found one out of the hundreds of agents they have evaluated as being “probably not carcinogenic.” I’m not accusing the IARC of practicing pseudoscience, but let’s just say that I’m sleeping better now that I know they’re the ones behind the California Prop 65 cancer warnings at fast food restaurants. I figure that as long as I don’t ingest 400 chalupas per day, I’ll probably be okay.
Due to the consensus of worldwide regulatory agencies (and with IARC’s conclusion put into context) I would already feel comfortable concluding that there is not sufficient evidence showing that Roundup causes cancer. However, let’s go down a level to the studies themselves and see what we find. The reason I didn’t start here is because individual studies can be very unreliable, especially when it comes to epidemiological studies (as opposed to controlled experiments). That said, one of the strongest experimental designs for these types of studies is the “prospective cohort study”, which follows a population of people with various exposure levels to the chemical over time and, only later, determines whether or not the groups show significant differences in health. While they can have their conclusions reversed due to unconsidered confounding variables (“Oops, people living close to power lines tend to be poorer and have less access to healthcare”), these types of studies at least avoid the problem of selective recall that plagues case-control studies: (“Hmm, I didn’t know what caused my tumor, but now that you mention it, I DO spend a lot of time on the cell phone!”). Following the surprising IARC conclusion, a study revisited and followed up on data from the large Agricultural Health Study (AHS). It found, in agreement with earlier conclusions, “no association was apparent between glyphosate and any solid tumors or lymphoid malignancies overall, including NHL and its subtypes.”
It certainly is looking like the evidence against Roundup is weak. However, a recent study in the journal Mutation Research threw a monkey wrench into things and associated weed killing products with non-Hodgkin lymphoma (NHL). It used the same AHS data above and combined it with a few less reliable case-control studies to conclude that people exposed to glyphosate have a 41% higher likelihood of developing NHL.
I’m a bit uncomfortable with the fact that it used the same data from a study that found no significant risk, added in less reliable data, and then concluded that there IS a risk. That seems like taking advantage of post-hoc wiggle-room. Another problem is that the 20-year time lag is the only one mentioned in the analysis. Why not report the results of the 15-year or 10-year lag since exposure? The 20-year lag was the only one that showed a relative risk greater than 0%. Coincidence? Read my upcoming book and you’ll suspect that this is Pitfall #5: Torturing Data. The study reports a 95% confidence interval as if they had a hypothesis, tested it, and found an increase in risk that would be unlikely if Roundup weren’t dangerous. In reality, when they skipped over data points that didn’t support their conclusion before landing on the one that did, the likelihood they would find something increased significantly. I can’t help but wonder if they would have even bothered to combine data from the less reliable studies if the AHS data showed significance on its own. I get the impression they found the result they did, because they went looking for it, and accordingly, their conclusion should be taken with a grain of salt. It would be analogous to asking “what are the chances I found an Easter egg in this particular place, given there were 20 possible places to search?” and not mentioning that you had searched a few other places before you found it. This may seem nit-picky when only a few results weren’t mentioned, but their whole conclusion of “statistical significance” hinges on it!
Observational studies like this are unreliable in the best of circumstances. They have the burden of showing that higher doses lead to a higher likelihood of illness (dose-response relationship). They have the burden of controlling for variables such as age, family history, body weight, and other things that may bias the results (confounding variables). For an extreme example, suppose there were a study that was much more compelling because it took blood samples of thousands of people and everyone with cancer had Roundup in their blood and everyone without cancer did not. A slam dunk case! Until later they find out that everyone with cancer also had DDT or some other chemical in their blood due to the fact that they were all farmers using a variety of insecticides. Suddenly, the case could fall apart.
Even if this study had carefully done everything possible and found that higher exposure to Roundup led to higher chances of developing NHL and also had a strong reason ahead of time that it would only show up after a 20-year lag, it would still be one observational study up against a consensus of health agencies around the world. People want to believe that science can confidently answer questions like “what causes cancer” by simply analyzing data. The truth is that, without randomized testing, science is severely hobbled. At best, it can draw tentative conclusions when data is carefully collected and analyzed by scientists trained in not fooling themselves and haven’t peeked at results before forming their hypotheses. Before you vote “guilty” in your next jury, remember that scientific consensus among multiple scientific organizations represents our best understanding of the world. In general, if you rely on “worldwide conspiracy and bribery” as the explanation for why scientific organizations are wrong, your argument is in trouble. No matter how compelling a conspiracy theory may be, the weight of scientific consensus should provide you with more than a reasonable doubt.
Disagree? Let me know what I got wrong and I’ll post updates. And keep those ideas coming for future blog entries!
