How Scientists Are Doing A Bait-And-Switch With Medical Data

    Researchers are "choosing their lottery numbers after seeing the draw", making medicine less reliable – and respected journals are letting them do it.

    Tom Chivers
    by Tom Chivers

    BuzzFeed Staff, UK

    Medical science is being undermined because researchers are changing the things they're measuring after looking at the data, a campaign group has warned.

    ThinkStock

    Research published in some of the world's most prestigious medical journals is rife with "outcome switching", they say, where the researchers choose what data to look at after the results come in. One scientist compares it to "choosing your lottery numbers after seeing the draw".

    It means that it is harder to tell whether a drug or a treatment is as effective as it claims, and may mean that doctors are failing to give people the best possible treatment for various diseases because they have been misled by bad data. It also may mean that the NHS and other health services are paying over the odds for expensive drugs that are no better than cheaper alternatives.

    The group found that 58 out of 67 articles they examined in the five most famous medical journals in the world had switched outcomes to one degree or another.

    The CEBM Outcome Monitoring Project, COMPARE, part of Oxford University's Centre for Evidence-Based Medicine, says that the top five medical journals all regularly publish articles with switched outcomes, and that it is a huge problem for science.

    JAMA, BMJ, Lancet, Annals of Internal Medicine, BMJ

    The journals are the British Medical Journal (BMJ), the New England Journal of Medicine (NEJM), the Journal of the American Medical Assocation (JAMA), the Lancet, and the Annals of Internal Medicine.

    When a research project begins, its authors record the things that they're going to measure at the end. "Outcome switching" is when they measure different things instead, without telling anyone.

    So, for instance, if you propose a study into the effects of alcohol on health, you could say that you were going to look at 10,000 people who drink and 10,000 people who don't and look at how many in each group die within five years.

    It would be "outcome switching" if, at the end of the trial, you count how many heart attacks there were, instead, without admitting in the article that you originally planned to count the number of deaths.

    This matters, because sometimes, you'll get good results just by chance. The more things you measure, the more likely you are to get a fluky good result.

    XKCD / Via xkcd.com

    Dr Ben Goldacre of Oxford University, who leads COMPARE, told BuzzFeed Science that: "Our statistical models are based on the assumption of one outcome being measured, so if you don’t report all the things you say are going to report, then your statistical models are fundamentally broken.

    "For instance, imagine a randomised trial on alcohol to determine whether a glass of wine a day is better for your health than abstinence. 'Better for your health' is a broad outcome, so you might measure deaths, heart attacks, blood tests, headaches, employment, subjective wellbeing all kinds of things, in all kinds of ways."

    But if you divide up the data lots of ways, and don't specify in advance what you're measuring, you'll probably get some impressive-sounding result. There's a lot of random noise in medical studies, meaning that sometimes you'll get what looks like a real effect just by chance.

    A scientific result is considered "statistically significant" if there's a less than one in 20 chance that it could happen by chance – if you measure 20 different things, you'll probably get one positive result, as this XKCD demonstrates.

    "If you measure a huge number of outcomes, and then allow yourself to pick and choose which you will report as your main finding, then suddenly the scientific integrity of your study is in big trouble," said Goldacre. "You can probably get some kind of positive finding, regardless of the true effect of your intervention."

    A professor of cognitive neuroscience told BuzzFeed Science: "Hidden outcome switching is like choosing lottery numbers after watching the draw."

    Chris Chambers of Cardiff University said: "Who would be surprised if you picked the winning numbers? In medical science the problem we face is much the same.

    "To be as sure as possible that a discovery is real, we pre-specify outcome measures, run the study, and then rely on statistical tests to tell us how likely a particular outcome was compared to chance."

    Not doing so, he said, "can have devastating downstream consequences for medical treatments", because the science that doctors are basing their treatments on becomes unreliable. "The issue boils down to transparency," he said.

    COMPARE looked at 67 articles published in the top five medical journals since October 2015, to see how many had changed outcomes without saying so. All but nine of them had.

    This goes against guidelines for transparency in research. The CONSORT guidelines, which all five of the journals have signed up to, state that any changes to outcomes should be publicly declared, and reasons given. "If you switch them, that's fine, but you've got to say that you switched them and why," said Goldacre.

    The group then sent letters to each of the journals, pointing out that the outcomes have been switched and asking them to publish clarifications.

    For instance, one trial in the Lancet said it would measure 22 outcomes; COMPARE wrote to them pointing out that it had only published results on 14 of them, and had added an extra one which wasn't mentioned. Another, in the Annals of Internal Medicine, had two prespecified outcomes; it published results for neither of them, and added a further 14 outcomes in its published study.

    The journals have responded very differently, Goldacre said. At one end of the spectrum, the BMJ has "set the standard".

    He told BuzzFeed Science that the BMJ had "published all our letters, accepted it had misreported outcomes, and issued corrections. That's what you'd expect a good journal to do."

    By contrast, the Annals of Internal Medicine's response was "extraordinary and confused", he said.

    They issued a letter signed by the editors, saying that "On the basis of our long experience reviewing research articles, we have learned that prespecified outcomes or analytic methods can be suboptimal or wrong."

    However, they did not explain why declaring when outcomes have been changed would be a bad idea. Goldacre said that "They essentially argue that outcome switching is fine and that they have the skill to allow people to do it, which breaks the promises they've made [to CONSORT] and the expectations their readers have that they are properly managing the problem."

    Chambers agreed, and called Annals' failure to respond "nothing short of astounding". "Quite frankly, the response of Annals to this basic scientific issue betrays a disappointing ignorance among the journal's editors about the purpose of trial registration," he said.

    Goldacre said: "We are confident that [Annals] are committed to addressing this problem, but there is a deep-rooted cultural problem in science and medicine about accepting these bad practices." COMPARE responded to the Annals editorial in a blog post.

    A spokesperson for Annals of Internal Medicine told BuzzFeed Science that they had addressed the issues in their letter.

    A spokesperson for the BMJ, meanwhile, told BuzzFeed Science: "The BMJ supports the aims of the COMPARE project. We are committed to publishing research papers in which the pre-specified outcomes listed in the trial registration are faithfully reported or the authors declare their intention to publish the outcomes elsewhere."

    CORRECTION

    The Annals of Internal Medicine published a letter signed by its editors in response to COMPARE's comments. An earlier version of this piece described this as an "anonymous editorial".