Science, Now Under Scrutiny Itself
By BENEDICT CAREYJUNE 15, 2015
The crimes and misdemeanors of science used to be handled mostly in-house, with a private word at the faculty club, barbed questions at a conference, maybe a quiet dismissal. On the rare occasion when a journal publicly retracted a study, it typically did so in a cryptic footnote. Few were the wiser; many retracted studies have been cited as legitimate evidence by others years after the fact.
Retracted Scientific Studies: A Growing List
“Until recently it was unusual for us to report on studies that were not yet retracted,” said Dr. Ivan Oransky, an editor of the blog Retraction Watch, the first news media outlet to report that the study had been challenged. But new technology and a push for transparency from younger scientists have changed that, he said. “We have more tips than we can handle.”
The case has played out against an increase in retractions that has alarmed many journal editors and authors. Scientists in fields as diverse as neurobiology, anesthesia and economics are debating how to reduce misconduct, without creating a police-state mentality that undermines creativity and collaboration.
“It’s an extraordinary time,” said Brian Nosek, a professor of psychology at the University of Virginia, and a founder of the Center for Open Science, which provides a free service through which labs can share data and protocols. “We are now seeing a number of efforts to push for data repositories to facilitate direct replications of findings.”
But that push is not universally welcomed. Some senior scientists have argued that replication often wastes resources. “Isn’t reproducibility the bedrock of science? Yes, up to a point,” the cancer biologist Mina Bissell wrote in a widely circulated blog post. “But it is sometimes much easier not to replicate than to replicate studies,” especially when the group trying to replicate does not have the specialized knowledge or skill to do so.
The experience of Retraction Watch provides a rough guide to where this debate is going and why. Dr. Oransky, who has a medical degree from New York University, and Adam Marcus, both science journalists, discovered a mutual interest in retractions about five years ago and founded the blog as a side project. They had, and still have, day jobs: Mr. Marcus, 46, is the managing editor of Gastroenterology & Endoscopy News, and Dr. Oransky, 42, is the editorial director of MedPage Today (he will take a position as distinguished writer in residence at N.Y.U. later this year).
In its first year, the blog broke a couple of retraction stories that hit the mainstream news media — including a case involving data faked by an anesthesiologist who later served time for health care fraud. The site now has about 150,000 unique visitors a month, about half from outside the United States.
Dr. Oransky and Mr. Marcus are partisans who editorialize sharply against poor oversight and vague retraction notices. But their focus on evidence over accusations distinguishes them from watchdog forerunners who sometimes came off as ad hominem cranks. Last year, their site won a $400,000 grant from the John D. and Catherine T. MacArthur Foundation, to build out their database, and they plan to work with Dr. Nosek to manage the data side.
Their data already tell a story.
The blog has charted a 20 to 25 percent increase in retractions across some 10,000 medical and science journals in the past five years: 500 to 600 a year today from 400 in 2010. (The number in 2001 was 40, according to previous research.) The primary causes of this surge are far from clear. The number of papers published is higher than ever, and journals have proliferated, Dr. Oransky and other experts said. New tools for detecting misconduct, like plagiarism-sifting software, are widely available, so there’s reason to suspect that the surge is a simple product of better detection and larger volume.
The increasing challenges to the veracity of scientists’ work gained widespread attention recently when a study by Michael LaCour on the effect of political canvassing on opinions of same-sex marriage was questioned and ultimately retracted.
Still, the pressure to publish attention-grabbing findings is stronger than ever, these experts said — and so is the ability to “borrow” and digitally massage data. Retraction Watch’s records suggest that about a third of retractions are because of errors, like tainted samples or mistakes in statistics, and about two-thirds are because of misconduct or suspicions of misconduct.
The most common reason for retraction because of misconduct is image manipulation, usually of figures or diagrams, a form of deliberate data massaging or, in some cases, straight plagiarism. In their dissection of the LaCour-Green paper, the two graduate students — David Broockman, now an assistant professor at Stanford, and Joshua Kalla, at California-Berkeley — found that a central figure in Mr. LaCour’s analysis looked nearly identical to one from another study. This and other concerns led Dr. Green, who had not seen any original data, to request a retraction. (Mr. LaCour has denied borrowing anything.)
Data massaging can take many forms. It can mean simply excluding “outliers” — unusually high or low data points — from an analysis to generate findings that more strongly support the hypothesis. It also includes moving the goal posts: that is, mining the data for results first, and then writing the paper as if the experiment had been an attempt to find just those effects. “You have exploratory findings, and you’re pitching them as ‘I knew this all along,’ as confirmatory,” Dr. Nosek said.
The second leading cause is plagiarizing text, followed by republishing — presenting the same results in two or more journals.
The fourth category is faked data. No one knows the rate of fraud with any certainty. In a 2011 survey of more than 2,000 psychologists, about 1 percent admitted to falsifying data. Other studies have estimated a rate of about 2 percent. Yet one offender can do a lot of damage. The Dutch social psychologist Diederik Stapel published dozens of studies in major journals for nearly a decade based on faked data, investigators at the universities where he had worked concluded in 2011. Suspicions were first raised by two of his graduate students.
“If I’m a scientist and I fabricate data and put that online, others are going to assume this is accurate data,” said John Budd, a professor at the University of Missouri and an author of one of the first exhaustive analyses of retractions, in 1999. “There’s no way to know” without inside information.
Here, too, Retraction Watch provides a possible solution. Many of the egregious cases that it posts come from tips. The tipsters are a growing cadre of scientists, specialized journalists and other experts who share the blog’s mission — and are usually not insiders working directly with a suspected offender. One of the blog’s most effective allies has been Dr. Steven Shafer, the current editor of the journal Anesthesia & Analgesia who is now at Stanford, whose aggressiveness in re-examining published papers has led to scores of retractions. The field of anesthesia is a leader in retractions, largely because of Dr. Shafer’s efforts, Mr. Marcus and Dr. Oransky said. (Psychology is another leader, largely because of Dr. Stapel.)
Other cases emerge from issues raised at post-publication sites, where scientists dig into papers, sometimes anonymously. Dr. Broockman, one of the two who challenged the LaCour-Green paper, had first made public some of his suspicions anonymously on a message board called poliscirumors.com. Mr. Marcus said Retraction Watch closely followed a similar site, PubPeer.com. “When it first popped up, a lot of people assumed it would be an ax-grinding place,” he said. “But while some contributors have overstepped, I think it has had a positive impact on the literature.”
What these various tipsters, anonymous post-reviewers and whistle-blowers have in common is a nose for data that looks too good to be true, he said. Sites like Retraction Watch and PubPeer give them a place to discuss their concerns and flag fishy-looking data.
These, along with data repositories like Dr. Nosek’s, may render moot the debate over how to exhaustively replicate findings. That burden is likely to be eased by the community of bad-science bloodhounds who have more and more material to work with when they pick up a foul scent.
“At this point, we see ourselves as part of an ecosystem that is advocating for increased transparency,” Dr. Oransky said. “And that ecosystem is growing.”
By BENEDICT CAREYJUNE 15, 2015
The crimes and misdemeanors of science used to be handled mostly in-house, with a private word at the faculty club, barbed questions at a conference, maybe a quiet dismissal. On the rare occasion when a journal publicly retracted a study, it typically did so in a cryptic footnote. Few were the wiser; many retracted studies have been cited as legitimate evidence by others years after the fact.
Retracted Scientific Studies: A Growing List
“Until recently it was unusual for us to report on studies that were not yet retracted,” said Dr. Ivan Oransky, an editor of the blog Retraction Watch, the first news media outlet to report that the study had been challenged. But new technology and a push for transparency from younger scientists have changed that, he said. “We have more tips than we can handle.”
The case has played out against an increase in retractions that has alarmed many journal editors and authors. Scientists in fields as diverse as neurobiology, anesthesia and economics are debating how to reduce misconduct, without creating a police-state mentality that undermines creativity and collaboration.
“It’s an extraordinary time,” said Brian Nosek, a professor of psychology at the University of Virginia, and a founder of the Center for Open Science, which provides a free service through which labs can share data and protocols. “We are now seeing a number of efforts to push for data repositories to facilitate direct replications of findings.”
But that push is not universally welcomed. Some senior scientists have argued that replication often wastes resources. “Isn’t reproducibility the bedrock of science? Yes, up to a point,” the cancer biologist Mina Bissell wrote in a widely circulated blog post. “But it is sometimes much easier not to replicate than to replicate studies,” especially when the group trying to replicate does not have the specialized knowledge or skill to do so.
The experience of Retraction Watch provides a rough guide to where this debate is going and why. Dr. Oransky, who has a medical degree from New York University, and Adam Marcus, both science journalists, discovered a mutual interest in retractions about five years ago and founded the blog as a side project. They had, and still have, day jobs: Mr. Marcus, 46, is the managing editor of Gastroenterology & Endoscopy News, and Dr. Oransky, 42, is the editorial director of MedPage Today (he will take a position as distinguished writer in residence at N.Y.U. later this year).
In its first year, the blog broke a couple of retraction stories that hit the mainstream news media — including a case involving data faked by an anesthesiologist who later served time for health care fraud. The site now has about 150,000 unique visitors a month, about half from outside the United States.
Dr. Oransky and Mr. Marcus are partisans who editorialize sharply against poor oversight and vague retraction notices. But their focus on evidence over accusations distinguishes them from watchdog forerunners who sometimes came off as ad hominem cranks. Last year, their site won a $400,000 grant from the John D. and Catherine T. MacArthur Foundation, to build out their database, and they plan to work with Dr. Nosek to manage the data side.
Their data already tell a story.
The blog has charted a 20 to 25 percent increase in retractions across some 10,000 medical and science journals in the past five years: 500 to 600 a year today from 400 in 2010. (The number in 2001 was 40, according to previous research.) The primary causes of this surge are far from clear. The number of papers published is higher than ever, and journals have proliferated, Dr. Oransky and other experts said. New tools for detecting misconduct, like plagiarism-sifting software, are widely available, so there’s reason to suspect that the surge is a simple product of better detection and larger volume.
The increasing challenges to the veracity of scientists’ work gained widespread attention recently when a study by Michael LaCour on the effect of political canvassing on opinions of same-sex marriage was questioned and ultimately retracted.
Still, the pressure to publish attention-grabbing findings is stronger than ever, these experts said — and so is the ability to “borrow” and digitally massage data. Retraction Watch’s records suggest that about a third of retractions are because of errors, like tainted samples or mistakes in statistics, and about two-thirds are because of misconduct or suspicions of misconduct.
The most common reason for retraction because of misconduct is image manipulation, usually of figures or diagrams, a form of deliberate data massaging or, in some cases, straight plagiarism. In their dissection of the LaCour-Green paper, the two graduate students — David Broockman, now an assistant professor at Stanford, and Joshua Kalla, at California-Berkeley — found that a central figure in Mr. LaCour’s analysis looked nearly identical to one from another study. This and other concerns led Dr. Green, who had not seen any original data, to request a retraction. (Mr. LaCour has denied borrowing anything.)
Data massaging can take many forms. It can mean simply excluding “outliers” — unusually high or low data points — from an analysis to generate findings that more strongly support the hypothesis. It also includes moving the goal posts: that is, mining the data for results first, and then writing the paper as if the experiment had been an attempt to find just those effects. “You have exploratory findings, and you’re pitching them as ‘I knew this all along,’ as confirmatory,” Dr. Nosek said.
The second leading cause is plagiarizing text, followed by republishing — presenting the same results in two or more journals.
The fourth category is faked data. No one knows the rate of fraud with any certainty. In a 2011 survey of more than 2,000 psychologists, about 1 percent admitted to falsifying data. Other studies have estimated a rate of about 2 percent. Yet one offender can do a lot of damage. The Dutch social psychologist Diederik Stapel published dozens of studies in major journals for nearly a decade based on faked data, investigators at the universities where he had worked concluded in 2011. Suspicions were first raised by two of his graduate students.
“If I’m a scientist and I fabricate data and put that online, others are going to assume this is accurate data,” said John Budd, a professor at the University of Missouri and an author of one of the first exhaustive analyses of retractions, in 1999. “There’s no way to know” without inside information.
Here, too, Retraction Watch provides a possible solution. Many of the egregious cases that it posts come from tips. The tipsters are a growing cadre of scientists, specialized journalists and other experts who share the blog’s mission — and are usually not insiders working directly with a suspected offender. One of the blog’s most effective allies has been Dr. Steven Shafer, the current editor of the journal Anesthesia & Analgesia who is now at Stanford, whose aggressiveness in re-examining published papers has led to scores of retractions. The field of anesthesia is a leader in retractions, largely because of Dr. Shafer’s efforts, Mr. Marcus and Dr. Oransky said. (Psychology is another leader, largely because of Dr. Stapel.)
Other cases emerge from issues raised at post-publication sites, where scientists dig into papers, sometimes anonymously. Dr. Broockman, one of the two who challenged the LaCour-Green paper, had first made public some of his suspicions anonymously on a message board called poliscirumors.com. Mr. Marcus said Retraction Watch closely followed a similar site, PubPeer.com. “When it first popped up, a lot of people assumed it would be an ax-grinding place,” he said. “But while some contributors have overstepped, I think it has had a positive impact on the literature.”
What these various tipsters, anonymous post-reviewers and whistle-blowers have in common is a nose for data that looks too good to be true, he said. Sites like Retraction Watch and PubPeer give them a place to discuss their concerns and flag fishy-looking data.
These, along with data repositories like Dr. Nosek’s, may render moot the debate over how to exhaustively replicate findings. That burden is likely to be eased by the community of bad-science bloodhounds who have more and more material to work with when they pick up a foul scent.
“At this point, we see ourselves as part of an ecosystem that is advocating for increased transparency,” Dr. Oransky said. “And that ecosystem is growing.”
No comments:
Post a Comment