Decanonising Science II

Should we have faith in science? Part II: peer-reviewed science papers
Thursday, February 26, 2015 - 12:51

Kirk Durston



The primary way scientific discoveries and advances are disseminated is through peer-reviewed papers published in scientific journals. The first step is to submit a paper to a journal. Those that survive preliminary filtering by the editor are sent out to be reviewed by qualified scientists in the field. On the basis of the reviewers’ recommendations, the paper is accepted or rejected. Only a fraction of papers submitted for publication make it through this peer-review process and are published.

One would hope that such a process would justify a high level of confidence in scientific publications, but recent findings suggest that our faith in peer-reviewed publications in mainstream journals of science may be on somewhat shaky ground.

The journal Nature, in a paper calling for increased standards in pre-clinical research, revealed that out of 53 papers presenting ‘landmark’ published findings in the field of haematology and oncology, only 6 could be confirmed by subsequent laboratory teams. For the 89% of papers that failed to have their results reproduced, it was found that blind control group analyses was inadequate or data had been selected to support the hypothesis and other data set aside.

Worse still, some of the papers that could not be experimentally reproduced, launched ‘an entire field, with hundreds of secondary publications that expanded on elements of the original observation, but did not actually seek to confirm or falsify its fundamental basis’.

Hundreds of other peer-reviewed, published science papers based on faulty initial papers!

Nature reported in October 2011 that although the number of submissions had increased by 44% over the past ten years, the number of retractions had increased by roughly 900%.

Austin Hughes, in a paper published in the Proceedings of the National Academy of Sciences, focusing on the origin of adaptive phenotypes laments, ‘Thousands of papers are published every year claiming evidence of adaptive evolution on the basis of computational analyses alone, with no evidence whatsoever regarding the phenotypic effects of allegedly adaptive mutations.’ He concludes that ‘This vast outpouring of pseudo-Darwinian hype has been genuinely harmful to the credibility of evolutionary biology as a science.’ Jerry Fodor and Massimo Piattelli-Palmarini write in New Scientist,

"Much of the vast neo-Darwinian literature is distressingly uncritical. The possibility that anything is seriously amiss with Darwin’s account of evolution is hardly considered. … The methodological skepticism that characterizes most areas of scientific discourse seems strikingly absent when Darwinism is the topic."

How can we distinguish the good papers from the poor? This can be very difficult without actually attempting to reproduce their findings. Short of that, apply the same critical thinking skills and healthy skepticism to scientific papers that you do for political, historical or religious claims. 21st century science can often be heavily influenced by poor experimental practices, unproven computational models, political agendas, competition for funding, and scientism (atheism dressed up as science). When going over a paper ask questions like, how large was the data set? What sort of statistical analysis was performed? Are there other papers that independently support or disconfirm these findings? What is not being discussed? One thing for sure, don’t accept something simply because ‘hundreds’ or even ‘thousands’ of papers say so, especially if Darwinian evolution is the topic. Practice critical thinking with the question in the back of your mind, 'Is this one of those papers that will be retracted?'.

Read Part III

Further reading:

http://www.nature.com/nature/journal/v483/n7391/full/483531a.html#t1

http://www.nature.com/news/publishing-the-peer-review-scam-1.16400

http://www.nytimes.com/2012/04/17/science/rise-in-scientific-journal-retractions-prompts-calls-for-reform.html?ref=science

http://dingo.sbs.arizona.edu/~massimo/publications/PDF/JF_MPP_darwinisms...


http://www.nytimes.com/2015/06/16/science/retractions-coming-out-from-under-science-rug.html?

Decanonising Science

Science, Now Under Scrutiny Itself
By BENEDICT CAREYJUNE 15, 2015

The crimes and misdemeanors of science used to be handled mostly in-house, with a private word at the faculty club, barbed questions at a conference, maybe a quiet dismissal. On the rare occasion when a journal publicly retracted a study, it typically did so in a cryptic footnote. Few were the wiser; many retracted studies have been cited as legitimate evidence by others years after the fact.


Retracted Scientific Studies: A Growing List

“Until recently it was unusual for us to report on studies that were not yet retracted,” said Dr. Ivan Oransky, an editor of the blog Retraction Watch, the first news media outlet to report that the study had been challenged. But new technology and a push for transparency from younger scientists have changed that, he said. “We have more tips than we can handle.”

The case has played out against an increase in retractions that has alarmed many journal editors and authors. Scientists in fields as diverse as neurobiology, anesthesia and economics are debating how to reduce misconduct, without creating a police-state mentality that undermines creativity and collaboration.

“It’s an extraordinary time,” said Brian Nosek, a professor of psychology at the University of Virginia, and a founder of the Center for Open Science, which provides a free service through which labs can share data and protocols. “We are now seeing a number of efforts to push for data repositories to facilitate direct replications of findings.”

But that push is not universally welcomed. Some senior scientists have argued that replication often wastes resources. “Isn’t reproducibility the bedrock of science? Yes, up to a point,” the cancer biologist Mina Bissell wrote in a widely circulated blog post. “But it is sometimes much easier not to replicate than to replicate studies,” especially when the group trying to replicate does not have the specialized knowledge or skill to do so.

The experience of Retraction Watch provides a rough guide to where this debate is going and why. Dr. Oransky, who has a medical degree from New York University, and Adam Marcus, both science journalists, discovered a mutual interest in retractions about five years ago and founded the blog as a side project. They had, and still have, day jobs: Mr. Marcus, 46, is the managing editor of Gastroenterology & Endoscopy News, and Dr. Oransky, 42, is the editorial director of MedPage Today (he will take a position as distinguished writer in residence at N.Y.U. later this year).

In its first year, the blog broke a couple of retraction stories that hit the mainstream news media — including a case involving data faked by an anesthesiologist who later served time for health care fraud. The site now has about 150,000 unique visitors a month, about half from outside the United States.

Dr. Oransky and Mr. Marcus are partisans who editorialize sharply against poor oversight and vague retraction notices. But their focus on evidence over accusations distinguishes them from watchdog forerunners who sometimes came off as ad hominem cranks. Last year, their site won a $400,000 grant from the John D. and Catherine T. MacArthur Foundation, to build out their database, and they plan to work with Dr. Nosek to manage the data side.

Their data already tell a story.

The blog has charted a 20 to 25 percent increase in retractions across some 10,000 medical and science journals in the past five years: 500 to 600 a year today from 400 in 2010. (The number in 2001 was 40, according to previous research.) The primary causes of this surge are far from clear. The number of papers published is higher than ever, and journals have proliferated, Dr. Oransky and other experts said. New tools for detecting misconduct, like plagiarism-sifting software, are widely available, so there’s reason to suspect that the surge is a simple product of better detection and larger volume.



The increasing challenges to the veracity of scientists’ work gained widespread attention recently when a study by Michael LaCour on the effect of political canvassing on opinions of same-sex marriage was questioned and ultimately retracted.
Still, the pressure to publish attention-grabbing findings is stronger than ever, these experts said — and so is the ability to “borrow” and digitally massage data. Retraction Watch’s records suggest that about a third of retractions are because of errors, like tainted samples or mistakes in statistics, and about two-thirds are because of misconduct or suspicions of misconduct.

The most common reason for retraction because of misconduct is image manipulation, usually of figures or diagrams, a form of deliberate data massaging or, in some cases, straight plagiarism. In their dissection of the LaCour-Green paper, the two graduate students — David Broockman, now an assistant professor at Stanford, and Joshua Kalla, at California-Berkeley — found that a central figure in Mr. LaCour’s analysis looked nearly identical to one from another study. This and other concerns led Dr. Green, who had not seen any original data, to request a retraction. (Mr. LaCour has denied borrowing anything.)

Data massaging can take many forms. It can mean simply excluding “outliers” — unusually high or low data points — from an analysis to generate findings that more strongly support the hypothesis. It also includes moving the goal posts: that is, mining the data for results first, and then writing the paper as if the experiment had been an attempt to find just those effects. “You have exploratory findings, and you’re pitching them as ‘I knew this all along,’ as confirmatory,” Dr. Nosek said.

The second leading cause is plagiarizing text, followed by republishing — presenting the same results in two or more journals.

The fourth category is faked data. No one knows the rate of fraud with any certainty. In a 2011 survey of more than 2,000 psychologists, about 1 percent admitted to falsifying data. Other studies have estimated a rate of about 2 percent. Yet one offender can do a lot of damage. The Dutch social psychologist Diederik Stapel published dozens of studies in major journals for nearly a decade based on faked data, investigators at the universities where he had worked concluded in 2011. Suspicions were first raised by two of his graduate students.

“If I’m a scientist and I fabricate data and put that online, others are going to assume this is accurate data,” said John Budd, a professor at the University of Missouri and an author of one of the first exhaustive analyses of retractions, in 1999. “There’s no way to know” without inside information.

Here, too, Retraction Watch provides a possible solution. Many of the egregious cases that it posts come from tips. The tipsters are a growing cadre of scientists, specialized journalists and other experts who share the blog’s mission — and are usually not insiders working directly with a suspected offender. One of the blog’s most effective allies has been Dr. Steven Shafer, the current editor of the journal Anesthesia & Analgesia who is now at Stanford, whose aggressiveness in re-examining published papers has led to scores of retractions. The field of anesthesia is a leader in retractions, largely because of Dr. Shafer’s efforts, Mr. Marcus and Dr. Oransky said. (Psychology is another leader, largely because of Dr. Stapel.)

Other cases emerge from issues raised at post-publication sites, where scientists dig into papers, sometimes anonymously. Dr. Broockman, one of the two who challenged the LaCour-Green paper, had first made public some of his suspicions anonymously on a message board called poliscirumors.com. Mr. Marcus said Retraction Watch closely followed a similar site, PubPeer.com. “When it first popped up, a lot of people assumed it would be an ax-grinding place,” he said. “But while some contributors have overstepped, I think it has had a positive impact on the literature.”

What these various tipsters, anonymous post-reviewers and whistle-blowers have in common is a nose for data that looks too good to be true, he said. Sites like Retraction Watch and PubPeer give them a place to discuss their concerns and flag fishy-looking data.

These, along with data repositories like Dr. Nosek’s, may render moot the debate over how to exhaustively replicate findings. That burden is likely to be eased by the community of bad-science bloodhounds who have more and more material to work with when they pick up a foul scent.

“At this point, we see ourselves as part of an ecosystem that is advocating for increased transparency,” Dr. Oransky said. “And that ecosystem is growing.”