Peer-Review and the Corruption of Science
Jonathan M. September 13, 2011 6:00 AM
The Guardian features an interesting opinion column by the renowned British pharmacologist David Colquhoun. The article bears the intriguing headline, "Publish-or-perish: Peer review and the corruption of science." The author laments that "Pressure on scientists to publish has led to a situation where any paper, however bad, can now be printed in a journal that claims to be peer-reviewed."
Colquhoun explains,
The blame for this sad situation lies with the people who have imposed a publish-or-perish culture, namely research funders and senior people in universities. To have "written" 800 papers is regarded as something to boast about rather than being rather shameful. University PR departments encourage exaggerated claims, and hard-pressed authors go along with them.
The author proceeds to list a few examples of the failure of the peer-review system to ensure robust and accurate journal content. He argues that part of the reason for the lapse in academic publication standards is the pressure on academics to publish many papers. If a scientist publishes frequently, that should actually call into question, rather than enhance, his credibility as a diligent and focused researcher.
Those of us who follow the professional literature (or even the blogosphere) may recall the Nowak et al. (2010) paper that appeared in Nature back in May of last year. It was regarded by many evolutionary biologists (most notably University of Chicago's Jerry Coyne) as a "misguided attack on kin selection."
Coyne noted,
If the Nowak et al. paper is so bad, why was it published? That's obvious, and is an object lesson in the sociology of science. If Joe Schmo et al. from Buggerall State University had submitted such a misguided paper to Nature, it would have been rejected within an hour (yes, Nature sometimes does that with online submissions!). The only reason this paper was published is because it has two big-name authors, Nowak and Wilson, hailing from Mother Harvard. That, and the fact that such a contrarian paper, flying in the face of accepted evolutionary theory, was bound to cause controversy.
I have often read papers, published in reputable journals, that I thought should not have passed through peer-review. Consider, for example, this paper, published in PLoS Biology in May of last year. Indeed, the esteemed atheist blogger PZ Myers wrote about it in a blog post headlined "Junk DNA is still junk" (to which I responded briefly here). The paper erroneously concluded "Overall, ...we find that most of the genome is not appreciably transcribed. [emphasis added]"
There is actually a pretty good response to this article here. The methodology of the PLoS Biology article is fatally flawed, for they use a program called "RepeatMasker", which screens out all the repetitive DNA. But given that about 50% of our genome is comprised of repetitive DNA, the conclusions drawn by the authors seems to be a little disingenuous to say the least! In fact, the official description of RepeatMasker itself states that "On average, almost 50% of a human genomic DNA sequence currently will be masked by the program."
As if that weren't bad enough, the researchers then base their results "primarily on analysis of PolyA+ enriched RNA." But we've known since 2005 that, in humans, PolyA- sequences are twice as abundant as PolyA+ transcripts. So the authors not only exclude half the genome from their research, but also completely ignore two thirds of the RNA in what remains!
By citing that paper PZ Myers didn't do his own credibility any favors. The point being made by Myers is a false one anyway because it is known that even DNA that is not transcribed can play important roles.
Then there was, of course, that recent paper in PNAS telling us that "There's plenty of time for evolution" (also paraded by Myers). The substance of the argument presented in this paper was terrible (for some of the reasons why, see here and here). Reading that paper when it came out, I was frankly astonished that it was able to pass through peer-review.
Back in June of 2009, a paper appeared in PNAS by Ghosh et al. purporting to demonstrate the production of endospores in the genus Mycobacterium (which includes many pathogens such as M. tuberculosis and M. leprae). Traag et al. (2010) document the problems with the paper:
Here, we report that the genomes of Mycobacterium species and those of other high G+C Gram-positive bacteria lack orthologs of many, if not all, highly conserved genes diagnostic of endospore formation in the genomes of low G+C Gram-positive bacteria. We also failed to detect the presence of endospores by light microscopy or by testing for heat-resistant colony-forming units in aged cultures of M. marinum. Finally, we failed to recover heat-resistant colony-forming units from frogs chronically infected with M. marinum. We conclude that it is unlikely that Mycobacterium is capable of endospore formation.
As ID proponents know only too well, the peer-review system has not only become corrupted in allowing substandard content into the academic market. It has also been turned into a gate-keeping system for imposing ideological conformity. Recently, an editor resigned over the publication of a seminal article by Roy Spencer and William Braswell. The paper's purpose was to demonstrate that one of the feedbacks that the Intergovernmental Panel on Climate Change has been treating as a positive feedback is really a negative feedback. You can read Roy Spencer's defense of his paper here.
In a similar incident in 2004, Smithsonian Institute evolutionary biologist Richard Sternberg was punished and pressured to resign following the publication of a pro-ID article by Stephen C. Meyer in a journal of which Sternberg was the editor.
In still another incident, a recent pro-ID paper authored by mathematician Granville Sewell was retracted from publication (after it had been subjected to peer-review and approved) as the result of a complaint from a blogger writing to the journal's editor. The journal, Applied Mathematics Letters has since apologized and paid $10,000 in compensation to Dr. Sewell.
What's to be done? Colquhoun makes the following recommendation:
There is an alternative: publish your paper yourself on the web and open the comments. This sort of post-publication review would reduce costs enormously, and the results would be open for anyone to read without paying. It would also destroy the hegemony of half a dozen high-status journals.
And, indeed, this is exactly how the Biologic Institute-associated journal Bio-Complexity operates. This peer-reviewed journal, dedicated to discussions surrounding the respective scientific merits of neo-Darwinian evolution and intelligent design, is published freely on the web and is open for comments and published responses, hence allowing -- even encouraging -- post-publication review.
Colquhoun further suggests,
...it would be essential to allow anonymous comments. Most reviewers are anonymous at present, so why not online? Second, the vast flood of papers that make the present system impossible should be stemmed. I'd suggest scientists should limit themselves to an average of two original papers a year. They should also be limited to holding one research grant at a time. Anyone who thought their work necessitated more than this would have to be scrutinized very carefully. It's well known that small research groups give better value than big ones, so that should be the rule.
The benefit of such a system, as Colquhoun notes, is that "With far fewer papers being published, reviewers, grant committees and promotion committees might be able to read the papers, not just count them."
Colquhoun is to be commended. The goal of the peer-review system ought to be the ensuring of factual accuracy and the highlighting of necessary revisions and corrections. Its goal should not be the enforcement of ideological and paradigmatic conformity, nor should it be the upholding of "consensus science." Post-publication review ought to be encouraged, and moves should be made to make journal content more frequently open-access.
Jonathan M. September 13, 2011 6:00 AM
The Guardian features an interesting opinion column by the renowned British pharmacologist David Colquhoun. The article bears the intriguing headline, "Publish-or-perish: Peer review and the corruption of science." The author laments that "Pressure on scientists to publish has led to a situation where any paper, however bad, can now be printed in a journal that claims to be peer-reviewed."
Colquhoun explains,
The blame for this sad situation lies with the people who have imposed a publish-or-perish culture, namely research funders and senior people in universities. To have "written" 800 papers is regarded as something to boast about rather than being rather shameful. University PR departments encourage exaggerated claims, and hard-pressed authors go along with them.
The author proceeds to list a few examples of the failure of the peer-review system to ensure robust and accurate journal content. He argues that part of the reason for the lapse in academic publication standards is the pressure on academics to publish many papers. If a scientist publishes frequently, that should actually call into question, rather than enhance, his credibility as a diligent and focused researcher.
Those of us who follow the professional literature (or even the blogosphere) may recall the Nowak et al. (2010) paper that appeared in Nature back in May of last year. It was regarded by many evolutionary biologists (most notably University of Chicago's Jerry Coyne) as a "misguided attack on kin selection."
Coyne noted,
If the Nowak et al. paper is so bad, why was it published? That's obvious, and is an object lesson in the sociology of science. If Joe Schmo et al. from Buggerall State University had submitted such a misguided paper to Nature, it would have been rejected within an hour (yes, Nature sometimes does that with online submissions!). The only reason this paper was published is because it has two big-name authors, Nowak and Wilson, hailing from Mother Harvard. That, and the fact that such a contrarian paper, flying in the face of accepted evolutionary theory, was bound to cause controversy.
I have often read papers, published in reputable journals, that I thought should not have passed through peer-review. Consider, for example, this paper, published in PLoS Biology in May of last year. Indeed, the esteemed atheist blogger PZ Myers wrote about it in a blog post headlined "Junk DNA is still junk" (to which I responded briefly here). The paper erroneously concluded "Overall, ...we find that most of the genome is not appreciably transcribed. [emphasis added]"
There is actually a pretty good response to this article here. The methodology of the PLoS Biology article is fatally flawed, for they use a program called "RepeatMasker", which screens out all the repetitive DNA. But given that about 50% of our genome is comprised of repetitive DNA, the conclusions drawn by the authors seems to be a little disingenuous to say the least! In fact, the official description of RepeatMasker itself states that "On average, almost 50% of a human genomic DNA sequence currently will be masked by the program."
As if that weren't bad enough, the researchers then base their results "primarily on analysis of PolyA+ enriched RNA." But we've known since 2005 that, in humans, PolyA- sequences are twice as abundant as PolyA+ transcripts. So the authors not only exclude half the genome from their research, but also completely ignore two thirds of the RNA in what remains!
By citing that paper PZ Myers didn't do his own credibility any favors. The point being made by Myers is a false one anyway because it is known that even DNA that is not transcribed can play important roles.
Then there was, of course, that recent paper in PNAS telling us that "There's plenty of time for evolution" (also paraded by Myers). The substance of the argument presented in this paper was terrible (for some of the reasons why, see here and here). Reading that paper when it came out, I was frankly astonished that it was able to pass through peer-review.
Back in June of 2009, a paper appeared in PNAS by Ghosh et al. purporting to demonstrate the production of endospores in the genus Mycobacterium (which includes many pathogens such as M. tuberculosis and M. leprae). Traag et al. (2010) document the problems with the paper:
Here, we report that the genomes of Mycobacterium species and those of other high G+C Gram-positive bacteria lack orthologs of many, if not all, highly conserved genes diagnostic of endospore formation in the genomes of low G+C Gram-positive bacteria. We also failed to detect the presence of endospores by light microscopy or by testing for heat-resistant colony-forming units in aged cultures of M. marinum. Finally, we failed to recover heat-resistant colony-forming units from frogs chronically infected with M. marinum. We conclude that it is unlikely that Mycobacterium is capable of endospore formation.
As ID proponents know only too well, the peer-review system has not only become corrupted in allowing substandard content into the academic market. It has also been turned into a gate-keeping system for imposing ideological conformity. Recently, an editor resigned over the publication of a seminal article by Roy Spencer and William Braswell. The paper's purpose was to demonstrate that one of the feedbacks that the Intergovernmental Panel on Climate Change has been treating as a positive feedback is really a negative feedback. You can read Roy Spencer's defense of his paper here.
In a similar incident in 2004, Smithsonian Institute evolutionary biologist Richard Sternberg was punished and pressured to resign following the publication of a pro-ID article by Stephen C. Meyer in a journal of which Sternberg was the editor.
In still another incident, a recent pro-ID paper authored by mathematician Granville Sewell was retracted from publication (after it had been subjected to peer-review and approved) as the result of a complaint from a blogger writing to the journal's editor. The journal, Applied Mathematics Letters has since apologized and paid $10,000 in compensation to Dr. Sewell.
What's to be done? Colquhoun makes the following recommendation:
There is an alternative: publish your paper yourself on the web and open the comments. This sort of post-publication review would reduce costs enormously, and the results would be open for anyone to read without paying. It would also destroy the hegemony of half a dozen high-status journals.
And, indeed, this is exactly how the Biologic Institute-associated journal Bio-Complexity operates. This peer-reviewed journal, dedicated to discussions surrounding the respective scientific merits of neo-Darwinian evolution and intelligent design, is published freely on the web and is open for comments and published responses, hence allowing -- even encouraging -- post-publication review.
Colquhoun further suggests,
...it would be essential to allow anonymous comments. Most reviewers are anonymous at present, so why not online? Second, the vast flood of papers that make the present system impossible should be stemmed. I'd suggest scientists should limit themselves to an average of two original papers a year. They should also be limited to holding one research grant at a time. Anyone who thought their work necessitated more than this would have to be scrutinized very carefully. It's well known that small research groups give better value than big ones, so that should be the rule.
The benefit of such a system, as Colquhoun notes, is that "With far fewer papers being published, reviewers, grant committees and promotion committees might be able to read the papers, not just count them."
Colquhoun is to be commended. The goal of the peer-review system ought to be the ensuring of factual accuracy and the highlighting of necessary revisions and corrections. Its goal should not be the enforcement of ideological and paradigmatic conformity, nor should it be the upholding of "consensus science." Post-publication review ought to be encouraged, and moves should be made to make journal content more frequently open-access.
No comments:
Post a Comment