Should we have faith in science? Part II: peer-reviewed science papers
Thursday, February 26, 2015 - 12:51
Kirk Durston
The primary way scientific discoveries and advances are disseminated is through peer-reviewed papers published in scientific journals. The first step is to submit a paper to a journal. Those that survive preliminary filtering by the editor are sent out to be reviewed by qualified scientists in the field. On the basis of the reviewers’ recommendations, the paper is accepted or rejected. Only a fraction of papers submitted for publication make it through this peer-review process and are published.
One would hope that such a process would justify a high level of confidence in scientific publications, but recent findings suggest that our faith in peer-reviewed publications in mainstream journals of science may be on somewhat shaky ground.
The journal Nature, in a paper calling for increased standards in pre-clinical research, revealed that out of 53 papers presenting ‘landmark’ published findings in the field of haematology and oncology, only 6 could be confirmed by subsequent laboratory teams. For the 89% of papers that failed to have their results reproduced, it was found that blind control group analyses was inadequate or data had been selected to support the hypothesis and other data set aside.
Worse still, some of the papers that could not be experimentally reproduced, launched ‘an entire field, with hundreds of secondary publications that expanded on elements of the original observation, but did not actually seek to confirm or falsify its fundamental basis’.
Hundreds of other peer-reviewed, published science papers based on faulty initial papers!
Nature reported in October 2011 that although the number of submissions had increased by 44% over the past ten years, the number of retractions had increased by roughly 900%.
Austin Hughes, in a paper published in the Proceedings of the National Academy of Sciences, focusing on the origin of adaptive phenotypes laments, ‘Thousands of papers are published every year claiming evidence of adaptive evolution on the basis of computational analyses alone, with no evidence whatsoever regarding the phenotypic effects of allegedly adaptive mutations.’ He concludes that ‘This vast outpouring of pseudo-Darwinian hype has been genuinely harmful to the credibility of evolutionary biology as a science.’ Jerry Fodor and Massimo Piattelli-Palmarini write in New Scientist,
"Much of the vast neo-Darwinian literature is distressingly uncritical. The possibility that anything is seriously amiss with Darwin’s account of evolution is hardly considered. … The methodological skepticism that characterizes most areas of scientific discourse seems strikingly absent when Darwinism is the topic."
How can we distinguish the good papers from the poor? This can be very difficult without actually attempting to reproduce their findings. Short of that, apply the same critical thinking skills and healthy skepticism to scientific papers that you do for political, historical or religious claims. 21st century science can often be heavily influenced by poor experimental practices, unproven computational models, political agendas, competition for funding, and scientism (atheism dressed up as science). When going over a paper ask questions like, how large was the data set? What sort of statistical analysis was performed? Are there other papers that independently support or disconfirm these findings? What is not being discussed? One thing for sure, don’t accept something simply because ‘hundreds’ or even ‘thousands’ of papers say so, especially if Darwinian evolution is the topic. Practice critical thinking with the question in the back of your mind, 'Is this one of those papers that will be retracted?'.
Read Part III
Further reading:
http://www.nature.com/nature/journal/v483/n7391/full/483531a.html#t1
http://www.nature.com/news/publishing-the-peer-review-scam-1.16400
http://www.nytimes.com/2012/04/17/science/rise-in-scientific-journal-retractions-prompts-calls-for-reform.html?ref=science
http://dingo.sbs.arizona.edu/~massimo/publications/PDF/JF_MPP_darwinisms...
http://www.nytimes.com/2015/06/16/science/retractions-coming-out-from-under-science-rug.html?
Thursday, February 26, 2015 - 12:51
Kirk Durston
The primary way scientific discoveries and advances are disseminated is through peer-reviewed papers published in scientific journals. The first step is to submit a paper to a journal. Those that survive preliminary filtering by the editor are sent out to be reviewed by qualified scientists in the field. On the basis of the reviewers’ recommendations, the paper is accepted or rejected. Only a fraction of papers submitted for publication make it through this peer-review process and are published.
One would hope that such a process would justify a high level of confidence in scientific publications, but recent findings suggest that our faith in peer-reviewed publications in mainstream journals of science may be on somewhat shaky ground.
The journal Nature, in a paper calling for increased standards in pre-clinical research, revealed that out of 53 papers presenting ‘landmark’ published findings in the field of haematology and oncology, only 6 could be confirmed by subsequent laboratory teams. For the 89% of papers that failed to have their results reproduced, it was found that blind control group analyses was inadequate or data had been selected to support the hypothesis and other data set aside.
Worse still, some of the papers that could not be experimentally reproduced, launched ‘an entire field, with hundreds of secondary publications that expanded on elements of the original observation, but did not actually seek to confirm or falsify its fundamental basis’.
Hundreds of other peer-reviewed, published science papers based on faulty initial papers!
Nature reported in October 2011 that although the number of submissions had increased by 44% over the past ten years, the number of retractions had increased by roughly 900%.
Austin Hughes, in a paper published in the Proceedings of the National Academy of Sciences, focusing on the origin of adaptive phenotypes laments, ‘Thousands of papers are published every year claiming evidence of adaptive evolution on the basis of computational analyses alone, with no evidence whatsoever regarding the phenotypic effects of allegedly adaptive mutations.’ He concludes that ‘This vast outpouring of pseudo-Darwinian hype has been genuinely harmful to the credibility of evolutionary biology as a science.’ Jerry Fodor and Massimo Piattelli-Palmarini write in New Scientist,
"Much of the vast neo-Darwinian literature is distressingly uncritical. The possibility that anything is seriously amiss with Darwin’s account of evolution is hardly considered. … The methodological skepticism that characterizes most areas of scientific discourse seems strikingly absent when Darwinism is the topic."
How can we distinguish the good papers from the poor? This can be very difficult without actually attempting to reproduce their findings. Short of that, apply the same critical thinking skills and healthy skepticism to scientific papers that you do for political, historical or religious claims. 21st century science can often be heavily influenced by poor experimental practices, unproven computational models, political agendas, competition for funding, and scientism (atheism dressed up as science). When going over a paper ask questions like, how large was the data set? What sort of statistical analysis was performed? Are there other papers that independently support or disconfirm these findings? What is not being discussed? One thing for sure, don’t accept something simply because ‘hundreds’ or even ‘thousands’ of papers say so, especially if Darwinian evolution is the topic. Practice critical thinking with the question in the back of your mind, 'Is this one of those papers that will be retracted?'.
Read Part III
Further reading:
http://www.nature.com/nature/journal/v483/n7391/full/483531a.html#t1
http://www.nature.com/news/publishing-the-peer-review-scam-1.16400
http://www.nytimes.com/2012/04/17/science/rise-in-scientific-journal-retractions-prompts-calls-for-reform.html?ref=science
http://dingo.sbs.arizona.edu/~massimo/publications/PDF/JF_MPP_darwinisms...
http://www.nytimes.com/2015/06/16/science/retractions-coming-out-from-under-science-rug.html?
No comments:
Post a Comment