Fleming's discovery of penicillin couldn't get published today. That's a huge problem
Updated by Julia Belluz on December 14, 2015, 7:00 a.m. ET
After toiling away for months on revisions for a single academic paper, Columbia University economist Chris Blattman started wondering about the direction of his work.
He had submitted the paper in question to one of the top economics journals earlier this year. In return, he had gotten back nearly 30 pages of single-space comments from peer reviewers (experts in the field who provide feedback on a scientific manuscript). It had taken two or three days a week over three months to address them all.
So Blattman asked himself some simple but profound questions: Was all this work on a single study really worth it? Was it best to spend months revising one study — or could that time have been better spent on publishing multiple smaller studies? He wrote about the conundrum on his blog:
Some days my field feels like an arms race to make each experiment more thorough and technically impressive, with more and more attention to formal theories, structural models, pre-analysis plans, and (most recently) multiple hypothesis testing. The list goes on. In part we push because want to do better work. Plus, how else to get published in the best places and earn the respect of your peers?
It seems to me that all of this is pushing social scientists to produce better quality experiments and more accurate answers. But it’s also raising the size and cost and time of any one experiment.
Over the phone, Blattman explained to me that in the age of "big data," high-quality scientific journals are increasingly pushing for large-scale, comprehensive studies, usually involving hundreds or thousands of participants. And he's now questioning whether a course correction is needed.
Though he can't prove it yet, he suspects social science has made a trade-off: Big, time-consuming studies are coming at the cost of smaller and cheaper studies that, taken together, may be just as valuable and perhaps more applicable (or what researchers call "generalizable") to more people and places.
Do we need more "small" science?
Over in Switzerland, Alzheimer's researcher Lawrence Rajendran has been asking himself a similar question: Should science be smaller again? Rajendran, who heads a laboratory at the University of Zurich, recently founded a journal called Matters. Set to launch in early 2016, the journal aims to publish "the true unit of science" — the observation.
Rajendran notes that Alexander Fleming’s simple observation that penicillin mold seemed to kill off bacteria in his petri dish could never be published today, even though it led to the discovery of lifesaving antibiotics. That's because today's journals want lots of data and positive results that fit into an overarching narrative (what Rajendran calls "storytelling") before they'll publish a given study.
"You would have to solve the structure of penicillin or find the mechanism of action," he added.
But research is complex, and scientific findings may not fit into a neat story — at least not right away. So Rajendran and the staff at Matters hope scientists will be able to share insights in this journal that they may not been able to publish otherwise. He also thinks that if researchers have a place to explore preliminary observations, they may not feel as much pressure to exaggerate their findings in order to add all-important publications to their CVs.
Smaller isn't always better
Science has many structural problems to grapple with right now: The peer review system doesn't function all that well, many studies are poorly designed so their answers are unreliable, and replications of experiments are difficult to execute and very often fail. Researchers have estimated that about $200 billion — or about 85 percent of global spending on research — is routinely wasted on poorly designed and redundant studies.
A big part of the reason science funders started emphasizing large-scale studies is because they were trying to avoid common problems with smaller studies: The results aren't statistically significant, and the sample sizes may be too tiny and therefore unrepresentative.
It's not clear that emphasizing smaller-scale studies and observations will solve these problems. In fact, publishing more observations may just add to the noise. But as Rajendran says, it's very possible that important insights are being lost in the push toward large-scale science. "Science can be small, big, cure diseases," he said. "It can just be curiosity-driven. Academic journals shouldn't block the communication of small scientific observations."
Updated by Julia Belluz on December 14, 2015, 7:00 a.m. ET
After toiling away for months on revisions for a single academic paper, Columbia University economist Chris Blattman started wondering about the direction of his work.
He had submitted the paper in question to one of the top economics journals earlier this year. In return, he had gotten back nearly 30 pages of single-space comments from peer reviewers (experts in the field who provide feedback on a scientific manuscript). It had taken two or three days a week over three months to address them all.
So Blattman asked himself some simple but profound questions: Was all this work on a single study really worth it? Was it best to spend months revising one study — or could that time have been better spent on publishing multiple smaller studies? He wrote about the conundrum on his blog:
Some days my field feels like an arms race to make each experiment more thorough and technically impressive, with more and more attention to formal theories, structural models, pre-analysis plans, and (most recently) multiple hypothesis testing. The list goes on. In part we push because want to do better work. Plus, how else to get published in the best places and earn the respect of your peers?
It seems to me that all of this is pushing social scientists to produce better quality experiments and more accurate answers. But it’s also raising the size and cost and time of any one experiment.
Over the phone, Blattman explained to me that in the age of "big data," high-quality scientific journals are increasingly pushing for large-scale, comprehensive studies, usually involving hundreds or thousands of participants. And he's now questioning whether a course correction is needed.
Though he can't prove it yet, he suspects social science has made a trade-off: Big, time-consuming studies are coming at the cost of smaller and cheaper studies that, taken together, may be just as valuable and perhaps more applicable (or what researchers call "generalizable") to more people and places.
Do we need more "small" science?
Over in Switzerland, Alzheimer's researcher Lawrence Rajendran has been asking himself a similar question: Should science be smaller again? Rajendran, who heads a laboratory at the University of Zurich, recently founded a journal called Matters. Set to launch in early 2016, the journal aims to publish "the true unit of science" — the observation.
Rajendran notes that Alexander Fleming’s simple observation that penicillin mold seemed to kill off bacteria in his petri dish could never be published today, even though it led to the discovery of lifesaving antibiotics. That's because today's journals want lots of data and positive results that fit into an overarching narrative (what Rajendran calls "storytelling") before they'll publish a given study.
"You would have to solve the structure of penicillin or find the mechanism of action," he added.
But research is complex, and scientific findings may not fit into a neat story — at least not right away. So Rajendran and the staff at Matters hope scientists will be able to share insights in this journal that they may not been able to publish otherwise. He also thinks that if researchers have a place to explore preliminary observations, they may not feel as much pressure to exaggerate their findings in order to add all-important publications to their CVs.
Smaller isn't always better
Science has many structural problems to grapple with right now: The peer review system doesn't function all that well, many studies are poorly designed so their answers are unreliable, and replications of experiments are difficult to execute and very often fail. Researchers have estimated that about $200 billion — or about 85 percent of global spending on research — is routinely wasted on poorly designed and redundant studies.
A big part of the reason science funders started emphasizing large-scale studies is because they were trying to avoid common problems with smaller studies: The results aren't statistically significant, and the sample sizes may be too tiny and therefore unrepresentative.
It's not clear that emphasizing smaller-scale studies and observations will solve these problems. In fact, publishing more observations may just add to the noise. But as Rajendran says, it's very possible that important insights are being lost in the push toward large-scale science. "Science can be small, big, cure diseases," he said. "It can just be curiosity-driven. Academic journals shouldn't block the communication of small scientific observations."
