- Why Current Publication Practices May Distort Science
By Neal S. Young, John P. A. Ioannidis, Omar Al-Ubaydli
PLoS Med 5(10): e201
Abstract: This essay makes the underlying assumption that scientific information is an economic commodity, and that scientific journals are a medium for its dissemination and exchange. While this exchange system differs from a conventional market in many senses, including the nature of payments, it shares the goal of transferring the commodity (knowledge) from its producers (scientists) to its consumers (other scientists, administrators, physicians, patients, and funding agencies). The function of this system has major consequences. Idealists may be offended that research be compared to widgets, but realists will acknowledge that journals generate revenue; publications are critical in drug development and marketing and to attract venture capital; and publishing defines successful scientific careers. Economic modelling of science may yield important insights.
Though the authors constantly talk about 'science' generally, upon closer look it turns out that their concern is actually with biomedical research. This becomes particularly clear if one looks at one of the author's previous essays “Why Most Published Research Findings Are False" (PLoS Med 2(8): e124), which is primarily concerned with lacking reproduction of data finds in various fields of life sciences, failure to pay sufficient attention to negative finds, and the tendency to temper with samples to 'improve' statistical significance with the effect of skewing results.
However, I do think that some of the problems the authors raise are present also in my field of research. For example, they point out “For much (most?) scientific work, it is difficult or impossible to immediately predict future value, extensions, and practical applications,” but nevertheless an early and quite rigorous selection and branding process takes place. Here with branding they refer to getting published in journals with high-impact factor, after a selection through peer review and editors. Since getting a high quality 'branding' is important, scientists adapt strategies to succeed according to these criteria. Meanwhile, journals “strive to attract specific papers, such as influential trials that generate publicity and profitable reprint sales” and try to increase their impact factor:
“Impact factors are widely adopted as criteria for success, despite whatever qualms have been expressed. They powerfully discriminate against submission to most journals, restricting acceptable outlets for publication. “Gaming” of impact factors is explicit. Editors make estimates of likely citations for submitted articles to gauge their interest in publication.”
They proceed with discussing an economic analogy in which the forced selection of some few research findings as being suitable for the desired publication in high-impact journals is an example for 'artificial scarcity': Even though a commodity (here, journal publication) exists in abundance, it is restricted in access, distribution or availability to make it rare and raise its value.
This has disadvantages that lie at hand. The importance of getting published can lead to increased conformity of research interests (“herding”) since outlying topics are risky, and it favours publishing of new and surprising findings over negative results that would attract less attention: “Negative or contradictory data may be discussed at conferences or among colleagues, but surface more publicly only when dominant paradigms are replaced.”
Many of this might sound familiar to you from my earlier post We have only ourselves to judge on each other, where I argued that increasing pressure (like financial and peer pressure) has the side-effect that marketing tactics become important for scientific topics, which goes on the expenses of objectivity and open criticism.
The authors then go on to closer investigate their criticism of the current publishing system and offer 10 options to deal with the problems that I want to briefly comment on:
Potential Competing or Complementary Options and Solutions for Scientific PublicationMy thoughts on these suggestions are:
- Accept the current system as having evolved to be the optimal solution to complex and competing problems.
- Promote rapid, digital publication of all articles that contain no flaws, irrespective of perceived “importance”.
- Adopt preferred publication of negative over positive results; require very demanding reproducibility criteria before publishing positive results.
- Select articles for publication in highly visible venues based on the quality of study methods, their rigorous implementation, and astute interpretation, irrespective of results.
- Adopt formal post-publication downward adjustment of claims of papers published in prestigious journals.
- Modify current practice to elevate and incorporate more expansive data to accompany print articles or to be accessible in attractive formats associated with high-quality journals: combine the “magazine” and “archive” roles of journals.
- Promote critical reviews, digests, and summaries of the large amounts of biomedical data now generated.
- Offer disincentives to herding and incentives for truly independent, novel, or heuristic scientific work.
- Recognise explicitly and respond to the branding role of journal publication in career development and funding decisions.
- Modulate publication practices based on empirical research, which might address correlates of longterm successful outcomes (such as reproducibility, applicability, opening new avenues) of published papers.
- Though I suspect the authors included this point to have the reader realize this is not a preferable option and thus action required, one should keep in mind that after all the system is so far not a complete disaster. People keep telling me peer review is failing, journals are dead, and other slogans of that sort. But I can easily imagine ways to make the situation even worse, and everybody who wants to 'improve' the system should make sure that improvement doesn't have unwanted side-effects that are even more distorting.
- I can't but think the authors missed an essential point with this claim. Even if one forgets about the impact factor, there is a huge pressure to publish, period. Scientists are publishing more and more, and one of the main reasons, so I think, peer review is struggling is that the more papers are produced, the less time there is to review them. Plus the problem that this time-consuming process isn't well acknowledge. Rapid distribution of all papers that 'contain no flaws' irrespective of their importance sounds like a great goal, but won't be feasible without changes in the publication and review culture. For example, one could consider my suggestions to have a division of labor in task.
- As much as I am in favor of publishing negative results, I don't see why they should be preferred over positive results. A result is a result and should be treated equally.
- Well, yes, of course. That's what science is all about. You don't decide whether a paper is worth publishing depending on whether you like the result or not.
- That is an interesting suggestion indeed that would directly discourage exaggerated claims. Just that I don't know how this could be done in practice.
- This is already the case. Increasingly more journals offer supplements of various type.
- Another very good suggestion. This can however only work if such reviews and summaries are a work acknowledged as an important contribution by the community. Otherwise it will just be regarded as a waste of time, time in which one could do 'real research'.
- That would indeed be good, but again it remains unclear to me how so. As usual however I am skeptic about setting incentives in a specific way because if done so without any sensible feedback mechanism, these incentives might develop their own life as well and eventually have backlashes in counterproductive strategies, that leaves people aiming to fulfil certain given secondary criteria instead of primary goals (for more on secondary criteria and primary goals, see The Marketplace of Ideas).
- That is basically what I usually refer to as creating awareness about the problem. Without that, nothing works. The paper does quite a good job with that.
- That's a question of science policy, together with the need to have more studies of what practices lead to desirable long-term outcomes. I totally agree this is a topic that needs to be payed more attention to, and that in a practical way such that results lead to change. As much as I like the general sense of the paper, on the practical side it is somewhat weak.
The authors conclude by asking whether we have created a system for the exchange of scientific ideas that serves best “the striving for knowledge,” the “search for truth” and “the free competition of thought”. It seems pretty clear that the present system does not serve best this purpose. The topic investigated in the article is a good example for what I mean with management and organization of knowledge. Scientific publishing is an essential ingredient to structuring and disseminating scientific knowledge, and we should pay close attention to this process not negatively affecting the way scientists chose or present research topics. I find it overly simplistic though to put the blame on the publishers. To a much larger extend the problem is caused by scientists accepting the system they are faced with and complying to the pressures excerted on them without asking about the long-term effects.
Neal S. Young, John P. A. Ioannidis, Omar Al-Ubaydli (2008). Why Current Publication Practices May Distort Science PLoS Medicine, 5 (10) DOI: 10.1371/journal.pmed.0050201