“The journal-operated system of peer-review, Lisi says, "is severely broken." On this point, he couldn't find a stronger ally than the science blogosphere. Most of the ScienceBloggers are unwavering advocates of the Open Access (OA) movement, and two of them—Bora Zivkovic, of PLoS ONE, and John Wilbanks, of Science Commons—devote all of their time to it.”
At least she mentions “Most OA advocates are quick to point out that open-access doesn't necessarily mean the end of publishers or peer-review.” Indeed. So that then leaves me to wonder what kind of allies on exactly what does Garrett find in the science blogosphere?
There are three reasons why statements like this upset me: they are a) unverified b) self-fulfilling and c) unconstructive. Let me elaborate:
- a) I actually find the peer review process useful, and I also think it is necessary. There are many people who otherwise would not get any qualified feedback on their work. I am lucky to have colleagues to discuss with, who I can ask for opinions, references, or keywords. But not everybody is that lucky. I certainly agree that peer review can be quite painful, and I have received my share of completely nonsensical reports by referees who evidently didn't read more than the abstract. On other occasions however referees have pointed out important issues, and made suggestions for improvement, and even for further studies. So where are all these people who allegedly think the peer review process is broken?
b) I frequently referee papers. I do my best trying to understand the author's work, and to write a useful review. This takes time, time I don't have for my own work, and the only thing I get for it is an automated Thank-you email from the publisher. I do it because I believe that peer review is an essential part of the organization of scientific knowledge and important for progress, but it only works if enough people participate constructively. What we'd need is to encourage people to take it more seriously, and not proclaim it is 'broken'.
c) What are the alternatives? Some people like to advocate 'open peer review,' which seems to mean you put your paper somewhere on the web and hope you'll get comments. This, excuse me, is hopelessly naive. The vast majority of papers would never get any comments. Heck, the vast majority of papers probably wouldn't even get read if it wasn't for peer review. Do me the favour and think two steps ahead. We would be running into a situation in which the well-known people and the well-established topics receive a lot of 'reviews' and a lot of attention, whereas the vast body of work will never get the necessary stamp of having been critically read by somebody with an adequate education. As a consequence, a large fraction of serious researchers would step down on the same level with all the weirdos and their backyard theories that never get published. Sorry, but I really don't want to be a scientist under such circumstances.
That having been said, I certainly don't think peer review works very well. My largest frustation is that people don't take it seriously. It has happened in many instances that I wrote a long report on a flawed paper and recommendend rejection, only to see later that the paper got published in a different journal in exactly the same version. Evidently, the authors were not even remotely interested in improving their work. The biggest problem however is just that we are writing way too many papers. Obviously, the more papers we write, the less time we have to read and comment on other people's papers. If you want to fix the peer review process there's then two easy things to do:
1) Lower pressure on researchers to produce papers.
2) Encourage refereeing e.g. by pointing out its relevance or by providing incentives.
Related: See my comment on the paper “Why Current Publication Practices May Distort Science” in which the role of publishing as a branding process was studied, and my posts Peer Review IV, Peer Review III, Peer Review II.