Peer Review VI

Peer review is at the heart of knowledge discovery. It is thus no surprise it is frequently subject of discussion on this and other science blogs. The other day I read a great post by Cameron Neylon, Peer Review: What is it good for? It occurred to me that instead of writing the 124th comment there, it might be more fruitful if we discuss here what I've - oddly enough - repeatedly suggested on other's blogs, but evidently not on my own.

To first make sure we're on the same page, here's how peer review works today. You write a manuscript and submit it to a journal, where it is assigned to an editor. Exactly what happens then depends somewhat on the journal. In some cases the editors sort out a lot of submissions already at this stage. At very high impact journals like Science and Nature, editors reject the vast majority of papers not because they're wrong but because they're not deemed relevant enough, but these journals are exceptions. In the general case, your manuscript will be assigned a number and be sent to one or several of your colleagues who (are thought to) work on related topics and who are asked to write a report on your paper. That's the peer review.

The assignment of people to your paper doesn't always work too great, so they have a chance to decline and suggest somebody else. At least in my experience in most cases the manuscript will be sent to two people. Some journals settle on one, some do three. In some cases they might opt for an additional report later on. In any case, the reviewers are typically asked for a written assessment of the manuscript, that's the most important part. They are also asked whether the manuscript is suitable for that particular journal. And typically they'll have to judge the manuscript on a 5 point scale on how interesting the content/good the presentation is etc. Finally, they'll have to make a recommendation on whether the paper is suitable for publication in the present form, whether revisions should be done, have to be done, or whether it's not suitable for publication. The reviewer's judgement is anonymous. The author does not generally know who they are. (That's the default. There is in principle nothing prohibiting the reviewer from contacting the author directly, but few do that. And in some cases it's just easy to guess who wrote a report.)

In most cases the report will say say revisions have to be done. You then revise the paper, reply to the queries and resubmit. The exchange is mediated by the editor and can go back and forth several times, though most journals keep this process to a few resubmissions only. The whole procedure can take anything between a few weeks and several years. In the end your paper might get rejected anyway. While in some cases one learns something from the reports, the process is mostly time-consuming and frustrating. The final decision on your manuscript is made by the editor. They will generally, but not always, follow the advice of the reports.

On the practical side, the manuscript- and review-submission is typically done through a web interface. Every publisher has their own labyrinth of usernames, passwords, access codes, manuscript-, article tracking-, and other -numbers. The most confusing part for me is that some journals assign different author and reviewer accounts to the same person. I typically write reports on one or two papers per month. That's more than ten times the number of manuscripts I submit. There are very few journals who pay reviewers for their report. JHEP announced some while back they would. Most people do it as a community service.

There are several well-known problems with this procedure. Most commonly bemoaned is that since the reviewers are anonymous they might abuse the report to their own advantage. That might be in the form of recommending the author cites the reviewer's papers (you're well advised to do that). In the worst case they might reject a paper not because they found a mistake but because it's incompatible with their own convictions. One can decline to review a paper in this case due to "conflict of interest," but well. We have all conflicting interests, otherwise we'd all be doing the same thing. What happens more often though is that the reviewer didn't read or at least not think about the paper and writes a sloppy report that might accept a wrong or decline a right paper mistakenly.

I have written many times that the way to address these community-based problems - to the extend it's possible - is to make sure researchers' sole concern is the advancement of science and not other sorts of financial pressure, public pressure or time pressure that might make it seem smart to down-thumb competitors. But that's not the aspect I want to focus on here.

Instead I want to focus here on the practical problems. One is that the exchange between the author and the reviewers is unnecessary slow and cumbersome. Submitting a written report and having it passed on by the editor must be a relic from the times when such a report was carved in stone and shipped across the ocean. It would be vastly preferable if journals would instead provide an online interface to communicate with the author that preserves the anonymity of the reviewer. This would allow the reviewer to ask some quick, clarifying questions and not only make their task easier, but also prevent misunderstandings. Typically, if the reviewer has misunderstood something in your manuscript you have to do a lot of tip-toeing to tell them they're wrong without telling them they're wrong because, you see, the editor is reading every word and knows who they are. The idea isn't that such a communication interface should replace the report, just that it could accompany the review process and that a written report is submitted after some determined amount of time.

Another practical problem with today's peer review process is that when your paper gets rejected and you submit it to another journal, you have to start all over again. This multiplies efforts on both, the authors' and the reviewers', sides. This is one of the reasons why I'm suggesting to decouple the peer review process from the journals. The idea is simply that the author would submit their manuscript not for publication to a journal, but first to some independent agency to obtain a report (or several). This report would then be submitted with the manuscript to the journal. The journal will still have to assess whether the manuscript is suitable for this particular journal, but the point is that the author has a stamp of legitimacy independent from a journal reference. It's up to the author then what to do with the report. You could for example just use the report (in some summarized form) with an arxiv upload.

There could be several of such peer review agencies, and they might operate differently. Eg some might pay the reviewer, some might not. Some might ask for a fee, some might not. One would see which works best. In the long run I would expect the reports' relevance to depend on the reputation of the agency.

This would address several problems at once. One is the by the open-access movement frequently criticized "power of journals." That's because in many, if not most, fields a journal reference is still considered a sign of quality of your work. But journal subscriptions are very costly, and thus the pressure to publish in journals often means scientific studies are not freely accessible for the public. The reason is simply that today subscription journals are the main providers of peer review. But there is actually no reason for this connection. Decoupling publication from the review process would remove that problem. It would also address the above mentioned problem that when your manuscript gets rejected from one journal you have to start all over again with the peer review. Finally, it would add more diversity to the procedure in that the reviewers and authors could chose which agency suits them best. For example, some might have an anonymous, and some an open review process. Time would tell which in the end is more credible.

There is the question of financing of course. Again, there could be several models for that. As I've said many times before, I think if it's a public service, it should be financed like a public service. That is to say, since peer review is so essential to scientific progress, such peer review agencies should be eligible to apply for governmental funding. One would hope though that, as today, most of the time and effort would be carried by the researchers themselves as a community service.

See also: Peer Review V, Peer Review IV, Peer Review III, and Peer Review II.