Open peer review and its discontents.

Some days ago, I commented on an arxiv paper that had been promoted by the arxiv blog (which, for all I know, has no official connection with the arxiv). This blogpost had an aftermath that gave me something to think.

Most of the time when I comment on a paper that was previously covered elsewhere, it’s to add details that I found missing. More often than not, this amounts to a criticism which then ends up on this blog. If I like a piece of writing, I just pass it on with approval on twitter, G+, or facebook. This is to explain, in case it’s not obvious, that the negative tilt of my blog entries is selection bias, not that I dislike everything I haven’t written myself.

The blogpost in question pointed out shortcomings of a paper. Trying to learn from earlier mistakes, I was very explicit about what that means, namely that the conclusion in the paper isn’t valid. I’ve now written this blog for almost nine years, and it has become obvious that the careful and polite scientific writing style plainly doesn’t get across the message to a broader audience. If I write that a paper is “implausible,” my colleagues will correctly parse this and understand I mean it’s nonsense. The average science journalist will read that as “speculative” and misinterpret it, either accidentally or deliberately, as some kind of approval.

Scientists also have a habit of weaving safety nets with what Peter Woit once so aptly called ‘weasel words’, ambiguous phrases that allow them on any instance to claim they actually meant something else. Who ever said the LHC would discover supersymmetry? The main reason you most likely perceive the writing on my blog as “unscientific” is lack of weasel words. So I put my head out here on the risk of being wrong without means of backpedalling, and as a side-effect I often come across as actively offensive.

If I got a penny each time somebody told me I’m supposedly “aggressive” because I read Strunk’s `Elements of Style,’ then I’d at least get some money for writing. I’m not aggressive, I’m expressive! And if you don’t buy that, I’ll hit some adjectives over your head. You can find them weasel words in my papers though, in the plenty, with lots of ifs and thens and subjunctives, in nested subordinate clauses with 5 syllable words just to scare off anybody who doesn’t have a PhD.

In reaction to my, ahem, expressive blogpost criticizing the paper, I very promptly got an email from a journalist, Philipp Hummel, who was writing on an article about the paper for spectrum.de, the German edition of Scientific American. His article has meanwhile appeared, but since it’s in German, let me summarize it for you. Hummel didn’t only write about the paper itself, but also about the online discussion around it, and the author’s, mine, and other colleagues’ reaction to it.

Hummel wrote by email he found my blogpost very useful and that he had also contacted the author asking for a comment on my criticism. The author’s reply can be found in Hummel’s article. It says that he hadn’t read my blogpost, wouldn’t read it, and wouldn’t comment on it either because he doesn’t consider this proper ‘scientific means’ to argue with colleagues. The proper way for me to talk to him, he let the journalist know, is to either contact him or publish a reply on the arxiv. Hummel then asked me what I think about this.

To begin with I find this depressing. Here’s a young researcher who explicitly refuses to address criticism on his work, and moreover thinks this is proper scientific behavior. I could understand that he doesn’t want to talk to me, evil aggressive blogger that I am, but that he refuses to explain his research to a third party isn’t only bad science communication, it’s actively damaging the image of science.

I will admit I also find it slightly amusing that he apparently believes I must have an interest talking to him, or in him talking to me. That all the people whose papers I have once commented on show up wanting to talk is stuff of my nightmares. I’m happy if I never hear from them again and can move on. There’s lots of trash out there that needs to be beaten.

That paper and its author, me, and Hummel, we’re of course small fish in the pond, but I find this represents a tension that presently exists in much of the scientific community. A very prominent case was the supposed discovery of “arsenic life” a few years ago. The study was exposed flawed by online discussion. The arsenic authors refused to comment on this, arguing that:

“Any discourse will have to be peer-reviewed in the same manner as our paper was, and go through a vetting process so that all discussion is properly moderated […] This is a common practice not new to the scientific community. The items you are presenting do not represent the proper way to engage in a scientific discourse and we will not respond in this manner.”

Naïve as I am, I thought that theoretical physics is less 19th century than that. But now it seems to me this outdated spirit is still alive also in the physics community. There is a basic misunderstanding here about necessity and use of peer review, and the relevance of scientific publication.

The most important aspect of peer review is that it assures that a published paper has been read at least by the reviewers, which otherwise wouldn’t be the case. Public peer review will never work for all papers simply because most papers would never get read. It works just fine though for papers that receive much attention, and in these cases anonymous reviewers aren’t any better than volunteer reviewers with similar scientific credentials. Consequently, public peer review, when it takes place, should be taken as least as seriously as anonymous review.

Don’t get me wrong, I don’t think that all scientific discourse should be conducted in public. Scientists need private space to develop their ideas. I even think that most of us go out with ideas way too early, because we are under too much pressure to appear productive. I would never publicly comment on a draft that was sent to me privately, or publicize opinions voiced in closed meetings. You can’t hurry thought.

However, the moment you make your paper publicly available you have to accept that it can be publicly commented on. It isn’t uncommon for researchers, even senior ones, to have stage fright upon arxiv submission for this reason. Now you’ve thrown your baby into the water and have to see whether it swims or sinks.

Don’t worry too much, almost all babies swim. That’s because most of my colleagues in theoretical physics entirely ignore papers that they think are wrong. They are convinced that in the end only truth will prevail and thus practice live-and-let-live. I used to do this too. But look at the evidence: it doesn’t work. The arxiv now is full with paid research so thin a sneeze could wipe it out. We seem to have forgotten that criticism is an integral part of science, it is essential for progress, and for cohesion. Physics leaves me wanting more every year. It is over-specialized into incredibly narrow niches, getting worse by the day.

Yes, specialization is highly efficient to optimize existing research programs, but it is counterproductive to the development of new ones. In the production line of a car, specialization allows to optimize every single move and every single screw. And yet, you’ll never arrive at a new model listening to people who do nothing all day than looking at their own screws. For new breakthroughs you need people who know a little about all the screws and their places and how they belong together. In that production line, the scientists active in public peer review are the ones who look around and say they don’t like their neighbor’s bolts. That doesn’t make for a new car, all right, but at least they do look around and they show that they care. The scientific community stands much to benefit from this care. We need them.

Clearly, we haven’t yet worked out a good procedure for how to deal with public peer review and with these nasty bloggers who won’t shut up. But there’s no going back. Public peer review is here to stay, so better get used to it.