Peer Review IV

[My weekend post on the recent survey on peer review reminded me I meant to write about a peer review problem that's only rarely mentioned: citations.]

Horizontal and Vertical Citations

Specifically, there seems to be a trend to increasingly cite horizontally instead of mostly vertically. What I mean with horizontal citations are citations of related works that go back to the same initial ideas or concepts, but are neither actually necessary to understand the content of a paper, nor have they investigated a closely related aspect of a problem. Citing horizontally is attached with a lot of politics. People cite others horizontally to be polite, because it seems smart for networking reasons, because they hope the favour will be returned, as a reply to annoying emails, or just because they believe conventions require it.

Typically it looks like this

"Recently, topic XYZ has been explored by many groups [6], [7], [8], [9], [10], [11], [12], [13], [14], [15], [16], [17], [18], [19], [20], [21], [22], [23], [24]."

As an example, see 0706.3155 ( "Collider Phenomenology of Unparticle Physics" by Cheung, Keung, & Yuan). In the more advanced version one quotes simply as

"Many studies have investigated the implications of ... [11]."

or

"Various considerations of ... have recently been developed in the literature [8] [9]."

And then clumps together 25 papers in citation [11], see e.g. 0801.1534 ( "Unparticle Self-Interactions and Their Collider Implications" by Feng, Rajaraman, & Tu). For an extreme version, try reference [8] of 0801.0018 ( "Unparticle physics at the photon collider" by Kikuchi, Okada & Takeuchi) which fills more than a whole page, and is probably just a complete list of what an arXiv keyword search brought up.

This kind of citation seems to be especially common in the first some years after a topic received interest, but has a cut-off length that I'd estimate to be somewhere around 50 papers where it simply is no longer feasible. For example in the first years when black holes at the LHC where hip ('01-'04 or so), citations of the sort "in a number of recent papers people have studied..." (e.g. hep-ph/0405054 "SUSY Production From TeV Scale Blackhole at LHC" by Chamblin, Cooper & Nayak) were quite common, but around 2005 references condensed to review articles and the few papers where the idea originated.

Also, in my impression the more established researchers take horizontal citing less seriously (who are you to request I cite your paper?).

In contrast to this I mean with vertical citations the papers that were actually used for a new publication, that are necessary to its understanding (whether they are sufficient to understanding is a different issue), or previous work on the same topic (even if unfortunately unknown to the authors during writing). Of course scientists need to pay proper credit to other people's works, and to back up arguments with references. But should they just group-cite 'various considerations'?

Reasons

This kind of citation was a very useful feature in the days before one could do a keyword search in a database, or click on 'cited by'. Horizontal citing serves the purpose to let the interested reader know who else has worked on a given topic and what other related studies have already been done. However, this is a good example where one sees how technological improvements together with the increase of our community can result in developments that have unwanted consequences.

Consequences

Whether we like it or not, the citation index of a researcher matters to his or her career. If many people cite horizontally out of politeness - possible often without even reading all papers themselves - it encourages fast publications on hip topics. These works contain more horizontal citations, which makes the topic look even more like the place to be. Most importantly, researchers have to act fast to be among the earliest papers because then they makes it onto the citation lists if those who come later. A mechanism like this is called positive feedback: interest causes increasing interest. Nowadays, one can literally make a living out of jumping on and off topics with a good timing.

An effect like this can considerably distort scientist's judgement on which areas they regard worth spending their time on.

Another annoying side effect is that people try to get citations, just because it seems possible, and because every single one improves their cite index. As a result, if I put a paper on the arxiv, the following day I will receive several emails of the type

"Dear Prof. Hossenfelder,
Today I read your interesting paper on X. I want to draw your attention to my interesting paper(s) on Y. [latex bibitem follows] "

Depending on temper people request more or less bluntly to be mentioned in my reference list. In some cases these references are interesting and might be useful for later papers. In rare cases I did indeed miss a previous publication on the same topic, which is as annoying as embarrassing. In most cases though, people seem to send these emails for no other reasons than that the title or abstract of my paper contains a word that appears also in their paper. I actually knew a guy who wrote a script to check the new arxiv submissions for keywords and to produce emails like the one above.

And you know what? I can't even blame people for doing this. Even if chances are low, if you send out enough annoying emails one or the other recipient will just cite you, and isn't that what matters*? It's one of these cases where the incentives lead people to focus on meeting secondary criteria (high citation index) instead of primary goals (good research). For more on primary goals and secondary criteria, see The Marketplace of Ideas.

Peer Review

Although peer review does to a certain degree ensure relevant previous publications are appropriately mentioned (restrictions apply), it rarely happens that it is pointed out to the author he has plenty of redundant papers on the reference list. What people put on the arXiv is their business, but if it was clear peer reviewed publications wouldn't support the citation of only weakly related papers this trend would calm down considerably. That's why I think peer review would be the place to address the issue.

How to

If you don't want to cite everybody, don't cite anybody. It sounds silly but it seems people get easily pissed off if one cites a colleague who has worked on topic X, but not themselves. If one doesn't cite the colleague either, it doesn't bother them. Just sticking stubbornly to the publications that were actually used and are relevant to a work seems to be acceptable (and if that isn't sufficient blame it on a page limit). It has the drawback however that colleagues are less likely to return the favor and cite you - Science or Sociology?.

Bottomline

In times where keyword searches and 'cited by' queries are possible, horizontal citations are unnecessary. They have however the side-effect of causing a positive feedback on fashionable topics that can distort objectivity.

Disclaimer

Nothing of what I've speculated here is backed up by an actual study, it's just my impression. It would be interesting to see an analysis of the citation distribution with regard to the cut-off length of clustered citations. I am not criticising the content of any of the papers I mentioned above (that in fact I didn't even read).

---

See also: Peer Review II, III and the related posts Science and Democracy I, II, and III.

---

* Footnote to the younger readers: That's meant in a sarcastic way, please don't take it seriously. You can easily spoil your reputation with that kind of behaviour. Nobody wants to work with somebody who is just incredibly annoying and self-centered.