Conform and be funded?

A recent issue of Nature magazine featured a study by Joshua Nicholson and John Ioannidis that looked at the citation count of principal investigators (PIs) funded by the US-American National Institute of Health (NIH).

Research grants: Conform and be funded
Joshua M. Nicholson, John P. A. Ioannidis
Nature 492, 34–36 (06 December 2012) doi:10.1038/492034a

Ionnadis is no unknown, he previously published a paper "Why Current Publication Practices May Distort Science" that we discussed here, and is author of the essay "Why Most Published Research Findings Are False". The Nature article is unfortunately subscription only, so let me briefly summarize what it says before commenting.

Nicholson and Ioannidis analyzed papers published between 2001 and 2012 in the life and health sciences, catalogued by the Scopus database. They looked those who had received more than 1,000 citations by April 2012 and an author affiliation in the United States. They found 700 papers and 1,172 authors matching this query.

The NIH invites PIs of funded projects to become members of study sections. The purpose of NIH study section is to evaluate scientific merit. Nicholson and Ioannidis found that from the 1,172 top-cited authors only 72 were currently members of study groups, and most of these 72 (as expected) currently received NIH funding. However, these 72 top-cited scientists are merely 0.8% of all section members. Maybe more insightful is that they further randomly selected 200 of the top-cited papers and excluded those with authors in a study group. From the remaining top-cited authors, only 40% are currently receiving NIH funding.

In a nutshell, this is to say that the majority of authors of research articles in the life and health sciences that were top-cited within the last decade do not currently receive NIH funding.

That's as far as the facts are concerned. Now let's see how Nicholson and Ioannidis interpret this finding and what they conclude. In the beginning of the article, they are careful to point out that scientific success is difficult to measure and the citation count should be regarded with caution:

"The influence of scientific work is difficult to measure, and one might have to wait a long time to understand it. One proxy measurement is the number of citations that scientific publications receive. Using citation metrics to appraise scientists and their work has many pitfalls... However, one uncontestable fact is that highly cited papers (and thus their authors) have had a major influence, for whatever reason, on the evolution of scientific debate and on the practice of science."

However, towards the end of the paper they write:

"The mission of the NIH is to support the best scientists, regardless of whether they are young, old or in industry... Such innovative thinkers should not have so much trouble obtaining funding as principal investigators. One cannot assume that investigators who have authored highly cited papers will continue to do equally influential work in the future. However, a record of excellence may be the best predictor of future quality, and it would seem appropriate to give these scientists the opportunity of funding their projects."

Note how now authoring a highly cited paper is synonym for being an "innovative thinker" and "may be the best predictor of future quality". In fact, they go even farther than that by arguing that all authors of highly-cited papers should have their projects NIH funded (apparently regardless of what this project is):

"Funding all scientists who are key authors of unrefuted papers that have 1,000 or more citations would be a negligible amount in the big picture of the NIH budget, simply because there are very few such people. This could foster further important discoveries that would otherwise remain unfunded in the current system."

I find the above closing paragraph of the article simply stunning. They seriously argue that something must be wrong with NIH funding -- according to their elaboration it's a "networked system" in which "exceptionally creative ideas may have difficulty surviving" -- because the NIH does not automatically fund projects of authors with papers who gathered more than 1,000 citations within the last decade.

Now I know nothing about funding problems in the life sciences. Maybe they have a good reason to hold a grudge against NIH peer review practice. Be that as it may, the facts do simply not support their arguments. I am tempted to say it actually speaks in favor of the NIH that they do not pay so much attention to the citation count because, as the authors write themselves, it's a questionable measure: It measures not only innovative thinking, but also fashions and just usefulness (reviews and illustrative diagrams tend to gather lots of citations), it moreover picks up social dynamics, popularity of the authors, or the amount of secondary work that is created, irrespective of whether that work is particularly insightful.

Many top-cited works are created because somebody has been fast enough to jump onto a topic about to take off. Is that a sign for not being "conform", as the title of the article suggests? I am trying to imagine that somebody would argue that all top-cited physicists should get their projects funded without peer review. And would try to publish this as an essay in Nature.