Against Measure

The recent Nature issue has an opinion piece by Julia Lane:

Let's make science metrics more scientific
Nature 464, 488-489 (25 March 2010)

Julia Lane is the director of the Science of Science and Innovation Policy programme with the National Science Foundation. This programme is a very timely initiative to support research with the aim to better understand the process of research itself.

If you've been reading this blog for a while, you know that this is exactly what I've been saying over and over again: we need a scientific approach to understand the academic system, much like we have one to understand the economic system. We are wasting time, money and effort because we don't understand the social dynamics of the communities and don't know what benefits knowledge discovery. Instead of a scientific investigation of these matters, we spend time with useless complaints. Clearly, funding agencies should be interested in how to use their resources more efficiently. It's thus no surprise the NSF has such a program. It's a surprise other agencies don't.

Thus, I agree on the general sense of Lane's article. Current metrics to assess scientific productivity are poor. They are used even though that is known. There are no international standards. Many measures for creativity and productivity are not used at all. Lane writes:

Existing metrics do not capture the full range of activities that support and transmit scientific ideas, which can be as varied as mentoring, blogging or creating industrial prototypes. [...]

Knowledge creation is a complex process, so perhaps alternative measures of creativity and productivity should be included in scientific metrics, such as the filing of patents, the creation of prototypes4 and even the production of YouTube videos. Many of these are more up-to-date measures of activity than citations.

She also points out that there are many differences between fields that have to be accounted for, and that the development of good metrics is an interdisciplinary effort at the intersection of the social and natural sciences. I would have added the computer sciences to that. I agree on all that. What I don't agree on is the underlying assumption that measuring scientific productivity is under all circumstances good and useful to begin with. Julia lane actually doesn't provide a justification, she just writes:

"Scientists are often reticent to see themselves or their institutions labelled, categorized or ranked. Although happy to tag specimens as one species or another, many researchers do not like to see themselves as specimens under a microscope — they feel that their work is too complex to be evaluated in such simplistic terms. Some argue that science is unpredictable, and that any metric used to prioritize research money risks missing out on an important discovery from left field. It is true that good metrics are difficult to develop, but this is not a reason to abandon them. Rather it should be a spur to basing their development in sound science. If we do not press harder for better metrics, we risk making poor funding decisions or sidelining good scientists."

True, this is no reason to abandon metrics. But let me ask the other way 'round: Where is the scientific evidence that the use of metrics to measure scientific success is beneficial for progress?

I don't have any evidence one way or the other (well, if my proposal under Ms Lane's program had been approved of, I might have). So instead I'll just have to offer some arguments why the mere use of metrics can be counterproductive. First, let me be clear that scientific research can be very different from one field to the other. We also previously discussed Alexander Shneider's suggestion that science proceeds in various different stages. The stages basically differentiate the phase of the creative process. For some of these stages, the use of metrics can be useful. Metrics are useful if it is uncontroversial what constitutes progress or good research. This will be the case in the stages where a research field is established. That's basically the paper-production, problem-solving phase. It's not the "transformative" and creative stage.

One has to be very clear on one point: metrics are not external to the system. The measurement does affect the system. Julia Lane actually provides some examples for that. Commonly known as "perverse incentives" it's what I've referred to as a mismatch between primary goals and secondary criteria: You have a primary goal. That might be fuzzy and vague. It's something like "good research" or "insight" or "improved understanding." Then you try to quantify it by use of some measure. If you use that measure, you have now defined for the community what success means. You dictate them what "good research" is. It's 4 papers per year. It's 8 referee reports and 1 YouTube video. It doesn't matter what it is and how precise you make it, point is that this measure in turn becomes a substitute for the primary goal:

"The Research Performance Progress Report (RPPR) guidance helps by clearly defining what agencies see as research achievements, asking researchers to list everything from publications produced to websites created and workshops delivered."

So you're telling me what I should be achieving? And then you want me to spend time counting my peas?

Measures for achievements are fine if you have good reason to believe that your measure (and you could adapt it when things go astray) is suitably aligned with what you want. But the problem arises in cases where you don't know what you want. Lane eludes to this with her mentioning of researchers who think their work is "too complex" (to be accurately measured) and that "science is unpredictable." But then she writes this is no reason to abandon metrics. I already said that metrics do have their use, so the conclusion cannot be to abandon them altogether. But merely selecting what one measures tells people what they should spend their time on. If it's not measured, what is it good for? Even if these incentives are not truly "perverse" in that they lead the dynamics of the system totally astray, they deviate researchers' interests. And there's the rub: how do you know that deviation is beneficial? Where's the evidence? You with your science metric, please tell me, how do you know what are the optimal numbers a researcher has to aim at? And if you don't know, how do you dare to tell me what I should be doing?

The argument that I've made previously in my post "We only have ourselves to judge each other" is that what you should be doing instead is to just make sure the system can freely optimize, at least within some externally imposed constraints that basically set the goals of research within the context of the society. The last thing you should be doing is to dictate researchers what is the right thing to do, because you don't know. How can you know, if they don't know?

And yes, that's right, I'm advocating laissez-faire for academia. All you out there who scream for public accountability, you have completely missed the point. It's not that scientists don't want to be accountable. There's just no sensible way to account for their work without that accounting hindering progress. Call that the measurement problem of academia if you like.

Bottomline: Before you ask for more scientific science metrics, deliver scientific evidence that the use of such metrics is beneficial for scientific progress to start with.