Measuring science the right way. Your way.

Today, I am happy to announce the first major update of our website SciMeter.org. This update brings us a big step closer to the original vision: A simple way to calculate what you, personally, think quantifies good research.

Evaluating scientific work, of course, requires in-depth studies of research publications. But in practice, we often need to make quick, quantitative comparisons, especially those of us who serve on committees or who must document the performance of their institutions (or both). These comparisons rely on metrics for scientific impact, that are measures to quantify the relevance of research, both on the administrative level and on the individual level. Many scientists are rightfully skeptical of such attempts to measure their success, but quantitative evaluation is and will remain necessary.

However, metrics for scientific impact, while necessary, can negatively influence the behavior of researchers. If a high score on a certain metric counts as success, this creates an incentive for researchers to work towards increasing their score, rather than using their own judgement for good research. This problem of “perverse incentives” and “gaming measures” is widely known. It has been subject of countless talks and papers and manifestos, yet little has been done to solve the problem. With SciMeter we hope to move towards a solution.

A major reason that measures for scientific success can redirect research interests is that few measures are readily available. While the literature on bibliometrics and scientometrics contains hundreds of proposals, most researchers presently draw on only a handful of indicators that are easy to obtain. This is notably the number of citations, the number of publications, the Hirsch-index, and the number of papers with high impact factor. These measures have come to define what “good research” means just because in practice we have no alternative measures. But such a narrow definition of success streamlines research strategies and agendas. And this brings the risk that scientific exploration becomes inefficient and stalls.

With SciMeter, we want to work against this streamlining. SciMeter allows everyone to create their own measures to capture what they personally consider the best way of quantifying scientific impact. The self-created measures can then be used to evaluate individuals and to quickly sort lists, for example lists of applicants. Since your measures can always be adapted, this counteracts streamlining and makes gaming impossible.

SciMeter is really not a website, it’s a web-interface. It allows you make your own analysis of publication data. Right now our database contains only papers from arXiv.org. That’s not because we only care about physicists, but because we have to start somewhere! For this reason, I must caution you that it makes no sense to compare the absolute numbers from SciMeter to the absolute numbers from other services, like InSpire or Google. The analysis you can do with our website should be used only for relative comparison within our interface.

For example, you will almost certainly find that your h-index is lower on SciMeter than on InSpire. Do not panic! This is simply because our citation count is incomplete.

Besides comparing authors with each other, and sorting lists according to custom-designed metrics, you can also use our default lists. Besides all authors, we also have pre-defined lists for authors by arXiv category (where we count the main category that an author publishes in), and we have lists for male and female authors (the same gender-id that we also used for our check of Strumia’s results).

The update also now lets you get a ten-year neural-net prediction for the h-index (that was the basis of our recent paper, arxiv version here). You find this feature among the “apps”. We have also made some minor improvements on the algorithm for the keyword clouds. And on the app-page you now also find the search for “similar authors” or authors who have published on any combination of keywords. You may find this app handy if you look for people to invite to conferences or, if you are a science writer, if you are looking for someone to comment.

SciMeter was developed by Tom Price and Tobias Mistele, and was so-far financed by the Foundational Questions Institute. I also want to thank the various test-users who helped us to trouble-shoot an early version and contributed significantly to the user-friendliness of the website.

In the video below I briefly show you how the custom metrics work. I hope you enjoy the update!