Friday, October 05, 2018

Gender-bias in Academia: The Case Strumia

Dr Alessandro Strumia is a physicist working at CERN, where he is Principal Investigator of an ERC Advanced research grant on the topic “Understanding the mass scales in nature.” Ie, he spends your tax money on the kind of research that I criticize in my book. He also recently published an analysis of publication and citation-rates in his field. At a workshop on gender diversity at CERN last weekend, he used this analysis to argue that women are underrepresented in physics because they are underperforming.

I did not attend the workshop and have not seen a recording of the talk, but I have seen the slides (a PDF version of which is here). The slides contain statements that are both inaccurate and exceedingly unprofessional.

For example, he begins his talk by stating that “smarter people are less affected by implicit bias,” but this is wrong. Studies have shown repeatedly that intelligence does not protect from thinking biases. Yes, intelligence is useful to overcome certain types of biases (mostly those that can be exposed with mathematical reasoning), but only once people are aware they are biased to begin with.

Strumia’s mistaken belief that intelligent people are less affected by cognitive biases does not remotely surprise me. I have encountered this very same attitude (“We are too smart to be biased!”) among almost all high-energy theorists and phenomenologists I have spoken with about the issue. That in itself is a bias, known as the “bias blind spot.”

But that Strumia is ill-informed about the very topic he speaks about at a scientific workshop is not the biggest problem with his presentation. Far worse is that he names and attacks two women, apparently because he is annoyed he did not get a job that he was shortlisted for. Nonsense like this just does not belong in a research presentation.

After complaints ballooned on social media on Monday, CERN pulled the slides from the net quickly and has since suspended Strumia. What will happen to his ERC grant is unclear. A large number of members of the particle physics community have meanwhile signed a statement declaring that they distance themselves from the content of Strumia’s talk.

Now, as you know, I have also recently taken up bibliometric analysis. And I admit I found some of the data Strumia showed interesting. We did, in our paper, also look at gender differences, but not for citation counts. We looked at an entirely different quantity, that of research broadness, and for this we did not find any gender differences.

The gender difference that Alessandro Strumia and his co-author Ricardo Torre find is huge. It’s a more than 100% difference in the total number of citations that researchers accumulate throughout their career.

I don’t think that the number of citations is a good measure for scientific performance, but if the difference between the genders was so large, it might mean that women and men chose their research projects in distinctly different ways. That would be interesting. I thus decided to look into this for a bit.

The key figure that Strumia presents on his slides is the total number of citations that researchers accumulate since the publication of their first paper:

Figure from slide 16 of Alessandro Strumia’s talk.

That the horizontal axis is labeled “scientific age” is unfortunate because this term has been coined to emphasize that the scientific age might differ from the chronological time passed since PhD or first paper. If a researcher takes a career-break, for example because of health reasons or for parental leave, their scientific age goes on hold. However, there is presently no standardized way to determine the scientific age, and in any case, you couldn’t do it from publication data alone, you’d also need biographic information.

Since women are more likely to take leave for child-raising, their citations should on the average increase somewhat slower, simply because they have more breaks in which they don’t publish. However, it seems unlikely that this would make such a huge difference. So, while the label on the axis is inaccurate, I don’t think it’s all that relevant.

When I saw this graph, however, another worry came to my mind immediately. When we did our previous analysis, we found that the vast majority of people who use the arXiv publish only one or two papers and are never heard of again. This is in agreement with the well-known fact that the majority of physicists drop out of academic careers.

I am not sure why this surprised me when it showed up in the data. Maybe because, if you work in the field, the drop-outs are pretty much invisible. They leave and you forget about them. But they are there, in the stats, big and fat.

Now, the total number of citations for such drop-outs will accumulate very slowly because they don’t publish new papers any more. And we know that women are more likely to drop out – that’s the “leaky pipeline” and reason why I find myself increasingly often, if not the only woman in the room, then at least the oldest woman in the room. And I’m only 42.

If you leave the drop-outs in the citation analysis, the leaky pipe will pull down the average of female authors more than of male authors.

I hence asked one of our PhD students, Tobias Mistele, to plot the same quantity as Strumia did for our data sample, but to only keep authors who have more than 5 papers in total, and who have published a paper in the last 3 years. This is sloppy way to shrink down the pool to “active researchers only.” It’s maybe not the most sophisticated way to do it, but it should give us an idea how large the contribution from the drop-outs is.

If we plot the number of citations for active researchers only, we see no noticeable difference between men and women:
arXiv data, active researchers only

When normalized to the number of authors per paper (as Strumia did), there is also no noticeable difference between men and women.

I must add a warning here. We do not use the same data set as Strumia and Torre. They use data from inspire, we use data from the arXiv. This means our data set does not reach back in time as far, and it includes disciplines besides high energy physics. So the absolute numbers are not directly comparable.

Another caveat I must add is that we are using a different method to identify male and female authors. We use the author-id algorithm that is explained in our earlier paper, and then try to match first names with a database for common anglo-saxon first names. Naturally, this means that the authors who remain in our sample are most likely to be of Western origin. By this method we assign a gender to 19% of authors. This is in contrast to Strumia and Torre who use a more elaborate gender-id procedure that allows them to match 60%. The remaining authors in our sample break down to 70,295 male und 53,165 female researchers. After applying the above mentioned cuts, we are left with 12,654 male and 8,177 female. That’s not a huge number, but decent.

Let me also mention that probably a similar effect is behind another finding in Strumia’s talk. He points out that women, on the average, are hired into faculty positions earlier. A paper that appeared on the arXiv yesterday argued that this is not a signal that women have an unfair advantage, but simply a consequence of women leaving at a higher rate. If they aren’t hired early, they’ll not be hired at all, which means the average age of hiring is smaller.

Finally, to state the obvious, this is a blogpost, not a paper. The above is a quick and dirty way to check whether removing dropouts significantly affects the large difference between men and women, and the answer seems to be yes. However, we will have to do a more careful analysis to arrive at definite conclusions. I haven’t checked my biases.

I want to thank Tobias Mistele for doing the graphs so quickly and Alessandro Strumia and Ricardo Torre for helpful communication.


Cy Myers said...

It would be interesting to see the unadjusted graph from your dataset as well. This would give a better idea of how close the data tracks to Strumia's, and understand how much the "active researchers" methodology changes the result.

Sabine Hossenfelder said...


Yes, unfortunately I don't have the data for that. I'm traveling & my student is on vacation now. Might take a few weeks until I can dig that up.

Louis Tagliaferro said...

There are likely many reasons why women are underrepresented in physics, social, cultural, psychological, and perhaps biological, however prejudicial thinking should never be acceptable as one of them.

Many people confusingly equate equality with humans being equal in abilities and possibilities, we are certainly not. Equality is about opportunity and treatment without bias, the rest is earned and/or granted by ones genetics.

Boris Borcic said...

Exactly my expectation when I saw Strumia's graph, that he'd likely failed to account for dropouts (and for why women would drop out more frequently)

Pavel Nadolsky said...

Thanks, Sabine, very interesting.
1. I wish to ask why the numbers of male and female researchers are about equal in your samples. This is very different from the actual situation in the HEP theory, where the ratio of male to female researchers is closer to 10:1. Did you apply some pre-selection?

2. It would not be surprising if the number of citations indeed grows slower with time for female researchers vs. the male researchers, independently of the researchers' individual abilities or actions. This actually happens in other fields. In art, the price of the artwork by female artists appreciates slower with time, regardless of what the artists do. The slowdown effect is large and perhaps comparable in the order of magnitude with the slowdown in the number of citations seen in Strumia's figure. See, in particular, Fig. 1 in, with a popular summary of this study in . The selection of "worthy" citations reflects one's ability to recall "memorable" work and may be driven by subtle, hard-to explain intuitions of the same psychological origin as when pricing a "valuable" art. To quote from the above study:

"Asking why women's art sells for less than men's elicits a long and complex answer, with endless caveats, entirely germane qualifiers and diverse, sometimes contradictory reasons. But there is also a short and simple, if unpopular, answer that none of those explanations can trump. Women's art sells for less because it is made by women."

Emily Bremer said...

A very nice blog post that takes into account the leaky pipe effect. Thanks for taking the time to do the analysis and please consider submitting to the arXiv

Heinrich Pas said...

Very interesting, Sabine. I like your pragmatic ansatz of making sense out of Strumia's data. I also find it exciting that your work might be a hint that my suspicion - that the family-unfriendly career path plays a large role among the reasons why there are so few women in fundamental physics - may be correct after all.

Yvan Dutil said...

These sampling bias work in both directions and need to be carefully taken into account.

Unknown said...

hi Sabine, congratulations for doing real work. This is how the workshop should have been done.

Of course, I had done an analogous check, and got the same qualitative trend shown in the talk, after restricting to hired/unhired authors. Adding the extra cuts considered by you (more than 5 papers, after ≈1995), the trend persists. My collaborator has not double-checked this claim.

I suggest that you can quickly roughly match to our database by restricting to hep arXiv categories (hep-ph, th, ex, lat; astro; nucl-th,ex, gr-qc).

A comment about the paper you quote at the end: it claims that "female astronomers are leaving the academic labor market, at a rate that is 3-4 times higher than male astronomers". We have data about M/F drop-outs (percentage of authors who wrote a paper and abandoned physics), that I avoided showing in order not to hurt the sensibility of young physicists, and because we don't really know how complete InSpire is about them. Trusting InSpire, the gender difference is a few 10%, much smaller than in the paper you quote. I see they used the Astrophysics Rumor Mill as database, while we use InSpire which is, at least, much much bigger.

Some details. We reconstruct the sex of 85% of authors, not 60%. Those missing mostly don't have names in InSpire (and we might have problems with Chinese authors, because some only have English transliteration, erasing most information about M/F). We add some country information, because various names like Gabriele or Andrea are M/F depending on the country. This, like any other mistake in M/F identification, goes in the direction of reducing a possible asymmetry.

PS: in our analysis we found your name among top-cited recent female authors.

Eric Rasmusen said...

It would be interesting to look at the variance of publications at differnet years out for men and women too, excluding complete dropouts. The general pattern is that males are more variable, but it might well not be true here, since this group is already pre-selected for being veyr very good (that is, they are people who got physics jobs). This would combine interestingly with the Assistant Professors As Options idea.

Vincent van der Goes said...

I wish I had a way to correct my own biases, or at least to learn what they are.

Jonathan Starr said...

Dr. Hossenfelder

This post made me smile ear to ear even before I read the whole thing. Thank you. I know every time some edge-lord outs themselves publicly everyone piles on but I always love to hear about a righteous prat fall.

It would be nice to see a good take down of the "leaky pipes/maternity + profit motive -> gender pay gap" argument. But when I've looked I seem to have trouble finding a treatment that is rigorous enough to use graphs and data sets. Mostly it's puff pieces written by people who are more comfortable with less-then-numeric approaches. The search pollution around any of these culture war things always confounds me and given your post I was hoping you'd know who I should be reading.

alexander.shen said...

And it would be interesting to analyze the data on the number of citations at the time of hiring - since there are conflicting opinion on the direction on bias, and any real data are interesting here

Egan said...

Thank you for debunking of the slides. It's interesting to know what was wrong in Strumia's analysis.

However I think we can't deny there is effectively a multitude of gender bias in STEM.
Sometimes it's in favor of men. Sometimes it's in favor of women.

I'll use as a particular example (in favor of women) the CNRS medals awarded to researchers each year (approximately 20 silver medals and 40 bronze medals per year).

If you go to the CNRS website it is said it employ 11 204 researchers with a women proportion of 32.9%.
Link :

So logically, and if there is no ability difference between men and women, we should find the same proportion in the awarded medals.

Let's see the numbers.
Here for silver medals :
Here for bronze medals :

2016 :
Total silver medal awarded : 16
Women : 8 (50%)

Total bronze medal awarded : 40
Women : 22 (55%)

2017 :
Total silver medal awarded : 20
Women : 10 (50%)

Total bronze medal awarded : 40
Women : 26 (65%)

2018 :
Total silver medal awarded : 20
Women : 9 (45%)

Total bronze medal awarded : 42
Women : 21 (50%)

Conclusion : Instead of awarding medals roughly 33% time to women, we see the CNRS committees try the respect the parity between men and women. It's a clear and unambiguous example of male discrimination in STEM. The medals are not awarded only for pure merit but also with a huge gender bias.

Uncle Al said...

Ban clothing from academic environments. Women would be a distraction, men would be a nuisance, and journalism would infest elsewhere - back to normal for everybody, and with no thermostat rancor.

Social problems are solved by doing less.

Unknown said...

Thanks, that was great!

lollerboller said...

So you delete the low publishers, which was the whole point of his data, and you are left with the norm. Most people do collaborations nowadays so when the low publishers are removed, the graphs are expected to be the same pretty much... If you had Strumias data I would think they go even more hand-in-hand when you make the decision to cut out the low publishers, and delete outliers. But what is the point ?

naivetheorist said...


"we found that the vast majority of people who use the arXiv publish only one or two papers and are never heard of again. This is in agreement with the well-known fact that the majority of physicists drop out of academic careers."

this is a reasonable interpretation but it is only one of several.

e.g. in the field of pop music, a vast majority of the individuals and groups (those whose records are top sellers) have only one or two such recordings in their careers. these individuals and groups)are known as "one-hit wonders'. this is also a well-known phenomenon in the field of book publishing.

there's no doubt that being in academia increases the number of papers one writes ('publish or perish' is a real thing, although it it is often more accurately described as to 'bring in grant money or perish') but it may also be true that the majority of researchers simply have only one or two 'good' ideas or projects worth publishing.

it would be very interesting to know if the common myth that professors effectively 'retire' from doing research once they attain tenure is a result of intellectual laziness (due to their having guaranteed lifetime employment) or intellectual burnout (due to the immense pressure they were under during the pursuit of such employment).

naive theorist

Liralen said...

As I've said before, probably ad nauseum, that those of us who perceive ourselves as biased against, would be smart to avoid professions where your performance is judged by how well you are liked instead of some more objective parameters.

Publish or Perish was a well-known problem by the time I entered college in 1974. According to the wiki, the earliest known usage of the phrase dates back to 1927.

Citation counts is a subset of Public or Perish, since you can't cite what hasn't been published.

I'm not the only member of a minority group to avoid academia for that reason.

That fact must certainly skew the results, but can't ever be measured. I even suspect the smartest of the minority groups will avoid academia until "Publish or Perish" is no longer a concern, or our view of ourselves as being biased against has changed.

Christophe Meudec said...

So is your conclusion that there is no discrimination in citation?

Tsetrot said...

Your plot is interesting, but I think it does not yet prove that Strumia was wrong. Can you make a plot conditioned to people having a permanent job? People having a permanent job are supposed to produce at a good level. If they stopped publishing it's their fault.

Unknown said...

Dear Sabine, thanks for your analysis, you raised good points. Seeing the unadjusted dataset would be useful, as mentioned. I hope you'll update us on this, time permitting. Again thanks for your analysis and comments, they are very insightful.

Sabine Hossenfelder said...


It's a good question, but unfortunately I can only answer I don't know. There may be various reasons for that, eg female names may be better covered in the table that we use or maybe women are more likely to use their full name to begin with so that they can be identified. We'll look at this, but it will take a while.

Sabine Hossenfelder said...


Regarding your point 2: Yes, priginally I was thinking it may be a real effect, which would in a sense be more interesting. Well, I hope that Strumia & Torre do some further analysis that will clarify the issue.

Sabine Hossenfelder said...


We're working on a paper and thinking of including the above gender breakdown. But it'll take a few more months for the paper to be finished.

Sabine Hossenfelder said...


If you find me among the top-cited recent female authors there's clearly something wrong with your analysis. More seriously, I am open to the possibility that this is a real effect. It doesn't seem implausible to me that women have other priorities than men in choosing and carrying through their research topics which reflects in the average citation rates. I am stunned though by how large the effect is.

Yes, we can shrink down our data to only a few arXiv categories. However, I'd be concerned that then the sample is so small that it becomes inconclusive because there's an element of luck in whether your research attracts attention or not. Unfortunately, I don't presently have a good idea for how to estimate the significance, other than doing some agent-based modeling. And with those models you tend to simply find what you put into the setup.

In any case, tbh, I am not terribly interested in the whole gender-thing, hep has bigger problems than that. But the ppl who work on this should look at it.

One more thing: It is quite possible that the reason we see no effect is due to our name-gender matching. As I mention, we have used a table for common anglo-saxon names. Since we throw out everyone we couldn't identify, this effectively means that we remove the vast majority of researchers in India, Asia, Eastern Europe, etc. This very possible has an effect on the male/female ratio. Best,


PS: A name would be helpful. There are a lot of "Unknowns" in my comment sections.

Sabine Hossenfelder said...


Yes, unfortunately a lot of studies on the subject are far from convincing, tiny samples, shaky methods. I definitely wish there was better data. The cynic in me sometimes suspects that maybe some people don't really want to know the truth.

Sabine Hossenfelder said...


We have removed those who left academia to find out whether the difference is anything more than the well-known fact that women leave in higher proportion.

Sabine Hossenfelder said...


No, this analysis does not tell you anything about discrimination.

Sabine Hossenfelder said...


I have no reason to think that Strumia is wrong in the sense that his data analysis is wrong. The question is whether that's an interesting and/or new finding. We don't have bibliographic information for authors, where are we supposed to take those from?

SRP said...

I admire physicists' typical confidence to plunge into any subject, but it might be a good idea to consult the specialized research on this general topic. A good review article covering everything from cognitive research to bibliometrics across numerous STEM and social science fields is:

Tam Hunt said...

Appreciate your commentary here, Bee, as always. I find it absurd and dangerous, however, whether Strumia's comments have zero merit or non-zero merit, that CERN would literally remove his slides from their workshop website. This is neanderthal-style censorship that should have no place in modern science. Would appreciate your calling this dangerous trend out if you can.

Klavs Hansen said...

I am a little confused with the conclusions of your analysis. It is a widespread opinion (at least in Sweden, where I have spent a good deal of time), that a lower citation rate for female scientists is a sign of discrimination. Now you seem to have eliminated this indication. In fact men and women are treated equally when it comes to citations, if judged by your almost identical curves in the carreer-corrected plot.
I understand that identical curves means that on basis of those data you can not claim that men are being discriminated against, but in all honesty it must work both ways.

TheBaluchiterium said...

Dear Dr. Hossenfelder,

the comparative graph you show which "does not present any noticeable difference between men and women" has been, as we read in the footnotes "based on a different data set" (at least you were honest enough to mention this difference). So this comparison does not debunk Dr. Strumia's graph at all, because he was talking about FUNDAMENTAL PHYSICS, and nothing else. And from the comments section I can deduce that most readers only read what they want to read and neglect the footnotes.

The fact that you use a different data set - of which you, as a professional, know that its content differs strongly from the dataset used by Dr. Strumia - suggests that you are making politics on this point, not honest research. This is deceptive tactics, because most of your readers on this blog will not know the difference between InSpire and ArXiv.

In order to clarify: InSpire is the database for unpublished work mainly in high-energy physics, whereas ArXiv covers many other scientific fields, in particular medicine and biology. That makes all the difference in the world.

Best regards
Timo Fleig
(A colleague in theoretical physics)

Sabine Hossenfelder said...


I believe, but do not know, that CERN pulled the slides primarily for two reasons: Firstly to show they were reacting quickly and not ignoring the issue, and second because Strumia made personal accusations - he actually had names on the slides. It's this latter point that really crosses a line. This is not "neanderthal-style censorship" it's to protect an innocent person, and I think that CERN did well removing the slides.

Sabine Hossenfelder said...

Dr Fleig,

Your accusation that I am not being honest and "use deceptive tactics" is sick shit. We use the arXiv database simply because we've used it before & it's a check we could do quickly. I have added all information clearly and explicitly. I never claimed to "debunk" anything. You even managed to quote a sentence I haven't written and that indeed makes no grammatical sense. And what the heck do you know about my readership? You have some thinking to do, man.

Peter Oakley said...
This comment has been removed by the author.
Peter Oakley said...

Thank you very much for this post. It may be reasonable to say only "this is wrong, he is a bad person" and be light on the details of why - because most will understand why. However I think refuting the logic and reasoning as you have done is much more powerful, particularly as a some people may yet sympathise with Strumia.

I find it hard to explain clearly but it is very profound for me. We can easily think bad things about the person, and that may well be right, but it is more important to think bad things about the arguments and reasoning.

Working with him to clarify and improve your response is right.
Thanking him is class.

JeanTate said...

We reconstruct the sex of 85% of authors

Really? I do hope the method for doing this is (or soon becomes) fully public. On its face, it seems an astonishing achievement! Perhaps authors in InSpire give their full, legal names; otherwise how to tell if Robin, say, or Chris, or Alex are M or F? The lead author of an astrophysics paper, soon to be published, on which I am a co-author, is Chen Wu; how to tell if that's M or F? Yes, if you had the Chinese character, you may get a good idea.

We add some country information, because various names like Gabriele or Andrea are M/F depending on the country.

Cool! So Jean who was born in Australia but now lives in Belgium changed from F to M? And Jean who was born in Switzerland but now lives in New Zealand went from M to F? And Jean who lives in Ottowa but commutes (2 hours each way) daily to Montréal ... well, s/he must be very confused.

Maybe the Mean Career Path figures could have another set added, the "we can't tell if M or F" one?

Kaleberg said...

They always warn women and minorities to choose fields with objective measurements, like money. If high energy physics was still about making sense of new data, there would be an objective way to judge work quality. Now that it's a beauty contest, quality rankings are much more subjective. When publications and citations are about pretty, it's hard to take them seriously. Even the fashion business has ditched this.

Thanks for looking at the data though. I think we're just seeing more of the usual - "No girls allowed." - "Why are there no girls here?"

Ajit R. Jadhav said...

Not *my* UG student, Bee, and *you* know what *I* mean.


Unknown said...

Sorry for disagreeing with most people on the thread, but I think there is a flaw here. Dam it, going against the narrative! sorry again! I hate to intrude when everyone was in self-congratulation mode.

The conclusion of your graph is that all things being corrected for, remaining female scientists - i.e. the ones that manage to produce 5 papers or more - do as well as the remaining male scientists. I think this is correct and I do not think anybody, even Strumia, ever denied that the best women physicists could be as good as men (or even better, see Marie Curie). The question is not on the individuals, it is on the average. Strumia is arguing that in the 'best' slice there are less women than men. I think your data confirms this interpretation.

The question is to figure out if women will, on average, do as well, better, or less well than men. What you are showing here is that more men (~12.000) remain active scientists than women (~8.000). You call this the leaky pipeline and argue that it is leaky because of social constraints. Well, let us forget about sociology - we really should BTW - and treat this data as we would treat any regular biological experiment, by sticking to the most parsimonious explanation.

In this case, the simplest explanation is that people drop out - women and men - because they were not doing well. This hypothesis is well supported by the observation that the papers of the women who drop out are less cited - hence your massive correction- and therefore more likely to be of lower quality. If the dropouts were as good as the remainders, their papers would keep being cited long after they dropped out and the effect reported by Strumia would no occur. Clearly, this is not the case.

The high dropout rate is, therefore, an argument in favor of Strumia argument, while your data shows that a sizeable fraction of women are competitive with men. It's all about averages, and this is all what Strumia was saying.

The argument is not, however, totally closed because an extra normalization is needed here: the graph should be Ncit/Npaper so as not to penalize scientists having produced few papers, but rather penalize scientists having few citations per citable unit.

TransparencyCNP said...

You ranked at the top of Google:

Sabine Hossenfelder said...


Yes, sorry. I actually try to avoid it. Not, UG though.

Sabine Hossenfelder said...


First, if you aren't able to use a Google account, could you *please* at least sign with a name or a pseudonym.

Second, what you say is a possible interpretation of the data, but I wasn't claiming the leaky pipe is entirely due to social influence. I was merely saying it's not a new finding.

Third, having said that, given that social hurdles exist for women that do not exist for men, it would be odd if these did *not* contribute to the leak. And, needless to say, this problem should be fixed.



Sabine Hossenfelder said...


PS: We also checked the normalized N_cit, and it doesn't have a noticeable difference between men and women either. However, I am not a big fan of combining different measures needlessly because that makes it harder to quantify the origin of correlations. Also, as I emphasized, this isn't a publication-level analysis and I want to do further checks.

Anonymous said...

Concerning the gender assignement, this is the method prescribed by Strumia&Torre:

We consider 70500 authors indexed in the InSpire database (see appendix A for details). The
Mathematica machine learning function Classify has been trained to automatically assign
to each name a tag: male, female or indeterminate (about 40%, that we ignore). Classify
sometimes fails (the female with highest PaperRank is Nicola Cabibbo, in 51th position): to avoid distortions we manually correct the most notable cases. Then, the female authors with
highest PaperRank are: M. Gaillard (in 114th position), H. Quinn (220th), R.E Kallosh (370th,
who becomes top-ranked female in 78th position restricting to papers written after 2000),
L. Randall (444th). HepNames does not include old authors like Noether, the Curie, Mayer.
Among the authors classified as male or female, 16% of the names in the data-base are
female, and receive 8.4% of the individual citations and 5.6% of the rank.

Classify sometimes fails (the female with highest PaperRank is Nicola Cabibbo, in 51th position) -- Indeed this statement does not fill me with great confidence (specially since the number of female Nicolas in general is probably smaller than the number of male Nicolas working outside of Italy).

There are indeed a few cross checks, that would be interesting and valuable to do. In first instance restricting to an American first-name dictionary is probably safer.

from the same paper:
Fig. 13 shows that the percentage of individual citations received by female authors is growing and is a factor of 2 higher in sub-fields dominated by large experimental collaborations (where bibliometrics cannot identify individual merit) than in more theoretical fields (where social effects are less important).
--> I would *also* claim, that social effects could be *larger* in the theoretical field, when it comes to citations (because you cite your "friends" or you write 20 emails every morning to get people to cite you), but perhaps this is just some sloppy wording or is based on a definition of "social effects" that is different from mine.

Sabine Hossenfelder said...

Which reminds me: "we manually correct the most notable cases" is a really bad idea. You don't go and fumble with your data by hand, this brings in personal bias. I didn't comment on this in my post because, given the total number of people in the sample, I doubt it makes a difference for the averages.

rms said...


simple is more parsimonious than parsimonious

rms said...


there are other explanations as simple as "people drop out - women and men - because they were not doing well":

because they find the field dumb
because they find the field below their worth (intellectually, economically..)
because they find the field interesting, but not that interesting
because they find the field interesting, but a dead end
because there are so many other interesting things to do
because there are too many assholes
because there are too few assholes
because it is boring

Topher said...

It saddens me to read here that CERN pulled the slides, suspended Strumia, and that his grant status is now challenged. If his research had shown no gender "performance" bias, there would be no such backlash, as that result would have been in line with the approved narrative for our society. Right or wrong, it isn't healthy scientifically or sociologically to silence the discussion and attempt to erase the evidence that it happened. The questions motivating Strumia's research have not gone away, so it would be better to address them in an open way. Not every conference talk must be correct to be worthwhile. The research merely has to be done in good faith, which I believe this is. It's the job of the community to criticize and try to pick apart this result, but not to squelch it.

Unknown said...

Anonymous writes: "Concerning the gender assignement, this is the method prescribed by Strumia&Torre...". That was the method used in version 1 of the paper. The slides use a more accurate method.

Sabine, is your plot dominated by big experimental groups, inside which gender asymmetries are lost? If yes, it would be interesting to either restrict to papers with few authors, or do a fractional counting of citations.

Alessandro Strumia

JeanTate said...

We reconstruct the sex of 85% of authors, not 60%. - Unknown, 5 October

male, female or indeterminate (about 40%, that we ignore) - Anonymous, 8 October

So, which is it?

The Mathematica machine learning function Classify has been trained to automatically assign to each name a tag: - Anonymous, 8 October

As anyone who has experience with ML can say, what an ML is trained on is vital; specifically, if the training set contains garbage, the results will also contain garbage.

And if training sets include bias, the results will also contain bias. For example, perhaps the training set is wonky, shall we say, when it comes to East Asian names, Chinese ones in particular. And if this ML fails with "Nicola", how confident can we be that it doesn't fail with "Chen" (a very common Chinese name)? Yes, the authors may recognize a "Nicola" failure, but would they even notice a "Chen" one?

The Strumia study is, to me, looking more and more like very poor quality, which any reviewer of integrity would reject ...

Flávio said...
This comment has been removed by the author.
sean s. said...

I'm gonna have to read this closely.

sean s.

Phillip Helbig said...

“We are too smart to be biased!”

Most such also describe themselves as being above arrogance.

Phillip Helbig said...

"We reconstruct the sex of 85% of authors"

I've heard of sex-reconstruction surgery, but that is something different!

Phillip Helbig said...

"Klavs Hansen said...

I am a little confused with the conclusions of your analysis. It is a widespread opinion (at least in Sweden, where I have spent a good deal of time), that a lower citation rate for female scientists is a sign of discrimination."

Without taking sides on this issue, do you seriously think that this issue could even be discussed in Sweden? Of course most people in Sweden believe that it must be due to discrimination.

This (one's culture) is also a huge bias.

(I speak Swedish and, spread out over 35 years, have spent almost a year in Sweden altogether.)

Kaleberg said...

The best I can tell about Strumia is that he is a second rate physicist feeling that he is owed a job based on his sex rather than his work. That seems to be the simplest explanation. If he had felt his work was superior, he would have gone on the attack in a physics forum, not one on gender balance.

JK said...

The extreme overreaction and appalling censorship behavior of CERN management and a minority subset (yes minority, the absences are way more notorious than the inclusions) of the scientific community over the Strumia affair has already created a significant counter-reaction and the condemnation has backfired. How do I know this? Because if you listen what people are saying outside of liberal academic circles, you find that most people has been shocked by a scientist getting expelled for expressing his (correct or not, that is irrelevant) conclusions based on some (flawed or not, again irrelevant) data analysis. By no less than CERN. And shocking it is indeed that academic research institutions are so politiziced.

I have skimmed through the comments of liberal academic blogs where you find majority support for what amounts to some sort of intellectual lynching of Strumia, and the contrast could not be bigger with the comments you find in the general press, the general public overwhelmingly find appalling that a Strumia was kicked out, and it has created a lot of sympathy for his views. I have seen people commenting that they are going to change their vote in the next elections, after voting leftwing parties all their lives, because the populist right are the only ones that are going to stop the "feminazism" and the positive discrimination policies. Extremism breeds extremism (but of course all extremists will deny they are).

This backfiring will be denied again by the most fervent metoo activists, but it is the truth and can be checked by anyone that has open eyes and ears.

Sabine Hossenfelder said...


The people who bother to comment are the ones who hold strong views. The majority can't be bothered one way or the other. It's good to keep this in mind. Personally I think the silent majority is a problem bigger than the extremists on either side.

Rob van Son (Not a physicist, just an amateur) said...

You would think that the question of gender bias in science is a perfect subject for a double blind study. And so it is:

Science faculty’s subtle gender biases favor male students

We also assessed faculty participants’ preexisting subtle bias against women using a standard instrument and found that preexisting subtle bias against women played a moderating role, such that subtle bias against women was associated with less support for the female student, but was unrelated to reactions to the male student.

Obviously, not everyone will be convinced by a mere double blind study:

Quality of evidence revealing subtle gender biases in science is in the eye of the beholder

Results from our three experiments, using general-public and university faculty samples, demonstrated that men evaluate the quality of research unveiling this bias as less meritorious than do women.

In this blog post, we see both effects neatly illustrated. There is a gender bias in science, with accomplished scientists arguing the existing differences are biological (i.e., blaming the victim), there are data showing this is not true, and based on very bad statistics reasoning, and there are (men) dismissing all the evidence that there is a bias against women.

PS, how would the authors of the original paper react to a social scientist publishing on theoretical physics?

Rob van Son (Not a physicist, just an amateur) said...

"By no less than CERN. And shocking it is indeed that academic research institutions are so politiziced."

There is another angle here. Strumia has put up a big banner over CERN saying "Women need not apply".

What do you think will happen when a scientist at a biology faculty claims "Creationism is True", or a scientist at a health institute who claims "Smoking does not cause cancer" or "HIV does not cause AIDS"? Or, to take it outside science, a journalist claiming "Truth is just an opinion"?

In all cases, these persons undermine the very missions of the institution they work in.

Extraordinary claims require extraordinary evidence. Strumia had very bad evidence and added conclusions that far outreached the "evidence" he had. Moreover, I assume he had to hire and supervise students and staff. How could he now have any legitimacy when evaluating female students and staff?

Phillip Helbig said...

"There is a gender bias in science"

Whatever the status of Strumia, your conclusion, quoted above, does not follow from your evidence. At most, you have shown that there is a bias among those taking part in the survey. This almost certainly varies with location, with epoch, and with the personalities of the people surveyed. It does not prove that "science", per se, is biased. I am not doubting the evidence you present, merely the grand conclusion derived from it.

Phillip Helbig said...

"I have seen people commenting that they are going to change their vote in the next elections, after voting leftwing parties all their lives, because the populist right are the only ones that are going to stop the "feminazism" and the positive discrimination policies."

Two points. The number of people who even know who Strumia is is not a significant factor in any election. Yes, sadly, there are people who think that a worse extremism is an antidote to extremism, cutting of their noses to spite their faces.

Rob van Son (Not a physicist, just an amateur) said...

@Philip Helbig
"At most, you have shown that there is a bias among those taking part in the survey. This almost certainly varies with location, with epoch, and with the personalities of the people surveyed. It does not prove that "science", per se, is biased. "

Lets look at this sentence by sentence.
"At most, you have shown that there is a bias among those taking part in the survey."

What you say is that samples cannot be used in social science. Which would include the data used by Strumia. It is not that the population of this study was somehow aberrant wrt "average" science faculty.

"This almost certainly varies with location, with epoch, and with the personalities of the people surveyed."

The above study showed that there was considerable bias in an important subset of science institutes. In addition to the above conclusion, it is now on those claiming women perform worse in science than men to come up with evidence that this is NOT caused by bias.

It is not that there is a dearth of studies showing such bias in every country studied. I did not want to swamp this blog with references, but typing in "gender bias in science" in Google Scholar already gives you close to 500 references to mull about.

The challenge is to find a location, epoch, and personalities that do NOT show a gender bias in science.

" It does not prove that "science", per se, is biased. "

In science you cannot prove an assertion. However, it is fairly easy to disprove the statement "women are not treated differently in science".

I have yet to find a group in science that does not show gender bias in some form or another. At a certain moment denial does not let the problem go away. Scientist reflect society. IF society has a gender bias, scientist will not be immune to it.

Marnie said...

Hello Sabine,

Thanks for your efforts.

I'm a woman with an undergraduate degree in physics and a MASc in electrical engineering from the University of British Columbia. I strongly considered doing a PhD in Physics at the University of British Columbia. My choice not to was as follows:

1. Overall job prospects looked poor in physics (compared to electrical engineering)

2. Subtle gender bias and assumptions about whether or not women could do mathematical and theoretical work was a turn off (this seemed to be a bigger problem in physics than in EE)

3. The emerging popularity of string theory at the time that I was considering a physics PhD was a turn off (as was another fad at the time, quantum intelligence.) I wanted to work on practical problems, and string theory and quantum intelligence seemed like a long way from that.

4. General unfriendliness of the professors and refusing to talk about issues like how having children could be combined with a career in physics was a big turn off.

5. Frequently not being taken seriously was a big turn off. The most glaring example of this was when I interviewed in 1994 at the forerunner of the company that would become D-Wave Systems. By this time, I had an undergraduate degree in Math and Physics, another undergraduate degree in electrical engineering (with minor in computer engineering), and an MASc in electrical engineering. During the interview, I was treated like a high school student or an item for purchase in a novelty store (not as a potential colleague.)

Lack of interest in physics or math was not the reason that I decided not to pursue a degree in physics. I still am fascinated with science, engineering and computer science.
But little has changed since the 1980s and 90s when I was in university. On the bright side, my 14 year old daughter is acing geometry and algebra. I'm thrilled that she isn't burdened quite so much as I was with assumptions about her mathematical ability.

Here are some things that still have to change (in STEM, including physics), before we can really say that women are welcome:

Assumptions that women like this and men like that (ie. don't assume that women like experimental work and men like theoretical work, or women like to work on socially relatable research topics while men don't)

Assuming that women can be paid less for similar work (still very common)

Expecting women to be socially accommodating while rewarding men for being aggressive

Punishing women in their performance evaluations for not being socially accommodating

Assuming women have less mathematical ability compared to men

Expecting people to have no family responsibilities (ie. that professors must have a full time stay at home spouse)

Refusing to accept that gender harassment and discrimination exist, and therefore, that any person reporting this must be a crazy.

That would be a good start.

SRP said...

Perhaps physics is unlike engineering, biology, psychology, and economics, but in those fields there appears to be significant hiring preference for women in the U.S.

sean s. said...

JV: “I have seen people commenting that they are going to change their vote in the next elections, after voting leftwing parties all their lives, because the populist right are the only ones that are going to stop the ‘feminazism’ and the positive discrimination policies.

Anyone who’s been “voting leftwing parties all their lives” knows that racism and sexism are positive discrimination policies; they’re not going to abandon the fight and start supporting populist, racist, sexist politicians so readily. Anyone who makes these “comments” probably has not voted for leftwing parties much if ever.

Positive discrimination is evil; discrimination against men (real or imagined) is not worse than discrimination against women (which is almost invariably real).

No doubt there will be some injustices in the fight to end discrimination. That’s unavoidable. But to give up is accept avoidable injustice.

A basic fact of persuasion is that there will always be those who can never be convinced; no matter how much we learn about discrimination against women, there will be those who don’t believe or who think it’s acceptable. They will proclaim their anger at feminists, but the fact is that the only way to mollify them is to acquiesce to gender-discrimination.

sean s.

Anonymous said...

(especially @JK)

I would like to remind again: In his slides, he had a personal attack on someone *named* explicitly, implying they got a job that he was (according to his citation metric) better qualified to do.

"This job" was a directorate level position, that of course includes management and diplomacy(footnote 1), dealing with ministries and so on, which are things possibly not very well measurable using a citation count.

Again, (and I am very sad, that this was not emphasized enough in the general press) he personally attacked a high-rank official of a research institution. If a high-rank official of CERN would be attacked personally as being unjustly hired, including defaming a member of the hiring committee, in a talk by someone employed as researcher in e.g. the Academy of Science in France -- very obviously you would expect the Academy to take the defaming slides down, otherwise you would be complacent, i.e. implying that these accusations are just or that they might be correct. As a scientific workshop organiser you might not be legally able or entitled to judge said correctness. This is what e.g. ombudsmen or courts are for.

With this line of reasoning, you do not need to discuss the other content of the slides. It is clear, that you do not want to keep these slides up!

There has been at least one other instance I know of (*not* involving CERN) on a lower level and following a personal conflict (that was tried to be resolved using an ombudsman). These slides were taken down. (Note that in that case, the slides were not *also* insulting a larger group of people and I think, twitter was not a big thing at the time - so no live-tweeting and picking this up by the media).

(footnote 1) Looking at the slides/general behaviour, you may as well make up your mind, whether they could form the base of an successful future application for a similar position.


[Sabine - if you think, this is too private or should not be discussed for the sake of any of the named people involved, feel free to message me and not publish this post - thanks]

sean s. said...

One of many Unknowns wrote that, “... the simplest explanation is that people drop out ... because they were not doing well. This hypothesis is well supported by the observation that the papers of the women who drop out are less cited ... and therefore more likely to be of lower quality.”[1]

This explanation should be testable and may be circular.

To test the explanation, we’d need to look only at the first one or two papers of all authors and see how frequently those are cited, especially before later papers are produced. It seems likely that even the best person’s first few papers are rarely cited, or may only be cited after they have produced a larger body of work.

A simple explanation is that people take time to develop their careers; those who drop out early never get to the point where they make the contribution that they could have if they’d have continued.

Unknown’s explanation may be circular because it assumes “dropping-out” happens because dropouts are not doing well, and lack of citation of early papers proves they were not doing well.

If women are dropping-out for reasons having nothing to do with their abilities, and if early papers are typically less likely to be cited, then Unknown’s explanation fails.

If the early papers of “remainders” are cited more often than those of “drop outs” early in their careers, then this claim might be credible; but if early papers are typically not cited frequently, then the explanation fails.

sean s.

[1] 5:48 PM, October 07, 2018

Phillip Helbig said...

"Expecting people to have no family responsibilities (ie. that professors must have a full time stay at home spouse)"

Of course, that applies to men as well. Perhaps even more so, because now it has become politically correct (and rightly so) to realize that women don't want to have to choose between career and family. Neither should men. Men might even experience more difficulty trying to combine the two, since it is less common that they raise the issue. If a woman leaves the field because she decided to have children instead, most people's reaction is probably "that shouldn't have been an either/or decision; what a shame" while in the case of a man it is perhaps more often "well, obviously he wasn't good enough anyway; that's just a lame excuse designed to provoke sympathy". :-|

TheBaluchiterium said...

Dr. Hossenfelder,

it is revealing that you do not reply to my point but use strong language instead.

The plot you made using ArXiv data has almost nothing to do with the plot by Dr. Strumia using InSpire data. But it leaves the impression that he was presenting a bad argument. What you should have done instead is take the InSpire data and then see if your suggested modification makes any important difference. But you are a professional, so you know that it probably would not have.

Then the question arises of why you post such an invalid comparison. Perhaps we can take a step back from accusations and you give a better reason than "we had the ArXiv" data for why you have created this post.

Sabine Hossenfelder said...

Dr Fleig,

You continue to accuse me of lying for no other reason that that you evidently don't like what I wrote. Yes, why not stop the accusations, seems a good idea.

TheBaluchiterium said...

Dr. Hossenfelder,

No, I am obviously not saying that you are lying. You present a comparison that is being misunderstood.

In your first reply to my comment, you wrote "I never claimed to "debunk" anything."

Now please take a look at the current version of Alessandro Strumia's Wikipedia page, I cite:
"Physicist Sabine Hossenfelder cited papers addressing some of Strumia's conclusions and provided an alternative analysis, claiming that after accounting for disproportionately higher rates of women leaving the field the sex differences became negligible. Refs.[25][26]"

This stands in stark contradiction to your assertion that you are "not claiming anything". As I observe, many people do think that you are making a claim here, so you will probably be happy to remove that sentence from the mentioned Wikipedia page, in your own interest.

BTW., I tried to remove that sentence in an earlier version of the page, but the change was reverted, suggesting that the conviction of you making claims has taken root.

Sabine Hossenfelder said...

Dr Fleig,

Now you are blaming me for what someone else writes on Wikipedia. Wow.

JimV said...

In response to Dr. Flieg:

The record shows that on July 16, 2018, Dr. Hossenfelder posted " A new tool for arXiv users". This post demonstrated that Dr. Hossenfelder and her associates in the project had obtained the necessary permissions and formatting information for the arXiv database, and developed the program routines and expertise to parse that database for various information. In the comments to that post she mentioned that this effort took half a year.

With that background, it seems natural to me that she would use that expertise, first to replicate Dr. Strumia's analysis with another, similar database which she had immediate access to, and having found general agreement, gone on to test another hypothesis as to what might cause the first result.

Wouldn't you or any scientist have done the same (if you were interested in the issue) as a starting point, rather than working another half-year on a different database? In her current post, she clearly points out her data source and methods, and the need for more exhaustive follow-up. Doesn't that satisfy the general requirements of scientific work thoroughly? To me it does, and you owe her an apology.

I have a general rule for myself, not to blame others unless I can't think of any reasonable excuse for their actions. It evolved during my time doing field service on turbines in power plants, and having to travel to unfamiliar places in rental cars following roadmaps (this was well before GPS). I had to slow down at intersections (without stop lights) in strange cities to read the street signs. So when somebody was ahead of me driving more slowly than I wanted to go, although it might annoy me, I allowed that maybe they too were in an unfamiliar place looking for their next turn. Of course, one could also assume bad motives on the part of the annoyer, but this will only increase your own mental turmoil while quite possibly being incorrect and unjust.

Sabine Hossenfelder said...


Thanks, I appreciate the support. Allow me to clarify that I do not officially work on bibliometric analysis; it's something I do outside my working hours. You could maybe call it a strange hobby. I feel like I better add this because according to my contract I spend 100% of my time on a different project, and on occasion one of our admin people tells me I work too much. So, it's not work ;)

opamanfred said...

Dr Fleig,
Come on...
Sabine wrote in her post: "I must add a warning here. We do not use the same data set as Strumia and Torre. They use data from inspire, we use data from the arXiv. This means our data set does not reach back in time as far, and it includes disciplines besides high energy physics. So the absolute numbers are not directly comparable."

Please tell us why you think it's not enough. To me, it seems more cautious than the average theory paper containing its fair share of unsubstantiated claims.
If others misrepresent her claims, surely it's not her fault, isn't it?

Anyway, the whole Strumia affair looks to me like a contest of gross misrepresentation and bad faith, on both sides, with few exceptions.

Hans Fynbo said...

As other have noticed there are considerable differences in different fields on arXiv. It has also been notived that, despite your comments or footnotes, many have misunderstood your plots as proving that Strumia’s analysis is flawed - e.g. on Wikipedia and twitter. In the interest of understanding what is right and wrong to conclude from this it would obviously be interesting to either use the same database, or perhaps simpler to just restrict your use of arXiv papers to those found in hep-X. In fact it could be interesting to run the analysis for different subgroups on arXiv to see how the curve changes in fields with overrepresentation of either men or women. Would this be difficult to do for you or your student?

Sabine Hossenfelder said...


No, not difficult, but it will take some time. We are looking into it.