Friday, October 05, 2018

Gender-bias in Academia: The Case Strumia

Dr Alessandro Strumia is a physicist working at CERN, where he is Principal Investigator of an ERC Advanced research grant on the topic “Understanding the mass scales in nature.” Ie, he spends your tax money on the kind of research that I criticize in my book. He also recently published an analysis of publication and citation-rates in his field. At a workshop on gender diversity at CERN last weekend, he used this analysis to argue that women are underrepresented in physics because they are underperforming.

I did not attend the workshop and have not seen a recording of the talk, but I have seen the slides (a PDF version of which is here). The slides contain statements that are both inaccurate and exceedingly unprofessional.

For example, he begins his talk by stating that “smarter people are less affected by implicit bias,” but this is wrong. Studies have shown repeatedly that intelligence does not protect from thinking biases. Yes, intelligence is useful to overcome certain types of biases (mostly those that can be exposed with mathematical reasoning), but only once people are aware they are biased to begin with.

Strumia’s mistaken belief that intelligent people are less affected by cognitive biases does not remotely surprise me. I have encountered this very same attitude (“We are too smart to be biased!”) among almost all high-energy theorists and phenomenologists I have spoken with about the issue. That in itself is a bias, known as the “bias blind spot.”

But that Strumia is ill-informed about the very topic he speaks about at a scientific workshop is not the biggest problem with his presentation. Far worse is that he names and attacks two women, apparently because he is annoyed he did not get a job that he was shortlisted for. Nonsense like this just does not belong in a research presentation.

After complaints ballooned on social media on Monday, CERN pulled the slides from the net quickly and has since suspended Strumia. What will happen to his ERC grant is unclear. A large number of members of the particle physics community have meanwhile signed a statement declaring that they distance themselves from the content of Strumia’s talk.

Now, as you know, I have also recently taken up bibliometric analysis. And I admit I found some of the data Strumia showed interesting. We did, in our paper, also look at gender differences, but not for citation counts. We looked at an entirely different quantity, that of research broadness, and for this we did not find any gender differences.

The gender difference that Alessandro Strumia and his co-author Ricardo Torre find is huge. It’s a more than 100% difference in the total number of citations that researchers accumulate throughout their career.

I don’t think that the number of citations is a good measure for scientific performance, but if the difference between the genders was so large, it might mean that women and men chose their research projects in distinctly different ways. That would be interesting. I thus decided to look into this for a bit.

The key figure that Strumia presents on his slides is the total number of citations that researchers accumulate since the publication of their first paper:

Figure from slide 16 of Alessandro Strumia’s talk.

That the horizontal axis is labeled “scientific age” is unfortunate because this term has been coined to emphasize that the scientific age might differ from the chronological time passed since PhD or first paper. If a researcher takes a career-break, for example because of health reasons or for parental leave, their scientific age goes on hold. However, there is presently no standardized way to determine the scientific age, and in any case, you couldn’t do it from publication data alone, you’d also need biographic information.

Since women are more likely to take leave for child-raising, their citations should on the average increase somewhat slower, simply because they have more breaks in which they don’t publish. However, it seems unlikely that this would make such a huge difference. So, while the label on the axis is inaccurate, I don’t think it’s all that relevant.

When I saw this graph, however, another worry came to my mind immediately. When we did our previous analysis, we found that the vast majority of people who use the arXiv publish only one or two papers and are never heard of again. This is in agreement with the well-known fact that the majority of physicists drop out of academic careers.

I am not sure why this surprised me when it showed up in the data. Maybe because, if you work in the field, the drop-outs are pretty much invisible. They leave and you forget about them. But they are there, in the stats, big and fat.

Now, the total number of citations for such drop-outs will accumulate very slowly because they don’t publish new papers any more. And we know that women are more likely to drop out – that’s the “leaky pipeline” and reason why I find myself increasingly often, if not the only woman in the room, then at least the oldest woman in the room. And I’m only 42.

If you leave the drop-outs in the citation analysis, the leaky pipe will pull down the average of female authors more than of male authors.

I hence asked one of our PhD students, Tobias Mistele, to plot the same quantity as Strumia did for our data sample, but to only keep authors who have more than 5 papers in total, and who have published a paper in the last 3 years. This is sloppy way to shrink down the pool to “active researchers only.” It’s maybe not the most sophisticated way to do it, but it should give us an idea how large the contribution from the drop-outs is.

If we plot the number of citations for active researchers only, we see no noticeable difference between men and women:
arXiv data, active researchers only

When normalized to the number of authors per paper (as Strumia did), there is also no noticeable difference between men and women.

I must add a warning here. We do not use the same data set as Strumia and Torre. They use data from inspire, we use data from the arXiv. This means our data set does not reach back in time as far, and it includes disciplines besides high energy physics. So the absolute numbers are not directly comparable.

Another caveat I must add is that we are using a different method to identify male and female authors. We use the author-id algorithm that is explained in our earlier paper, and then try to match first names with a database for common anglo-saxon first names. Naturally, this means that the authors who remain in our sample are most likely to be of Western origin. By this method we assign a gender to 19% of authors. This is in contrast to Strumia and Torre who use a more elaborate gender-id procedure that allows them to match 60%. The remaining authors in our sample break down to 70,295 male und 53,165 female researchers. After applying the above mentioned cuts, we are left with 12,654 male and 8,177 female. That’s not a huge number, but decent.

Let me also mention that probably a similar effect is behind another finding in Strumia’s talk. He points out that women, on the average, are hired into faculty positions earlier. A paper that appeared on the arXiv yesterday argued that this is not a signal that women have an unfair advantage, but simply a consequence of women leaving at a higher rate. If they aren’t hired early, they’ll not be hired at all, which means the average age of hiring is smaller.

Finally, to state the obvious, this is a blogpost, not a paper. The above is a quick and dirty way to check whether removing dropouts significantly affects the large difference between men and women, and the answer seems to be yes. However, we will have to do a more careful analysis to arrive at definite conclusions. I haven’t checked my biases.


I want to thank Tobias Mistele for doing the graphs so quickly and Alessandro Strumia and Ricardo Torre for helpful communication.

107 comments:

  1. It would be interesting to see the unadjusted graph from your dataset as well. This would give a better idea of how close the data tracks to Strumia's, and understand how much the "active researchers" methodology changes the result.

    ReplyDelete
  2. Cy,

    Yes, unfortunately I don't have the data for that. I'm traveling & my student is on vacation now. Might take a few weeks until I can dig that up.

    ReplyDelete
  3. There are likely many reasons why women are underrepresented in physics, social, cultural, psychological, and perhaps biological, however prejudicial thinking should never be acceptable as one of them.

    Many people confusingly equate equality with humans being equal in abilities and possibilities, we are certainly not. Equality is about opportunity and treatment without bias, the rest is earned and/or granted by ones genetics.

    ReplyDelete
  4. Exactly my expectation when I saw Strumia's graph, that he'd likely failed to account for dropouts (and for why women would drop out more frequently)

    ReplyDelete
  5. Thanks, Sabine, very interesting.
    1. I wish to ask why the numbers of male and female researchers are about equal in your samples. This is very different from the actual situation in the HEP theory, where the ratio of male to female researchers is closer to 10:1. Did you apply some pre-selection?

    2. It would not be surprising if the number of citations indeed grows slower with time for female researchers vs. the male researchers, independently of the researchers' individual abilities or actions. This actually happens in other fields. In art, the price of the artwork by female artists appreciates slower with time, regardless of what the artists do. The slowdown effect is large and perhaps comparable in the order of magnitude with the slowdown in the number of citations seen in Strumia's figure. See, in particular, Fig. 1 in https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3083500, with a popular summary of this study in https://www.theartnewspaper.com/news/female-artists-really-do-earn-less-than-men-survey-finds . The selection of "worthy" citations reflects one's ability to recall "memorable" work and may be driven by subtle, hard-to explain intuitions of the same psychological origin as when pricing a "valuable" art. To quote from the above study:

    "Asking why women's art sells for less than men's elicits a long and complex answer, with endless caveats, entirely germane qualifiers and diverse, sometimes contradictory reasons. But there is also a short and simple, if unpopular, answer that none of those explanations can trump. Women's art sells for less because it is made by women."

    ReplyDelete
  6. A very nice blog post that takes into account the leaky pipe effect. Thanks for taking the time to do the analysis and please consider submitting to the arXiv

    ReplyDelete
  7. Very interesting, Sabine. I like your pragmatic ansatz of making sense out of Strumia's data. I also find it exciting that your work might be a hint that my suspicion - that the family-unfriendly career path plays a large role among the reasons why there are so few women in fundamental physics - may be correct after all.

    ReplyDelete
  8. These sampling bias work in both directions and need to be carefully taken into account.

    ReplyDelete
  9. hi Sabine, congratulations for doing real work. This is how the workshop should have been done.

    Of course, I had done an analogous check, and got the same qualitative trend shown in the talk, after restricting to hired/unhired authors. Adding the extra cuts considered by you (more than 5 papers, after ≈1995), the trend persists. My collaborator has not double-checked this claim.

    I suggest that you can quickly roughly match to our database by restricting to hep arXiv categories (hep-ph, th, ex, lat; astro; nucl-th,ex, gr-qc).

    A comment about the paper you quote at the end: it claims that "female astronomers are leaving the academic labor market, at a rate that is 3-4 times higher than male astronomers". We have data about M/F drop-outs (percentage of authors who wrote a paper and abandoned physics), that I avoided showing in order not to hurt the sensibility of young physicists, and because we don't really know how complete InSpire is about them. Trusting InSpire, the gender difference is a few 10%, much smaller than in the paper you quote. I see they used the Astrophysics Rumor Mill as database, while we use InSpire which is, at least, much much bigger.

    Some details. We reconstruct the sex of 85% of authors, not 60%. Those missing mostly don't have names in InSpire (and we might have problems with Chinese authors, because some only have English transliteration, erasing most information about M/F). We add some country information, because various names like Gabriele or Andrea are M/F depending on the country. This, like any other mistake in M/F identification, goes in the direction of reducing a possible asymmetry.


    PS: in our analysis we found your name among top-cited recent female authors.

    ReplyDelete
  10. It would be interesting to look at the variance of publications at differnet years out for men and women too, excluding complete dropouts. The general pattern is that males are more variable, but it might well not be true here, since this group is already pre-selected for being veyr very good (that is, they are people who got physics jobs). This would combine interestingly with the Assistant Professors As Options idea.

    ReplyDelete
  11. I wish I had a way to correct my own biases, or at least to learn what they are.

    ReplyDelete
  12. Dr. Hossenfelder

    This post made me smile ear to ear even before I read the whole thing. Thank you. I know every time some edge-lord outs themselves publicly everyone piles on but I always love to hear about a righteous prat fall.

    It would be nice to see a good take down of the "leaky pipes/maternity + profit motive -> gender pay gap" argument. But when I've looked I seem to have trouble finding a treatment that is rigorous enough to use graphs and data sets. Mostly it's puff pieces written by people who are more comfortable with less-then-numeric approaches. The search pollution around any of these culture war things always confounds me and given your post I was hoping you'd know who I should be reading.

    ReplyDelete
  13. And it would be interesting to analyze the data on the number of citations at the time of hiring - since there are conflicting opinion on the direction on bias, and any real data are interesting here

    ReplyDelete
  14. Thank you for debunking of the slides. It's interesting to know what was wrong in Strumia's analysis.

    However I think we can't deny there is effectively a multitude of gender bias in STEM.
    Sometimes it's in favor of men. Sometimes it's in favor of women.

    I'll use as a particular example (in favor of women) the CNRS medals awarded to researchers each year (approximately 20 silver medals and 40 bronze medals per year).

    If you go to the CNRS website it is said it employ 11 204 researchers with a women proportion of 32.9%.
    Link : http://www.cnrs.fr/mpdf/spip.php?article201

    So logically, and if there is no ability difference between men and women, we should find the same proportion in the awarded medals.

    Let's see the numbers.
    Here for silver medals : https://www.cnrs.fr/fr/talents/cnrs?medal=39
    Here for bronze medals : https://www.cnrs.fr/fr/talents/cnrs?medal=40

    2016 :
    Total silver medal awarded : 16
    Women : 8 (50%)

    Total bronze medal awarded : 40
    Women : 22 (55%)

    2017 :
    Total silver medal awarded : 20
    Women : 10 (50%)

    Total bronze medal awarded : 40
    Women : 26 (65%)

    2018 :
    Total silver medal awarded : 20
    Women : 9 (45%)

    Total bronze medal awarded : 42
    Women : 21 (50%)

    Conclusion : Instead of awarding medals roughly 33% time to women, we see the CNRS committees try the respect the parity between men and women. It's a clear and unambiguous example of male discrimination in STEM. The medals are not awarded only for pure merit but also with a huge gender bias.

    ReplyDelete
  15. Ban clothing from academic environments. Women would be a distraction, men would be a nuisance, and journalism would infest elsewhere - back to normal for everybody, and with no thermostat rancor.

    Social problems are solved by doing less.

    ReplyDelete
  16. So you delete the low publishers, which was the whole point of his data, and you are left with the norm. Most people do collaborations nowadays so when the low publishers are removed, the graphs are expected to be the same pretty much... If you had Strumias data I would think they go even more hand-in-hand when you make the decision to cut out the low publishers, and delete outliers. But what is the point ?

    ReplyDelete
  17. bee:

    "we found that the vast majority of people who use the arXiv publish only one or two papers and are never heard of again. This is in agreement with the well-known fact that the majority of physicists drop out of academic careers."

    this is a reasonable interpretation but it is only one of several.

    e.g. in the field of pop music, a vast majority of the individuals and groups (those whose records are top sellers) have only one or two such recordings in their careers. these individuals and groups)are known as "one-hit wonders'. this is also a well-known phenomenon in the field of book publishing.

    there's no doubt that being in academia increases the number of papers one writes ('publish or perish' is a real thing, although it it is often more accurately described as to 'bring in grant money or perish') but it may also be true that the majority of researchers simply have only one or two 'good' ideas or projects worth publishing.

    it would be very interesting to know if the common myth that professors effectively 'retire' from doing research once they attain tenure is a result of intellectual laziness (due to their having guaranteed lifetime employment) or intellectual burnout (due to the immense pressure they were under during the pursuit of such employment).

    naive theorist

    ReplyDelete
  18. As I've said before, probably ad nauseum, that those of us who perceive ourselves as biased against, would be smart to avoid professions where your performance is judged by how well you are liked instead of some more objective parameters.

    Publish or Perish was a well-known problem by the time I entered college in 1974. According to the wiki, the earliest known usage of the phrase dates back to 1927.

    Citation counts is a subset of Public or Perish, since you can't cite what hasn't been published.

    I'm not the only member of a minority group to avoid academia for that reason.

    That fact must certainly skew the results, but can't ever be measured. I even suspect the smartest of the minority groups will avoid academia until "Publish or Perish" is no longer a concern, or our view of ourselves as being biased against has changed.

    ReplyDelete
  19. So is your conclusion that there is no discrimination in citation?

    ReplyDelete
  20. Your plot is interesting, but I think it does not yet prove that Strumia was wrong. Can you make a plot conditioned to people having a permanent job? People having a permanent job are supposed to produce at a good level. If they stopped publishing it's their fault.

    ReplyDelete
  21. Dear Sabine, thanks for your analysis, you raised good points. Seeing the unadjusted dataset would be useful, as mentioned. I hope you'll update us on this, time permitting. Again thanks for your analysis and comments, they are very insightful.

    ReplyDelete
  22. Pavel,

    It's a good question, but unfortunately I can only answer I don't know. There may be various reasons for that, eg female names may be better covered in the table that we use or maybe women are more likely to use their full name to begin with so that they can be identified. We'll look at this, but it will take a while.

    ReplyDelete
  23. Pavel,

    Regarding your point 2: Yes, priginally I was thinking it may be a real effect, which would in a sense be more interesting. Well, I hope that Strumia & Torre do some further analysis that will clarify the issue.

    ReplyDelete
  24. Emily,

    We're working on a paper and thinking of including the above gender breakdown. But it'll take a few more months for the paper to be finished.

    ReplyDelete
  25. Unknown,

    If you find me among the top-cited recent female authors there's clearly something wrong with your analysis. More seriously, I am open to the possibility that this is a real effect. It doesn't seem implausible to me that women have other priorities than men in choosing and carrying through their research topics which reflects in the average citation rates. I am stunned though by how large the effect is.

    Yes, we can shrink down our data to only a few arXiv categories. However, I'd be concerned that then the sample is so small that it becomes inconclusive because there's an element of luck in whether your research attracts attention or not. Unfortunately, I don't presently have a good idea for how to estimate the significance, other than doing some agent-based modeling. And with those models you tend to simply find what you put into the setup.

    In any case, tbh, I am not terribly interested in the whole gender-thing, hep has bigger problems than that. But the ppl who work on this should look at it.

    One more thing: It is quite possible that the reason we see no effect is due to our name-gender matching. As I mention, we have used a table for common anglo-saxon names. Since we throw out everyone we couldn't identify, this effectively means that we remove the vast majority of researchers in India, Asia, Eastern Europe, etc. This very possible has an effect on the male/female ratio. Best,

    B.

    PS: A name would be helpful. There are a lot of "Unknowns" in my comment sections.

    ReplyDelete
  26. Jonathan,

    Yes, unfortunately a lot of studies on the subject are far from convincing, tiny samples, shaky methods. I definitely wish there was better data. The cynic in me sometimes suspects that maybe some people don't really want to know the truth.

    ReplyDelete
  27. lollerboller,

    We have removed those who left academia to find out whether the difference is anything more than the well-known fact that women leave in higher proportion.

    ReplyDelete
  28. Christophe,

    No, this analysis does not tell you anything about discrimination.

    ReplyDelete
  29. Tsetrot,

    I have no reason to think that Strumia is wrong in the sense that his data analysis is wrong. The question is whether that's an interesting and/or new finding. We don't have bibliographic information for authors, where are we supposed to take those from?

    ReplyDelete
  30. I admire physicists' typical confidence to plunge into any subject, but it might be a good idea to consult the specialized research on this general topic. A good review article covering everything from cognitive research to bibliometrics across numerous STEM and social science fields is:
    http://journals.sagepub.com/doi/abs/10.1177/1529100614541236

    ReplyDelete
  31. Appreciate your commentary here, Bee, as always. I find it absurd and dangerous, however, whether Strumia's comments have zero merit or non-zero merit, that CERN would literally remove his slides from their workshop website. This is neanderthal-style censorship that should have no place in modern science. Would appreciate your calling this dangerous trend out if you can.

    ReplyDelete
  32. I am a little confused with the conclusions of your analysis. It is a widespread opinion (at least in Sweden, where I have spent a good deal of time), that a lower citation rate for female scientists is a sign of discrimination. Now you seem to have eliminated this indication. In fact men and women are treated equally when it comes to citations, if judged by your almost identical curves in the carreer-corrected plot.
    I understand that identical curves means that on basis of those data you can not claim that men are being discriminated against, but in all honesty it must work both ways.

    ReplyDelete
  33. Dear Dr. Hossenfelder,

    the comparative graph you show which "does not present any noticeable difference between men and women" has been, as we read in the footnotes "based on a different data set" (at least you were honest enough to mention this difference). So this comparison does not debunk Dr. Strumia's graph at all, because he was talking about FUNDAMENTAL PHYSICS, and nothing else. And from the comments section I can deduce that most readers only read what they want to read and neglect the footnotes.

    The fact that you use a different data set - of which you, as a professional, know that its content differs strongly from the dataset used by Dr. Strumia - suggests that you are making politics on this point, not honest research. This is deceptive tactics, because most of your readers on this blog will not know the difference between InSpire and ArXiv.

    In order to clarify: InSpire is the database for unpublished work mainly in high-energy physics, whereas ArXiv covers many other scientific fields, in particular medicine and biology. That makes all the difference in the world.

    Best regards
    Timo Fleig
    (A colleague in theoretical physics)

    ReplyDelete
  34. Tam,

    I believe, but do not know, that CERN pulled the slides primarily for two reasons: Firstly to show they were reacting quickly and not ignoring the issue, and second because Strumia made personal accusations - he actually had names on the slides. It's this latter point that really crosses a line. This is not "neanderthal-style censorship" it's to protect an innocent person, and I think that CERN did well removing the slides.

    ReplyDelete
  35. Dr Fleig,

    Your accusation that I am not being honest and "use deceptive tactics" is sick shit. We use the arXiv database simply because we've used it before & it's a check we could do quickly. I have added all information clearly and explicitly. I never claimed to "debunk" anything. You even managed to quote a sentence I haven't written and that indeed makes no grammatical sense. And what the heck do you know about my readership? You have some thinking to do, man.

    ReplyDelete
  36. This comment has been removed by the author.

    ReplyDelete
  37. Thank you very much for this post. It may be reasonable to say only "this is wrong, he is a bad person" and be light on the details of why - because most will understand why. However I think refuting the logic and reasoning as you have done is much more powerful, particularly as a some people may yet sympathise with Strumia.

    I find it hard to explain clearly but it is very profound for me. We can easily think bad things about the person, and that may well be right, but it is more important to think bad things about the arguments and reasoning.

    Working with him to clarify and improve your response is right.
    Thanking him is class.

    ReplyDelete
  38. We reconstruct the sex of 85% of authors

    Really? I do hope the method for doing this is (or soon becomes) fully public. On its face, it seems an astonishing achievement! Perhaps authors in InSpire give their full, legal names; otherwise how to tell if Robin, say, or Chris, or Alex are M or F? The lead author of an astrophysics paper, soon to be published, on which I am a co-author, is Chen Wu; how to tell if that's M or F? Yes, if you had the Chinese character, you may get a good idea.

    We add some country information, because various names like Gabriele or Andrea are M/F depending on the country.

    Cool! So Jean who was born in Australia but now lives in Belgium changed from F to M? And Jean who was born in Switzerland but now lives in New Zealand went from M to F? And Jean who lives in Ottowa but commutes (2 hours each way) daily to Montréal ... well, s/he must be very confused.

    Maybe the Mean Career Path figures could have another set added, the "we can't tell if M or F" one?

    ReplyDelete
  39. They always warn women and minorities to choose fields with objective measurements, like money. If high energy physics was still about making sense of new data, there would be an objective way to judge work quality. Now that it's a beauty contest, quality rankings are much more subjective. When publications and citations are about pretty, it's hard to take them seriously. Even the fashion business has ditched this.

    Thanks for looking at the data though. I think we're just seeing more of the usual - "No girls allowed." - "Why are there no girls here?"

    ReplyDelete
  40. Not *my* UG student, Bee, and *you* know what *I* mean.

    --Ajit

    ReplyDelete
  41. Sorry for disagreeing with most people on the thread, but I think there is a flaw here. Dam it, going against the narrative! sorry again! I hate to intrude when everyone was in self-congratulation mode.

    The conclusion of your graph is that all things being corrected for, remaining female scientists - i.e. the ones that manage to produce 5 papers or more - do as well as the remaining male scientists. I think this is correct and I do not think anybody, even Strumia, ever denied that the best women physicists could be as good as men (or even better, see Marie Curie). The question is not on the individuals, it is on the average. Strumia is arguing that in the 'best' slice there are less women than men. I think your data confirms this interpretation.

    The question is to figure out if women will, on average, do as well, better, or less well than men. What you are showing here is that more men (~12.000) remain active scientists than women (~8.000). You call this the leaky pipeline and argue that it is leaky because of social constraints. Well, let us forget about sociology - we really should BTW - and treat this data as we would treat any regular biological experiment, by sticking to the most parsimonious explanation.

    In this case, the simplest explanation is that people drop out - women and men - because they were not doing well. This hypothesis is well supported by the observation that the papers of the women who drop out are less cited - hence your massive correction- and therefore more likely to be of lower quality. If the dropouts were as good as the remainders, their papers would keep being cited long after they dropped out and the effect reported by Strumia would no occur. Clearly, this is not the case.

    The high dropout rate is, therefore, an argument in favor of Strumia argument, while your data shows that a sizeable fraction of women are competitive with men. It's all about averages, and this is all what Strumia was saying.

    The argument is not, however, totally closed because an extra normalization is needed here: the graph should be Ncit/Npaper so as not to penalize scientists having produced few papers, but rather penalize scientists having few citations per citable unit.

    ReplyDelete
  42. You ranked at the top of Google:
    https://imgur.com/a/sWaypYP

    ReplyDelete
  43. Ajit,

    Yes, sorry. I actually try to avoid it. Not, UG though.

    ReplyDelete
  44. Unknown,

    First, if you aren't able to use a Google account, could you *please* at least sign with a name or a pseudonym.

    Second, what you say is a possible interpretation of the data, but I wasn't claiming the leaky pipe is entirely due to social influence. I was merely saying it's not a new finding.

    Third, having said that, given that social hurdles exist for women that do not exist for men, it would be odd if these did *not* contribute to the leak. And, needless to say, this problem should be fixed.

    Best,

    B>

    ReplyDelete
  45. Unknown,

    PS: We also checked the normalized N_cit, and it doesn't have a noticeable difference between men and women either. However, I am not a big fan of combining different measures needlessly because that makes it harder to quantify the origin of correlations. Also, as I emphasized, this isn't a publication-level analysis and I want to do further checks.

    ReplyDelete
  46. Concerning the gender assignement, this is the method prescribed by Strumia&Torre:

    We consider 70500 authors indexed in the InSpire database (see appendix A for details). The
    Mathematica machine learning function Classify has been trained to automatically assign
    to each name a tag: male, female or indeterminate (about 40%, that we ignore). Classify
    sometimes fails (the female with highest PaperRank is Nicola Cabibbo, in 51th position): to avoid distortions we manually correct the most notable cases. Then, the female authors with
    highest PaperRank are: M. Gaillard (in 114th position), H. Quinn (220th), R.E Kallosh (370th,
    who becomes top-ranked female in 78th position restricting to papers written after 2000),
    L. Randall (444th). HepNames does not include old authors like Noether, the Curie, Mayer.
    Among the authors classified as male or female, 16% of the names in the data-base are
    female, and receive 8.4% of the individual citations and 5.6% of the rank.

    Classify sometimes fails (the female with highest PaperRank is Nicola Cabibbo, in 51th position) -- Indeed this statement does not fill me with great confidence (specially since the number of female Nicolas in general is probably smaller than the number of male Nicolas working outside of Italy).

    There are indeed a few cross checks, that would be interesting and valuable to do. In first instance restricting to an American first-name dictionary is probably safer.

    from the same paper:
    Fig. 13 shows that the percentage of individual citations received by female authors is growing and is a factor of 2 higher in sub-fields dominated by large experimental collaborations (where bibliometrics cannot identify individual merit) than in more theoretical fields (where social effects are less important).
    --> I would *also* claim, that social effects could be *larger* in the theoretical field, when it comes to citations (because you cite your "friends" or you write 20 emails every morning to get people to cite you), but perhaps this is just some sloppy wording or is based on a definition of "social effects" that is different from mine.

    ReplyDelete
  47. Which reminds me: "we manually correct the most notable cases" is a really bad idea. You don't go and fumble with your data by hand, this brings in personal bias. I didn't comment on this in my post because, given the total number of people in the sample, I doubt it makes a difference for the averages.

    ReplyDelete
  48. @Unknown

    simple is more parsimonious than parsimonious

    ReplyDelete
  49. @Unknown

    there are other explanations as simple as "people drop out - women and men - because they were not doing well":

    because they find the field dumb
    because they find the field below their worth (intellectually, economically..)
    because they find the field interesting, but not that interesting
    because they find the field interesting, but a dead end
    because there are so many other interesting things to do
    because there are too many assholes
    because there are too few assholes
    because it is boring

    ReplyDelete
  50. It saddens me to read here that CERN pulled the slides, suspended Strumia, and that his grant status is now challenged. If his research had shown no gender "performance" bias, there would be no such backlash, as that result would have been in line with the approved narrative for our society. Right or wrong, it isn't healthy scientifically or sociologically to silence the discussion and attempt to erase the evidence that it happened. The questions motivating Strumia's research have not gone away, so it would be better to address them in an open way. Not every conference talk must be correct to be worthwhile. The research merely has to be done in good faith, which I believe this is. It's the job of the community to criticize and try to pick apart this result, but not to squelch it.

    ReplyDelete
  51. Anonymous writes: "Concerning the gender assignement, this is the method prescribed by Strumia&Torre...". That was the method used in version 1 of the paper. The slides use a more accurate method.

    Sabine, is your plot dominated by big experimental groups, inside which gender asymmetries are lost? If yes, it would be interesting to either restrict to papers with few authors, or do a fractional counting of citations.

    best,
    Alessandro Strumia

    ReplyDelete
  52. We reconstruct the sex of 85% of authors, not 60%. - Unknown, 5 October

    male, female or indeterminate (about 40%, that we ignore) - Anonymous, 8 October

    So, which is it?

    The Mathematica machine learning function Classify has been trained to automatically assign to each name a tag: - Anonymous, 8 October

    As anyone who has experience with ML can say, what an ML is trained on is vital; specifically, if the training set contains garbage, the results will also contain garbage.

    And if training sets include bias, the results will also contain bias. For example, perhaps the training set is wonky, shall we say, when it comes to East Asian names, Chinese ones in particular. And if this ML fails with "Nicola", how confident can we be that it doesn't fail with "Chen" (a very common Chinese name)? Yes, the authors may recognize a "Nicola" failure, but would they even notice a "Chen" one?

    The Strumia study is, to me, looking more and more like very poor quality, which any reviewer of integrity would reject ...

    ReplyDelete
  53. This comment has been removed by the author.

    ReplyDelete
  54. I'm gonna have to read this closely.

    sean s.

    ReplyDelete
  55. “We are too smart to be biased!”

    Most such also describe themselves as being above arrogance.

    ReplyDelete
  56. "We reconstruct the sex of 85% of authors"

    I've heard of sex-reconstruction surgery, but that is something different!

    ReplyDelete
  57. "Klavs Hansen said...

    I am a little confused with the conclusions of your analysis. It is a widespread opinion (at least in Sweden, where I have spent a good deal of time), that a lower citation rate for female scientists is a sign of discrimination."


    Without taking sides on this issue, do you seriously think that this issue could even be discussed in Sweden? Of course most people in Sweden believe that it must be due to discrimination.

    This (one's culture) is also a huge bias.

    (I speak Swedish and, spread out over 35 years, have spent almost a year in Sweden altogether.)

    ReplyDelete
  58. The best I can tell about Strumia is that he is a second rate physicist feeling that he is owed a job based on his sex rather than his work. That seems to be the simplest explanation. If he had felt his work was superior, he would have gone on the attack in a physics forum, not one on gender balance.

    ReplyDelete
  59. The extreme overreaction and appalling censorship behavior of CERN management and a minority subset (yes minority, the absences are way more notorious than the inclusions) of the scientific community over the Strumia affair has already created a significant counter-reaction and the condemnation has backfired. How do I know this? Because if you listen what people are saying outside of liberal academic circles, you find that most people has been shocked by a scientist getting expelled for expressing his (correct or not, that is irrelevant) conclusions based on some (flawed or not, again irrelevant) data analysis. By no less than CERN. And shocking it is indeed that academic research institutions are so politiziced.

    I have skimmed through the comments of liberal academic blogs where you find majority support for what amounts to some sort of intellectual lynching of Strumia, and the contrast could not be bigger with the comments you find in the general press, the general public overwhelmingly find appalling that a Strumia was kicked out, and it has created a lot of sympathy for his views. I have seen people commenting that they are going to change their vote in the next elections, after voting leftwing parties all their lives, because the populist right are the only ones that are going to stop the "feminazism" and the positive discrimination policies. Extremism breeds extremism (but of course all extremists will deny they are).

    This backfiring will be denied again by the most fervent metoo activists, but it is the truth and can be checked by anyone that has open eyes and ears.

    ReplyDelete
  60. JK,

    The people who bother to comment are the ones who hold strong views. The majority can't be bothered one way or the other. It's good to keep this in mind. Personally I think the silent majority is a problem bigger than the extremists on either side.

    ReplyDelete
  61. You would think that the question of gender bias in science is a perfect subject for a double blind study. And so it is:

    Science faculty’s subtle gender biases favor male students
    http://www.pnas.org/content/109/41/16474

    We also assessed faculty participants’ preexisting subtle bias against women using a standard instrument and found that preexisting subtle bias against women played a moderating role, such that subtle bias against women was associated with less support for the female student, but was unrelated to reactions to the male student.

    Obviously, not everyone will be convinced by a mere double blind study:

    Quality of evidence revealing subtle gender biases in science is in the eye of the beholder
    http://www.pnas.org/content/112/43/13201

    Results from our three experiments, using general-public and university faculty samples, demonstrated that men evaluate the quality of research unveiling this bias as less meritorious than do women.

    In this blog post, we see both effects neatly illustrated. There is a gender bias in science, with accomplished scientists arguing the existing differences are biological (i.e., blaming the victim), there are data showing this is not true, and based on very bad statistics reasoning, and there are (men) dismissing all the evidence that there is a bias against women.

    PS, how would the authors of the original paper react to a social scientist publishing on theoretical physics?

    ReplyDelete
  62. "By no less than CERN. And shocking it is indeed that academic research institutions are so politiziced."

    There is another angle here. Strumia has put up a big banner over CERN saying "Women need not apply".

    What do you think will happen when a scientist at a biology faculty claims "Creationism is True", or a scientist at a health institute who claims "Smoking does not cause cancer" or "HIV does not cause AIDS"? Or, to take it outside science, a journalist claiming "Truth is just an opinion"?

    In all cases, these persons undermine the very missions of the institution they work in.

    Extraordinary claims require extraordinary evidence. Strumia had very bad evidence and added conclusions that far outreached the "evidence" he had. Moreover, I assume he had to hire and supervise students and staff. How could he now have any legitimacy when evaluating female students and staff?

    ReplyDelete
  63. "There is a gender bias in science"

    Whatever the status of Strumia, your conclusion, quoted above, does not follow from your evidence. At most, you have shown that there is a bias among those taking part in the survey. This almost certainly varies with location, with epoch, and with the personalities of the people surveyed. It does not prove that "science", per se, is biased. I am not doubting the evidence you present, merely the grand conclusion derived from it.

    ReplyDelete
  64. "I have seen people commenting that they are going to change their vote in the next elections, after voting leftwing parties all their lives, because the populist right are the only ones that are going to stop the "feminazism" and the positive discrimination policies."

    Two points. The number of people who even know who Strumia is is not a significant factor in any election. Yes, sadly, there are people who think that a worse extremism is an antidote to extremism, cutting of their noses to spite their faces.

    ReplyDelete
  65. @Philip Helbig
    "At most, you have shown that there is a bias among those taking part in the survey. This almost certainly varies with location, with epoch, and with the personalities of the people surveyed. It does not prove that "science", per se, is biased. "

    Lets look at this sentence by sentence.
    "At most, you have shown that there is a bias among those taking part in the survey."

    What you say is that samples cannot be used in social science. Which would include the data used by Strumia. It is not that the population of this study was somehow aberrant wrt "average" science faculty.

    "This almost certainly varies with location, with epoch, and with the personalities of the people surveyed."

    The above study showed that there was considerable bias in an important subset of science institutes. In addition to the above conclusion, it is now on those claiming women perform worse in science than men to come up with evidence that this is NOT caused by bias.

    It is not that there is a dearth of studies showing such bias in every country studied. I did not want to swamp this blog with references, but typing in "gender bias in science" in Google Scholar already gives you close to 500 references to mull about.

    The challenge is to find a location, epoch, and personalities that do NOT show a gender bias in science.

    " It does not prove that "science", per se, is biased. "

    In science you cannot prove an assertion. However, it is fairly easy to disprove the statement "women are not treated differently in science".

    I have yet to find a group in science that does not show gender bias in some form or another. At a certain moment denial does not let the problem go away. Scientist reflect society. IF society has a gender bias, scientist will not be immune to it.

    ReplyDelete
  66. Hello Sabine,

    Thanks for your efforts.

    I'm a woman with an undergraduate degree in physics and a MASc in electrical engineering from the University of British Columbia. I strongly considered doing a PhD in Physics at the University of British Columbia. My choice not to was as follows:

    1. Overall job prospects looked poor in physics (compared to electrical engineering)

    2. Subtle gender bias and assumptions about whether or not women could do mathematical and theoretical work was a turn off (this seemed to be a bigger problem in physics than in EE)

    3. The emerging popularity of string theory at the time that I was considering a physics PhD was a turn off (as was another fad at the time, quantum intelligence.) I wanted to work on practical problems, and string theory and quantum intelligence seemed like a long way from that.

    4. General unfriendliness of the professors and refusing to talk about issues like how having children could be combined with a career in physics was a big turn off.

    5. Frequently not being taken seriously was a big turn off. The most glaring example of this was when I interviewed in 1994 at the forerunner of the company that would become D-Wave Systems. By this time, I had an undergraduate degree in Math and Physics, another undergraduate degree in electrical engineering (with minor in computer engineering), and an MASc in electrical engineering. During the interview, I was treated like a high school student or an item for purchase in a novelty store (not as a potential colleague.)

    Lack of interest in physics or math was not the reason that I decided not to pursue a degree in physics. I still am fascinated with science, engineering and computer science.
    But little has changed since the 1980s and 90s when I was in university. On the bright side, my 14 year old daughter is acing geometry and algebra. I'm thrilled that she isn't burdened quite so much as I was with assumptions about her mathematical ability.

    Here are some things that still have to change (in STEM, including physics), before we can really say that women are welcome:

    Assumptions that women like this and men like that (ie. don't assume that women like experimental work and men like theoretical work, or women like to work on socially relatable research topics while men don't)

    Assuming that women can be paid less for similar work (still very common)

    Expecting women to be socially accommodating while rewarding men for being aggressive

    Punishing women in their performance evaluations for not being socially accommodating

    Assuming women have less mathematical ability compared to men

    Expecting people to have no family responsibilities (ie. that professors must have a full time stay at home spouse)

    Refusing to accept that gender harassment and discrimination exist, and therefore, that any person reporting this must be a crazy.

    That would be a good start.

    ReplyDelete
  67. Perhaps physics is unlike engineering, biology, psychology, and economics, but in those fields there appears to be significant hiring preference for women in the U.S.

    http://www.pnas.org/content/112/17/5360

    ReplyDelete
  68. JV: “I have seen people commenting that they are going to change their vote in the next elections, after voting leftwing parties all their lives, because the populist right are the only ones that are going to stop the ‘feminazism’ and the positive discrimination policies.

    Anyone who’s been “voting leftwing parties all their lives” knows that racism and sexism are positive discrimination policies; they’re not going to abandon the fight and start supporting populist, racist, sexist politicians so readily. Anyone who makes these “comments” probably has not voted for leftwing parties much if ever.

    Positive discrimination is evil; discrimination against men (real or imagined) is not worse than discrimination against women (which is almost invariably real).

    No doubt there will be some injustices in the fight to end discrimination. That’s unavoidable. But to give up is accept avoidable injustice.

    A basic fact of persuasion is that there will always be those who can never be convinced; no matter how much we learn about discrimination against women, there will be those who don’t believe or who think it’s acceptable. They will proclaim their anger at feminists, but the fact is that the only way to mollify them is to acquiesce to gender-discrimination.

    sean s.

    ReplyDelete
  69. (especially @JK)

    I would like to remind again: In his slides, he had a personal attack on someone *named* explicitly, implying they got a job that he was (according to his citation metric) better qualified to do.

    "This job" was a directorate level position, that of course includes management and diplomacy(footnote 1), dealing with ministries and so on, which are things possibly not very well measurable using a citation count.

    Again, (and I am very sad, that this was not emphasized enough in the general press) he personally attacked a high-rank official of a research institution. If a high-rank official of CERN would be attacked personally as being unjustly hired, including defaming a member of the hiring committee, in a talk by someone employed as researcher in e.g. the Academy of Science in France -- very obviously you would expect the Academy to take the defaming slides down, otherwise you would be complacent, i.e. implying that these accusations are just or that they might be correct. As a scientific workshop organiser you might not be legally able or entitled to judge said correctness. This is what e.g. ombudsmen or courts are for.

    With this line of reasoning, you do not need to discuss the other content of the slides. It is clear, that you do not want to keep these slides up!

    There has been at least one other instance I know of (*not* involving CERN) on a lower level and following a personal conflict (that was tried to be resolved using an ombudsman). These slides were taken down. (Note that in that case, the slides were not *also* insulting a larger group of people and I think, twitter was not a big thing at the time - so no live-tweeting and picking this up by the media).

    (footnote 1) Looking at the slides/general behaviour, you may as well make up your mind, whether they could form the base of an successful future application for a similar position.

    Kaylo

    [Sabine - if you think, this is too private or should not be discussed for the sake of any of the named people involved, feel free to message me and not publish this post - thanks]

    ReplyDelete
  70. One of many Unknowns wrote that, “... the simplest explanation is that people drop out ... because they were not doing well. This hypothesis is well supported by the observation that the papers of the women who drop out are less cited ... and therefore more likely to be of lower quality.”[1]

    This explanation should be testable and may be circular.

    To test the explanation, we’d need to look only at the first one or two papers of all authors and see how frequently those are cited, especially before later papers are produced. It seems likely that even the best person’s first few papers are rarely cited, or may only be cited after they have produced a larger body of work.

    A simple explanation is that people take time to develop their careers; those who drop out early never get to the point where they make the contribution that they could have if they’d have continued.

    Unknown’s explanation may be circular because it assumes “dropping-out” happens because dropouts are not doing well, and lack of citation of early papers proves they were not doing well.

    If women are dropping-out for reasons having nothing to do with their abilities, and if early papers are typically less likely to be cited, then Unknown’s explanation fails.

    If the early papers of “remainders” are cited more often than those of “drop outs” early in their careers, then this claim might be credible; but if early papers are typically not cited frequently, then the explanation fails.

    sean s.

    [1] 5:48 PM, October 07, 2018

    ReplyDelete
  71. "Expecting people to have no family responsibilities (ie. that professors must have a full time stay at home spouse)"

    Of course, that applies to men as well. Perhaps even more so, because now it has become politically correct (and rightly so) to realize that women don't want to have to choose between career and family. Neither should men. Men might even experience more difficulty trying to combine the two, since it is less common that they raise the issue. If a woman leaves the field because she decided to have children instead, most people's reaction is probably "that shouldn't have been an either/or decision; what a shame" while in the case of a man it is perhaps more often "well, obviously he wasn't good enough anyway; that's just a lame excuse designed to provoke sympathy". :-|

    ReplyDelete
  72. Dr. Hossenfelder,

    it is revealing that you do not reply to my point but use strong language instead.

    The plot you made using ArXiv data has almost nothing to do with the plot by Dr. Strumia using InSpire data. But it leaves the impression that he was presenting a bad argument. What you should have done instead is take the InSpire data and then see if your suggested modification makes any important difference. But you are a professional, so you know that it probably would not have.

    Then the question arises of why you post such an invalid comparison. Perhaps we can take a step back from accusations and you give a better reason than "we had the ArXiv" data for why you have created this post.

    ReplyDelete
  73. Dr Fleig,

    You continue to accuse me of lying for no other reason that that you evidently don't like what I wrote. Yes, why not stop the accusations, seems a good idea.

    ReplyDelete
  74. Dr. Hossenfelder,

    No, I am obviously not saying that you are lying. You present a comparison that is being misunderstood.

    In your first reply to my comment, you wrote "I never claimed to "debunk" anything."

    Now please take a look at the current version of Alessandro Strumia's Wikipedia page, I cite:
    "Physicist Sabine Hossenfelder cited papers addressing some of Strumia's conclusions and provided an alternative analysis, claiming that after accounting for disproportionately higher rates of women leaving the field the sex differences became negligible. Refs.[25][26]"

    This stands in stark contradiction to your assertion that you are "not claiming anything". As I observe, many people do think that you are making a claim here, so you will probably be happy to remove that sentence from the mentioned Wikipedia page, in your own interest.

    BTW., I tried to remove that sentence in an earlier version of the page, but the change was reverted, suggesting that the conviction of you making claims has taken root.

    ReplyDelete
  75. Dr Fleig,

    Now you are blaming me for what someone else writes on Wikipedia. Wow.

    ReplyDelete
  76. In response to Dr. Flieg:

    The record shows that on July 16, 2018, Dr. Hossenfelder posted "SciMeter.org: A new tool for arXiv users". This post demonstrated that Dr. Hossenfelder and her associates in the project had obtained the necessary permissions and formatting information for the arXiv database, and developed the program routines and expertise to parse that database for various information. In the comments to that post she mentioned that this effort took half a year.

    With that background, it seems natural to me that she would use that expertise, first to replicate Dr. Strumia's analysis with another, similar database which she had immediate access to, and having found general agreement, gone on to test another hypothesis as to what might cause the first result.

    Wouldn't you or any scientist have done the same (if you were interested in the issue) as a starting point, rather than working another half-year on a different database? In her current post, she clearly points out her data source and methods, and the need for more exhaustive follow-up. Doesn't that satisfy the general requirements of scientific work thoroughly? To me it does, and you owe her an apology.

    I have a general rule for myself, not to blame others unless I can't think of any reasonable excuse for their actions. It evolved during my time doing field service on turbines in power plants, and having to travel to unfamiliar places in rental cars following roadmaps (this was well before GPS). I had to slow down at intersections (without stop lights) in strange cities to read the street signs. So when somebody was ahead of me driving more slowly than I wanted to go, although it might annoy me, I allowed that maybe they too were in an unfamiliar place looking for their next turn. Of course, one could also assume bad motives on the part of the annoyer, but this will only increase your own mental turmoil while quite possibly being incorrect and unjust.

    ReplyDelete
  77. JimV,

    Thanks, I appreciate the support. Allow me to clarify that I do not officially work on bibliometric analysis; it's something I do outside my working hours. You could maybe call it a strange hobby. I feel like I better add this because according to my contract I spend 100% of my time on a different project, and on occasion one of our admin people tells me I work too much. So, it's not work ;)

    ReplyDelete
  78. Dr Fleig,
    Come on...
    Sabine wrote in her post: "I must add a warning here. We do not use the same data set as Strumia and Torre. They use data from inspire, we use data from the arXiv. This means our data set does not reach back in time as far, and it includes disciplines besides high energy physics. So the absolute numbers are not directly comparable."

    Please tell us why you think it's not enough. To me, it seems more cautious than the average theory paper containing its fair share of unsubstantiated claims.
    If others misrepresent her claims, surely it's not her fault, isn't it?

    Anyway, the whole Strumia affair looks to me like a contest of gross misrepresentation and bad faith, on both sides, with few exceptions.

    ReplyDelete
  79. As other have noticed there are considerable differences in different fields on arXiv. It has also been notived that, despite your comments or footnotes, many have misunderstood your plots as proving that Strumia’s analysis is flawed - e.g. on Wikipedia and twitter. In the interest of understanding what is right and wrong to conclude from this it would obviously be interesting to either use the same database, or perhaps simpler to just restrict your use of arXiv papers to those found in hep-X. In fact it could be interesting to run the analysis for different subgroups on arXiv to see how the curve changes in fields with overrepresentation of either men or women. Would this be difficult to do for you or your student?

    ReplyDelete
  80. Hans,

    No, not difficult, but it will take some time. We are looking into it.

    ReplyDelete
  81. In response to Jim V and opamanfred:

    Yes, Dr. Hossenfelder mentions caveats. Again, I am not saying that her post is a flat-out lie. But comparing arXiv data with inspire data is just plain nonsense, as every insider knows. Of course the situation in medicine and biology is vastly different from the situation in physics and such a comparison mixing different fields in the base data has nothing to do with Strumia's point.

    And, as you can see from what many people take from her comparison, she IS being misunderstood, and she makes no effort to correct that. Even the people who set up Strumia's Wiki page think she has made a claim which she herself denies.

    @Dr. Hossenfelder: Goodness, of course I am not making you responsible for what others write. But if someone wrote that about ME, on Wikipedia, I would be quite keen on making sure that it is correct - or I would make sure that it is removed.

    ReplyDelete
  82. @ Hans Fynbo,
    quite right, good that you make the same point as I have already.

    @ Dr. Hossenfelder: Interesting that you are looking into a distinction of fields in the arXiv data. Why didn't you do so right away?
    I'd be very curious to learn about the outcome. Want a bet?

    ReplyDelete
  83. Dr Fleig,

    We didn't look into the distinction right away because I was traveling, giving several public lectures, indeed I was officially on vacation, and my student was on vacation too. Besides, as I mentioned, I am not actually working on this. I did this voluntarily, literally while on a plane, as a public service. Your complaints are nauseous and reflect very badly on you. You clearly don't like I even had something to say about this. You clearly just want me to go away and shut up. Well, bad news, I will not.

    I don't have the time to run after people who want to misunderstand what I write.

    The arxiv data have no significant differences in citations when restricted to hep-th, or hep-ph, or hep-ex. You'd have lost that bet. And it wasn't hard to guess because the arXiv is dominated by papers in those disciplines. If there was a significant difference in those areas, we'd have seen this in the full data.

    "comparing arXiv data with inspire data is just plain nonsense"

    We didn't compare arXiv data with inspire data. Can you not even read graphs? Strumia claims the presence of an effect that, if it exists, should be both in the inspire and arxiv data. It is not. Please give me a reason why Inspire data should have a gender gap and the arXiv not.

    ReplyDelete
  84. It might help to have a look at the arxive stats to understand how biased inspire vs arxive is: https://arxiv.org/help/stats/2017_by_area/index

    I do not see any significant contribution of biology, the largest contributions in recent years are from condensed matter, astro, hep, computer science.

    Kaylo

    ReplyDelete
  85. PS. I do not see medicine and bio seems to be about 1.5%. Again, this is recent years, cumulative is much less, given hep was 100% of all submissions in the 90s.

    Kaylo

    ReplyDelete
  86. Kaylo,

    The arXiv covers more areas than does inspire, but most of these areas have a similarly low number of women as does hep. If there really is an effect as large as Strumia claims, it should show up in the arXiv data too.

    ReplyDelete
  87. Sabine --> exactly the point I was trying to make!!!

    ReplyDelete
  88. Dr. Hossenfelder,

    you wanted to say something about this because you seem to have a political agenda. Then you create a post that fools many people into believing you have debunked one of Strumia's claims. You yourself realized that that is not the case, after receiving criticism. You didn't make an effort to correct for that. Now you say that restricting to hep-th would give a graph that contradicts Strumia's, too. So please show that graph. And probably it would be fair to try and publish your findings (like Strumia did) to prove that you are not just preaching to Hossenfelder's fan-club.

    I am quite aware that gender bibliometrics is not your professional field, and neither is it Strumia's. But if you want to make a point here you got to be more rigorous. Another problem may well be that the sets of publications on Inspire and arXiv do not sufficiently overlap, even if physics would be dominant on arXiv. Means that many (probably most) people do not double post their papers (I never do). EITHER they submit to Inspire OR to arXiv. Again, this will ruin any valid comparison.

    ReplyDelete
  89. Dr Fleig,

    You assume I have a political agenda. I do not know which. Do you even realize that I would benefit if what Strumia claims is correct?

    You come here to complain about what I wrote even though I was very clear spelling out exactly what we have done and what not. Are you sure that it's not you who is having a political agenda, that agenda being that you want me to shut up?

    I will publish what I like and when I like it, and if you don't like that then I suggest you do the analysis yourself. If anything then your comments encourage me to not share any preliminary information with anyone.

    "many (probably most) people do not double post their papers (I never do). EITHER they submit to Inspire OR to arXiv. Again, this will ruin any valid comparison

    What the heck are you talking about?

    ReplyDelete
  90. Dr Fleig,

    Thinking about this some more, your recent comment shows that you do not have the faintest clue what databases we are even talking about. You come here to complain that the analysis we did supposedly doesn't make sense and that we should instead in the blink of an eye have done something else, and you have no idea what we are even talking about? Wow. Really you should do some self-assessment about your behavior. It's highly unprofessional.

    ReplyDelete
  91. I had never heard of INSPIRE until this post. (INSPIRE never heard of me either, of course.) From their web page I see this:

    INSPIRE Project overview

    CERN, DESY, Fermilab and SLAC have built the next-generation High Energy Physics (HEP) information system, INSPIRE. It combines the successful SPIRES database content, curated at DESY, Fermilab and SLAC, with the Invenio digital library technology developed at CERN. INSPIRE is run by a collaboration of CERN, DESY, Fermilab, IHEP, and SLAC, and interacts closely with HEP publishers, arXiv.org, NASA-ADS, PDG, HEPDATA and other information resources.

    INSPIRE features include

    . Searchable fulltext for all arXiv content
    . Figures and searchable figure captions extracted from recent arXiv articles
    . Fulltext search and snippet display for recent arXiv articles and material from some publishers

    It would seem to me from this and data from previous commenters that the claim by Dr. Flieg that there should be a significant difference between the characteristics of authors in the INSPIRE and arXiv databases is unfounded and could only be based on cognitive bias.

    In my estimation, the one who has done the most to discredit Dr. Strumia's analysis on this site is Dr. Fleig, by association.

    ReplyDelete
  92. JimV,

    Thanks, I appreciate the support and also the information. Let me clarify though that the arXiv database does differ in important ways from inspire, as I have mentioned: The arXiv hosts (almost) no papers dated prior to 1993 (some people have uploaded older papers, but it's so rare as to not have any statistical relevance) and the arXiv also covers disciplines not on inspire. Besides this, the arXiv gets its citation- and reference-information from inspire, whereas we take ours from paperscape (as explained in our paper).

    However, I don't think these are relevant differences for the subject under discussion. The disciplines represented on the arXiv have similar female/male ratios as the ones on inspire, and besides this the data is dominated by the overlap-disciplines for historical reasons. For this reason the trend that Strumia finds should be in the arXiv data too simply because the overlap is so large.

    Inspire has the benefit that it has some (limited) biographical information which the arXiv does not have, which possibly makes the difference. In any case, I'll tell you more about this once we have a better idea what's going on.

    ReplyDelete
  93. "I'll tell you more about this once we have a better idea what's going on."

    As I said, and as you realize yourself, everything that was going on here was unprofessional, including my remarks (I am not a bibliometrics expert). My main point of criticism, however, was that your post created beliefs about claims in the outside world, BEFORE you knew what was going on. That is what you should have avoided.

    I rest my case.

    ReplyDelete
  94. I can't stop other people from believing what they want to believe.

    ReplyDelete
  95. Sean S. You said: "To test the explanation, we’d need to look only at the first one or two papers of all authors and see how frequently those are cited, especially before later papers are produced. It seems likely that even the best person’s first few papers are rarely cited, or may only be cited after they have produced a larger body of work.

    A simple explanation is that people take time to develop their careers; those who drop out early never get to the point where they make the contribution that they could have if they’d have continued.

    Unknown’s explanation may be circular because it assumes “dropping-out” happens because dropouts are not doing well, and lack of citation of early papers proves they were not doing well.

    If women are dropping-out for reasons having nothing to do with their abilities, and if early papers are typically less likely to be cited, then Unknown’s explanation fails.

    If the early papers of “remainders” are cited more often than those of “drop outs” early in their careers, then this claim might be credible; but if early papers are typically not cited frequently, then the explanation fails."


    It made me wonder, instead of evaluating the stay-ins, whether it would make sense to evaluate citability of the drop-outs by gender.

    While no difference would not give us more information, which is most likely due to the assumption of early career work being of lower quality than later potential work. If women dropouts were cited more, this would be an indicator that more talented women left the field than men of similar capability. A control for this could be comparing by gender the citations those academics who had persisted past your 5 yr mark, but only comparing their early career work .

    Both groups being the same would suggest that the general assumption of early career work being of lower quality holds, even for talented remainders. A difference in citations may suggest something different...

    (a moderator may be how much researchers cite their own work, or study a specified topic instead of branching out into several topics. A continued development of one's very specific question generally serves to promote one's earlier work in the same area. Both traits of promoting other people's work and having more variety of topics would reduce one's efforts in promoting own work. Traits that female researchers may bring to the table.)

    Sorry if I was mistaken when reading your post if this was already conducted!

    ---
    On another note, I'd like to mention this interesting comment by an extraordinarily open male friend, who said that men feel more comfortable working with other men than with women. Some folks have mentioned implicit bias here. The fact is that most of us are not aware of why we feel certain ways, so we tend to rationalize it. In Strumia's case, it seems his rationalization of his discomfort over-powered his prudence. But I do think this discomfort (difference in interaction style and sexual attraction) is a root cause that goes beyond buck-up-and-accept-women. Modern co-ed work conditions have only been ongoing since the mid-1900s. Beyond aging into an almost sexless form, openly addressing these discomforts and expected norms, with the willingness to work together (such as considering dress codes...etc.), may go a long way to coming to a better understanding of how we can more effectively work together?

    ReplyDelete
  96. Grace En-Tien Chang;

    It made me wonder, instead of evaluating the stay-ins, whether it would make sense to evaluate citability of the drop-outs by gender. ... If women dropouts were cited more, [than similar men] this would be an indicator that more talented women left the field than men of similar capability. A control for this could be comparing by gender the citations those academics who had persisted past your 5 yr mark, but only comparing their early career work .

    That seems like a reasonable test. I am not aware of any such study being done yet.

    men feel more comfortable working with other men than with women.

    Human beings in general are more comfortable with the familiar; and our culture does tend to segregate children’s activities by gender. Those of us men who were forced to work with/around/for women generally find working with women comfortable.

    You are correct; “openly addressing these discomforts and expected norms, with the willingness to work together” is necessary.

    sean s.

    ReplyDelete
  97. Nothing you show here actually disputes Stumia's basic thesis, that the expected results are not what you expect from the mainstream hypothesis. What you've shown is that men and women who don't drop out appear to be treated equally, and the lower representation of women seems to be entirely explained by them dropping out. So unless you can show the dropout rate is due to discrimination you've basically confirmed Stumia's thesis.

    ReplyDelete
  98. Wait, so you used a dataset that includes BIOLOGY and MEDICINE to refute a sample set about high energy physics? Yes you did include that it included sciences outside high energy physics. You did not point out that some of those have a MAJORITY of female Bachelor's degree. If you're not being dishonest you're communicating REALLY badly and allowing people to believe that the datasets would be similar regarding male vs. female patterns when they really wouldn't.

    ReplyDelete
  99. Michael,

    Your comments are (a) internally incontradictory (first you claim we confirmed Strumia's result, then you claim we cannot confirm it) (b) make false claims (I did not offer an interpretation of the finding -- you did) and (c) wrong (the fraction of biology papers is so small it has an absolutely negligible effect).

    ReplyDelete
  100. Of course if you cull the bad results you can improve the track...

    ReplyDelete
  101. @Sabine Hossenfelder a) There is no contradiction between saying that you cannot confirm something with a dataset and that you can refute it. For instance suppose that you compared violent crime rates at households where the police knew that there was a gun in the house in the last year with other households. The police might well know there is a gun there because of criminal acts by the householders which either involve a gun or cause a search to happen. So the dataset couldn't show that having a gun in the house makes violent crime more likely. But the police knowing there's a gun in the house were negatively correlated with violent crime that's good evidence.

    b) You said that the effect goes away when you do X that's a conclusion because it assumes that the effect didn't go away when you changed the dataset. If you're not going to come to a conclusion, at least tentatively, why bother?

    c) Ok so you're clearly communicating badly, not necessarily dishonestely. There are medical and biology papers in you dataset. You are of course aware of the massively different gender balances between these fields and physics. So you why not simply state a summary of the percentage of papers in each field and there gender balances? BTW you're STILL communicating badly. You deny that BIOLOGY papers are not a significant part of the dataset, but I referred to biology and MEDICINE. So are you just too lazy to write two extra words or does this really shift the data?

    ReplyDelete
  102. Michael,

    I never said anything about "refuting" - you did. What we did is to look for the same effect in another data set. I showed you the results. And that's that. I am not "denying" anything. The paper share of the arXiv database is no secret, why don't you go and look it up yourself. Speak of "lazy". The last time I looked there wasn't such a thing as medicine on the arXiv. Do you even have the slightest idea what you are talking about?

    ReplyDelete
  103. @Sabine Hossenfelder You made a claim not me. The paper share of ArXiv database may not be a secret but it's something that you know that most people don't. Why the hell should I have to look it up, taking time out of my day when I'm talking to someone who DEFINITELY KNOWS IT. It's not my job to look up possible reasons why your claims might be invalid. I've already educated you on the difference between not being able to confirm something from a dataset and being able to disprove it. That should have been something you learned years ago. Work on your communication skills. If you're not prepared to simply post numbers that YOU definitely know, like the percentage of biology and medical papers on arXiv, why bother posting anything? All you're doing is playing a game where people say "That's invalid because of this" and you say "No it isn't because of this thing that I didn't bother to tell you.". It's irritating. Stop it.

    ReplyDelete
  104. Hi Michael,

    please read through this thread carefully, as I wrote above:

    It might help to have a look at the arxive stats to understand how biased inspire vs arxive is: https://arxiv.org/help/stats/2017_by_area/index

    I do not see any significant contribution of biology, the largest contributions in recent years are from condensed matter, astro, hep, computer science.

    I do not see medicine and bio seems to be about 1.5%. Again, this is recent years, cumulative is much less, given hep was 100% of all submissions in the 90s.

    And as Sabine added, all disciplines that are significantly present in the arxive dataset have similar male-domination compared to HEP alone. This has been discussed above.

    Kaylo

    ReplyDelete
  105. And a P.S.:

    Why the hell should I have to look it up, taking time out of my day when I'm talking to someone who DEFINITELY KNOWS IT. It's not my job to look up possible reasons why your claims might be invalid.

    --> Sorry, but given that biology is 1.5% (just in recent years, so <1% for the total data set) and medicine is ZERO, there is no need to point this out explicitly necessarily... This argumentation would require also making statements on the amount of e.g. papers on History or Linguistics which are equally zero.

    And as said, the values of zero and much below 1.5% have been discussed above.

    kaylo

    ReplyDelete
  106. Michael,

    I have expressed myself very clearly, it's just that you want to put things into my mouth I didn't say. You also mistakenly seem to think I am under some obligation to respond to your ill-informed and inappropriately aggressive comments. This is not so.

    "Why the hell should I have to look it up, taking time out of my day when I'm talking to someone who DEFINITELY KNOWS IT."

    You didn't ask a question, you simply made a wrong statement, which I corrected. Why the heck would you try to bother checking that your statements are actually correct before you make them? It could be useful to avoid making wrong statements.

    Besides, if you had bothered to read the earlier comments or at least have asked Google you could have prevented your confusion. The relevant link, which was already posted above and which is not hard to find is this:

    https://arxiv.org/help/stats/2017_by_area/index

    Now excuse me, I have better things to do.

    ReplyDelete