Friday, February 26, 2016

"Rate your Supervisor" comes to High Energy Physics

A new website called the "HEP Postdoc Project" allows postdocs in high energy physics to rate their supervisors in categories like "friendliness," "expertise," and "accessibility."

I normally ignore emails that more or less explicitly ask me to advertise sites on my blog, but decided to make an exception for this one. It seems a hand-made project run by a small number of anonymous postdocs who want to help their fellows find good supervisors. And it's a community that I care much about.

While I appreciate the initiative, I have to admit being generally unenthusiastic about anonymous ratings on point scales. Having had the pleasure of reading though an estimated several thousand of recommendation letters, I have found that an assessment of skills is only useful if you know the person it comes from.

Much of this is cultural. A letter from a Russian prof that says this student isn't entirely bad at math might mean the student is up next for the Fields Medal. On the other hand, letters from North Americans tend to exclusively contain positive statements, and the way to read them is to search for qualities that were not listed.

But leaving aside the cultural stereotypes, more important are personal differences in the way people express themselves and use point scales, even if they are given a description for each rating (and that is missing on the website). We occasionally used 5 point rating scales in committees. You then notice quickly that some people tend to clump everyone in the middle-range, while others are more comfortable using the high and low scores. Then again others either give a high rating or refuse to have any opinion. To get a meaningful aggregate, you can't just take an average, you need to know roughly how each committee member uses the scale. (Which will require endless hours of butt-flattening meetings. Trust me, I'd be happy being done with clicking on a star scale.)

You could object that any type of online rating suffers from these problems and yet they seem to serve some purpose. That's right of course, so this isn't to say they're entirely useless. Thus I am sharing this link thinking it's better than nothing. And at the very least you can have some fun browsing through the list to see who got the lowest marks ;)

9 comments:

Phillip Helbig said...

While I understand where it's coming from, rounding to 1 or 0 decimal places, instead of a dozen or so, would look better. :-)


Susskind, Leonard (1)
Average

Degree of expertise in general physics 10
Degree of expertise in general mathematics 4
Degree of expertise in their field 8
Degree of contribution to their papers 9
Accessibility to collaborate and communicate with postdocs 9
Availability to interact and discuss 9
Friendliness 9

Really? I would give myself at least a 4 in maths, and I'm sure that Lenny is better than I am here.

Probably not a problem here, but note that in some scales, 1 is the best and 10 is the worst. (A confusion of this sort is probably the origin of the legend that Einstein was not a good pupil.)

Sabine Hossenfelder said...

Phillip,

I find the phrase "general mathematics" almost impossible to parse. What the heck is "general mathematics"? Are stochastic differential equations "general mathematics"? Field extensions? Morse theory? If you ask a random person on the street, they would probably rate me a tenner just for knowing what the exterior algebra is ;)

Yes, as I mentioned in my post, it's an unfortunate omission that the website doesn't clearly state what the number means - they don't even label the ends of the scale.

I have also heard that the rumor of Einstein being bad at math originated in the difference between the Swiss and German school grades. Best,

B.

John said...

There's a solution to this! Instead of just taking the mean of each person's score, use some statistical approach that takes into account the variation of each person's vote. For instance, the literature on item response theory seems apropos.

Uncle Al said...

" "friendliness," "expertise," and "accessibility."" When ISO 900x empowered the LHC to fabricate $21 million in precisely duplicated, exhaustively documented, defective magnet quench dump brazes, would that have been better with a little glitter and a smile?

"Best efforts will not substitute for knowledge" W. Edwards Deming. While everybody else is properly gathering at railheads to organize personnel and matériel, the winner is racing through the Ardennes. God save us from the congenitally inconsequential.

hepts said...

In my opinion, there are better ways to get the necessary information. It is very easy in HEP to find out who a senior person has worked with recently, and get in tough with those people. With the rumor mill, institute websites etc it is also easy to identify those who might have been less successful at working with their advisor.
On top of that, getting in touch with local postdocs might give you additional important information, like whether the potential advisor is currently negotiating for another job, planning a long sabbatical, if the funding of the group was increased or cut recently, whether the position were successful securing the next career step... all that is just an email or Skype chat away!

Also, use your network: Most likely at least one of the people who wrote a letter of reference for you knows your potential new advisor. As local postdocs, they might know someone who knows someone...
Check the advisors website. Do they list previous students and postdocs, and where they are now? What is the track record of the place in general? And many more things that can be done.

Jim said...

Dr. Arahony must be astonishingly transparent, that his abilities are known to 15 decimal places. Physical science meets social science....

naivetheorist said...

sabine:

the reason for the widely-held but erroneous belief that Einstein was bad at math in school is given here

http://www.nytimes.com/1984/02/14/science/einstein-revealed-as-brilliant-in-youth.html?pagewanted=all

but if you look at Einstein's professional publications, it is clear that he was in fact, not all that good at doing math. his greatness resides in his perseverance and his extraordinarily deep physical insight.

btw - happy blog anniversary.

richard

Maurice said...

Why exactly did u turn him down initially?! You ask for donations, don't you? Does he ask for something more in return than just that u continue your blog?

Sabine Hossenfelder said...

You posted this comment in the wrong thread. I was assuming it's a grant that, as most grants in academia, can only be spend on a very narrow number of purposes, most of which only cause the recipient more work. Thus my example with the conference.