Friday, August 21, 2009

Anonymity in Science

I recently read a very provocative essay in Inside Higher ED

To put his point of view into perspective, I think it is useful to know where the author comes from, so here is some info from his website

Jeffrey R. Di Leo is Dean of the School of Arts & Sciences, and Associate Professor of English and Philosophy at the University of Houston-Victoria. He is also president of the Southern Comparative Literature Association.
Professor Di Leo has a dual Ph.D. in Philosophy and Comparative Literature from Indiana University. His teaching and research interests include ethics, contemporary innovative literature, classical American philosophy, aesthetics, philosophy of language, and literary and cultural theory.


Needless to say, I read his essay with the eyes of a physicist and found plenty to disagree. But

First some words on anonymity in general.

In an earlier post, I commented on anonymity in the blogosphere. Here too, I want to distinguish between anonymity and pseudonymity. The former is when the source of information is completely unknown, the latter is when the information is assigned to a name - a pseudonym - that is not connected to an actual identity but serves to address the person and also makes it possible to recognize them later. Most commenters on this blog are psydonymous rather than anonymous in that they have chosen a nickname and return with that same name.

I strongly discourage posting as "Anonymous" because as soon as there's two anonymousses in a comment section it becomes impossible to lead a decent conversation. In the long run, a pseudonym also allows me to learn what your background is and thus what reply to your questions might be useful. This saves all of us a lot of time and reduces misunderstandings.

Now back to Di Leo's essay against anonymity

Di Leo has a problem with anonymity of the peer review process. His main point is that "Anonymity is Anti-Dialogue," "Anonymous propositions are fundamentally monological, not dialogical," and that critical dialogue is the essence of progress in science. While this is correct, peer review is not actually anonymous but pseudonymous since the author can typically correspond with "reviewer 1" and "reviewer 2," mediated by the editor. Leaving that aside, Di Leo dislikes anonymity in general, and the one-sided anonymity of peer review in particular, since usually the reviewers know the identity of the author, but not vice versa.

Di Leo regards double-blind reviews in which the reviewer doesn't know the author's identity (which he calls "totally anonymous") as "not as problematic" as the standard partially anonymous peer review. However, as I said several times before, double-blind reviews are practically unfeasible in the 21st century. Not only because in an increasing number of fields papers can be found on pre-print servers long before they got published, but also because in specialized disciplines it isn't hard to figure out who wrote a paper on what topic. People have a distinct writing style (or absence thereof), and it is often not much of a secret what they have been working on recently. Let's thus forget about that option.

Di Leo then goes on to explain that "anonymous assessment is antithetical to the very idea of the academy," that "academia has created a culture and ethics of uncritical consent and has hidden it behind the cloak of collegiality," and further bold assessments about the status of academia. He writes
"The common rationale for academic anonymity is quite clear: if one were required to accompany one's assessment with one's true identity, one would not speak the truth."


So let me comment on this.

First, science is done by humans. We know that humans are not perfectly rational, and that the human perception and opinion making process is easily biased. Disregarding such weaknesses leads to disasters as you can see for example in the ailing status of our financial systems. You can talk about the "idea of the academy" all you like, we don't live in the world of ideas. Reality isn't ideal, and the reason why utopias of all kind fail is that they envision an unrealistic human behavior. The academic system shouldn't be build upon somebody's ideal for scientific dialogue but upon reality. If we have good reason to think that some reviewers might feel the need to pay attention to the author's status and their own relation to it if their identity was revealed, then we can bemoan they feel this need but denying its existence isn't helpful.

Second, I completely agree that critical dialogue is essential to science, that anonymity makes it impossible, and that pseudonymity hinders it. However, I don't think peer review is the place to lead that dialogue. Peer review is the place to decide what fulfills the requirements of being published in a certain journal as a certain type of article. More often than not, the dialogue is lead through these publications and not in the review process.

Third, nobody forbids the reviewer to contact the author of a paper and reveal their identity should they wish so; I actually know people who do that frequently. It is thus factually wrong to say the peer review process "prohibits or prevents dialogue" as Di Leo states. One main reason why many reviewers do not engage in more dialogue with the author is not that they are afraid of it, but simply that they don't have the time. There are weeks when I receive 3 or 4 requests for referee reports. I can't sensibly lead an extended conversation with all of these authors, otherwise I wouldn't be doing anything else. I read the paper, write my report, and that's it. Some of the authors I know anyway, and they know my opinion. If you manage to reduce the number of written papers by a factor 100 or so we can talk about extended dialogue.

Fourth, the above statement that reviewers would not speak the truth if their identity was revealed and thus something must be wrong with the community is only partly correct. I strongly doubt any serious scientist would deliberately omit pointing out factual errors in a paper, whether anonymous or not. But that is only part of the review process. There are many journals for which factual correctness is a necessary requirement but far from being sufficient for publication. What also matters is whether a paper is appropriate for a journal, and how interesting it is. And that is, like it or not, to a large extend a personal assessment. It might very well be you find a paper more interesting if it was written by a Nobel Prize winner and that isn't even irrational. It is this part of the assessment that might make the reviewer feel uneasy if it wasn't anonymous because it is socially and politically very involved. Sure, scientists shouldn't take it personally if some of their colleagues don't find their work tremendously exciting, but some do.

Di Leo also briefly comments on blogs and scientific debate
"If one, for example, posts on his or her blog a statement concerning one’s belief in gremlins, one is not obligated to respond to persons who disagree with this statement. However, in the academy, students, faculty, and administration are expected to answer to questions about their opinions."

While he means to point out the difference between blogs and scientific debates, he actually points out what's different between science and belief. You are not obligated to respond to persons who disagree with your believes, because you can believe whatever you want, may that be gremlins or the flying spaghetti monster. But that's not science. Likewise it's not science to ignore facts relevant to your research that have been brought into your attention, but it shouldn't matter by whom or where.

Where and from whom you obtained information might however affect how much attention you'll pay to it to begin with, and that is another reason against unnecessary use of anonymity. We live in the information age. Today's problem isn't obtaining information but filtering it, and the source of information is a very widely used and very effective filter.

Bottomline

Anonymity has its place in science, but it should be used sparsely and only if absolutely necessary. Anonymity has the disadvantage of making dialogue more complicated, I agree with Di Leo on that. But it has the advantage of avoiding unnecessary social and political baggage. I thus think anonymous peer review will remain essential to scientific publishing.

I would welcome however if the communication in the peer review process wasn't so cumbersome. It is feasible for example that the author and the reviewer lead a discussion without each exchange being mediated by the editor, eg in some interface similar to a chat. After some weeks, the reviewer could summarize his impression from the exchange. This would allow it to easily clarify some general questions or possible misunderstandings.

On the general theme, the problems Di Leo addresses are at the core caused by the four pressures that I have discussed here. Financial pressure, time pressure, peer pressure, and public pressure all skew scientists' opinions and make it necessary for them to pay attention to things other than their true scientific assesment. The solution is thus to reduce these pressures as far as possible.


29 comments:

Plato said...

Let's take Pi for instance and Howard Burton's biography of Lazardis's push to develop such a place as, "academy like?:)"

Mike's excited and feels something is on the horizon and so, how would he go about providing the space for dialogue and sustainability for future innovation? Hire Howard or, some other notion which predates Howard's existence?

Not all of us can attend and we can only gain the tail end by blogging journalists like yourself Bee whose interest is about science.

So the format of papers written and discussed are dialogued in presentation. Whose name is who, as to importance, or rather, what is remembered as to the finer sticking points that such and such a person by name made it?

More then I think toward the conceptual understanding that will be remembered, and not the name of the person who made it "one step further?"

Does this provide for ego 's "to dominate" rather then ideas exchanged freely and fluid.

Selfless then. You see.

The creative format is and exchange is very important and not just in the creation of Plato's story like, but analytical too in Aristotelean debate and record keeping, in rebuffing the position, by taking a different point of view.

Sets up an interactive relationship with the "outside world and progressive internalized movement toward apprehension of further concpets in relation, as well, to redux movement toward the next plateau.

Neil' said...
This comment has been removed by the author.
Neil' said...

Excellent and thoughtful post overall as I've come to expect, but I think Bee brushed off a major feature helpful to many writers. I maintain that double-anonymous reviewing is a good option for those of us who are controversial or nobodies, and want our ideas to be taken on their own merit. Let's not forget about that option. No matter how well one pleads for the advantages of knowing who someone is, or waxes optimistic on reviewers etc. not having prejudices - they do, as Bee admits. Hence, it is very important for writers to have the option of being unnamed. I can take my chances with someone guessing - I want at least to not hand them my name on a silver platter.

As for reviewers being anonymous, I have mixed feelings. I say, let them decide for themselves and the authors for themselves, respectively.

Zephir said...

If anonymity is wrong for science, I cannot understand, why double-blind peer-review is so highlighted by most of prestigious scientific journals.

Bee said...

Neil: I did not brush off the option of double-blind review, I said it is not feasible and explained why. If it was possible, I would be in favor of it, but please tell me how it's supposed to work in practice? Best,

B.

Bee said...

Plato: Unfortunately, it is more often the names that are remembered than the concepts. There was this guy Einstein, right. What was it again that he got the Nobel Prize for? It is another weakness of the human mind that the average person tends to remember people and their stories better than abstract concepts (and certainly they prefer to talk about the stories). Thousands of journalists have made excessive use of this, but it comes with the price (we discussed this earlier) that many aspects of the scientific process, in particular the importance of the community, remain misunderstood by the public. Instead the picture that prevails is that of the lonely genius (surfer in Maui or so, you know what I mean ;-)

Best,

B.

Bee said...

Zephir: Is it?

Jean-Paul Billon said...

Well Bee. At the end of 80's I made a first important discovery in the realm of propositional automated theorem proving, with valuable industrial benefits for digital circuit simulation. It was hard to get some attention to my work because as an almost 40 something old man with no academic history, people knowing that did not give any credibility to such a breakthrough, and it took some years to get them understand the discovery was genuine and indeed very simple and useful. I did another breakthrough in 1996 on first order logic with equality theorem proving. In both cases, anonymous peer review processes had helped me to get published and my work known and exploited. Note that the company paying me didn't believe that such an old searcher of them as I was could have done such a breakthrough and forbade me to go in any conference to defend my work, pushed also by a scientific adviser member of the French academy of sciences who did not like the trend of my work. I was happy that clever people in Germany started a whole carrier exploiting my results. So, I think that anonymous peer reviewing can do a lot for those who are "nobodies" but propose interesting new ideas.

Plato said...

Hi Bee,

Yes, what you say is true. If I refer back I see where I have used such quotes to reinforce perceptions in relation to Einstein's name.

The "conceptuality of geometry" that can break one free of Euclidean perceptions was a most freeing moment for me in that "previous history by names" were individuals who worked to solve the fifth postulate, now became an "aha Einstein moment" for me?:)

So I would say that to be "free from the shackles of our own prisons," means, that in realization, we have been changed by what is now produced from discovery and has been introduced as "scientifically new."

This can only be done if it resonates to the very core and unites all that came before it. Not the name?:) Kuhn's revolution comes to mind. It's what Kuhn embodied.

So now looking at the shape in the Arizona glass case are not just objects, but reveal a very close connection to that "very dynamical nature inside."

That surfer has created a very dynamical mandalic mapped view of all the interactions. I can't imagine the view he must have inside:)

Best,

Uncle Al said...

Communication must be attributable but not necessarily traceable. Science is brought forward more by mortuaries than by maternity wards. Autoritätsdusel ist der größte Feind der Wahrheit, Albert Einstein, 1901. After 1916 he denied quantum mechanics.

Ya gotta look. Vote Uncle Al a big juicy 10!

Phil Warnell said...
This comment has been removed by the author.
Phil Warnell said...

Hi Bee,

Certainly another interesting perspective on your continuing theme regarding the validity and place of anonymity. In reading your post I couldn’t help but equate the anonymous aspect of peer review with the long standing concepts of Western justice and the images that are associated with it. After all peer review as like justice is a process of judgement that entails repercussion. This anonymity aspect here to has long been a source of conflict captured in the images of Lady Justice as to ask if she should be blindfolded or not.

What’s often misunderstood by the blindfold is that it appears to serve to prevent justice, since after all how can she see the scales and after be certain who she is striking down with the sword, if the judgement so warrants. What many fail to recognize is the blindfold is actually symbolic of objectivism and not with anonymity as per say.

The other thing that should be remembered in such comparison is that one’s peers, although they can represent as being both defenders and accusers; it is nature itself that represents the lady with the scales. So in my way of thinking, anonymity is not relevant to the task or do your peers represent to be anything but the sword; which we all know like in the case of justice is only wrong as people being the ones that wield it by reason of their judgement and are wrong only when they act without her proxy, which is known as not seeing the truth.

Best,

Phil

Bee said...

Hi Jean-Paul,

Thanks for sharing your story. Just for clarification: the peer review process was single- or double-blind? Best,

B.

Bee said...

Hi Phil,

That is an interesting association. As to the question whether the judgement is made by Nature or by our peers, as I have said several times before, the final judgement is without doubt made by Nature. However, it might take a long time for that judgement to come in (especially if you can't convince anybody to test your ideas). In the meantime, it's the judgement of the community that decides which ideas survive sufficiently long, or are tested at all (and whether you have a job that allows you to work them out). This judgement is especially important in areas where experiments are hard, expensive, or both - thus, no surprise theoretical physics is one of the fields most sensitive to the issue of peer review. In the long run, if data remains sparse, the mere assessment of peers by peers can lead to a sociological backreaction and secondguessing of what others might find interesting, that can also lead to a quite unhealthy pre-selection of new ideas. Best,

B.

Neil' said...

Bee, re double-blind review: journals often offer it and have for years, but I'm sure few take advantage of it - maybe that's why it seems odd to you. AFAICT it works like this: The submitter S asks for DBR. The Editor E sends the paper with the name and affiliation removed, to the reviewer R1. (Forgive the cutesy, topical simulation of physics language.) R1 has to look at the paper on its merits, with no chance of conscious or subconscious bias from "Oh, that guy!" or "Where the heck is that place?" etc. S takes his or her chances with guesses but at least isn't handing ID on a platter.

R1 sends her comments back to E and S, as "comments from R1" etc. Then S can revise as needed etc. If R1, R2, ... and E concur on approval, then now maybe S has a chance to get accepted, that otherwise wouldn't happen.

And yes the bias is real. There was an academic sting study, I'll look, that resubmitted previously accepted papers but pretended they were from nobodies at backwater colleges etc. Many were rejected.

Zephir said...

Postulate: From general perspective (which I always reccomend to consider at the first place) anonymous collectivistic approach leads to the lost of personal motivation (which lead to the fall of communism, BTW) and it slows down generation of new ideas. While too individualistic approach fragmentizes science and it slows down acceptation of new ideas.

Theorem: In my opinion most effective approach it would be to keep peer-review as blind, as possible. BUT after publishing of article, it's peer-review should become available together with names of reviewers.

Zephir said...

/*.. with no chance of conscious or subconscious bias ...*/
There is always bias given by fact, if we choose reviewer, whose scope of interests overlaps with scope of interest of author, it becomes biased due the possible conflict of interest or existence of personal coalition.

If we separate the scope of interests, we increase risk of incompetence of reviewer. This risk is the more pronounced, the more science becomes specialized - which effectively means, above some critical density of information peer-review process isn't effective anymore.

Jean-Paul Billon said...

Hi Bee

The peer review process was double-binded of course. That's why I got published while being a nobody, and in spite of the adversity of some well known and influential individual trying to stop my work (and in fact trying to kill any work made by anyone in my country in the domain of truly automated theorem proving). Not only my identity was hidden to the reviewers, but also my nationality, and this helped my work to escape the attempts that were made to keep hidden any advances on truly automated theorem proving made in my country (the reason why is that the influential individual I referred to above wanted to replace the work done on automated theorem proving in various labs in my country by his own approach based on manual proving with an intuitionist logic verifier, and he eventually completely succeeded).

Pope Maledict XVI said...

A lot of these problems would be solved if editors wrote to referees saying, "Just tell us whether this thing is right or wrong. Don't worry about how important it is --- leave that to us." Apart from being more fair, this would [ideally] lead to more consistent standards, and stop the publication of crap papers by well-known people.

The real core of the problem is that referees for important journals wield tremendous power: the author's whole career could be in the balance. Asking human beings to avoid abusing such power when there is zero risk to them --- it's just asking for trouble. Better to demarcate their role very clearly and strictly: you are here to find mistakes, not to tell us your philosophy, and we don't care which other papers you think good or bad.

Plato said...

I believe the context Jeffrey R. Di Leo used reference to Plato's academy was in essence not quite correct in his assessment?

The traits I reveal in my previous two comments reveal a better understanding of Plato's views then what was revealed and used in context by reference to support "against anonymity." See: What's in a Name

Best,

Zephir said...

/*..you are here to find mistakes, not to tell us your philosophy, and we don't care which other papers you think good or bad...*/
Problem is, such finding is way easier, if it relies on background philosophy of referrer and or wider context of author, as expressed in his other articles. For example for relativist every notion of Aether is unacceptable, which resulted in excommunication of Aether concept from science for many years. Referrers should suffer by the same public control, like authors - which doesn't mean, peer-review process iself should be public from obvious reasons.

Neil' said...
This comment has been removed by the author.
Neil' said...

If we separate the scope of interests, we increase risk of incompetence of reviewer.
OK, "scope of interests" is good, but that is not separated by the name and affiliation/address of the submitter being withheld from the Reviewer. The Reviewer does not need that, and should not have it per the submitter's wishes if at all. It is not a valid part of the presentation itself, it encourages the ad hominem fallacy.

Plato said...

Hi Phil,

I liked your comparison to Lady Justice, blindfolded or not. You are closer too Plato's view as well in this relationship in terms of objectivity.

As I was scouring through for information on Welcher Weg, Pair production and Gamma ray photon, I came across something close too, that Lee spoke about in terms of Justice that I thought might be added for consideration.

Lee Smolin:What we are dealing with is a sociological phenomenon in the world of academic science. I do think that the ethics of science have been to some degree corrupted by the kind of groupthink explored in chapter 16, but not solely by the string-theory community. For one thing, it is the academic community writ large that makes the rules. In a court of law, a good lawyer will do anything within the law to advance the cause of his clients. We should expect that the leaders of a scientific field will likewise do everything within the unwritten rules of academia to advance their research program.

This is not to raise the issue of whether or not a research area should be considered according to basis of the argument of Lady Justice and opposing points of view in that court of law, but to point to the objectivity needed for or against, based on it's own merit in a "scientific discussion."

And thusly, it must remain in the Halls of Science to further debate this point, or to infuriate and contrive, by another "group think" according too, a design of it's own law and disseminate amongst the masses it's own brand of Justice?

So, let them "weight in again" while us "silent types" watch from the court room seats.:)

Best,

Phil Warnell said...

Hi Plato,

I’m glad that what I said made a little sense beyond simply myself. This objective aspect they share in common of course is rooted in doubt. However, if one compares the accused in the eyes of justice and the eyes of nature, the focus of the mandate differs here, for the accused in law must be proven to be guilty beyond a reasonable doubt, while the physicist’s theory must relieve all reasonable doubt to be considered true. In justice evidence must be primary supplied by one's accusers, where in science the scientist (proponent) is expected to provide it. In a sense, it is nature itself who is the accused and not the scientist, for they stand if anything as being the ones required relieving all reasonable doubt. It should be no wonder then to discover that Francais Bacon’s primary occupation was that in being a lawyer and he referred to nature as a lady. A lady he was infatuated with, as I suspect all good scientists must be:-)

“But if there be any man who, not content to rest in and use the knowledge which has already been discovered, aspires to penetrate further; to overcome, not an adversary in argument, but nature in action; to seek, not pretty and probable conjectures, but certain and demonstrable knowledge — I invite all such to join themselves, as true sons of knowledge, with me, that passing by the outer courts of nature, which numbers have trodden, we may find a way at length into her inner chambers.”

-Francis Bacon- Novum Organum (1620)


Best,

Phil

Plato said...

Hi Phil,

Well said.

As you know Socrates had a good ear for the truth, and as has been told, he wondered the streets seeing if such truth can issue forth from the commoner, as well as the legislator, as he knew that truth "did exist in each person" and that it was a matter of "hearing it."

Unfortunately for Socrates this truth was never heard as it was overlapped with common issues of the day?;)

Michael Shermer has a counter opinion in Scientific America Shakespeare Interrupted pg 30 in relation too, "who Shakespeare was," while I still contend, that it was Bacon indeed. Court room dram or not. As a lawyer Bacon and a politico, knew his position precarious and to this, his motivation to conceal?

I did not see Shakespeare's name in the anonymity list of "against Anonymity.":)

Best,

Kay zum Felde said...

Hi Bee,

that things with the four pressures is quite interesting. I didn't red this back when posted, since I was in hospital.

But I've found one result of one of these pressures. My paper is 'under consideration' but the journal looks for a referee for a while now. May be my paper is not in the 'mainstream' ?

Best

Kay

Bee said...

Hi Kay,

Possibly it's just bad luck. They might have contacted some referees that were traveling/sick/on leave, whatever. But yes, it can be a problem. That's why I don't think the idea of "open peer review," in which you just put your paper out and let people comment, is any good. The vast majority of papers likely would never get any review. (The majority if papers probably gets never even read.) Best,

B.

Anonymous said...

There's a new interesting programme about CERN and the big bang experiment. They have among others interviewed Otto Rössler!

Here's the link:
http://www.vbs.tv/watch/motherboard/...-and-big-bangs


"Black Holes And Big Bangs

VBS explores CERN, the world’s largest particle physics laboratory, to determine whether its 17 miles of tubing buried under the Franco-Swiss border will reveal the origin of mass in the universe – or generate an earth-devouring black hole. (We share a condescending chortle with the proton-smashing scientists about the concerns of the scientific illiterate)."