Worthwhile Canadian Initiative: Adverse selection and single-blind peer review

The American Economics Association has announced that, as of July 1, 2011, its journals will be moving from double-blind to single-blind peer review. The identity of a paper's author will now be revealed to any potential referees. The Association gives three reasons for its decision:

"Easy access to search engines increasingly limits the effectiveness of the double-blind process in maintaining anonymity".
It "increases the administrative cost of the journals"
It makes it "harder for referees to identify an author's potential conflicts of interest arising, for example, from consulting".

The first reason does not stand up under closer scrutiny. A survey of Worthwhile Canadian Initiative readers (results here) found that our readers overwhelmingly agreed with the statement "I can always find out who has written a paper by searching online." But they do not always carry out that search. I asked our readers: "When I'm asked to do a "blind" review on a paper, I search on-line to find out who the author actually is." The most frequent response was "sometimes", followed by "no" and then "yes."

A person asked to do a double-blind review can either do that review blind, judging the paper only on its intrinsic merits, or carry out a single-blind review, by doing some research to discover the author of the paper.

A person asked to do a single-blind review has only one option: to undertake the review in full knowledge of the author's identity. There is no free disposal of knowledge; identity, once revealed, cannot be unknown.

But some people may prefer to do double-blind reviews. A reviewer may not want their decision to be influenced by confirmation bias, the tendency to interpret evidence so that it fits our preconceptions, or other psychological quirks.

In this era of electronic editorial management, it would be easy enough to create a box for referees: "if you wish to know the identity of the author of this paper, click here." Surely, as economists, we believe that choice is good, and reducing people's options is a bad thing?

The second reason for the move to single-blind reviewing, that anonymizing papers creates administrative costs, is perhaps more persuasive. However given that the AEA currently holds unrestricted net assets of $17.9 million, and posted a $465 thousand operating surplus in 2010, it does not seem to me that cost considerations should be decisive.

The third rationale, that it allows referees to identify conflicts of interest such as consultancies, is also puzzling. How is a referee supposed to know about the author's consulting business? Surely that is a matter for the author to disclose to the editor. Now it might make sense to accompany the move to single-blind refereeing with a requirement that the referee state any relationship to the author (past or present colleague/student/teacher/co-author/friend), in order to identify potential author/referee conflicts of interest. Yet this disclosure would have to be accompanied by some guidelines as to what constitutes an unacceptably close author/referee relationship, so that conflicts were treated in a consistent fashion. And in small intellectual communities, eliminating anyone who has a potential conflict of interest with the author would shrink the pool of potential referees down to a small puddle.

If the stated reasons for moving to single-blind peer review do not stand up to close scrutiny, what could have motivated the change? I find it hard to believe that the change is a response to a groundswell of support for single-blind review - the majority of respondents to the WCI survey opposed the move to single-blind refereeing.

To understand the change, one most first understand who benefits from it. The WCI survey respondents identified two groups that stood to benefit the most from single-blind refereeing: authors at top-ranked institutions, and authors with well-established reputations.

A simple explanation of the move to single-blind refereeing is that the editors of the journals, who are mostly well-established academics at top-ranked schools, advocated single-blind review on the grounds that it would benefit themselves and their colleagues/students. I do not find that argument convincing on the grounds that it is hard work to edit a journal. People take on editing duties in part because it's the right thing to do, a way to give back to the profession. When people are acting altruistically in editing a journal, an explanation for their choices based entirely on selfishness does not ring true.

An alternative explanation is based on the idea of adverse selection.

Let's suppose that a journal editor's objectives are to maximize the impact of his or her publication, that is, the number of times that articles are cited and read, and the average quality of the articles published, and also to minimize the effort required to produce the journal.

In a world where more articles are published than anyone could possibly read, and it is costly to ascertain the quality of research, people use the author's reputation as a signal of what work is worth reading and citing.

Therefore, a journal wishing to maximize its impact will try to attract articles by high-reputation authors.

It might also be true that articles by established authors/authors at "top-ranked institutions" are better, on average, than other articles. In this case, a journal editor wanting to increase the quality of his or her journal will wish to attract articles from established authors, and (possibly) discourage submissions from obscure authors. Established authors tend to do good work, and extracting a few gems from the mass of articles submitted by obscure authors is costly.

Suppose also that authors are uncertain about the quality of their research. For stronger results, one could assume that authors tend to be overconfident. Authors know their own reputation and the reputation of their university. And suppose that authors believe (as Nick Rowe suggested in comments on an earlier post) that referees are rational Bayesian decision makers. That is, everyone believes that referees use information about the quality of the author to inform their recommendation on a manuscript.

Under these assumptions, authors at top ranked institutions/established authors will prefer to submit to journals with single-blind refereeing, and authors at lower ranked institutions/authors with no reputation or a poor one will prefer to submit to journals with double-blind refereeing.

So this is my preferred explanation of the continued popularity of single-blind review: double-blind review disproportionately attracts submissions from authors at low- or mid-ranked institutions, and authors with no prior track record. In other words, there's adverse selection. Journal editors adopt single-blind review in order to encourage more submissions from established authors.

If you are opposed to the AEA's decision to adopt single-blind refereeing, you can sign the petition here: http://www.petitiononline.com/AEA/petition.html.