Referees: the ultimate scarce resource

Every paper published in an academic journal goes through "peer review". It is sent out to two or three external reviewers, who evaluate the manuscript for originality, sound research methods, and so on. Unfortunately the quantity of manuscripts submitted to journals outstrips the number of people willing to review them, and the system is in danger of breaking down.

Herb Emery of the University of Calgary just sent me this article by his biologist colleagues, Jeremy Fox and Owen Petchey, describing the problem:

The peer review system is breaking down and will soon be in crisis: increasing
numbers of submitted manuscripts mean that demand for reviews is outstripping supply. This is a classic “tragedy of the commons,” in which individuals have every incentive to exploit the “reviewer commons” by submitting manuscripts, but little or no incentive to contribute reviews. The result is a system increasingly dominated by “cheats” (individuals who submit papers without doing proportionate reviewing), with increasingly random and potentially biased results as more and more manuscripts are rejected without external review.

They consider (and reject) the often proposed solution of monetization: paying referees and demanding a submission fee from authors. This, they argue, is a barrier to young academics who may not have the up-front funds to pay for submissions.

Instead, they argue for a new form of currency, "Pubcred."

The majority of journals now use online systems to handle both submissions and reviews. These systems could be used to link number of reviews with number of submissions. We propose that authors “pay” for their submissions with credits, called PubCreds, “earned” by doing reviews. Submission of a manuscript costs three  PubCreds, while a completed peer review pays one PubCred. Every individual would have an account held in the central “PubCred Bank.” Their account would be  credited when they carry out a peer review, and debited when a manuscript is submitted. Individuals could view their account balance and transaction history on the PubCred Bank web site. We suggest that the PubCred Bank also log requests to review that have been declined, and the reason for declining (the reasons for this are explained below). Critically, submission of a manuscript to a journal would be possible only if an individual’s account balance contained sufficient PubCreds (we discuss this amount below).

The obvious draw-back to this system is that academics tend to write and submit articles in their youth and referee in their old age. What about young authors who have not yet earned enough publication credits?

There may occasionally be good reasons why, even collectively, the authors of a manuscript have not earned enough PubCreds to pay for a submission. Perhaps the sole author is a graduate student, or an academic who happens not to have been asked to review for a long time. We suggest that all authors have a limited “overdraft” facility on their PubCred accounts. We imagine that few if any academics will need to go into debt by more than 10–15 PubCreds, even at the beginning of their academic careers. A period of data collection prior to startup could determine whether this is correct.

One could also imagine a secondary market developing for Pubcreds, with those who referee more often than they submit "selling" their credits either for cash (if credits were made transferrable) or for co-authorship credit.

The liquidity issue is one potential problem with the PubCred system. The other issue is the quality of referees' reports. It's easy enough to write a report, it's hard to write a good one. Since, at present, there is no reward for refereeing other than the intrinsic satisfaction of knowing that one has done a good job, there is no reason to write anything other than a good quality report. Would creating monetary or other incentives to accept refereeing assignments lead to a decrease in the quality of refereeing? Fox and Petchey suggest "Reviewers providing late, superficial, sloppy, or inappropriate reviews should receive no PubCreds for doing so." Perhaps that would be enough to solve the problem.

Fox and Petchey's proposal is one of many designed to ease the problem of scarce referees. The Shakespeare Quarterly has just experimented with on-line peer review. It's a process familiar to everyone who reads this blog on a regular basis - post an idea, and have dozens of people comment on it and scrutinize it. If the idea stands up under public scrutiny, it's worth publishing in an academic journal (though, one could ask, what is the point of doing so?).

Another idea is to solve the problem by generating less research. Proposals along these lines typically go as follows: "Some academics are better researchers than others. In fact, academics in Useless Group should stop doing research and instead devote their energies entirely to teaching." Oddly enough, Useless Group is always someone else - colleagues at lower-ranked universities, older colleagues, junior colleagues, colleagues in other disciplines, and so on.

Finally, the need for referees could be reduced by redefining what is meant by research - perhaps blogging should count?

 

Comments

[Jack]

I think fundamentally the task of refereeing is not valued highly enough in the profession. If publishing in the AER is the highest goal one might achieve, refereeing for the AER should also be very highly regarded (but not equivalent, by all means). This suggests a gradual shift in the academic culture to reward good refereeing. (Although it's hard to identify and evaluate good refereeing, since it is double-blind, in general better journals ask better reviewers--though it's hard to tell how much is because of research expertise and how much is due to refereeing quality.)

Refereeing is already indirectly rewarded in the profession because it is a determinant of journal associate and co-editorships, which are highly regarded and rewarded. But again, these rewards are also determined by research productivity, and they may be too uncertain and long-run to function as effective rewards for diligent reviewer work.

The credits system is a good idea, but one that could probably be gamed, so I'm not sure it's the best solution.

[Luis Enrique]

I think you have to start you career with a loan of pubcreds. I'm an early career researcher, and frankly if I had to review as many papers as I'm going to submit, the quality of the peer review process would suffer. Whereas if I'm lent 10 creds that I have to repay once I'm less clueless, that might work better. Mind you, lots of early careers crash and burn so they'd be lots of defaults!

[Andrew F]

Interesting idea. I think you'd need some way for young academics to earn credits other than refereeing. I'm not sure how loans could work. The problem is also that older referees who have built up a large stock of credits would have diminishing incentive to referee. It might even have perverse effects, like paying volunteers a nominal wage actually decreases their utility and could reduce how much their volunteer.

[Jeremy Fox]

Flattered that WCI has posted on PubCreds. As an ecologist and a co-author of the PubCreds idea, I certainly appreciate feedback from actual economists on what's fundamentally an economics idea.

Owen Petchey and I have a blog about PubCreds, where we address the feedback we receive: . The concerns Frances raises are all quite natural, and have been raised by others. In brief response (check out our blog if you want the long versions):

-Our reasons for rejecting a real money system include, but weren't limited to, the possibility that some authors might lack the funds to pay.

-In addition to allowing 'overdrafts', there are various other plausible ways one might address the problem of young authors lacking PubCreds, including but not limited to directing more requests to review towards young authors.

-I don't think PubCreds should be transferable, and would consider transferring PubCreds in exchange for co-authorship to be unethical. On our blog we discuss other ways in which the system might be gamed.

-I don't think there'd be issues with quality of reviews if editors can deny 'payment' for low-quality reviews. It would probably be a good idea (and straightforward) for journals to publish guidance as to what constitutes a good review. Indeed, I think the system would actually force an increase in the quality of reviews (and also their timeliness, if late reviews earned reduced or no payment).

-I agree with Jack's comment that while rewards for reviewing exist, they seem to be either little-valued by referees, or else too uncertain to greatly affect referee behavior. Surveys indicate that referees do like receiving the minimal rewards they receive (such as having their names published in lists of referees). But that of course doesn't mean that referees value those rewards enough to alter their behavior in order to obtain them.

A couple of other economically-oriented comments. First, there are strong analogies between PubCreds and the famous Capitol Hill Baby Sitting Co-op. Fortunately, there's reason to think that at least some of the problems the Co-op ran into, such as seasonal mismatches of supply and demand, wouldn't be an issue for PubCreds. Second, an economics journal, published by Berkeley Electronic Press, already runs a hybrid of a PubCred-type system and a real money system. Basically, when you submit you can either agree to review two papers for the journal in future, or else pay a real money fee (via credit card) to buy your way out of that obligation. If you agree to do the reviews but later back out, your credit card is charged with a fine. I'm curious what readers of this blog think of this hybrid system.

[Frances Woolley]

Thinking a bit further on PubCred v. monetary systems:

- an advantage of PubCred over monetary systems is that the transactions costs of international cash payments can be huge. Since I haven't been able to work out how to get a US tax number, whenever a US publisher offers me money to do refereeing I either get the paperwork side of it cleared first or ask for some kind of non-monetary payment (e.g. books, donation to a favourite charity in lieu of cash).

I think Fox and Petchey are a bit optimistic about (a) the competence and (b) the generosity of people running on-line manuscript submission software. Some journals know me as Frances Woolley (frances_woolley [at] carleton.ca) others as Frances R Woolley ([email protected]) some as Francis Woolley or Frances Wooley - and I have a fairly unusual name. Matching names is a non-trivial business. My experience of dealing with publishers does not fill me with a huge amount of optimism about their ability to sort this kind of problem out.

As for the generosity of people running on-line manuscript submission software in terms of their willingness to submit information to PubCred without imposing large charges for this service - yes, well.

Another concern I have is privacy and reputation effects. What does your PubCred balance signal about your academic reputation? Do you want that information to be public? Would this violate Canada's Freedom of Information and Privacy Act?

On the Berkeley system - one issue with tying refereeing obligations to particular submissions is that after a rejection - and the majority of papers are rejected - one hardly feels like writing warm and fuzzy referee reports on other people's papers. Incidentally BEP still haven't asked for the two reports that I owe them (I have refereed for them, but prior to submitting).

[Mike Moffatt]

"The obvious draw-back to this system is that academics tend to write and submit articles in their youth and referee in their old age. What about young authors who have not yet earned enough publication credits?"

This surprises me - my current ratio is 4:1 for reviews:submissions. Either I'm reviewing too much or not submitting enough!

[Frances Woolley]

Jeremy: "I don't think there'd be issues with quality of reviews if editors can deny 'payment' for low-quality reviews."

I personally don't find it that easy, as an editor, to say 'low quality review' or 'high quality review'. On the editorial software that I use, there is an option whereby we can rate the quality of a review. I never use it because the evaluation of reviews is a tricky and subjective business. The less good reviews that I get are often from people who don't really understand what is expected of them, or who have tried but failed to come to grips with a paper that they just don't have the expertise to handle. I wouldn't want to deny those people payment.

[Jon]

This strikes me as strange, the incentive to review is to preserve space for your own papers.

The real problem may be that the papers ( in economics ) are much much too long. They contain too much routine review material. Top science journals accept only a few pages at most; 5-7 pages is the norm at mid-tier journals with SIs being mostly off book and accessible for download only in either case.

[Jeremy Fox]

Regarding the problem of uniquely identifying authors and reviewers, I don't know that you have to, any more than the real money banking system does. All you need, I think, is to uniquely identify accounts. See http://www.ipetitions.com/petition/fix-peer-review/blog/2677

Whether PubCred balances should be publicly viewable is an interesting question. As someone who does a lot more reviewing than submitting, I'd be more than happy to have my balance publicly viewable (or make it viewable, if the system left the choice up to me)! But I can certainly see how others might feel differently, and fortunately the system should still work (or not work!) if PubCred balances aren't publicly viewable. I have no idea if there's be legal issues involved in making PubCred balances publicly viewable; perhaps naively, I'd be surprised if there were.

Regarding review quality, my own experience as an editor has been that reviews for which I wouldn't want to pay are a quite distinctive subset of all reviews. They're generally very short, superficial, and sloppy, and don't say anything that would be of any help to either an editor or an author. If the editor can refuse payment for such reviews, he ensures that all the reviews he receives will be helpful to him and to the authors, which I think is basically all you want (at least, it's all I want as an editor, or as an author).

The issues of who pays for the infrastructure and ongoing operations of the system, and getting the various journal owners to participate and share their data (a big collective action problem), are, I think, the biggest issues. All I can say is that it's early days, and that conversations with publishers and the leading scientific societies in our field are ongoing...

[Frances Woolley]

This just arrived in my inbox:

Nice post on the blog. I went to put up a post and then i realized I am not registered with the blog to post. If you think what I wrote is useful, then feel free to put it up. As a proud member of the “useless” group I was tempted to point out that the top journals in economics don’t risk editorial decisions to referees outside of the inner circle. I won’t say which journal I had this experience with, but I was told once by the associate editor that people like me were used to referee papers that the editors knew they would reject but they needed a report. The real submissions were refereed by those whose judgment they would trust. Early in my career I asked the editors of Exploration in Economic History and the Journal of Economic History to consider using me as a referee since I was trying to show that I had some stature in my field. Both editors made it clear that they rely on a select group for these tasks. I am happy to say that I have managed to referee for these journals but well after I needed the credit on my vita to get tenure or anything important.

[JW Mason]

"Fox and Petchey suggest "Reviewers providing late, superficial, sloppy, or inappropriate reviews should receive no PubCreds for doing so." Perhaps that would be enough to solve the problem."

But wait! who will review the quality of the reviews, and what's *their* incentive to do a good job of it? Clearly we need an additional system to properly incentivize the review-reviewers. Call them Pubcredcreds. But wait! what stops the review-reviewers from racking up pubcredcreds with sloppy review-reviews....

Seriously, what is it about economists that draws them to this fantasy of a world without intrinsic motivation?

[JW Mason]

(And of course, the whole reason we use outside reviews is precisely because the journal editors *don't have the expertise* to evaluate all submissions themselves. If they could reliably assess the quality of an outside review, there wouldn't be need for outside reviews in the first place.)

[Stephen Gordon]

Because clearly, intrinsic motivation isn't getting the job done. If it was, there wouldn't be a problem to try to understand and solve.

[Patrick]

"Canada's Freedom of Information and Privacy Act"

No. This applies to Gov't. There are other laws (e.g. Personal Information Protection and Electronic Documents Act) that apply to the private sector. IANAL, but I doubt that there would be any problem in this case.

[Frances Woolley]

JW Mason - "Seriously, what is it about economists that draws them to this fantasy of a world without intrinsic motivation?"

Just to clarify - Fox and Petchey are biologists.

[anon]

"They consider (and reject) the often proposed solution of monetization: paying referees and demanding a submission fee from authors. This, they argue, is a barrier to young academics who may not have the up-front funds to pay for submissions."

Aren't submission fees often covered by research grants? (Definitely true for the article processing fees that some journals charge for accepted articles.) Also, journals would probably waive the requirement in special cases, such as first-time authors, independent/non grant-funded researchers, etc. These authors would be very price-sensitive and have positive effects for the journal (subscriptions, prestige, citation rating etc.) so waiving their fees would make sense economically.

[Kevin milligan]

Isn't there a large incentive to try to make yourself look smart to the journal editor? That's what motivates me to do a good job most times--or at least the inverse, ie doing a good enough job to not look like an idiot to the editor.

[Stephen Gordon]

Yes, that's usually my motivation. But of course, it only works for people whom I'm trying to impress. As time goes on, that list gets shorter.

A related issue is getting people to participate on SSHRC adjudication committees. It's a lot of work, and for no greater reward than making sure good research gets funded.

[Frances Woolley]

Kevin - "Isn't there a large incentive to try to make yourself look smart to the journal editor?"

In the existing system, there's little incentive to say yes and then do a poor job. But if one had to referee in order to submit, wouldn't it be tempting to say yes and then do a so-so job, especially for journals outside your immediate research area? Note that Fox and Petchey are proposing credits that are transferable within journals, so refereeing for one journal would buy you submission credits at another - I do a lousy review for Journal of Development Economics (which I'll never submit to) and then use my credit to submit to CJE.

Anon - "Aren't submission fees often covered by research grants?" - yes, for junior researchers, no for graduate students.

"These authors would be very price-sensitive and have positive effects for the journal" - not obvious. Hardly anyone buys individual subscriptions any more, and prestige and citations are driven more by already established scholars. So a classic citation-increasing strategy employed by journals is to commission state of the art pieces by well known researchers, because survey pieces are frequently cited (as in "for a review of recent studies on labour supply elasticities, see ____")

A major problem with monetization is that established academics have a high value of time - a good report might take 3 or 5 hours. A payment of $50, thought of in terms of an hourly wage rate, is insulting. I would think $100 would be the minimal kind of payment you'd need to induce any kind of behavioural response.

[Patrick]

"$50 ... insulting"

I can see it not being enough to induce anyone to do extra work but supposing refereeing counts as part of the regular duties of an established academic, if the choice is to do X for base wage or to do refereeing for base wage + $50, then why would you leave money on the table? What else could you be doing that pays more?

[Frances Woolley]

Patrick: "What else could you be doing that pays more?"

Just to clarify - I do a lot of refereeing - if PubCred was in place I'd have a large positive balance. But I do it because I enjoy it - it's a great way to keep up with what people are doing and, as Kevin Milligan said earlier, to build good relationships with professional colleagues. But $50 wouldn't be enough to motivate me to do a report that I wouldn't do otherwise. Because I could be doing lots of things that pay more.

The number one thing I could be doing that pays more is my own research - which, of course, makes the world a better place ;-) O.k., well, it brings me professional prestige (a pay-off I care about), and is fun. Or I could be blogging - there's a non-zero probability Paul Krugman will read something I write on this blog, and that's not true of anything else I write.

But just thinking in crass monetary terms...

$50 for five hours of refereeing is about minimum wage, so I could earn that working in retail.

What pays more? Consulting ($250 or possibly much more for similar amounts of time effort). Cleaning my own house instead of hiring a cleaning person (saves $100 after tax = say $160 before tax). Painting my own porch instead of hiring a painter (for comparable amounts of time, probably saves $200 after tax = say $320 before tax).

In December I'm stepping down as co-editor of Review of Economics of the Household, as it's solely because it is so stressful and difficult to find people to do referee reports - you write and people don't respond, or they say no, and so you ask more people - do you write to your friends and take advantage of their good will or write to strangers and hope to catch them on a good day or...? If it wasn't for that, I'd really enjoy editing the journal.

[marcel]

The version of the Lucas Critique applied to money targeting was....damn! Mental blank. British guy. But it was more empirical than theoretical.

I think you mean Goodhart's Law.

(I tried posting this several days after the original blog post, and it wouldn't take. Do you close comments?)

[marcel]

I should have mentioned, the previous comment was for a Nick Rowe post from a few days back, but I got a pop-up warning message that it wasn't allowed, I suspect because I only now got around to seeing his post, and commenting on it.

[Nick Rowe]

edast: Yep! That's the one! Goodhart's Law. (I could even picture his face, just blanked on the name).

I didn't close comments.

Sometimes it won't take my comments either. i have to log out, then log back in again.

[Patrick]

Frances, clearly it doesn't make sense at the margin but what if it's something you have to do anyway? So the way to think about it is not 50 / # hrs it's 50 + base wage. Would the outlook change if it was cosidered in determining your merit increment (sorry for the UofA terminology. Not sure what you call it at Carleton)?

[Frances Woolley]

Patrick -

I've probably been overstating my case. There probably are a number of people out there who would say yes at a wage of $50 and no at a price of zero. A lot of journals give referees a free subscription and they wouldn't do that if incentives didn't matter. Certainly as an editor I'd really like to be able to say "if you do this report we will offer you ____" even if ____ was a $10 gift certificate at Starbucks.

On the base salary issue: I don't know if I want to make too big a deal of this because someone other than you might be reading, but - provided I do my required teaching and administrative duties - I get my base pay whether I spend my marginal time doing research or going canoeing. So the marginal calculation is: which do I prefer $50+doing a referee report OR the best alternative use of my time. In fact the lower my base pay, the more likely I am to do the report for cash, because I'd need the money more.

If it counted for merit pay that would change things. Carleton - like a lot of universities here out east (from the U of A perspective) - has no merit pay.

It would also make a difference if I was the sole breadwinner responsible for the financial support of the family - and had a partner at home to take care of the kids. Actually, when publishers come and say to me "we're looking for someone to do X" when X is low to medium prestige job for low to medium pay I always say "o.k., we need to think of someone who has kids and a partner who doesn't have a paid job..."

But the interesting thing is this - I don't think I've ever been offered $50 to, say, review a manuscript or act as an external examiner or... It's always either at least $100 or nothing.

[Richard Morrisroe]

I would argue that this policy (especially the monetary incentives) might have a perverse effect. I'm sure many of you are familiar with the notion of intrinsic motivation versus extrinsic motivation. For those that aren't, essentially the research suggests that if you reward people for doing something they enjoy, then they actually do it less.

That being said, the fact that there are not enough reviewers may mean that most people don't enjoy doing reviews, so the previous may not apply.

As a graduate student with almost four papers done and heading for submission soon, I would love this system. That may be because sloppy research in my field (psychology) really annoys me, and if i could play a part in correcting some of the more egregious errors before they got printed, i would be glad to do so.

I think its an interesting idea, but would definitely want to see it piloted in small areas of each discipline before introducing it across the board.

On the notion of the lack of reviewers, this may have something to do with the current job market for academics. You have many grad student all submitting papers, and most of these students will not end up as researchers, so will never get a chance to contribute reviews back to the commons. This problem may be a sign that we need more researchers who stay in the field long term. That being said, given that i am a grad student who wants to work in academia, i am probably horribly biased.