How the quest for ratings distorts research

Imagine yourself in the position of a senior university administrator. Your university employs hundreds of academics, all of whom claim that their research is of vital, earth-shattering import. You have neither the time nor the expertise to evaluate the quality of the work done by individual faculty members. How do you sort out the good from the bad?

Counting publications doesn't work: quality matters.

Relying on the assessment of outside experts is imperfect: the evaluators are almost certainly well acquainted with the evaluatees, and in the small world of academic research what goes around comes around. It's risky to be too harsh on someone when in a year or two the relative positions of evaluator and evaluatee might be reversed. Plus calling in an outside expert is expensive.

So administrators and others are increasing relying on bibliometrics: the number of times an article is cited or referenced by other academics, the impact factor of the journal in which an article appears, and so on. 

Such measures have value - a work that is cited is probably also being read (o.k., skimmed) and likely contains ideas or research findings that are of interest to someone.  So to some extent bibliometrics measure quality.

Yet impact factors differ systematically across disciplines and between fields within disciplines.

Some disciplines have higher impact factors than others. For example, the highest impact economics journal, Journal of Economic Literature, comes in at 6.9 citations per article, while the top sociology and political science journals both come in at around 3.7  - in other words, an article published in 2008 was cited on average 3.7 times in 2009. (These numbers were taken directly from the Social Sciences Citation Index database, and include only citations in journals included in that data base - citations in books, working papers, or government documents don't count)

Some fields have higher impact factors than others. The highest impact field journal is currently the Journal of Financial Economics (4.0). By way of contrast, the top ranking labour journal, Journal of Human Resources comes in at 1.9, while the Journal of Public Economics scores a measly 1.2.

And some countries have higher impact factors than others. The American Economic Review scores 3.2, trouncing the Canadian Journal of Economics (0.582), Australian Economic Papers (0.373), Scandinavian Journal of Economics (0.558), Spanish Economic Review (0.667) or Singapore Economic Review (0.224).

These have implications, first, for the evaluation of research. Measuring research quality by impact factors means that a top financial economics article is "better" than a top public economics article, an American-focused article is "better" than an article dealing with a specifically Australian or Spanish or Singaporean issue.

But is it? Articles get cited when another article on the same subject is published. When lots of articles are published on a given topic, such as housing market bubbles, the most influential bubbles articles will get lots of citations. It's just like TV: ratings are determined by the size of the audience, and the greater the potential audience, the greater the potential ratings.

Yet the number of articles published on a topic is an imperfect measure of its social value. The number of published articles on a subject is a function of the number of academics studying that particular problem, and the number of journals publishing in that area.

But what determines the number of academics who study a particular problem? Academics are hired to teach, so where teaching demand is high, as for business or education or English programs, the number of academics working in that area will be large. But students' choice of major is affected by all sorts of factors unrelated to the value of cutting-edge research in that field, for example, it's fun, it's easy, or it teaches useful work-related skills.

Academics also - to some extent - have freedom to choose their own research programs. Problems that are enjoyable to study will receive more research attention, regardless of their social value. I have a highly policy-relevant paper on long term care insurance, for example, that I can't finish because whenever I start working on it I imagine myself in a nursing home, and get deeply depressed.

Finally, some research areas provide more opportunities for fame and fortune outside academia - utilities regulation, for example, rather than say micro theory - which may or may not be an indication of the social value of that research area.

And what determines the number of journals published in an area? That depends upon the willingness of academics to take on the work of managing a journal, and the potential for journal profits - which come from either sponsorship from a professional society, such as the Canadian Economics Association, or from library subscriptions. Since US (and, to a lesser extent, European) university libraries have historically been the ones with the most cash, journals that appeal to the US market have thrived.

It's just like medical research. Problems afflicting rich people offer greater potential profits than problems afflicting poor people - and profit potential determines research funding determines research output, even if research into problems afflicting poor people would save more lives.

Comments

[Nick Rowe]

That reminds me: I must do another post about the Fed, and stop writing about the Bank of Canada. Our citation index will slip otherwise.

[Jim Sentance]

If they can't do it properly, why have senior administrators evaluating faculty? Don't they have better things to do?

[Jack]

Here's something I've wondered for a while: when the time comes to evaluate performance (tenure or full professorship) why don't candidates submit, in addition to their package, the annotated CV of another professor elsewhere who was promoted to the desired rank. For example, if I'm going for tenure at UBC and administrators don't know how to read my CV, but they see that someone with the same output was promoted at UofT a couple years ago, it's informative.

Of course one might say the candidate would ''game'' the process by submitting the weakest possible promoted CVs, but administrators would anticipate this and see the accompanying CVs as a ''lower bound'' for tenure (or full professorship).

[Dorian Taylor]

Looks like you are describing the Matthew effect to a tee.