Results from a natural experiment: Economists are different

Last year, in my report on the inner workings of the adjudication process for research funding applications at the Social Science and Humanities Research Council (SSHRC), I made the following complaint:

For some reason that has never been clearly explained to me - and I've done this twice - SSHRC insists on a fixed 'success ratio' of 40%. No matter how much money a committee has, or how good the applications are, 40% of the applications will receive funding. No more, no less.

This year, that rule was changed - thus setting the stage for an interesting natural experiment.

The proximate reason for the experiment was a budget cut. Although SSHRC received pretty much the same number of applications asking for pretty much the same amount of money, this year's budget was about 25% less than in 2006. Since the feds are running an embarrassingly large surplus these days, the most likely explanation for this reduction is that Canada's New Government^TM believes - not unreasonably - that SSHRC gives money to knuckle-headed Marxists who wouldn't vote Conservative in a million years.

That would bring the success rate down to 30% if the old rules were applied. There's been a recent SSHRC initiative to 'empower the committees', so the decision to give the committees the discretion to choose a success rate might be explained that way. But since the decision to drop the rule was made after the budget problems manifested themselves, it's at least possible that SSHRC wanted to be able to deflect some of the flak to the academics on the adjudication committees.

So what did the committees do with their new-found freedom? I calculated two sets of series using the data (available here) for the last two competitions. For each committee and for each year, I calculated the average grant requested, the average grant awarded, and then constructed a measure of the 'funding rate': the average award as a percentage of the average request. (The reason for going through all this is that there are significant variations in the cost of doing research across disciplines.) In 2006, the budget envelope was set so that a success rate of 40% would be consistent with a funding rate of 75%. All committees hit the success ratio (they had no choice in the matter), but since there were variations in project quality and project costs across committees, the funding rate ranged between 65% and 80% of the average request.

This year, the budget envelopes were set for a 75% funding rate and a 30% success rate, but committees could exchange a lower funding rate in order to obtain a higher success rate. (I seem to recall that there might have been a minimum success rate, but as we shall see, it wasn't a binding constraint for us.)

Here's a plot of the success and funding rates for the 2006 and 2007 competitions; the X's mark the SSHRC targets used to determine the budget envelopes:

In 2006, the committee decisions were distributed along the fixed 40% success rate;  Economics is somewhere in the cluster around the SSHRC target . But in 2007, we see the committees choose different points along the new budget line in which higher success rates come at the cost of lower funding rates.

There were 2 broad categories of responses to the 2007 problem, and one very distinctive outlier:

1. Committees that appear to have aimed for - with varying degrees of success - the 30% target:  9 had success rates less than 31%.
2. Committees that appear to have aimed higher than 30%, with some managing to come quite close to last year's 40% target: 11 distributed along the interval 32%-39%.
3. Economics was in a category by itself, with a success rate of over 49%.

Before the economics committee started work, the chair (me, as it turns out) pointed out the nature of the new budget constraint we faced, and asked  the committee its opinion on how it should proceed. The immediate and unanimous response was that we should aim for a success rate that was as high as possible. So we set to work with that goal in mind: cutting budgets to the bone (and beyond) in order to find funding for as many projects as we could.

Clearly, we were the only committee to make that decision. Why?

One possible explanation might be that the economists' budgets were all grossly inflated, so it was easy to cut them back in order to get a high success rate. But this doesn't seem to be the case. Here are the average requested budgets, expressed as a percentage of the average request in economics:

• 86%: Philosophy
• 87%: History
• 89%: Literature 2
• 94%: Literature 1
• 97%: Classics
• 100%: Economics
• 111%: Fine Arts
• 112%: Finance
• 119%: Cultural Studies
• 120%: Political Science
• 129%: SSHRC average
• 129%: Management
• 134%: Law
• 140%: Sociology
• 142%: Geography
• 151%: Psychology
• 154%: Linguistics
• 157%: Interdisciplinary
• 159%: Education 1
• 161%: Anthropology and archeology
• 164%: Psychology 2
• 166%: Health studies/Social work
• 167%: Education 2
• 178%: Education 3

As I mentioned earlier, the cost of research varies across disciplines: archaeological digs don't come cheap. But it'd be hard to conclude from these numbers that economists are inflating their budgets - or if they are, then not extravagantly more than in other fields.

Which brings us back to the question of why economists were such an outlier. I suspect that if I could come up with a really good story, I could get a JPE article out of all this, and not just a blog post.

Comments

[happyjuggler0]

Sometimes I think economists try too hard to figure things out logically and don't bother to ask people in the field why they do what they do, or don't do. This looks like such a situation to me. Why not ask the other fields why they chose the criteria they chose? You still might not get anywhere, but I suspect you won't be able to deduce it with pure logic and zero empiricism.

[Stephen Gordon]

The short answer is that people lie. It's possible that not-entirely-noble considerations are at play here, and information about them will not be freely offered.

[Christine]

Thanks for the info - always interesting to hear about inner workings of secretive committees.

But then again, given your comment above, how do we know you're not lying?????

[Stephen Gordon]

I have witnesses! Okay, their names aren't published yet on the SSHRC site, but they will be. Soon. Any day now.

[Nick Rowe]

An interesting paper by Lonnie Magee and Mike Veall on this subject:
http://papers.ssrn.com/sol3/papers.cfm?abstract_id=357220 or
ftp://repec.iza.org/RePEc/Discussionpaper/dp617.pdf
The fact that 40% (or some fixed percentage) of applications get accepted creates a strong "keynesian" multiplier process (fewer applications means fewer acceptances which means fewer applications next year etc.) which could seriously harm disciplines like Economics, where it is easier to judge the likelihood of success, or where there may be slightly greater opportunity costs of applying to SSHRC.

Draw a diagram, with "number of applications" on the vertical axis, and "number of acceptances" on the horizontal axis. SSHRC behaviour is represented by a ray from the origin with a slope of 1/0.40 (for a 40% acceptance rate). Applicants' behaviour is represented by an upward-sloping curve (more acceptances lead to more applications). Equilibrium (if it exists) is where the two curves intersect, just like the Keynesian Cross diagram from ECON 1000.

First, an interior equilibrium may not exist (or may not be stable). For example, if no economist will apply unless the chances are greater than 40%, the equilibrium has zero applications from economics.

Second, even if a stable equilibrium does exist, small shifts in the applicants' curve will cause large changes in the equilibrium (a multipler). So if economists are just a little bit less likely to apply than other disciplines, you get a large reduction in equilibrium applications and acceptances for economics. (If you look at the data, you see that economics gets a lot fewer SSHRCs than other "crap-shoot" or "low opportunity cost" disciplines.)

Something should be done!