Last year, in my report on the inner workings of the adjudication process for research funding applications at the Social Science and Humanities Research Council (SSHRC), I made the following complaint:
For some reason that has never been clearly explained to me - and I've done this twice - SSHRC insists on a fixed 'success ratio' of 40%. No matter how much money a committee has, or how good the applications are, 40% of the applications will receive funding. No more, no less.
This year, that rule was changed - thus setting the stage for an interesting natural experiment.
The proximate reason for the experiment was a budget cut. Although SSHRC received pretty much the same number of applications asking for pretty much the same amount of money, this year's budget was about 25% less than in 2006. Since the feds are running an embarrassingly large surplus these days, the most likely explanation for this reduction is that Canada's New GovernmentTM believes - not unreasonably - that SSHRC gives money to knuckle-headed Marxists who wouldn't vote Conservative in a million years.
That would bring the success rate down to 30% if the old rules were applied. There's been a recent SSHRC initiative to 'empower the committees', so the decision to give the committees the discretion to choose a success rate might be explained that way. But since the decision to drop the rule was made after the budget problems manifested themselves, it's at least possible that SSHRC wanted to be able to deflect some of the flak to the academics on the adjudication committees.
So what did the committees do with their new-found freedom? I calculated two sets of series using the data (available here) for the last two competitions. For each committee and for each year, I calculated the average grant requested, the average grant awarded, and then constructed a measure of the 'funding rate': the average award as a percentage of the average request. (The reason for going through all this is that there are significant variations in the cost of doing research across disciplines.) In 2006, the budget envelope was set so that a success rate of 40% would be consistent with a funding rate of 75%. All committees hit the success ratio (they had no choice in the matter), but since there were variations in project quality and project costs across committees, the funding rate ranged between 65% and 80% of the average request.
This year, the budget envelopes were set for a 75% funding rate and a 30% success rate, but committees could exchange a lower funding rate in order to obtain a higher success rate. (I seem to recall that there might have been a minimum success rate, but as we shall see, it wasn't a binding constraint for us.)
Here's a plot of the success and funding rates for the 2006 and 2007 competitions; the X's mark the SSHRC targets used to determine the budget envelopes:
In 2006, the committee decisions were distributed along the fixed 40% success rate; Economics is somewhere in the cluster around the SSHRC target . But in 2007, we see the committees choose different points along the new budget line in which higher success rates come at the cost of lower funding rates.
There were 2 broad categories of responses to the 2007 problem, and one very distinctive outlier:
- Committees that appear to have aimed for - with varying degrees of success - the 30% target: 9 had success rates less than 31%.
- Committees that appear to have aimed higher than 30%, with some managing to come quite close to last year's 40% target: 11 distributed along the interval 32%-39%.
- Economics was in a category by itself, with a success rate of over 49%.
Before the economics committee started work, the chair (me, as it turns out) pointed out the nature of the new budget constraint we faced, and asked the committee its opinion on how it should proceed. The immediate and unanimous response was that we should aim for a success rate that was as high as possible. So we set to work with that goal in mind: cutting budgets to the bone (and beyond) in order to find funding for as many projects as we could.
Clearly, we were the only committee to make that decision. Why?
One possible explanation might be that the economists' budgets were all grossly inflated, so it was easy to cut them back in order to get a high success rate. But this doesn't seem to be the case. Here are the average requested budgets, expressed as a percentage of the average request in economics:
- 86%: Philosophy
- 87%: History
- 89%: Literature 2
- 94%: Literature 1
- 97%: Classics
- 100%: Economics
- 111%: Fine Arts
- 112%: Finance
- 119%: Cultural Studies
- 120%: Political Science
- 129%: SSHRC average
- 129%: Management
- 134%: Law
- 140%: Sociology
- 142%: Geography
- 151%: Psychology
- 154%: Linguistics
- 157%: Interdisciplinary
- 159%: Education 1
- 161%: Anthropology and archeology
- 164%: Psychology 2
- 166%: Health studies/Social work
- 167%: Education 2
- 178%: Education 3
As I mentioned earlier, the cost of research varies across disciplines: archaeological digs don't come cheap. But it'd be hard to conclude from these numbers that economists are inflating their budgets - or if they are, then not extravagantly more than in other fields.
Which brings us back to the question of why economists were such an outlier. I suspect that if I could come up with a really good story, I could get a JPE article out of all this, and not just a blog post.