Affirmative action for policy-relevant research?

The Social Sciences and Humanities Research Council of Canada (SSHRC) has replaced its old "standard research grants" with new "insight grants."

Both the old and new grants fund "investigator framed" research - that is, ideas and topics generated by researchers themselves. (SSHRC has other programs that target specific research areas, for example, the automotive industry)

Investigator framed research works best when researchers know more than politicians or bureaucrats about where there are opportunities for knowledge gains, and what the emerging issues that will matter in the future are. 

The disadvantage of investigator framed research is that some topics might end up being under-researched (see here and here). Canada-specific topics that are hard to publish in top-tier international journals. Topics that require detailed knowledge of Canadian policies or institutions. Topics with few fringe benefits attached. (Circumpolar conferences are rarely held in Rio.)

The new insight grant program aims to align investigator framed research with Canada's policy needs by identifying "priority research areas."

There is a section on the insight grant application form that looks like this:

It's a bit like affirmative action (or employment equity). Tick the box if you are a member of a designated group - and then you will receive funding priority. Or will you?

The SSHRC website describes how the priority areas factor into the funding decision in its FAQs section:

Do proposals linked to a priority area have an increased chance of receiving funding?

Yes. SSHRC will continue to fund the highest quality applications to each of its funding opportunities, across all eligible disciplines and areas of research, including research falling within the priority areas. In addition, SSHRC management may choose to dedicate additional funds to support meritorious proposals falling within one of the priority areas.

So, as I understand it (and I may be wrong) there will be two rounds of research funding. In the first round, the best, say, 25 percent of applicants will be funded, regardless of research area. In the next round, only applicants in the priority research areas will be considered, and the best, say, 10 percent of applicants within the priority areas will receive funding.

How this affects applicants' incentives depends upon a number of things.

The first is the amount of money in each pool: more funds for priority research areas create stronger incentives to apply. In fact, SSHRC has not made any definite funding commitments. Official statements say only that SSHRC may choose to dedicate additional funds to priority research areas.

The second is applicants' knowledge of their ranking relative to other applicants. Suppose that you are confident that you are within the top 10 percent of SSHRC applicants. Then you can be pretty sure that you will receive funding regardless. So there is no incentive to change or tweak your research area in order to improve your chances of funding.

Do applicants, in fact, know how they rank relative to others? In economics, historically, they have done. The economics standard research grants were awarded largely on the basis of an applicant's track record, that is, publication history. Count publications in top-tier and good mid-level journals - your own and other people's -  and you can work out your odds of success.

When applicants know where they rank, the people who have the most incentive to switch to priority research areas are marginal applicants - people who have a relatively low chance of getting funded in the first, quality-only, round, but a good chance of getting funded in the second, priority-area, round. The top applicants have no incentive to switch; they will be funded anyways. 

(Indeed, this could, conceivably, lead to a stigmatization of the priority areas: "You only got funded because you're doing research on the digital economy." The stigmatizing effect of affirmative action is one reason why such programs are sometimes opposed by the very groups that the programs aim to help.)

The new insight grants, however, are not adjudicated in the same way as the old standard research grants. Economics, instead of having its own committee, is grouped together with business, management, and related fields. Ranking the Journal of International Economics against the Journal of International Business Studies is more difficult than comparing two economics journals. This increased uncertainty would increase "top" applicants' incentives to incorporate priority research areas into their funding program, as it increases the odds that a top applicant will find herself in the second round.

A final factor that matters is the cost of switching. For people who do empirical Canadian research anyways, the costs of incorporating one of the priority research areas might be fairly low.

Indeed, that is one of my main concerns with this initiative. It rewards talk: My new research program - Blogging and Canoeing: Heteroskedastic Intersectionalities - will bring the digital economy to Northern Canada, providing opportunities for innovation and leadership within Aboriginal communities, while contributing to the environmental sustainability of Northern development. Will people really change their area of research focus, or will it all end up being just so much window dressing?

Overall, though, this is one kind of affirmative action that some economists, at least, should favour. In 2010-11, 6.2% of SSHRC standard research grant funding went to economists, which is comparable to the 5.5% going to political science and 5.9% going to literature and modern languages. It is far less, however, than the 12.4% share going to education and 11% going to management and business. If economists have a comparative advantage in the priority research areas, that could be used to increase our funding share. 

So now it's time to get back to that Blogging and Canoeing insight grant proposal...

Comments

[Stephen Gordon]

It will be interesting to see how the adjudication committees respond. For example, suppose that there's a policy-centred proposal that is near the funding cut-off. They might be tempted to *not* fund it on the grounds that it would be at the top of the list for the second round.

[John]

great post.

Would be good to compare our system to the us and how it affects the research being done... that would be quite an interesting paper...

[Frances Woolley]

Stephen "They might be tempted to *not* fund it on the grounds that it would be at the top of the list for the second round."

Yes, as economists we always look for the perverse incentives, don't we! One thing that's not clear is whether or not that second round will actually exist, and if so how much money will be attached to it. I was just double checking, and this is what the detailed description of the insight evaluation program says:

SSHRC continues to recognize the value of research throughout all of the social sciences and humanities, as well as in its current priority areas. Applicants are asked to indicate in their application form whether their research falls within one of SSHRC’s priority areas. This information is gathered for reporting purposes and may also assist SSHRC in identifying applications for either additional support, a more tailored adjudication, or both.

Note the *may* (not "will")

[Frances Woolley]

John - thanks for the kind words. It would be interesting to look at the US system, unfortunately I don't know that much about it.

[Stephen Gordon]

Of course, this may have no effect at all on economics. It's really not hard to make the link between even the most esoteric subject and policy analysis: "proper policy analysis needs proper analytical tools, and this project will provide some".

There was a SSHRC session at the CEAs in June, and from what the SSHRC representative said, these measures weren't written with economics in mind.

[Jim Rootham]

It's not clear that your characterization of the process is accurate. I'd like to propose a slightly different model. Note: This is just from a reading of the text presented, not from any more detailed experience or information.

The way I see it happening is that the budgeted amounts get standard review and allocation. The remaining proposals get resorted on the basis of how much do we want that question answered. Those that reach an absolute level on that scale get funded. There may be a qualifying cutoff for the quality of the researcher, and competing proposals targeting similar questions will likely be adjudicated on researcher quality.

[Frances Woolley]

Jim "The remaining proposals get resorted on the basis of how much do we want that question answered"

With the insight grants, the grants get scored by expert assessors and the adjudication committee on three aspects of the proposal: "Challenge—The aim and importance of the endeavour (40%)" "Feasibility—The plan to achieve excellence (20%)" "Capability—The expertise to succeed (40%)". The scores within each of these areas are added together to get a total score. E.g. someone with a great proposal and a lousy track record would get say 38/40 on the challenge part and 10/40 on the capability part.

The priority research areas aren't actually mentioned in any of these categories, e.g. under "Challenge" projects are evaluated on, for example, "potential influence and impact within and/or beyond the social sciences and humanities research community" - there's nothing there about the priority research areas.

Having gone to the trouble of getting expert assessors and doing all of that scoring, they're highly unlikely to throw out the first round scores during any second stage assessment. Reading through one of these proposals takes hours.

Stephen - one thing I'm really curious about is how this joint management-business-economics committee is going to work. Will there be sub-committees? How many economists will be on the big committee/any sub-committees? How will the funds be divided between, say, accounting, finance, and economics? I know bilingual people have a high probability of being asked to serve on these committees so do you have any inside knowledge?

[Stephen Gordon]

Not really - just what I heard at the CEA session.

What had really concerned me is that we'd be fighting for funding with elements in the social sciences that hate economics. I was somewhat relieved that economists would be joint with management and business types. There's some commonality there - if only because most of them will have taken a principles course.

[Nick Rowe]

Stephen: "I was somewhat relieved that economists would be joint with management and business types."

But those guys are so much better at marketing and entrepreneurship than we are. I would say economists are #@*&ed!

[Frances Woolley]

Sorry, Stephen, I agree with Nick on this one.

[Determinant]

OK. Competition. Even better, you know the nature of the competition so you know the rules. So get better at marketing. Hire a lobbyist. Have a seminar on marketing economics research to SSHRC. Come on, you people throw these answers at the rest of us when we make comments like this about our own fields and industries.

Stop staring at your navels and do something! Come on geese, you have to have this sauce like the rest of us!