« When you need to know about the "don't knows" | Main | What Does Health Spending Buy You? »

Comments

Feed You can follow this conversation by subscribing to the comment feed for this post.

This is right on - both in terms of the need for a "fair and reasonable tenure process" and the difficulty of articulating appropriate criteria. A couple of additional thoughts. One problem may be the lack of good data on normal academic performance at a junior level. Who is the tenure track professor typical in a field? Someone with a number of years of post-graduate school academic experience or someone without? Where you start from can affect what you can reasonably be expected to achieve a few years later. In terms of possible available comparators, what is a normal level of citations for someone at the pre-tenure level within a field?

On that note, could the number of citations a person has received be used to evaluate tenure? In American legal circles, e.g. where journals are not peer reviewed, some argue that citations are the only reliable measure of academic excellence. I also wonder how much data there is on what journals are available for publication and about their acceptance data/profile. While some field may have that data publicly available, some don't. Or if they do, it isn't compiled or all that easily to access. There is also the problem of identifying standards and criteria across the university as a whole. If they are too specific to a department/field that's potentially arbitrary. On the other hand, standards that would be more than reasonable in, e.g., law (where national focus is reasonably expected and there is less hierarchy within the journals, with the result that a prof seeking tenure should really have published quite a lot in those "good" journals) might be quite unreasonable in economics (neither being true) .

Would one way to approach this be more contractual/person specific (and I'm ignoring away collective bargaining issues for the moment and also the need for across university standards)? Could a department head/dean and an incoming faculty member agree on some relatively specific targets for that person to achieve in order to be awarded tenure, targets specific to field and specialty within field? I can see lots of issues with that of course - gaming not least among them - but it might be a good way to raise standards without simply engaging in fantasy about what is institutionally possible.

Alice: "One problem may be the lack of good data on normal academic performance at a junior level." Yes, absolutely. Moreover what is normal is changing - as European and now Asian universities are becoming much more serious players in the publications game, it's getting harder to place in the global top 10 or 20.

Citations work well for promotion to full, but I don't know if they're as useful for tenure, just because of the publication lags in econ are relatively long.

I can see an argument for looser criteria. Consider the example of theoretical physics. A fairly public war has erupted over the past number of years between the establishment and some non-crank dissidents (e.g. Smoling, Woit) who make the case that the field has been hijacked by string theory, which has no realistic hope of producing a single falsifiable prediction, and has been spinning its wheels for at least 20 years now. As a result, if a young physicist wants to get published and eventually get tenure, they pretty much have to do string theory. Anything else is just going to be ignored. I gather that macro has suffered a similar affliction (at least according to Krugman).

Since the purpose of tenure is to provide cover to academics doing controversial or novel work, does it really make sense to only award tenure on the basis of criteria that amount to satisfying the existing old-timers club - which typically really doesn't react well to having it's orthodoxy challenged. As an example, didn't Akerloff have trouble getting "The Market for Lemons" published? It's not hard to imagine a world where it didn't get published...

So it looks to me like your hosed either way.

Oops. Typo. That's *Smolin*. Lee Smolin at the Perimeter Institute.

Thanks for this post, Frances. As a tenure letter writer, I always appreciate having the clear guideline. I can evaluate the case much more cleanly. I also get to avoid having to write the always-awkward 'well, s/he looks good enough for *your* U'. Instead I can write 'candidate clearly does/doesn't meet the standard'.

On the other hand, having sat through tenure committee meetings I appreciate having less quantified standards. Being active and known as a scholar in your field is often more important than an extra publication. That's hard to quantify, but it is definitely a standard I look for candidates to meet.

Thanks for this post, Frances; your point about the implications of arithmetic surprised me and is clearly important.

But I think we need to think harder about your summary that "Yet I just do not see how, without clear, explicit and consistently enforced tenure criteria, it is possible to have a fair and reasonable tenure process." I can imagine situations in which "fair and reasonable" do not require "clear and explicit." As a thought experiment, imagine tenure decisions made by an all-knowing person -- God or a similarly-wise member of the Royal Society of Canada. We might not know or understand the basis of the decision, but it might be fair, reasonable and even wise.

I think we can also understand the problem in terms of a simple model. Every candidate has an intrinsic quality q. We agree that q should be above a certain threshold in order to grant tenure. However, we don't observe q; we only observe noisy signals of q {p1, p2, p3,....} and everyone sees a different value. If we just pick one of those signals as our estimate of q, we get a signal that's noisier than it needs to be. I think that captures part of the frustration with clear rules; we think they will necessarily lead to "suboptimal" decisions.

I think clear rules also lead to strategic behaviour on the part of the candidates. That can be a very good thing if we want them to focus their energies on a particular target (top journals or number of articles or citations, etc.) But it can also have perverse effects as they focus on the "lowest-hanging fruit"; they aim for the theme- or conference-based issue of the top journal, or just try to publish a comment, having deduced that this is the easiest route to publication in a given journal. (This is an even bigger concern for citation-based criteria.)

Simon: Thanks for those comments. They confirm the point that Alice made earlier - data would be useful. If we knew the distribution of q and the distribution of the signals of q, tenure decisions would be much easier to make.

On your more general point: if the only rule is that there are no rules, that's fine, but people have a right to know. In that case, the tenure criteria would say something like: "the tenure decision will be made by the tenure committee on the basis of their own assessment of the candidate's research, teaching, and service, together with letters from two external reviewers. Successful candidates are expected to have demonstrated, to the satisfaction of the tenure committee, their ability to consistently conduct and publish quality research, and demonstrate the ability to teach effectively."

The worst case scenario, though, is when there are rules, but no one knows exactly what they are, and they keep changing, because they're never written down, formally agreed upon, or made explicit.

Kevin, thanks for the comments. When I think about your record, it raises the issue of the potential conflict between economics research that is in the national interest (e.g. your tax/benefit simulator, which is an amazing resource), and "top X" type research criteria. The requirement to publish in journals X, Y and Z can create a disincentive to carry out research on CAnadian economic issues.

From the perspective of someone in a tenure track, I think clear rules are certainly appreciated and reduce the likelihood of favoritism or cronyism. Loose rules can be interpreted charitably, or not. To be fair, they can also allow for subjective but relevant variables to play a role: a candidate on the bubble who is a good department citizen gets tenure, while a candidate on the bubble who is a lousy department citizen does not. I would hope such subjective variables (e.g., "being active and known") would play the role of tie-breakers, and not be the main drivers behind the decision.

Lastly, I see the danger of gaming a system by aiming for the easiest journal in its class, but assuming that candidates want to remain marketable in the broader job market until they get tenure (and maybe even beyond), they will weigh the benefit of getting the easiest "A" publication against the cost: to know that in the broader market, that publication is generally seen as much weaker than other A's.

In finance, a study a few years ago showed that the modal number of publications five (or six?) years after the PhD from a list of respected finance journals (core eight, I think) was one, and that averaging one article per year would put the researcher in the top percentile of the profession.

I forgot to mention things get even messier when P&T committees involve faculty from very different fields, who have their own, very different standards. The same finance journal found that, using a similar arithmetic as Prof. Woolley's, a correct conversion rate from physics to finance would be 80:1 !

Nice piece, Fran. We are constantly revising our written tenure standards, to meet university criteria, provide guidelines to tenure-track colleagues, and to ensure that what is written down reflects actual procedure, and vice versa!. I think the arithematic in this piece should go to every econ dept in the country.

Francis - I echo Linda's comment!

At my new University we don't have tenure in my department (one or two exceptional cases). But the requirements are still there to be retained after your second review period to go through a formal process like this. As a junior faculty member, I worry less about hard requirements than transparent requirements. Setting goals is good. Sure, there may be a case where adherence to the goals is the enemy of a good resolution -- that is why we have committees, to sort out the hard cases that are in the grey zone.

But if every case is a hard case that is a serious problem.

I do think it is harder to make a snap early career judgement in fields like Economics and Finance with long review times and few papers per professor (Jack's example was scary). Physics professors should have enough papers for the law of large numbers to kick in. But if the number of papers is 2-3 then a lot of this is going to be luck. It's not there merit is irrelevant to journal decisions (it's actually a huge piece) but that there are more high quality papers than slots. So there is always an element of luck as to whether a meritorious paper will get a high enough prioirty score.

Joseph "It's not that merit is irrelevant to journal decisions (it's actually a huge piece) but that there are more high quality papers than slots. So there is always an element of luck as to whether a meritorious paper will get a high enough prioirty score. "

Precisely! This also reinforces Patrick's point above about the difficulty of getting truly innovative contributions recognized. If Akerlof wasn't famous for lemons, would Economics of Identity found a home in a top journal? I very much doubt it.

Linda, Dave - thanks!

"The requirement to publish in journals X, Y and Z can create a disincentive to carry out research on Canadian economic issues."

The disincentive is huge.

Further to Simon's comments, another serious problem with quantitative standards is they encourage people to write minor variants on the same paper over and over.

Consider candidate A, who has seven publications in mid-ranked journals, all of which use the same dataset and the same basic econometric approach to address the same basic question. The papers differ in minor ways, so the candidate is not self-plagiarizing, but is very close to the line.

Candidate B has four publications in similarly ranked journals, but they address distinct questions using distinct methods.

In most cases I'd view candidate B as having made the larger total contribution, but it'd hard to square that assessment with rules stating I must judge candidates by counting their quality-adjusted page count. A new faculty hire aware of this incentive structure will be encouraged to publish like candidate A rather than B.

Kevin and Francis:

Could that disincentive be fixed through clearer tenure requirements?

e.g., An applied economist must publish in X,Y, or Z journals PLUS publish at least one article in Canadian Public Policy (or demonstrate an equivalent contribution on Canadian economic issues).

@Joel: I wouldn't want to restrict our hires to publishing on Canadian issues. I consider myself an applied theorist, and though I have some CJE papers, none of my papers is directly "Canadian content". I think we need a way to recognize that publishing on Canadian issues is a contribution, even when it appears in a "lower ranked" journal perhaps because of the topic, rather than forcing people into that box.

"The requirement to publish in journals X, Y and Z can create a disincentive to carry out research on Canadian economic issues."

The disincentive is huge.

I put for the idea on a previous thread where this came up that the Government of Canada could make economic research grants conditional on research using Canadian date and subjects to create a countervailing incentive. I was loudly booed for my idea by the economics professors who post on this site.

Jack: "In finance, a study a few years ago showed that the modal number of publications five (or six?) years after the PhD from a list of respected finance journals (core eight, I think) was one, and that averaging one article per year would put the researcher in the top percentile of the profession. "

Would love to get a reference for that.

As an outsider, I agree with Patrick: "So it looks to me like your hosed either way".

I do think that both informal and formal, metric-based solutions are easily gamed (although perhaps by different sets of people). Formality and explicitness is often taken up as a cure for informal procedures that are perceived to be biased towards an in-crowd, and I've negotiated union contracts in that spirit a long time ago. However, I increasingly see metric-driven approaches succumbing to the same problems.

Linda's comment: about constantly revising to "ensure that what is written down reflects actual procedure" may be the best we can hope for in a flawed world.

If Akerlof wasn't famous for lemons, would Economics of Identity found a home in a top journal? I very much doubt it.

I've no idea, but (again, as an outsider) I am a big fan of the Economics of Identity papers - they seem to explain a lot of things that other approaches have not been able to tackle.

Tom - I'm a big fan of Akerlof and Kranton too, but similar ideas had been around for a while - e.g. Nancy Folbre's early work (that didn't make it into the QJE - but did get her a MacArthur genius grant).

I guess the point with this post was simply to point out that metrics need to be realistic - e.g., for a mid-ranked institution, correspond to say the top 1/2, top 1/3 or whatever of the distribution, rather than the top 2, 5 or 10%. Also, if tenure committees make decisions on the basis of metrics the candidates have a right to know what those are.

.

I think one of the other problems with this issue is that the tenure decision is not merely a decision on whether one receives the benefits of tenure, but also on whether one is terminated from one's employment, which is the usual effect of a denial of tenure. The standards that seem appropriate for the first sort of decision often seem quite wrong from the second sort of decision, and yet they are by necessity the same. In my experience committees can be unenthusiastic about awarding tenure but equally unenthusiastic about terminating the employment of someone who, while not fabulous, is also not the sort of person you'd fire if you were working at a corporation or law firm.

Alice, good point.

"I guess the point with this post was simply to point out that metrics need to be realistic - e.g., for a mid-ranked institution, correspond to say the top 1/2, top 1/3 or whatever of the distribution, rather than the top 2, 5 or 10%. Also, if tenure committees make decisions on the basis of metrics the candidates have a right to know what those are."

The way you phrase it here makes me think the problem is analogous to simplistic price discrimination by firms.

A school at the say 70th percentile should grant tenure to those candidates who establish themselves at (roughly) the 70th percentile and above in the discipline. However when they make a hire whose potential is at the 80th or 90th percentile they want that hire to achieve as close to that potential as possible to maximize the benefit to the reputation of the department, grant money brought in, etc. A tenure process where the true "price" of tenure is hidden incentivizes all candidates to achieve as high as possible to ensure their chances of tenure are as high as possible. Hiding the true lowest acceptable output to achieve tenure allows departments to maximize the gains from each of their varyingly-able new hires.

All of this makes it sound like universities would be better off just getting rid of tenure entirely.

Colin: "Hiding the true lowest acceptable output to achieve tenure allows departments to maximize the gains from each of their varyingly-able new hires."

Excellent point. With numeric standard of the X articles in Y journals form, what can happen is that the minimum becomes the maximum, and people do no more. (Though given the "lottery ticket" element of submitting an article to a journal, aiming for no more than the minimum is very risky).

The problem with this line of reasoning, however, is that it neglects the effect of precedent: the lowest true acceptable output is revealed when a marginal candidate is granted tenure.

In an age of readily downloadable C.V.s, it takes two minutes to work out what the last 5 successful candidates had published at the time of their tenure bid (ratemyprofessors even gives a pretty reasonable approximation of the individual's teaching scores). A person who is denied tenure when other candidates with similar profiles have been granted it can say 'hey, this isn't fair' - and call a lawyer. I think this is why universities are adopting these explicit standards - it's the only way to raise the bar.

I think you hit the nail on the head. In the absence of clearly defined tenure standards, universities are asking for lawsuits. On the other hand, having explicit standards that, in practice, are not complied with doesn't resolve the problem, in some ways it only makes it worse.

I suspect the only workable way of doing this is rather than setting an explicit set of standards that would-be tenured professors have to achieve (publish X articles in type y journals), you set out a range of the indicia of success that you expect a successful candidate achieved. Those criteria might include having published X articles in type Y journals, but presumably they would also include having published important work in less prestigious journals (say, on Canadian issues, or in more esoteric fields), having made other important contributions to your field (Kevin's tax/benefit simulator might be an example, his contributions in areas of public policy might be another), your teaching record, being a good departmental citizen, etc. A successful candidate likely wouldn't ace (and wouldn't be expected to ace) all of those indicia (and if they did, well, let's face it, they'd be getting tenure under any system), but they give guidance both to candidates and evaluators as to what the department is looking for and what they should give weight to in accessing tenure decisions.

Having clear, but flexible, criteria helps address some of the problems with either hard standards or no standards. It reduces the incentive to "game" the system, since no one criteria guarantees tenure (i.e., the candidate who squeezes 2 or 3 papers into the most marginal of type Y journals, but is otherwise an a**hole, has no legitimate expectation that he'll be granted tenure). Similarly, candidates who fail to achieve any, or most, of those indicia have little basis for complaint (they might complain, of course, but if you've got a clear set of reasonable criteria and they failed to satisfy most of them, no court is going to take their complaint seriously). True, it doesn't resolve the "hard" cases ("why is my research on living standards in the early Arab Empire, less important than so-and-so's paper on modelling currency shocks in n-currency space?"), which will always be judgement calls, but it does reduce likely disputes to those judgement call cases.

In practice, this is what departments are going to do anyhow, so you might as well set the practice out explicitly.

@Prof. Woolley: The reference is: Terry L. Zivney and William J. Bertin (1992). Publish or Perish: What the Competition is Really Doing Journal of Finance, Vol. 47, No. 1 (Mar., 1992), pp. 295-329. To be fair, the article is 20 years old, things have probably changed somewhat in the meantime, but I would be very surprised if it has gotten easier to publish in the core journals (or any respectable journal, for that matter)
stable jstor url is: http://www.jstor.org/stable/2329099 .

@Chris Auld: Your example with candidates A and B reminds me of the story of Chicago economist George Stigler, when asked by a journalist why he had published "only" 40-some articles while his colleague Harry Johnson (a Canadian, BTW) had published over 200: "Yes, but mine are all different!"

Why should this process be so much more complex than what private firms have to do when deciding whether to give people raises/promotions/etc.?

Because tenure=employment for 20, 30, 40 years in the best job you can imagine.

While I can see the benefits of having well-defined criteria for tenure, let me play the devil's advocate here. Imagine that we were asked to develop similarly well-defined criteria for hiring. Would it be sufficient to say that you you need a PhD from type X department (and perhaps Y publications and/or r&r's in type Z journals)?

What I'm trying to get at is that our hiring decisions are almost entirely based on 'potential', and it is arguable that our tenure decisions should be as well (although to some less degree since tenure candidates do have a few years behind them - although, given the exceedingly slow publishing process in economics, this relatively short period is not really that informative).

"Because tenure=employment for 20, 30, 40 years in the best job you can imagine."

Sure, that explains why it matters to the would-be tenured professor. On the other hand, the stakes for the department are probably less significant than for a private firm hiring, say, a futures trader who, if so inclined, could bankrupt the company (see, for example, http://www.nickleeson.com/ or http://en.wikipedia.org/wiki/J%C3%A9r%C3%B4me_Kerviel), os I think Alex's point is still valid. I suppose you might inadvertently hire a plagiarist, but while that might earn you a good ribbing at the annual AEA convention, that's not like the bring down the department.

econometrician: " Imagine that we were asked to develop similarly well-defined criteria for hiring."

We are, and we do it. At Carleton, the university will not approve a position unless the department can say "this is what the person will do and this is why we need a new person." Job ads are vetted. Hiring committees have external members to make sure that things are above board. The entire process is subject to extensive scrutiny.

Bob Smith - " On the other hand, the stakes for the department are probably less significant than for a private firm hiring, say, a futures trader who, if so inclined, could bankrupt the company" Ouch!

What are the stakes, realistically? I would define "deadwood" as follows: "a person whom the department could replace with a contract instructor and receive an unchanged amount of teaching, research and administrative output." The difference between a tenured professor's salary and a contract instructor's salary is, say, on average, about $75,000 to $100,000+. Giving tenure to deadwood is a pretty expensive proposition - it won't bankrupt the company, but it's sure a big drain.

"Because tenure=employment for 20, 30, 40 years in the best job you can imagine."

Hmm. Maybe along with doing the math from the standpoint of what the standards imply about the average candidate is is worth doing it for the profession too. (Just being an off-the-cuff contrarian for a bit of fun.)

Is it possible there are more people who want "the best job you can image" than we actually socially want to provide tenure slots for? (With those currently holding the job due to the cost of eliminating tenure being larger than the cost of simply letting them continue to earn abnormal returns.)

Is it possible that each department has an incentive to leave their standards a bit too low ( lower than they would like to set them ) because they know their department needs to continue being seen as young and fresh. Basically that the PR value of a new tenure hire to the rest of the department helps them directly while the cost is spread over the whole university.

"The difference between a tenured professor's salary and a contract instructor's salary is, say, on average, about $75,000 to $100,000+. Giving tenure to deadwood is a pretty expensive proposition - it won't bankrupt the company, but it's sure a big drain."

Agreed, but there are probably risks of a similar magnitude (albeit of a different nature) in any employment relationship.

When reading your original post, I was thinking of the evaluation processes at law firms for determining whether they want to hire back articling students or bring associates into the partnership, decisions that are akin to the tenure decision (both in their implications for the lawyers and the possible repercussions for the firms - you can fire the deadwood, but perhaps not after you get sued for their work or they cost you a client).

The traditional metric is billable hours, but that's subject to the same risks of gaming (well, gaming isn't quite the right word, maybe inaccuracies) as journal publications. If X bills a lot more than Y, but is less efficient in his work (which makes clients grumpy when they get the bill), are they really a better lawyer? What if X bills more, but does low-value work, while Y bills less, but knocks clients socks off (so they'll keep bringing work back), who do you want as your partner? And you have the same realities arising from external contraints - sometimes the billable hours just aren't there (anytime after 2008) - as well as the generic factors that are common to every workplace (can you stand working with this guy/girl, do they pull their weight in departmental/firm administration, do they do their share of teaching/business development). All firms have formal evaluation criteria (no surprise, lawyers, even unsuccessful ones, are nothing if not litigious), but they're not applied mechanically.

Bob - I think this gets back to Alice's point about what is different about tenure. It's not like a typical promotion decision - denial of tenure = dismissal. A law firm can just decide not to promote an associate to partner, and eventually the person is probably going to get the message and go away. In other words, there is no definite point at which a person must be either promoted to partner or fired. By way of contrast, in universities, there is a specified time - say the fall of the 5th, 6th or 7th year - at which a person is either given tenured or terminated.

"all firms have formal evaluation criteria" - that's really all I'm suggesting, nothing more.

dBonar - Lorne Carmichael has some interesting work on the economics of tenure.

Dbonar: I'm not sure it is about a department granting tenure more easily to look young and fresh. The incentive to lower tenure standards in a department also may be based on what the perceived probability of getting a new hire is. Departments normally do not control the resource allocation decision for making a new hire. If a department thinks that it will not be given a replacement hire if it denies tenure to a candidate, then there is a strong incentive to interpret criteria more loosely. Positions generally have to be competed for with other departments and there is no guarantee you will be able to replace departures. If a departure is not replaced, it does leave a staffing hole in the department.

Livio: You put it better than I did. I don't think the issues I raised are unique to tenure, but really are a wider organizational problem.

In part, I think of it as similar to pulling together a divisional budget from many managers and their teams. Each managers (roughly, department) has plan (tenure candidates) he wants to see funded. Most of the plans have merit and the team members and managers who support them in an absolute sense are right. With infinite resources, a large percentage of the proposals should be funded. But the division isn't given an infinite budget in either cash or manpower, so decisions need to be made.

Some managers will deny that there is a budget constraint and argue that all -- in particular their team's -- worthwhile projects should be funded.

Other managers will realize the need to switch from arguing for their projects on an absolute basis to arguing from a relative basis. Unfortunately they often have a hard time correctly assessing plans from different areas of the division, which can lead to both trying to build influence with the people building the budget and to semi-consciously overlooking the weaknesses in their own team's projects so they can better hide those weaknesses.

The comments to this entry are closed.

Search this site

  • Google

    WWW
    worthwhile.typepad.com
Blog powered by Typepad