« A Very Brief History of Federal Cash Transfers: Canada 1867 to 2017 | Main | Milk is mind-bogglingly cheap. »

Comments

Feed You can follow this conversation by subscribing to the comment feed for this post.

"an unacceptably low average"?? Presumably you give weight to an element because, in your professorial opinion, it has high human capital and is appropriate to the course level. If 'almost all' prove unsatisfactory, it may indicate,inter alia;
-element is unsuited to course level
-it ain't the brightest class
-students didn't allocate effort to match rubric weight
-your pedagogy slipped a little )^_^(
none of these devalue pre-circulated rubrics
Without clear criteria, stated in advance, it's difficult to believe that evaluation of essays will ever be accepted as 'fair & accurate'
On a slight tangent, based on personal experience, I think the 'average' student appreciates a weighting fn even on analytic exams, where there is always a 'right answer': it helps allocation of effort when there's a strict time limit.

What I would have appreciated in school was clear expectations of some less formal aspects: "Paper should pretend Freud and Marx are correct descriptions of their respective domains.", "Questions in class shouldn't show the lecturer to be wrong.", "Posted office hours are a snare and a delusion." among others. Granted, these didn't all apply to all of the individuals in question, but all had at least one.

John - excellent! Also: "That high school English teacher who encourages you to write your innermost thoughts in your journal? Don't."

A number of the challenges with your rubric (albeit not all) can be managed by expanding the rubric to better reflect the difference between A+, A, B, and C level work. For example, rather than lit review - 10 with a number of criteria, you could include brief descriptions of different quality lit reviews:

A+ --> draws from more than 3 relevant econ articles; identifies original connections and contrasts among the articles; organization is both clear and based around the most significant ideas; considers significant empirical and theoretical contributions.

A --> draws from at least 3 relevant econ articles; identifies connections and contrasts among the articles, some of which are original; organization is based around ideas; considers empirical and theoretical contributions

B --> draws from 3 econ articles, at least two of which are important to the field; identifies connections and contrasts; organization is based at least somewhat around ideas; considers empirical or theoretical contributions

C --> draws from no more than 3 econ articles, most of which are not important to the field; identifies connections or contrasts; organization is by article; unable to recognize the difference between empirical or theoretical contributions

Adam, yes, that's much better, but also much more effort!

Though this also illustrates the dangers of being super-explicit with expectations. E.g., with your "A" criteria - how can I assess if a connection between articles is original if the articles aren't in my field? Is there an objective criteria by which to decide which are "the most significant" ideas? Which are significant contributions? With regards to your B level criteria - why is important to the field here and not for the As? Do I want to get into an argument with a student about whether or not an economic article is "important to the field" - and how can undergrad or even masters level students be expected to know which articles are important to the field and which ones are not?

Also how do you compare one student who has three fantastic articles but does a lousy job of analyzing them, and another student who has three lousy articles but does a fantastic job of synthesizing them?

In some ways what you've suggested is an aspirational rubric - what I would, in my dreams, hope to get from undergraduate students - rather than a practical one. At the same time, it's really good to share aspirations - to say what a beautiful literature review would look like.

This is a nice discussion of a fairly difficult issue. And the comments add value; the problem I always had with developing rubrics was the need to make them specific enough to be useful and flexible enough to allow for me to be (pleasantly) surprised.

«Imagine, for a moment, that students acquire valuable human capital during their time at university. Imagine that the grades on a student's transcript reflect his or her level of human capital.»

That's pretty hard to imagine for me: the overwhelming value of a university degree is the degree itself, or more precisely, in getting admission to the university that gives the degree, and there are books and papers that show the compelling quantitative evidence for that.

The grades on the transcript are mostly a formality, and smart students spend a lot of effort getting admission to a valuable university, and thereafter go through the motions, and make the least effort needed to get the degree, as they have "clear expectations of some less formal aspects" as another commenter says and this relates to:

«Part of the human capital gained in a university education is knowledge of the unwritten rules, the social norms and conventions, of intellectual life.»

Indeed, and the students who care about the content of their courses are usually a minority, the insecure, or the swots and the nerds, and the proposal here to make evaluation criteria more transparent may be good for them though, as they tend to get hang up about these things.

Pious homilies from entities like "centres for teaching excellence" are meant mostly as marketing exercises, to make-believe that quality of teaching really matters to careers or that "people" like potential employers really care about it.

Blissex: From anecdotes, a number of people seem to take the signaling value of a degree (especially at the doctorate level) as coming from or possessing a culture of achievement, of not giving up or settling for a lower outcome when facing adversity, but making extra effort to overcome it. And probably assuming that this will last at least for a few years while the initial work track record "proof of merit" is established and/or disillusion sets in.

But one may consider this included in "social norms and conventions".

«the signaling value of a degree (especially at the doctorate level) as coming from or possessing a culture of achievement, of not giving up or settling for a lower outcome when facing adversity, but making extra effort to overcome it.»

I must agree that for a doctorate or an MBA that signaling value matters in some countries, in the USA for example; but in several other countries doctorates in particular result in lower lifetime earnings and more difficult careers, as they signal something else instead, and many employers other than universities don't want "otherworldly" "know-it-alls" :-). It can cut both ways :-).

It depends on context indeed; I also admit that in some countries transcripts and marks matter significantly at least in some professions as in those cases employee insecurity, and swottiness and nerdiness, are valued by many employers.

Apart from whichever signaling value they ascribe to credentials, in general employers care little about the actual content of the sources, beyond a fairly low minimum, for most non-professional degrees. Put another way most employer HR departments want to make their jobs simple by delegating staff selection to a single metric delivered for free to them by universities :-).

Usually students, or at least their mothers, know well which metric (admissions, transcripts, ...) employers actually value as a signal.

Blissex:

"That's pretty hard to imagine for me: the overwhelming value of a university degree is the degree itself, or more precisely, in getting admission to the university that gives the degree"

Even more precisely, getting admission to *and graduating from* the university that gives the degree.

In a signalling world, a professor's job is to sort more able from less able students. We add value by identifying and failing low ability students. . Hence the arguments for/against setting out clear expectations are totally different in a signalling world. Does lack of clear expectations improve the informational content of the "completed university" signal? One could, in fact, argue that setting out clear and detailed instructions that any idiot could follow means that any idiot can pass - and the signalling value of a university credential is diminished.

With signalling, the entire "clear expectations help students learn" discussion becomes moot.

cm - people who start university but fail to complete a degree have worse outcomes by almost any measure than people who never start university in the first place. So I think there is something to the giving up/effort story.

Frances to cm: would it be because those who never tried have a better understanding of their abilities, a cognitive trait they can use all their life or are those who fail to complete are so marked by failure they get discouraged for a long time?

Now, this is time consuming, but one way to look at it is if you read all the papers through once without a pen in hand, you can see where there is systemic weakness and then construct your rubric from there (or if the class is large enough, just a sample). Then you can reread with your weights and then look on what you might want to focus on as a teacher in terms of setting those professional norms. (Repentant former graduate student in Literature).

Jedgar--The problem is that the usual practice is to construct the rubric *in advance* and *share it with your students* so that they are not in the dark about what they are supposed to be doing. This is especially the case when the activity (e.g., a statistics-based research paper) is something they've never done before. A colleague (and former teacher) of mine who did this developed his research paper rubric from his experience as a journal referee. He had the students read journal articles and apply the rubric to what they read. Then he had them begin the process of writing their own research papers (this was, actually, in a PhD program).

I just gave a third year programming final exam. At this school, it is typical to give a textual description "Program must compute the daily 5 year - 5 year ahead expected inflation from the given data" followed by the correct output "Correct output is 1.0, 1.1, 1.2, 1.1, ..." One of the problems on the exam given yesterday had only the textual description. Several students freaked out asking how to ensure correct output without knowing the correct output. Of course, outside exams, if I have a program producing the correct output, I don't ask someone for a new one! It is helpful to give students some indication of what is expected, but at some point they have to see what problem solving looks like.

Jacques Rene - "those who never tried have a better understanding of their abilities" Possibly. Or they're willing to make a decision and stick with it. Definitely something like that.

Donald: "He had the students read journal articles and apply the rubric to what they read."

That's a really neat idea.

Frances - do you have any reference for your comment that students who start university but fail to graduate have worse outcomes? I find that very interesting and would like to read more.

Here's a different sort of expectation that I have: 5-10% of students in a large class will fail by the end of semester.

I don't think that's a good expectation to share with students! Indeed, there is probably good pedagogical reason to cultivate the perception that I believe all the students will do very well -- even though this is a misperception of what I believe. Given what we know about stereotype threat, etc, there is lots of reason to think that students will achieve more if we can convince them that we have high expectations of them, even when those expectations are unrealistically high.

Ruth - I was looking at the relationship between life satisfaction and education in the canadian community health survey just the other day - the people who have some university/college but not a certificate have lower levels of happiness than people who have completed high school but not gone onto university. This and related phenomenon - that there's something about graduating rather than just simply attending school - are often termed the "sheepskin effect". I looked for a nice reference for you, though, and couldn't find one quickly.

Toby - " Indeed, there is probably good pedagogical reason to cultivate the perception that I believe all the students will do very well"

This is a fascinating idea - that laying out unrealistic expectations may increase student learning.

Blissex: "I must agree that for a doctorate or an MBA that signaling value matters in some countries, in the USA for example; but in several other countries doctorates in particular result in lower lifetime earnings and more difficult careers, as they signal something else instead, ..."

Yes I was talking about the US, and my opinion is this is part of a more general phenomenon of "brand awareness" (if not "brand focus").

"Usually students, or at least their mothers, know well which metric (admissions, transcripts, ...) employers actually value as a signal."

And here we go for the killer argument ... mom knows better. No, I don't actually think in general mothers have a good/better than others assessment of this. But you are certainly right when broadening the argument to "social environment counselors".

Frances/Jacques:

"cm - people who start university but fail to complete a degree have worse outcomes by almost any measure than people who never start university in the first place. So I think there is something to the giving up/effort story."

"Frances to cm: would it be because those who never tried have a better understanding of their abilities, a cognitive trait they can use all their life or are those who fail to complete are so marked by failure they get discouraged for a long time?"

BTW short introductions - I'm a software developer with CS degree, and my understanding of other domains is of the armchair/life experience variety - no other formal "education" beyond mandatory college classes.

"Failing college/uni/tertiary education" is correlated with pretty much two things (may be considered more than two):

* Life adversity (family issues, health problems, having to abort college for financial reasons, unexpected mother-/fatherhood, etc.) - these are causations that very likely have impact on general socioeconimic success.

* Being talked/coerced into college despite lack of inclination or ability - I know first hand examples of talented peers who were high-achieving throughout high school, and then in uni reverted to mean. Following cases with analysis:
* An arguably brilliant and generally nice guy who was socially abrasive/inept to the point of deliberately letting "authority issues" escalate to levels most people wouldn't - he was good, but got into a situation where he couldn't complete his PhD probably because of limitations related to this
* Another IMO above-average smart guy who excelled through high school under the pressure of his parents (and while living with them) - when on his own, his motivation to deal with adverse (socially imposed) conditions plummeted, and he pursued an "easy/secure government job" definitely below his "skill" level - perhaps the smart choice in the long term (I may have known several individuals in similar straits who took a lower-risk and nominally "lower-skill" exit)

On the second point, I'm from former East Germany, and what I convey happened in the 90's during our social revolution and reunification, which injected another dimension of uncertainty and disturbance into everybody's circumstances. Otherwise people may have made other choices (in a presumably starkly different social environment - hypothetical as it may be).

Jacques: There is self-selection, of the 'cowardly' or 'wise' type (for my lack of coming up with better descriptions), i.e. people avoiding things where they perceive either little chance of success or little chance of *enjoying* the (so-called?) success. E.g. there are certain advancement opportunities (BS or even real) where you are asked to put in significantly more work for an imaginary or tangible payoff - then there are the dimensions what you are "guaranteed" to get out (probably conditional on "succeeding") vs. entirely speculative payoffs, e.g. the opportunity to participate in the next level of audition/competition, etc. And not unimportantly, what *other* people (much likely those who encourage you to participate!) are getting out of it. And comparing to peers who "don't participate".

So maybe it is not so much that people get unfairly discouraged but plainly see that some things are not for them, psychologically, or there is no perceived advantage. Maybe they could have been successful, but the uncertainty or abhorrence of the imagined lifestyle weigh more heavily.

Of course a lot of this is probably my own projections.

"Sheepskin effect": You're only as good as your last record. You can go out of HS with a bang: "Mom I graduated!" or out of college with a whimper:"Mom, I failed at university..."
There are medals for bravery and others for service but there is no diploma for two years of study (though some universities will tranform your fisrt two years into a certificate) :"Mom I have my degree!"

Yep, telling students exactly what to expect will prepare them for real world analysis. Not!

Definitely.

Don't say short answer when you mean short essay.

Jacques: I think it is something else. People are pushed into a narrative that doesn't come from themselves but is externally imposed. Left to their own devices, most (at least many) are directionless. There has been a loss of "community" and other forms of social structure that gives individuals and groups a direction. Most people go to college because that has become a minimum requirement for what was passing as "good" jobs not requiring back breaking labor only a few decades ago (i.e. the prior generation's experience), and it is heavily pushed on them by everybody. Of course they are interested mostly in the degree paper! Because that's what the employers are interested in also.

Most jobs don't actually require this level of teaching, outside of technical specialties that are probably a minority of jobs overall. Most jobs require some foundations, but most skill and experience is acquired on the job. Back in my day, only around 10% or so of pre-HS students would (be allowed to) go on to high school and then college directly. A larger proportion would be put through a combined vocational+high school program. Some number of them would also go to college eventually, "upgrading" e.g. from a technician role to an engineer role (from an individual perspective, perhaps out of desire, but in the bigger picture because of a social need in a society moving from industrialization on to higher technology). This also made for some subculture of people taking pride (and frankly sometimes turning into a sense of superiority) in having learned a trade as opposed to those who just went to college without on-the-floor practical experience.

Now much of the "technical" work, together with manufacturing, has been offshored and continues to be. What remains (so far?) in country are (in no particular order) administrative roles, sales, operations, healthcare, legal, law enforcement, construction, transportation, durable goods maintenance/repair, etc.: the whole infrastructure of keeping the local economy and workings of society going - job functions that are location or language/culture specific.

Most (not all) of these job roles have in common that they are relatively light on "classroom teachable" skill - they are more of the vocational/learning on the job type. I want to make it clear that I *don't* mean "low skill" - but by far most skill acquisition is practical, not from textbooks. "More college" will not help here, certainly not to address the "training" aspect.

cm: I agree with you about university being a means to an end for many students - and that's fair enough. I do think there are more "classroom teachable" skills out there than your comment suggests - whether or not universities are actually teaching them is another matter entirely.

"Classroom teachable" is also not really what I meant - more accurate would be that the subject matter is not structured and formalized by scientific theories, either because it cannot be, or just because nobody would attempt it, or it is not general/workplace independent enough. And thus it is arguably outside the purview of uni/college teaching - their function is not to train the workforce for specific companies or institutions.

There is still plenty scope for classroom teaching in the vocational domain - but there also the taught material must be general for the domain. Also when companies/institutions offer internal classes (because eventually the staff *needs* this training for the organization to work), these classes will be held in classrooms. But they will be specific to the organization's need.

Yet another aspect of the whole "why college" problem is that in many fields the required "skill"/foundation level *has* risen - but the human material hasn't "improved" to kids can be taught a higher level in the same number of years. Then just in principle the lifetime years of learning/training must be increased. Whether what is currently practiced is very effective is another question.

To some extent you can replace breadth/depth on the basics with higher level material (e.g. drilling multiplication tables vs. higher level concepts), but this is very limited - higher level concepts cannot be understood without enough basics.

Now (or since a while?) cursive handwriting is being phased out - it's now placing letter by letter. Who writes anymore, right? Now you need "computer skills", i.e. presumably navigating touch screens and "apps" provided by major internet companies, as that's what replaces "traditional" teaching methods apparently.

When I was in high school, calculations transitioned from pen+paper and slide rule to electronic calculators - I still got to see both. A few years later I started hearing complaints that graduates had difficulties with, or were plainly unable to do written division and mental arithmetic in general. Now again, who needs that, where this skill would have been necessary, now computers and software packages are used.

Verify your Comment

Previewing your Comment

This is only a preview. Your comment has not yet been posted.

Working...
Your comment could not be posted. Error type:
Your comment has been posted. Post another comment

The letters and numbers you entered did not match the image. Please try again.

As a final step before posting your comment, enter the letters and numbers you see in the image below. This prevents automated programs from posting comments.

Having trouble reading this image? View an alternate.

Working...

Post a comment

Your Information

(Name and email address are required. Email address will not be displayed with the comment.)

Search this site

  • Google

    WWW
    worthwhile.typepad.com
Blog powered by Typepad