Never mind the mean, feel the variance: some thoughts on grading schemes

Imagine two course sections with the following grading schemes:

Section A: 4 assignments worth 5% each for a total of 20%; 35% midterm; 45% final.

Section B: 4 assignments worth 10% each for a total of 40%; 60% final.

In my experience, students often reason: "Section B places more weight on assignments. I can work with my friends and use other resources, and get good marks on the assignments. Therefore it'll be easier to get a good grade in Section B."

But professors know students work together on assignments, and it's almost impossible to tell who has done the work and who has just copied it. They put weight on the assignments, so that students have an incentive to complete them. But profs don't want the assignments to have much influence on the students' final grade.

The best way to diminish the effective influence of assignments is to give everyone more or less the same grade - say 80 percent.  That way assignments just scale up the class average, and students' relative position in the class is primarily determined by their score in the examinations.

But if everyone gets good marks on assignments, won't the grades in Section B be higher, on average, because in that section assignments are worth 40%?   

Actually, no. Seasoned university professors are usually pretty good at setting up their courses to generate a particular distribution of grades. Every year, a given professor will award more or less the same percentage of As, Bs, and so on. Regardless of the weight going to assignments, the class average will be whatever the professor wants it to be. If the professor wants the class average to be 60%, she sets a killer final; if she wants it to be 80%, she asks predictable questions lifted straight from the text and the lecture notes. The best predictor of this year's grade distribution is last year's grade distribution - not the weight placed on assignments - because last year's grade distribution is the best indicator of what the professor wants the grades to be. 

If I was a student, I would pick Section A over Section B. Remember that - because there is usually relatively little variance in assignment scores - students grades are mostly determined by how well they do on the exams. In Section A there is a midterm exam, so students get a pretty good sense early on of how well they are doing in the course, and can seriously step up the effort level - or drop the course - if things are going badly. Also, with both a midterm and final exam, there is less chance of the entire course being derailed by a catastrophic event (e.g. sickness, sadness, lack of sleep, overwork). 

Students often ask "what was the class average?". An equally important question is: "what was the class variance?" The variance in a course component - in combination with the weight placed on that component in the grading scheme - determines how much it matters.

Comments

[Avon Barksdale]

This is all a symptom of the dumbing down the university curriculum - making university an extension of high school where professor are expected to “teach”. “Here's a gold star for doing your assignment.” Meaningful feedback comes from know that you can solve all the problems in the textbook. Move to 100% final exams, no marked assignments or problems sets. Let the student determine his/her own level of involvement. People who ask what the course average was shouldn't be there anyway. The best way to diminish the effect of assignments is not to have them. But, I am sure if a professor tried that the student body would erupt and the dean's office would get involved. Oh well, it doesn't matter, I guess. Most of what goes on at university is about buying a signal. Where I work, we separate the wheat from the chaff pretty effectively, making sure to exclude the ones that should never have graduated in the first place.

[Tom Brown]

I had a prof (for a lot of classes) who used a unique procedure for assigning grades on his midterms. These were small classes, keep in mind... but he'd look at his sheet on which he'd recorded the scores and then start writing them on the board like this:

98 8D
-----------------
92
91 :D
91
-----------------
84
83
77
75 :)
75
74
73
-----------------
68
67
66 :|
59
----------------
39 :(

There was always at least one frown at the bottom! ... But he'd just basically eyeball it right there in front of us. I'm pretty sure I "earned" at least one frown... but I don't recall dropping.

[Brett Reynolds]

Why would a professor want a particular distribution? This kind of norm-referenced testing makes sense for placement purposes, but for achievement testing, shouldn't the professor design the test to sample broadly from the types of knowledge and skills that were taught in the course and then take whatever distribution falls out of that?

[Jon]

The department puts pressure on adjuncts in particular to give low grades. The students satisfaction comes from high grades and student interest is important to holding an adjuncts position.

Ergo, the department is prescribing a distribution.

You can be darn sure the incentives are pretty strong to deliver.

[Frances Woolley]

Brett:

"shouldn't the professor design the test to sample broadly from the types of knowledge and skills that were taught in the course and then take whatever distribution falls out of that?"

Yes and no. Yes, one should design the test to sample knowledge from the types of knowledge and skills that were taught. And, no, one should not take whatever distribution falls out of that.

Here's an intermediate micro example. Students are expected to be able to derive a demand function given a utility function and a budget constraint. But some utility functions are harder for students to work with than others. A Cobb-Douglas utility function will give a class average of 75% or higher - though one can mess it up a bit by writing it as a log function when you've always done it in class the other way, or vice versa. Perfect substitutes and perfect complements will routinely generate much lower class averages. And if you really want to mess with your students, you can give them u=x^2+y^2 which looks like cobb-douglas but isn't.

There are many different ways to design tests - as long as the prof is designing the test and setting up the marking scheme the prof has a great deal of freedom to determine what the class average will be.

[Frances Woolley]

Avon: "Move to 100% final exams, no marked assignments or problems sets"

In Ontario, where we have a Liberal government, there has been a push in the K-12 world for separating out "behavioural outcomes" from "learning outcomes." In this world, giving marks simply for completing an assignment would be a bad thing, because that's a "behavioural outcome" not a "learning outcome."

So you can take comfort from the fact that your dislike of my approach is shared by many of those in the educational establishment.

At the same time, I'm not sure if you've really got the point here. The point of this post is that even with marked assignments, profs can easily easily set up the marking scheme so that students' grades are, effectively, largely determined by the final exam. Unless a prof is not particularly good at setting up exams, there's no real difference between 90% of grades on exams and 100% of grades on exams.

Re midterms: people learn better when they get immediate feedback. More feedback means more learning. A 100% final takes away many of these opportunities for feedback, so t's a bad idea on for pedagogical reasons.

[Majromax]

@Brett:
> shouldn't the professor design the test to sample broadly from the types of knowledge and skills that were taught in the course and then take whatever distribution falls out of that?

Define "achievement testing" in the context of a university course?

For most courses, the list of material to learn is fairly long and there's no obvious real-world benchmark for success. Learning calculus isn't like building a bridge, where you can test the failure strength of the students' project, nor is it like memorizing a list of nations and their capitals, where success and failure are fully binary states.

You could ague that "achievement testing" means that a grade of x% means the student understands x% of the material, but that just pushes the question back by a single step. How do we measure conceptual knowledge as a percentage?

Squaring that circle, we could go so far as to say that we can *assign* weights to curriculum items, and in so doing create a reasonably objective assessment scheme... and then we've just created a final exam.

[Avon Barksdale]

No, I understood your point – that you can create a marking scheme that largely determines outcomes through the final even with assignments. But, if the goal is to make the final largely determine the grade, then why not just use only the final exam in the first place?

The idea that universities provide pedagogy is the reason why I throw most of the CVs I get in the garbage. These people can't learn on their own. They take little initiative and almost no responsibility for their own learning. 100% finals separate those who should get a university degree from those who shouldn't. Using 100% final exams would shrink the university population dramatically; only the serious students and those who wanted to learn would be there. It's fine to use your professor as a resource, but the university is not there to grade homework. But, today, a university degree is more of a signal and government redistribution to the middle class than a reflection of an independent learner. It makes my job a bit harder, but I just keep throwing most of the CVs I get in the garbage. It doesn't fool some.

[Majromax]

> But, if the goal is to make the final largely determine the grade, then why not just use only the final exam in the first place?

Because as it turns out students are simple creatures, and they respond to the incentive of trivially-marked assignments by actually doing the assignments. In turn, that means that they pay a more consistent amount of attention in class throughout the term.

This is a win-win. The students have an official, formal feedback mechanism during the course, so interested students can learn more. The instructors also tend to have a more equal workload; a 100% final weighting will result in long office-hour lineups in the week or so prior to the exam.

The upshot is then that for any targeted grade distribution, students will actually know more with a mixed weighting than with a 100% final weighting.

> 100% finals separate those who should get a university degree from those who shouldn't.

That contradicts your point. If as you posit any particular grade distribution is achievable regardless of weighting, then 100% finals don't separate anything.

> It's fine to use your professor as a resource, but the university is not there to grade homework.

Whither marks at all, then? A 100% final is just homework under a restrictive setting, and the precision implied by numeric marking supposes far more discernment than is achievable if we posit it as just some Platonic "achievement" measurement.

[Frances Woolley]

Avon - what Majromax said.

[whitfit]

Avon Barksdale says:

"only the serious students and those who wanted to learn would be there. It's fine to use your professor as a resource, but the university is not there to grade homework."

I am not sure what field you are in, but where I work, a lot of learning happens through feedback. When I started in my field, my writing would be marked up like a wall of graffiti, and that is how I learned how to do what I do effectively. Why shouldn't there be feedback in school? Institutionalized feedback through midterms and assignments can help with this (especially since most university final exams never go back to the student, so provide no feedback beyond a number grade).

It is absurd to think that learning is just a question of being serious enough. It also requires people (professors, superiors, clients) who will critique your work and suggest how it can be improved, among many other things.

[Patrick]

FWIW, during my incarceration at McGill engineering in the early 90's, it seemed to me that the Profs dealt with the rampant 'divided and conquer' of homework assignments by making them long and really hard (relative to the questions they intended to ask on exams). God help you if you were a bit of a loner. You'd probably ace the final, but it didn't always make-up the difference if the assignments were a big chunk of the total. For this reason, less weight on the assignments always appealed to me, but we rarely got to pick and choose sections, and if we did the Profs usually agreed to use the same marking scheme.

[Ryan V]

As a student, I always appreciated courses that gave homework and midterms, but would ignore those marks if they were lower than the final. What are your thoughts on those types of schemes, Frances?

Avon: 100% finals separate those who should get a university degree from those who shouldn't.

Majromax: That contradicts your point. If as you posit any particular grade distribution is achievable regardless of weighting, then 100% finals don't separate anything.

A prof may be able to achieve the same distribution with a 100% final, but the placement of students within that distribution could be very different.

[Frances Woolley]

Ryan V: "A prof may be able to achieve the same distribution with a 100% final, but the placement of students within that distribution could be very different."

All depends upon the variance on the assignment grades.

[Avon Barksdale]

The point of a 100% final is to eliminate those who don't have the discipline to do the learning on their own. Handing out or suggesting problem sets is fine, even with solutions, just don't make it make it mandatory. Let the student decide how much work she will put into the course. The nickel and dime approach to grades, to get them to do the work, is why I see so many incompetent graduates. They can't write, they can't think, and they can't learn on their own. As you said yourself Frances, your students can follow plans, but they can't come up with ideas. The initiative of independent learning and deep curiosity are the only solutions. It's the students that you hardly need to be there for, the ones who are so driven to understand that they will put in the hard yards in the library for every course. The 100% final keeps those students and dumps the rest.

But like I said, university is now an extension of high school, representing a transfer to the middle class. Of course the top 20% of the university will still do well both at school and in life, but they were the only ones who should have been at university in the first place.

[Chris J]

Avon: I disagree. I studied physics and what we learned on our problem sets was how to attack really hard problems. You don't
learn that on your own. You learn that by trying it, then working a bit with classmates, and then getting it and doing tbe problem again as youbwrite your answer. What I am really saying is the assignment is he way that the prof structures how the class works together.

Secondly, the assignment defines the standard. By getting graded assignments back, with scores, the expectations are clearly defined.

I have also been in courses where the final was brutal but if you did all your assignments you would at least pass.

[Avon Barksdale]

Chris,

I studied physics, too. In fact I have a PhD in physics. The professor can still give problem sets and assignments, even with solutions, but not make them mandatory. The professor can set expectations that way, it's no problem, and still keep the final 100% of the grade. You will quickly find out who is serious about learning.

[Nick Rowe]

Avon: I am in two minds about assignments.

One argument in favour: Think about signal processing. If we have multiple signals, each of which is noisy, but where the signals are not perfectly correlated with each other, we should put strictly positive weight on each of those signals. Even if some signals are noisier than others.

[Chris J]

Avon,

Excellent. A fellow physicist who came up with what is without question the best handle in the comments thread.

[Chris J]

But on a more serious point... the assignments allow the prof to create two parallel grade schemes. If you do well on the assignments, you will do ok in the course even if you bomb the final.(Queen's prof quoted in student newspaper: "I just set the final exam. You guys got no chance in Hell." The prof had promised that a genuine attempt on the assignments would allow a pass. This was a fourth year nuclear physics class not Intro to Mechanics.)

Having assignments allows the prof to prep a hard final.

Assignments allow profs to assign problems that take 3 to 6 hours to do. You can't put problems like that on exams.