The depressing job of grading exams

I have a very good job, and I'm well paid to do it. The only part of my job I don't like is grading exams. I have just finished grading 30 exams for my graduate course and 300 exams for my first year course.

A pile of 300 exams sitting on the table is just too daunting a pile even to begin. So I break it into three, and only put 100 on the table, and tell myself that's my job for the day. But even 100 is too never-ending. So I count off 10 exams, and tell myself when I have graded that 10 I can go and make a cup of tea, or check in on WCI, or get something to eat. Then I go back and grade another 10. That's the only way I can do it. That's my way of fooling myself into getting the job done. I know it's irrational, because the job takes longer when I have to count out piles of 10 as well as grade the exams.

I normally don't mind doing boring jobs, because normally when you're doing a boring job you can let your mind wander. But that doesn't work with grading exams. You have to concentrate on the job.

My job is to teach those students economics. Or to get them interested enough in economics that they teach themselves economics. Or a bit of both. So when they fail an exam question, that means that I have failed too. I think that's why it's a depressing job, because I'm forced to look at my own failure, again and again.

And that's one of the reasons I don't let the TAs do all the grading. The TAs grade the multiple choice and short-answer sections of the first year exam, but I grade the more open-ended long-answer/essay section myself. Partly because I think I'm better at doing it than the TAs. But mostly because I need to see all the details of where I have failed as a teacher.

I also get to see where I have failed to set a good exam question. That's depressing too. A good exam question generates answers that let me know exactly what the student does and does not understand. Sometimes weak students will write deliberately obfuscating answers, to try to hide their weakness. But sometimes a good student may miss the point of a question, or may fail to explain clearly enough what they understand. And the hardest part of the job is trying to figure out which is which, and try to give a reasonably accurate grade.

For example, when drawing the monopoly diagram, many students drew the MR, MC, and ATC curves all crossing at the same point, which could only happen by fluke. So when they show that point as determining the monopolist's profit-maximising quantity, do they mean Q is determined where: MR cuts MC (the right answer); or where MR cuts ATC; or where MC cuts ATC? And do they realise this could only happen by fluke? Maybe they think it has to be that way? Maybe they deliberately drew it that way to try to hide their ignorance? Or maybe they are good students who just drew it that way by fluke?

I tell them never to draw fluky diagrams. I tell them to draw big diagrams which reduces the chance of flukes. But some, especially the weaker students, draw small diagrams with a thick pencil, so flukes always seem to happen.

A good exam question should test a range of abilities, so that the very best students are stretched to the limit but the weaker students can still get part of it right. But that's not enough. A good exam question should also make it clear when the student is ducking the question. That's never a problem on multiple choice questions. It's less of a problem on closed-ended questions which have a right and wrong answer. But it's a problem on many open-ended exam questions.

Good exam questions are like a scarce natural resource. There are only so many you can mine, and you can't keep on using the best ones year after year.

But maybe the depressing job of grading exams is still better than the stressful job of writing them.

Comments

[Frances Woolley]

Nick: "So when they fail an exam question, that means that I have failed too"

This post could be called "Why all profs should grade their own exams." Seeing where one has failed, over and over again, is one way of learning one's weaknesses as a teacher, and thus becoming a better one. I find it also makes me more critical of the course material. Given a choice between concluding "I've failed again" and "this concept is fundamentally impossible to understand" sometimes I pick the later.

Have you thought about telling student what will be on the exam before-hand, or at least increasing the amount of information you give them? That might reduce the total pain burden of final exams.

Will you have time now to repsond to some of the comments about middle earth that are beyond what I can respond to?

[Nick Rowe]

Thanks Frances. Yep, grading your own exams is a form of feedback. It makes the course more interactive, in a strange way. It's too late to help the students who wrote that exam, of course, but it ought to help us improve to help next year's students.

I try to keep what is on the exam secret. I tell them the format, and what sort of answers I'm looking for. They can look at old exams. But I reckon studying for the final exam is part of learning, and I want them to learn everything, and I can't test everything on one exam.

You realise you have caused a major demarcation dispute by writing a money/macro post?!!! You are supposed to be a mIcro-economist! I am even more annoyed that it's a good post, with none of the usual howlers you get when microeconomists try to do macro/money. My normal theory is that macroeconomists can do micro (at least up to a point), but that microeconomists don't get macro at all. (Which is why so many intro texts are written by macro people.) Oh well.

[Edeast]

If you really want to increase the feedback and student stress levels, put an oral portion on the exam. My undergraduate molecular cell biology exam, had students leave for 5-7 min interviews during the exam. The range of topics was limited but the depth was the scary part.

[CBBB]

If you really want to increase the feedback and student stress levels, put an oral portion on the exam.

This is a very good idea. Exams are not that great of a way of finely ranking students. Sometimes a good student can make a careless mistake that isn't related to overall understanding and be severely punished (depending on the marker), or on the other hand sometimes exams are made a bit too easy or made in a way that helps out students who just memorize material without really understanding much of it.

[Nick Rowe]

That is a good point of oral exams. You can ask followups too. "Did you mean to draw the MC, ATC, and MR curves all intersecting at the same point, or was that just a fluke?"

They are very time-intensive though, which would be a very big problem with 300 students. And stressful all around. We do them for some PhD comp exams (as a supplement to the written exam), and for thesis defences, but not otherwise.

[edeast]

It definitely wouldn't work for first year.
I went to a small undergrad school, and it was a 3rd/4th year small class. I think she(the prof) did it to prep us for grad school. To make it manageable, choice of 5 or so topics known ahead of time, and you choose which one you'd be interviewed on. Surprise isn't as big part of it, but you really had to know/understand what you were talking about.
Philosophy was the other class which did something similar, more of a thesis defense against a peer and a prof.

[Jeremy Fox]

Yeah, same here Nick. I grade my own exams, for the same reasons you do, and I hate it, for the same reasons you do. I'm out of town for the holiday, back on Jan. 3, where a big stack of them are waiting for me to crank through before our registrar's grade submission deadline.

The only context in which I enjoy it is when I've changed how I teach something, and the students' exam performance shows that the change worked. A few years ago, I made a big change in how I teach introductory mathematical modeling to ecology students, ditching about 25% of my lectures in favor of having the students work in pairs in class on practice problems. It was nice to see that it actually worked. Unfortunately, it worked at the in-retrospect-predictable cost of causing them to get worse on questions asking them to define key terms. But the cost was worth it for the gain in their understanding of the material.

[Peter N]

Your method of splitting up the work is NOT a waste of time. Studies have shown,for instance, that judges give lighter sentences early in the day and immediately after lunch. You wouldn't want to be paper 99 in a continuously graded batch of 100.

[RebelEconomist]

I feel your pain, Nick. When I did that job, my wife used to find me rocking and mumbling repetitively as I marked! Three hundred scripts, and therefore probably 300 students, is probably too many. It always used to amuse me how economists would teach how a mixture of different types of consumption would generate the most utility, and then take on their own work in huge monolithic blocks in an attempt to get it over with!

Happy 2013!

[Ralph Musgrave]

I'm much in awe of people who battle on with boring depressing jobs. Those jobs HAVE TO BE DONE: whether its picking up litter, emptying trash bins or marking exam papers. Good on you. People who have interesting jobs don't deserve any pay at all.

[Jim Sentance]

I did a number of boring, repetitive jobs working my way through my undergrad, but nothing as mind-numbingly excruciating as marking exams. All I can say is thank god for Scotch.

[Jacques René Giguère]

Jim Sentance: December 1979: the first paper began thus: "This must have been a good budget. Minister Crosby worked so hard." Scotch. You can't do that job sober. One of these days, I'll ask my anesthetist friend for fentanyl...

[stewarjt]

It's admirable you grade 300 exams. Is a class that large conducive to student learning?

My idea of teaching and learning is that I let students know exactly what material I expect them to assimilate. Then I help them learn. The exam is designed to reward students who did the reading twice before class, prepared study questions and participated in class discussion, i.e., practiced the analysis. The test isn't a test of memorization and regurgitation. The true test of whether any student learns is if s/he can reproduce the analysis on an exam, in class or in office hours.

[Nate O.]

If you haven't already, make friends with your school's education department. "Teaching" is a very specific technical skill, unrelated to your field. Even experienced teachers could benefit from a better understanding of pedagogy "best practices."

No exam is 100% reliable or 100% valid, but there are ways to improve them. Fewer items make a less reliable test, so err on the long side. It might be helpful to give a certain number of "free" incorrect answers, to compensate for unreliable questions (unclear wording, mistakes made in lecture, etc.)

Avoid constructions like "none of the above" or asking negative questions. Ask for the right answer, not "which is not the right answer?"

Obscure students name when grading to eliminate bias. In some courses, I've received higher grades just because the teacher liked me, and thought I was a "good student."

Regarding the issue with the monopoly curves intersecting, provide a blank axis with scale numbers, so they have to draw it to the scale you desire. Make your expectation of the scale explicit, so they don't try to squeeze it in the corner. If that doesn't work, maybe provide blank diagrams and have them ID the correct one or something. Or provide a diagram, and have them interpret it.

[Ken Purdye]

I congratulate you on looking at all the papers yourself. I tried to do a part time masters and was failed in micro. When I asked to discuss with the prof, it became very clear that the TA had graded the paper. As a "mature" student, my answers were tempered with business experience, somewhat critical of the theory I had been taught. This clearly didn't appeal to the TA. The prof, of course, defended the process and the mark. I guess I went to the wrong university. Keep looking at the exams yourself. Please, for future students.

[Asif Dowla]

Nick:

I face the opposite problem. When I ask what will be the monopolist's profit where MR, MC, and ATC intersect, 50% of the class answers 0! I tell them to practice diagrams and draw them to understand the logic. Most of them don't do it. Heck, my own son was trying to study for Micro via flashcards. It made me so angry. During the thanksgiving break, I sat down with him and forced him to think through the cost curves. After he went back to campus, he sent me a picture showing all his rendering of diagrams. Even there I picked out an error, albeit a small one. His MC curve was linear instead of looking like Nike's Swish. I sent him a text message with the correction.

He aced his final, but not enough to pull up his grade to an A. To his defense, he was taking an honors section and this was his first semester of college and being away from home. He is taking Macro (honors) next semester. We plan to use Google chat on a weekly basis. Students don't know how to effectively study economics. It is our job to teach them.

[Steve]

Nick,
I use the ten-papers-followed-by-a-small-reward method as well. I wonder how common this is. I also start by grading the copy of the student who has asked the sharpest questions and/or pointed out the most mistakes during the course. It's usually one of the best two or three copies in the class. Puts me in a good mood and gives me the illusion that I'm getting through to some of the students, even if the best ones would probably do about as well with no input at all from me.

[Derek]

I find that to be consistent, it is best to mark one question on each exam at a time. I try to change the order each time though, so the same students are not always first (when I may be more likely to miss something) or last (when I just want to get done so I can take a break, lol).

[Frances Woolley]

Derek - absolutely - I find it's much faster, also I'm more likely to spot two papers or assignments that are suspiciously similar. Though it may not be as much of an issue for the kind of shorter questions that Nick's marking.

[Jack]

Is there evidence that open-ended questions are necessary in a field like economics? As a newbie professor, it doesn't seem to me that I can better test students using open-ended questions rather than a sequence of multiple-choice questions. At least for courses of 300, it seems a worthwhile question to ponder.

[Jacques René Giguère]

Jack: in my experience, introducyion students don't know enough to express themselves on open-ended open questions.

[Satnosh]

Humm,

When I completed grading for pile of 180 papers, The first that I came to my mind - is we should put more efforts in writing questions. As I realized that even good students, those who prepares well - also missed question. For open ended Question also - I felt that some hints need to be given about what is expected from the student. Otherwise they will try to write everything and anything.

Nice Post - loved it.