« Wicksteed, stocks and flows | Main | Visualizing GDP Performance »


Feed You can follow this conversation by subscribing to the comment feed for this post.

Francis,.... Stop correcting papers and take some time to go out to a pub......

Sandy - Now that's a sensible comment!

If both skills are useful and relevant to the particular class, then it would make sense to have both types of test items on the test.

Some exam questions have irrelevant information included (like including a number that isn't needed in calculating the answer). I think that's sometimes a good thing, because the real world contains lots of information that is and is not relevant, and it's an important skill to decide which is which.

I wonder if the "realistic" exam question 1 could be understood in those terms?

(And all I can think of is that very old episode of Family Ties:

Q: "I don't understand what you are saying Prof."

Econ Prof: "Well suppose I produce a shirt, and I'm trying to decide what price to put on the shirt..."

Q: "A shirt! What colour is the shirt?"

Econ Prof: "Oh, the colour of the shirt doesn't matter..."

Q: "The colour of a shirt doesn't matter?!! Professor! You don't know anything about economics!"

Definitely formulation 1. At least if your goal is to educate students so that they can apply economic principles to the real world. It doesn't matter what gooseberries are.

One of my friends (a math post-doc who was educated in the UK) has been complaining lately that exams here are formulated to test how well the students remember what they were taught, instead of testing how well they're able to apply the principles they were taught. Particularly, his complaint is that in the UK, you essentially get a minimum pass for remembering, can get the equivalent of a B by being able to combine the principles together in fairly obvious ways, and can only get an A if you can combine them in non-obvious ways. Here you can get an A just for remembering everything.

If someone is getting hung up on the gooseberries, they didn't learn the principles they were taught, even if formulating the question is a more formal theoretical phrasing might jog their memory.

As a student, one gooseberry problem is fine but after 30 of them it gets easier and easier to make a stupid mistake because while looking back the problem while doing part d you misread something. The second problem helps avoid that mistake and test whether or not the student actually knows the concepts he should know. Adding extra fluff to the problem just makes it easier to make a stupid mistake and doesn't really test whether the student can apply concepts to the real world like previous commenters have suggested.

Nick - excellent!

Neil - my gut response is to agree with you - but doesn't an application (e.g. farming, sports, canoeing) that some students are more comfortable/familiar with than others bias the exam in favour of some groups of students?

Joseph - does the second approach really test the concepts better? E.g. the second question specifically identifies K as an input, so the student doesn't have to think about what an input to production actually is. My worry is that a good percentage of students would be able to answer the second type of question perfectly and still not have any sense of how value added taxes worked in the real world.

Either one is ok. Depends on the purpose of the question. My population ecology exams typically include some questions based on real world examples. For instance, I might show students some real population dynamic data, along with some background information on the species, and ask them to develop two alternative hypotheses regarding the mechanisms driving the dynamics and an experiment that would distinguish those hypotheses. With these questions, I'm testing the students' ability to interpret real world information, and apply general concepts to specific, real world situations.

I'll also ask some questions based on fake examples, often based on the ecology of a made-up species like jackalopes or the Loch Ness Monster. With these sorts of questions, I'm typically testing the students' understanding of a particular concept, and I want the students to focus on that concept without getting confused by irrelevant details. If I asked such questions using real species, there'd be a risk that students would try to draw on their background knowledge of those species to answer the question, and so would end up trying to answer a question other than the one I asked.

Good point. This is partially why I feel like exams are such a poor way of testing a lot of the concepts that we would want to test. If the object is to test if a student specifically knows how to apply VAT formulas through a series of production then two is probably better. I don't know if one really tests how to apply VATs in real life either. Often the situation isn't that cut and dry. To my eyes, I can instantly translate problem one into problem two, but I know quite often it is not that easy. Perhaps a slightly more involved version of problem one would be better but of course that limits the number of problems one can ask on the exam.

The classes that I always felt I got the most out of were ones that had rather involved problem sets that don't always have cut and dry answers to everything but gave you a good idea of how complex the real application of an idea could be, and that paired these problem sets with exam problems that were much simpler like problem two.

Some finance/accounting guys sit in the cubes next to me (there's another does of reality for you). Based on the conversation I hear over the cube walls I'd venture to say that the question is irrelevant. In reality, what really happens is: try to get SAP to generate the report that calculates the taxes only to find it won't print then call corporate IT in [insert far off city] and wait a week for the SAP consultant to fly in at $300/hr + expenses so she can fiddle with [insert arcane configuration setting] meanwhile have lots of meetings on how to 'manage expectations'. Don't forget to order in sandwiches ('cause your so dedicated to be having meetings over your lunch break!). ;)

I think it also depends on the type of student. In introductory graduate courses in biostatistics (ie, for not particularly quantitative health-sciences folks) we were trying to teach them to go from a problem to the statistical formulation, so type 1 was better. It was important, however, that the numbers and variables in the question really were realistic, so that any knowledge the students had about the setting didn't mislead them. That's both because it detracts from the question and because it makes them grumpy and inclined to give low evaluations.

For example, if it's likely that any of the students are gooseberry growers, you would want $20 worth of fruit to be at least a roughly plausible first-year crop from four $25 plants.

In this particular example, I think that question 2 is vastly better? Why? Because I did not know what gooseberry is, and most importantly - I do not know if gooseberry bush is considered as food in canadian tax system or not. For instance change the example to "rye seed" as input and "rye flour" as output. Good luck.

If you want to simulate real world environment, then create exam questions where people at least have access to the internet so that they may check some facts they are not sure about. Otherwise, I would recommend to stick with conceptual exams and test their real-world skills using seminar works and such where students may work on them independently.

JV: This is off-topic, and irrelevant to your perfectly valid point, but I can't resist. One of the neat things about Canada, for me, is that I can buy rye bread (and sometimes even rye flour) in most supermarkets. (I picked up a taste for it in Germany and Austria years ago).

Thomas: "it makes them grumpy and inclined to give low evaluations." True enough. The classic example is the "True, false uncertain: If A then B" type question, which inevitably generates a large number of "Not A" answers.

J.V. Dubois: "I do not know if gooseberry bush is considered as food in canadian tax system or not."

This gets right at the heart of the issue. My guess is that a significant percentage of the students think like you do. With the exception of herbs such as rosemary and sage, *bushes are not edible*. A berry-producing bush could not possibly be considered food, because it can't be eaten.

Sitting here writing an exam, it is impossible for me to imagine that there are people who think "gooseberry bush. I could eat that, it must be food." Notice that it's not necessary to know what a gooseberry is, only what a bush is.

You're right, rye seed is far more ambiguous - are the seeds edible or not? They might not be, if they've been commercially treated with various stuff. As was noted in an earlier thread, there is an exemption to the exemption for rabbits - they are food, and thus not taxable, except when they are sold as pets.

There is another practical problem to consider. I need to generate a certain percentage of As, Bs, Cs, and Ds. In order to get some Cs and Ds, I have to have some questions that a significant percentage of students fail to answer accurately.


What kind of mark would you give me if my answer was that farmer Rose does everything off the books, pays for the bush, and is paid for the berries, in cash, and doesn't pay, charge or remit HST or claim an input tax credit?

Do we have to add the assumption that Farmer Rose complies with the law?

I think JV's point is a good one. The risk with using a "real world" example is that some smart-ass student might come-up with a technically correct "real world" answer that misses the point of the exercise. So, they might say, assuming Farmer Rose is a "small supplier" for HST purposes she pays x on the bushes, collects nothing on the berries and doesn't claim an ITC. It's a technically correct answer (given the assumption) but evades the point of the exercise.

I'll give you a real life example. I took an accounting tax course a few years ago, and we were given a hypothetical problem and asked to work out the taxpayer's tax liability. The point of the exercise was to take us through the various rules to see if we understood them. Unfortunately, they'd been using the same question for 4 or 5 years without changing the date. While my accounting colleagues dutifully worked out the supposed liability, My answer was simple: "nothing, the limitation period expired 3 years ago". That was a perfectly correct (if lazy) answer, and one which completely (and intentionally) missed the point of the exercise.

Bob Smith - If someone said something about small suppliers I would be absolutely thrilled and delighted! Though it would be a foolish student who didn't say "If Farmer Rose is a small supplier, then... if he is not, then..." State your assumptions, and work through the question based on those assumptions. That's almost always acceptable. And good for you on the accounting question - that's totally fair.

The bigger point is: what general knowledge can one assume in a third year university course? To what extent is my job to teach students about life in general as opposed to economics as opposed to accounting?

I can sympathize Prof Woolley. My personal preference is for a Q1 style. My background is engineering and I think it might be easier for us to generate real world examples that don't leave someone confused. At the core, I think critical thinking is the key and IMO, the real questions are better at making people think. The challenge is that the questions are harder to formulate and harder to grade. For example, even without knowing, students could have made some gooseberry assumptions and provided an answer to your question. The problem is that their assumptions may lead them to an analysis different that you had hoped to test them on. Obviously with this range of answers, marking becomes a super bear. Assessment is hard. I very often found that test results were not in line with what I perceived as the students level of knowledge. But, a 2-4 hour interview with each student is hardly practical. I think it is true that in the real world there is a lot of hand waving and approximation, but that shouldn't stop is from trying to pass along the underlying theories.

We are inundated with so much information these days, that critical thought is more important than ever. Yet, it seems to be less present than ever. Keep pushing them Prof Woolley. They may not thank you for it, but they should because they'll be better for it.

As an aside, I enjoyed this TEDx Warwick talk from Dr. McMahon on teaching and learning.
- Michael Boudreau

Frances: Just for the fun, I was trying to find how gooseberry bushes are treated in canadian tax system. And I must say that I failed. The closest I get to truth is reading this memorandum: http://www.cra-arc.gc.ca/E/pub/gm/4-4/4-4-e.pdf And I must say that it is unclear on the topic, as fruit tree are not mentioned as being taxed or exempt from tax. By the way, rye seed is also a little bit unclear. It is clearly exempt from the tax - unless you pack it in small packages to be used for feeding pets, such as birds.

However while searching the internets I found UK tax law, that in section 3.5 clearly states that gooseberry bushes are zero rated: http://customs.hmrc.gov.uk/channelsPortalWebApp/channelsPortalWebApp.portal?_nfpb=true&_pageLabel=pageVAT_ShowContent&propertyType=document&id=HMCE_CL_000133 So for all UK economic proffesors out there, do not use this exam question because the answer may be possibly quite different.

Also, would you take an answer like "If gooseberry bushe is taxed as food then X otherwise Y."? Also, this kind of answer may easily rob you of time, not to mention other possible effects - such as causing stress and panic, or fool you outright because you think it is just some mind game.

Bob: There is quite a lot of possible "smart-ass" answers even in . There were some internet memes about them. I liked this one: http://www.flickr.com/photos/ovc/2265572618/

Also, there can be very nasty questions - like asking you to solve an equation without solution. Answer that there is no solution can be an easy one, unless you get some students trying to find an error in their arithmetic process until they run out of time that would help them solving other questions.

As a journalism student studying economics, and having written this exact exam earlier this week, I prefer the first type of question, hands-down. I don't think the second type of question is necessarily harder - if anything, it's easier, as it clearly defines all the terms as I learned them in the textbook, so it's just a matter of remembering how to connect the dots.

The first type of question forces me to apply what I learned to a "real-world" example - and seeing how I hope to use my economics knowledge in the real world, rather than just on economics exams, it's a better measure of my understanding and application of the concepts. Plus, it's far more interesting - frankly, I think a big reason why so many journalism and arts students are turned off economics is because it's perceived as the abstract study of dry equations about input prices and VAT rates, rather than as a lens for understanding why the world works the way it does.

I can't speak for someone planning on studying economics or accounting in an academic setting, but for my purposes I'd much rather think about gooseberry bushes.

I just think question 2 is a little easier to read because the symbols K and X stand out more than "gooseberry bushes" and "gooseberries".

Frances: "I need to generate a certain percentage of As, Bs, Cs, and Ds. In order to get some Cs and Ds, I have to have some questions that a significant percentage of students fail to answer accurately."

That is perfectly reasonable requirement. All I want to say is that I do not think that it should be based on irrelevant topic. Actually, with your gooseberry question you may achieve the opposite effect. People who know that rye seeds are exempt from tax even if they are not edible may have hard time deciding if gooseberry bush is exempt or not. People who have no idea about how the tax system works and just think about the issue like just from the information available in the question, like "bush is not edible, thus it is taxed with zero rate" will have it easy. I do not know if this is the way to construct a question for smart people.

OMG, it should be "bush is not edible, thus it is taxed with non-zero rate".

I vote overwhelmingly for Q1. The main reason I hated physics class was that I got sick of reading problem statements like, "a mass of M kg is in motion at X m/s down a plane of R degrees with friction Q". If your math skills are not highly intuitive, and mine are not, this gets mind-numbing. I know econ students are all brilliant at math, so perhaps that isn't an issue for them.

On the other hand, Q1 would get me thinking about gooseberry fool, in which case I would likely lose focus and get the answer wrong.

Personally, I think testing both would be the perfect situation because it tests both their ability to handle things in a real world context and also tests their ability to deal with the abstract world. However, if I had to pick one option it would be the first version.

Q 1 is better from a cognitive point of view.
Human brain is not really wired for abstraction.
Up to very recently, we had to reason on the lines of "Grunk hungry. Grunk sees zeeba. Grunk kill zeeba. Grunk eat zeeba. Grunk happy".
Explain at the outset what a gooseberry bush is and whether it is an edible under the law to avoid confusions arising from ignorance of the gooseberry bush concept. I teach a course in International commerce ( not Int'l economics but commerce)and each time I explain that under Customs regulations, a wrist watch bracelet is not a bracelet but a watch, everybody is non-plussed...). But the brain react better with real life.

Nick: whatever you and Frances say defines what's on topic so: you need rye bread to really enjoy Montreal smoked meat. Ever been to Schwartz's?

Shangwen: "I know econ students are all brilliant at math" - as Joel mentions earlier, the course that this exam is from is one that is taken by students from journalism, our public affairs and policy management program, etc., as well as econ majors, so not all of the students are good at math. There's a huge variation in the skill sets students bring to the classroom.

Gooseberry fool - yummmm..... Not commonly eaten in Canada, but I absolutely love it.

Jacques Rene - Schwartz's, yes, years ago, fantastic. Also Ben's at least once, but that's another story.

I agree in principle that it's a good idea to fill in the gaps, and not disadvantage students who are unfamiliar with particular terminology. That's a vital part of creating a fair and equitable evaluation process. But what are the limits? Is it fair to assume, for example, that people know that New Brunswick is a province in Canada with a high unemployment rate? Is it fair to assume this if New Brunswick has been mentioned in class a couple of times?

At the end of the day, people gain from learning what a wrist watch bracelet is.

@Patrick - SAP and its competitors are necessary for figuring how much you bought, how much tax you paid, in what provinces, which sales were zero rated. And what taxes you collected, on how much in sales, how must to exempt entities (provincial governments and their agencies that don't use HST, status "Indians" on reserve...), through which corporations (even small companies are often 8 or 10 separate corporations with different GST numbers), etc, etc.

But it doesn't just spit out your tax return. Once you get the numbers out, you have to figure out the tax implications. Your data will be wrong because the data entry clerk that punched in half the invoices didn't take Frances' econ class, and included the GST as part of the purchase price. If your system is well configured and your staff well trained, you'll be able to run two or three reports and make them balance. In reality, you'll probably start with that, and then run 10 more reports to figure out why you don't appear to have collected enough GST to cover off the amount you sold.

It's fun. But don't think that theory is irrelevant in the age of computers. The computer just gathers data, the accountant turns it into something meaningful.

I like real-world questions on assignments; that way the students can learn the concepts through concrete examples. On the exam you need to use culturally non-specific examples, apply trees and apples, maybe, so your foreign students aren't tripped up.

Chris J: "On the exam you need to use culturally non-specific examples, apply trees and apples"

Apples are culturally non-specific? That's a bit of a Euro-centric perspective!

Actually, Chris J, I'm really glad you posted that, because it shows just how hard it is to know what is culturally specific and what isn't.

Guys, I think that you did not get the point. In most countries any and all plants, trees and seeds used to produce human food or that are used to feed livestock are zero rated. In most countries anybody who would think that gooseberry bush is taxed by non-zero would have gotten the answer wrong. And it is quite possible that it is actually true for Canada at least under some circumstances that were not mentioned in the actual question.

It is not about cultural/language difference, like somebody not knowing what bush or apple is. It is about country specific answer. Also it is not about "smartass" answers. It is about errors stemming from genuine an legitimate uncertainty artificially brought into the exam question. For me, clearly stating if gooseberry bush is taxed or not would solve the issue. But then what you did is that you just selected different names for variables. Instead of "producer" you used word "farmer Rose", instead of input "K" you used "gooseberry bush", instead of output "X" you used "berries" and instead of Canadian Revenue Agency you used tax collection agency.

I studied the science, so I can live with any name for the variable. What is confusing is if you want to assert certain hidden attributes to variables based on their name. In math it does not matter if the name of the variable is "X" or "Y" or "input_variable". However it matters if it is only you that know that there is hidden convention that "X" means real number and "input_variable" must be positive. In your example it is as if in some previous example you established that farmer Rose is bad ad calculus and he always makes specific rounding error. Or if you say that effectivity of tax collection of Canadian Revenue Agency is 95%. This is the sort of potential error that you can bring unintentionally into any example if you use "real world" examples.

J.V. Dubois - excellent point, and would have been an issue with the rye seed/rye example. However in Canada living trees - and presumably bushes also - are taxable:

However, some agricultural products are not zero-rated. These include supplies of goods such as cut flowers, foliage, bedding plants, sod, living trees, firewood, fur and animal hides, feathers, down, processed wool, maple-sugar candy, gravel, stones, rock, soil, and the urine from pregnant mares, which are taxable. See http://www.cra-arc.gc.ca/E/pub/gm/4-4/4-4-e.html#P101_4444

The more serious issue is that gooseberries and currents in general are banned in some parts of the US (and Farmer Rose might not want them either) due to the possibility of spreading rust to white pine trees. (Something I only learned a few days ago, when researching gooseberry bushes).

J.V. Dubois - the more general point is that, unless a question is basically a purely deductive logical exercise, there will be uncertainty - assumptions will need to be made and stated.

I had my share of strange questions. In the end it boils down to meta thinking like - "I have only 5 minutes to solve this question. There is no way teacher actually meant for me to incorporate this outside "meta" knowledge into the example." So now how real world this is. And another fact is that exams are extremely stressful. You need to come up with correct result relying only on artificial conditions set before you. And of course every exam can decide about if your several-year effort will be rewarded by making it to prom. I know about people who got it wrong because they were distracted by squeaking fan or fly in the room. In such an environment it is so easy to be distracted by this whole meta game of what professor actually meant by this question.

PS: I had this image in front of my eyes during the whole conversation. And now I found it. I do not know if it is real, but it is hilarious: http://funnyexam.com/answers/9541-love-the-teacher-s-wtf

PS2: Nick: Completely agree on the rye bread there. Rye flour is pretty common in Austria and there is no match for a homemade bread made from pre-ferment half rye half wheat flour that my wife bakes. Best bread I have ever eaten and I did my share of travel.

A bit more about using real world examples: I had no problems with algebra or functions & relations. Calculus was another story. I could work my way through the mechanical steps of calculating the derivative of an equation and what not, but I found it hard to absorb because neither the Q-year refresher I took, nor the first-year course I fought my way through ever gave a concrete real world example of why I would want to do this. It wasn't until a third-year stats course I had to take that I encountered a real-world practical application at which point it all made sense.

So, teaching concepts is fine but you need to tie it to the real world at some point.

"I need to generate a certain percentage of As, Bs, Cs, and Ds. In order to get some Cs and Ds, I have to have some questions that a significant percentage of students fail to answer accurately."

That's interesting framing. You could well have written, "I need to generate a certain percentage of As, Bs, Cs, and Ds. In order to get some As and Bs, I have to have some questions that a significant percentage of students fail to answer accurately."

What I like about the applied question is that sub-part 'd' refers to input tax credits. The word 'input' is never mentioned anywhere else. So that really catches an extra-level of understanding, if the student has it.

There seem to be two concerns: whether someone lacking in domain knowledge will be disadvantaged by the first type of question, and whether the question might accidentally introduce legitimate domain-specific complexities that disadvantage multi-disciplined students. I'm mostly going to talk about the latter.

The thing with these types of "critical thinking" problems is that they really intend for students to "critically think" along a prescribed path. For this reason I will refer to them as "obfuscated regurgitation" problems. As long as the students are "mature" in taking these sorts of questions and understand what's expected of them, this should be fine. When I took first year macro there was an exam question along the lines of "explain why [some country] was mistaken to intervene to increase domestic food production." I understood what the question wanted from me and regurgitated from the correct part of the first year macro lecture notes. Someone less "mature" in the academic setting might have misinterpreted this obfuscated regurgitation problem for an actual critical thinking problem. Such a person might have started talking about food security, dumping, susceptibility to global commodities markets, or specific real-world examples. Consequently they would have gotten the question wrong.

I think either type of question should be fine as long as you introduce everything earlier in the course. If you want to talk about gooseberries in the exam, then just make sure they appear in the homework first. If you want to ask an obfuscated regurgitation problem, then make sure the student have had such a question asked and marked beforehand so they know what's expected of them.

Update: I now have the exam papers back from the TA, who marked this particular question.

The beauty of this question - either in formulation 1 or in formation 2 - is that it doesn't matter what the student puts for (a) and (b), it is still possible to get (c) and (d) correct.

The answer to (c) is: answer to (b) minus answer to (a); if negative, enter zero.

The answer to (d) is: answer to (a).

People did pretty poorly on (c) and (d). In part this is they were tripped up by the idea of remitting zero, when I'd specifically said "remit=write a cheque for". (But if I hadn't said remit=write a cheque for, then there would have been issues because people wouldn't have remembered what remit means), and in part (even though I'd done similar examples in class - though not one with a zero rated output) it's really difficult to grasp the mechanics of how a value added tax works unless you've actually sat down and done a GST or HST return.

Now back to the marking marathon...

The comments to this entry are closed.

Search this site

  • Google

Blog powered by Typepad