"He's a smart guy, almost as smart as he thinks he is..." Review on ratemyprofessors.com
The ratemyprofessor.com website comes in for a lot of criticism. Some allege that the reviews are bogus. Others argue that it provides no useful information for students, just laurels for hot, easy teachers. Another common criticism is that too few students post for the site to give useful information. Moreover, those who do post tend to be exceptionally happy or excessively disgruntled.
While these criticisms have merit, official teaching evaluations are also flawed. Official evaluations are likewise drawn from a selective sub-sample of students: those who drop, withdraw or simply don't bother to attend lecture often do not complete the evaluation form. Professors who grade generously score higher on official evaluations, too.
Ratemyprofessors.com has one big advantage over official evaluations: it is publicly accessible. Students use it because, often, there is no alternative source of information. Yet how reliable are the reviews on ratemyprofessors?
The University of Toronto student union publishes all undergraduate professors' teaching evaluations in its Anti-Calendar. I took a small sample (22) of economics professors, and matched their ratings on ratemyprofessors with their scores on the university's official teaching evaluations.
The immediate challenge I faced is that the official evaluations and ratemyprofessors measure quite different things. The closest match was between the ratemyprofessors category "clarity" (1=confusing, 5=crystal clear) and the University of Toronto category "explains" (Explains concepts clearly and with appropriate use of examples. 1=very poor, 7=outstanding). Even in this small sample, there was a statistically significant (p=0.08) relationship between the two sets of scores.
The correlation between the two is not perfect. However what is clear from reading the anti-calendar is that one professor's evaluations will vary a great deal from course to course, and even sometimes for sections within that course (in doing these calculations, I took a weighted average of evaluations in all undergraduate courses taught). Overall, I would say that the ratemyprofessors "clarity" score gives some information about a professor's official "explains" score.
Another pair of concepts that one might expect to be correlated are the ratemyprofessors "easiness" score (1=hard, 5=easy) and the University of Toronto "difficulty" score ("Compared to other courses at the same level, the level of difficulty of the material is… 1=very low 7=very high"). The correlation between the two, however, is weak (r=-0.06) and one does not significantly predict the other:
I suspect that selection may be an issue here. The students who fill out the official form are the ones who found the course easy enough to stick with; the ratemyprofessors contributors include students who drop. This explains why the two are different, but raises the question: which one is more accurate?
Ratemyprofessors has a third dimension along which it evaluates professors: helpfulness. Unfortunately there is no category that is an obvious match for helpfulness - such as availability for office hours, or willingness to answer questions - on the University of Toronto teaching evaluation form. As an alternative, I decided to compare the ratemyprofessors summary measure "quality" - an average of clarity and helpfulness - with one of the summary measures on the University of Toronto form. That form has a couple of summary measures. I used "Retake" or "Considering your experience with this course, and disregarding your need for it to meet program or degree requirements, would you still have taken this course?"
There is not much a relationship between the ratemyprofessors "quality" score and the University of Toronto "retake" score. It could be that the ratemyprofessors scores are flawed, or it could be that these are not comparable measures of teaching effectiveness. I strongly suspect that the retake question is confusing, and hard for students to answer.
Thomas Timmerman's much larger study of ratemyprofessors finds "the summary evaluation correlates highly with summary evaluations from an official university evaluation." Brown, Baillie and Fraser also find a strong (r=0.5) between official measures of clarity and helpfulness and ratemyprofessors scores - although interestingly they find, like I do, a weaker correlation between the official and the ratemyprofessor ease/difficulty scores.
Ratemyprofessors isn't perfect, but if it's the only publicly available source of information about the quality of instruction, people will use it. The only way to diminish its influence is to provide an alternative.