One way to measure students' satisfaction with their educational experiences is to ask graduates, if they could choose again, if they would select the same program.
Canada's National Graduates Survey (NGS) has been asking some variation on that question since 1982. The questions asked, and the percentage of students saying that they would select the same field of study again, are shown in the table below. Although the NGS includes both college and university graduates, for ease of interpretation the table below shows results for university graduates only.
| Year | NGS question, asked two years after graduation | Population | Percent choosing same |
| 1982 | "Given your experience since completing the requirements for the ____, would you have selected the same educational program, a different program, or not taken any post-secondary program?" | University grads n=13,718 | 71.8% |
| 1990 | "Given your experiences since completing the _____ program in 1990, would you have selected the same field of study or specialization? | University grads n=18,480 | 76.0% |
| 1995 | "If you could choose again, would you select the same field of study or specialization that you completed in 1995?" |
University grads n=20,848 |
73% |
| 2005 | "If you could choose again, would you select the same field of study or specialization that you completed?" |
Bachelors or higher n=10,313 |
80.5% |
| 2013 | "If you could choose again, would you select the same field of study or specialization that you completed?" (Note: asked 3 years after graduation) | Bachelors or higher n=10,382 | 76.9% |
| Calculated by F. Woolley from NGS PUMFs |
This looks like a good news story. The percentage of graduates who would choose the same field of study again has been holding steady, or even rising over time. So universities must be doing something right! Wrong.
The 1982 NGS question gave graduates the option of selecting an entirely different program, or not taking any post-secondary education. Framed that way, only 71.8% of graduates would choose the same program again - a lower percentage than found in any subsequent survey. From 1990 onwards, the question was framed much more narrowly, in terms of choosing the same field of study or specialization. The options of not taking higher education or selecting an entirely different program were taken off the table, and the percent who would choose the same again bumped up.
Another, more subtle change in the NGS happened between 1995 and 2005. Up to and including 1995, the "if you could choose again" question came at the end of the questionnaire, after respondents had been probed about their employment history, subsequent educational experiences, and primed with questions such as "During your program, how satisfied were you with the availability of your professors outside scheduled class time?" In 2005 and 2013, the "if you could choose again" question was asked right at the beginning of the survey, after a series of neutral questions about, for example, distance education. Again, the portion responding that they would select the same program increased. I hypothesize that graduates' reported satisfaction with their field of study is higher before they are probed deeply about their university and post-graduation experiences.
When I see a series of changes to how a question is asked, like the ones introduced in the 1990 and 2005 NGS, and each change is associated with an bump up in the number of positive responses, I get suspicious, and start asking questions. Who influences the NGS survey design? Would they benefit if graduates appeared to be satisfied with their education?
According to Alex Usher, the NGS is paid for by Employment and Skills Development Canada. Although ESDC could have influenced the design of the "if you could choose again" question, it's not obvious why they would want to. The design of the NGS, like every Statistics Canada survey, is also influenced by consultations with various user groups, but I have been unable to find out any specifics about the NGS advisory groups. Perhaps somewhere there is a political science or public administration student looking for a thesis topic - if so, the politics of statistics would make a great one.
The above table also shows a drop off between 2005 and 2013 in the number of graduates responding that they would select the same field of study again. Does that reflect a real change in student satisfaction, or is it, too, an artifact of the data collection process? The question did not change between the 2005 and 2013, nor did the positioning of the question on the questionnaire. What did change, however, was the amount of time that elapsed between when the respondent graduated and when they took the survey. The 2005 survey was carried out two years post graduation; the 2013 survey was carried out three years afterwards.
This has two potential impacts on the student satisfaction question. First, as time goes by, graduates gain more information about the value of their education. Second, since people starting their careers move frequently, the longer window between graduation and the administration of the NGS survey might mean that more graduates could not reached by Statistics Canada. Indeed, the 2013 NGS response rate of 46.4 percent was considerably lower than the 2005 response rate of 65.3 percent (info taken from the NGS user guides). If graduates who are still living with their parents are disproportionately likely to be both (a) un- or underemployed and (b) in a place where the Statistics Canada can track them down this would explain the drop in satisfaction rates between 2013 and 2005.
It would be wonderful to have consistent time series data on the satisfaction of graduates with their post-secondary education. Unfortunately the National Graduates Survey fails to provide it.
Apropos of nothing much, I will take a moment to point out that some of the most desired new employees at my employer tend to be high school drop-outs. They're willing to work the hours we give them, and ask for more, whereas their student contemporaries are not. And we get them young!
Posted by: Erik Lund | April 05, 2018 at 02:29 PM
Erik - but that raises a whole load of questions about what demographics and technological change will do to the demand for labour, for graduates, and by extension that demand for education. A reporter asked me today, in response to reports of a shortage of workers in Iowa, "if there's only 2.5 percent unemployment, where will new workers come from?" From the education sector is one place.
Posted by: Frances Woolley | April 05, 2018 at 03:14 PM