« Why is Canadian GDP Growth Higher Under Liberal Governments? | Main | Tatonnement: in Walras, PSST, and monetary disequilibrium »


Feed You can follow this conversation by subscribing to the comment feed for this post.

Hear hear. The job of civil servants is to carry out the orders of the political representatives. When told to conduct a voluntary NHS, they should do the best they possibly can given the resources provided.

However, I don't think that this obligation extends to carrying the 'communication strategy' water for the government. They ought not to be forced to say things that are simply untrue.

I don't know whether to hope that Mr. Smith actually does know these statements he is making are untrue, or that he thinks his job is to say untrue things to make his Minister more comfortable. Neither is terribly palatable.

Yeah. If (when?) the government decides that it will conduct policy under the assumption that the world is flat, civil servants can't publicly say otherwise. But that doesn't mean that a civil servant can go on to claim that there's no evidence that the world is round.

Mr. Smith has no place at StatsCan if he really thinks that "there’s no scientific basis for saying this is going to be fundamentally flawed." He is either incompetent, or corrupt. I'm not sure which is worse.

I hope that academics will organize and make a lot of noise, as a response. It's not gonna happen, but it should.

I guess based on probability theory, there *is* a non-zero chance that by pure randomness, the results of the voluntary long-form survey will accurately reflect the population of Canada. He's right that we can't rule that chance out.

Ben: And Stephen is right that anyone hinging their argument on certainty has no business being within miles of a position that has 'statistician' in its title.

Even if by some miracle it should happen to be the case that the response pattern to the voluntary census mimics, in every relevant statistical sense, the response pattern to a mandatory census, we would have no way of determining as much. We will have to treat the respondents to subsequent censii as unobservably different than respondents to previous censii.

Smith is right that the new versions will not be "fundamentally flawed"---we work with self-selected samples all the time. But that's also beside the point: the argument against the voluntary census is not that it will be utterly worthless, but rather that it will be inferior and incomparable to previous versions.

Smith's subsequent remarks suggest he isn't aware that the census is used for anything other than estimating frequencies of various ethnic groups, and he is wrong even in that limited and straightforward context to argue that the mandatory census was not a superior instrument.

I don't think it's just StatCan that has been infected with a does of political careerism amongst civil servants and as so well demonstrated by the Smith character. Check the meteoric rise of some officials in other key departments. Bright people but junior or middle grade and not Ass DM caliber. Good thing there is no really serious federal policy initiatives, other than destroying StatCan, being undertaken by what's left of the federal gov't.

Chris: That's a very good point. Even if everything does work out, there would be no way of knowing.

The Canadian Financial Capabilities Survey, that I'm working with right now, is a voluntary survey. The response rate is 56%. But there's fairly large interprovincial differences in response rates - e.g. Alberta, Sask have 61% response rates, BC has a 49% rate.

The Labour Force Survey, which seriously pursues its interviewees (and which people have some kind of obligation to answer), has a response rate of 90% (random self-referencing link thrown in to boost our google search ranking).


It appears to be a concerted attack: http://www.winnipegfreepress.com/breakingnews/study-claiming-crime-on-the-rise-gets-hung-out-to-dry-by-experts-in-the-field-116112184.html

Thanks for sounding the alarm, but, um, "'I' before 'E' except after 'C'" (seige)

Good lord. Fixed.

Please address his argument:

“There is no scientific reason why you would say before it even starts, before I see results, that there’s going to necessarily be a significant problem with the count of Inuit or Métis or immigrants beyond the levels we’ve seen in the 2006 census.”

I thought I did. All of the available evidence shows that there *will* be a significant problem.

There's no scientific reason why we would say even before it starts, that the sun will necessarily rise tomorrow. But it's still a terrible idea to conduct ourselves as though it won't.

He says there will be no greater problem with the count of Inuit or Métis or immigrants beyond the levels we’ve seen in the 2006 census. Please address that argument.

Hi Norman,

I did address that argument in a LTTE I submitted to the Globe on Sunday morning. May have missed their Monday paper deadline, or they may just have rejected it. In any case, here is what I wrote, for posterity:


Dear Editor,

Chief Statistician Wayne Smith is quoted in your article (Chief Statistician asked to rethink Census for 2016) saying there is “no scientific basis” for concerns about the reliability of the coming voluntary National Household Survey. Mr. Smith is wrong. There is ample evidence, from Canada and abroad, that voluntary surveys produce biased results when not corrected against a population benchmark such as a census. From this gathered evidence, we can induce that there is a strong likelihood of problems arising with the National Household Survey. Inductive reasoning has been a core tenet of scientific and statistical practice since at least the time of David Hume.

When given a task by our elected representatives, civil servants have an obligation to carry out the task even if it may be poorly conceived. However, whether this obligation extends to acceding to scientifically dubious assertions of the elected representatives seems to be an open question.

Actually, the choice of the Aboriginal/Métis example is a bit disingenuous. Previous censuses *already* had problems with those groups.

But for the other groups, we'll be introducing problems where they didn't exist before.

Good thing there is no really serious federal policy initiatives, other than destroying StatCan, being undertaken by what's left of the federal gov't.
Apart from the destruction of climate research? Apart from the destruction of efficient crime fighting strategies? Apart from...

In other words, while you don't like the examples he chose, you do not dispute his statement as quoted.

If the journalist misrepresented him here

"Mr. Smith is adamant, however, that critics cannot know for sure that the results from the optional long form, even for smaller sub-groups of the population, will be inferior to what was collected via the mandatory approach in 2006."

Then he's being disingenuous at best and deliberately obfuscating at worst. It the journalist correctly represented his views, then he's just as wrong as I said in the OP.

The statement "no scientific basis" is false. Induction is a scientific principle.

The statement "critics cannot know for sure" is literally true, but is not scientific. As I tweeted, these are weasel words befitting of a politician or lawyer, not a statistician.

Seems like the best way to save the census is to get the worst response rate possible.

If you love the census, don't fill it out.

"There is no scientific reason why you would say before it even starts, before I see results, that there’s going to necessarily be a significant problem with the count of Inuit or Métis or immigrants beyond the levels we’ve seen in the 2006 census."

Be generous to Smith. Agree with him that there are potentially significant problems that will appear when we see the results. This raises the question: why in the world did we switch to a system that potentially has significant problems from one that was sound?

Alan: The problem with that strategy is that it gives the government the perfect excuse: "It's not *our* fault; blame the boycott organized by our opponents."

Patrick: That's being *too* generous. At best, he's not being honest about the evidence on the sample selection problem. At worst, he's denying it exists.

"critics cannot know for sure that the results [...] will be inferior"

Imagine if that was the test of public policy: "We are going to do this unless you can *prove with 100% certainty* that it's going to make things worse."  Of all the ridiculous things to say...

The full transcript of the interview is now up at:

Is census data usable? ‘Our thinking has evolved,’ chief statistician says
Globe and Mail Update
Posted on Monday, February 14, 2011 4:57PM EST

Boycott or not: why would I fill it out? Takes time, makes no difference to me. In an election, at least my favorite party gets some extra funding, and even for parties who can't win, having a higher vote count makes a difference. I don't see what good it does for me to participate in a totally flawed survey. Seems like a waste of time.

So here's a soundbite from the CPC's newly appointed chief statistician on one of the methods for how the country might achieve at least its now, er, conservatively projected 50% response rate for the NHS:

"In the past, support has been expressed in the form of census beer. It's been painted on the side of trucks: census messages."

So, in light of the out-of-context line from one the CPC's new attack ads quoting the Lib. leader re: the Cndn. flag, one might be sorely tempted draw a mock-up of the envisioned "NHS beer ad" campaign, & quip:

"The NHS: a failing imitation of a census label."

Yikes. After reading the full transcript, I have to set aside the possibility that the Globe misrepresented Wayne Smith's views.

This is worse than I thought.

Right off the bat, he's dreaming if he thinks he's going to get a 50% response rate. Marketing people jump for joy if they get 20% response rates from the general public.

This might be the stupidest thing I've ever read:

"If the response falls to 50 per cent, there's one inevitable consequence of that. And that's exactly why the sample has been bumped to one in three from one in five. To compensate for that effect to the extent possible.

At a 50-per-cent response rate with a one in three sample, the sampling error would still be somewhat worse in the National Household Survey than it was in the 2006 census, the long form.

If the response is actually 60 per cent on the NHS it’s a wash. They are basically just as accurate, one as the other.

And if it goes to 70 pct actually the NHS sampling error is smaller than it was in 2006."

Just because the sample is larger, does *not* mean the sampling error is smaller. This was shown by Literary Digest back in the 1930s:


"In retrospect, the polling techniques employed by the magazine were to blame. Although it had polled 10 million individuals (only about 2.4 million of these individuals responded, an astronomical sum for any survey),[3] it had surveyed firstly its own readers, a group with disposable incomes well above the national average of the time (shown in part by their ability still to afford a magazine subscription during the depths of the Great Depression). The magazine also used two other readily available lists: that of registered automobile owners and that of telephone users. While such lists might come close to providing a statistically-accurate cross-section of Americans today, this assumption was manifestly untrue in the 1930s. Both groups had incomes well above the national average of the day, which resulted in lists of voters far more likely to support Republicans than a truly typical voter of the time."

The response rate was gigantic, but since the sample was biased, the results were garbage. Standard Stats 101 stuff.

Yep. This is extremely discouraging. If the Chief Statistician doesn't seem to understand the problem of sample selection bias, then we are all in deep, deep trouble.

Here's his bio. Like our illustrious PM, he's got a Master's in econ. He has been with StatCan for 30 years, but on the communications side of things. Which he doesn't seem that good at; towards the end of the interview, when his back got up over the crit. from the international community, he sounded just like a lot of Conservative bloggers, with his little judo move of twisting someone's words to deflect a legitmate q. ("So what you're saying is...")


Just finished reading the rest. Oh man. The bit Mike quotes is a real whopper. Like an undergraduate who didn't do the readings and is trying to fake his way through the quiz with nonsense that sounds like statistics.

After looking at his bio, my only thought is: Wouldn't we want the chief statistician to actually be a statistician?!?

This is madness. One talking point is how a mandatory long form is an invasion of privacy. Another is that we should consider a register based system where the government maintains vast comprehensive databases of personal information, which would be a far greater blow to personal privacy than a long form census could ever be.

He apparently doesn't understand or have any concept of cost accounting. To suggest that you can't reasonably attribute any cost to the NHS is ridiculous.

His own expert don't share his optimism. A friend pointed me to this. Save it now before it's gone.


Non-sampling error
However, it is believed that the most significant source of non-sampling error for the National Household Survey will be non-response bias. All surveys are subject to non-response bias, even a Census with a 98% response rate. The risk of non-response bias quickly increases as the response rate declines. This is because, in general, non-respondents tend to have characteristics that are different than those of the respondents and thus the results are not representative of the true population. Given that the National Household Survey is anticipated to achieve a response rate of only 50% there is a substantial risk of non-response bias.

Statistics Canada is very much aware of these risks and their associated adverse effects on data quality. The Agency is currently adapting its data collection and other procedures to mitigate as much as possible against these risks. In particular, we will be using data on response patterns from the 2006 Census and information generated during data collection in 2011 to guide our field follow-up effort to minimize non-response bias. As well, where possible, 2011 Census data will be used as auxiliary information in National Household Survey estimation procedures to partially offset some of the remaining biases. However there is certain to be some residual, significant bias that will be impossible to measure and correct.

Comparability of data over time
Any significant change in the methods of a survey can affect the comparability of data over time. There is a real risk that this will be the case for the National Household Survey. There will always and inevitably be an element of uncertainty as to whether and to what extent a change in a variable reflects real change or an artefact arising from the change in methodology from the mandatory long-form census to the voluntary National Household Survey.
The French version is even harsher:
Il est clair qu’on ne pourra jamais déterminer avec certitude si et dans quelle mesure la modification d’une variable est attribuable à un changement réel ou découle de la transition du recensement à formulaire détaillé obligatoire à l’Enquête nationale auprès des ménages à participation volontaire.

We have never previously conducted a survey on the scale of the voluntary National Household Survey, nor are we aware of any other country that has. The new methodology has been introduced relatively rapidly with limited testing. The effectiveness of our mitigation strategies to offset non-response bias and other quality limiting effects is largely unknown. For these reasons, it is difficult to anticipate the quality level of the final outcome.

The significance of any quality shortcomings depends, to some extent, on the intended use of the data. Given that, and our mitigation strategies, we are confident that the National Household Survey will produce usable and useful data that will meet the needs of many users. It will not, however, provide a level of quality that would have been achieved through a mandatory long-form census.

It's disgraceful, really.

The CRA has all the info the Gov. really needs...kill statcan...just think of all that money we could save!

The comments to this entry are closed.

Search this site

  • Google

Blog powered by Typepad