Statistics Canada under siege

If there's one thing that has prevented me from despairing completely about the débâcle that is and will be the 2011 census, it's been my faith in the professionalism and expertise of the people who work at Statistics Canada. Their present political masters may be deaf to reason, but this is only a temporary state of affairs. There will eventually be a change of government and when that happens, the expertise of the StatsCan professionals will still be there.

So this is very, very bad news:

Chief Statistician asked to rethink census for 2016: When it comes to the 2011 census, however, [Chief Statistician Wayne] Smith said he is sure about one thing: there’s no justification for critics to say that moving to a voluntary long-form survey will wreck the quality of the data.

“I’m asking Canadians to suspend judgment because there’s no scientific basis for saying this is going to be fundamentally flawed.”

...Mr. Smith is adamant, however, that critics cannot know for sure that the results from the optional long form, even for smaller sub-groups of the population, will be inferior to what was collected via the mandatory approach in 2006.

“There is no scientific reason why you would say before it even starts, before I see results, that there’s going to necessarily be a significant problem with the count of Inuit or Métis or immigrants beyond the levels we’ve seen in the 2006 census.”

This is wrong, and badly so. So wrong that when his political masters suggested in public that the analysts at Statistics Canada were of this opinion, Munir Sheikh - Mr Smith's predecessor as Chief Statistician - felt obliged to resign in order to speak freely and correct the record:

I want to take this opportunity to comment on a technical statistical issue which has become the subject of media discussion. This relates to the question of whether a voluntary survey can become a substitute for a mandatory census.

It can not.

(I reread this letter often. Resignations on a point of principle are rare, especially for a career civil servant.)

In point of fact, Statistics Canada has done quite a bit of research documenting the fundamental flaws associated with voluntary surveys; see Kevin Milligan's guest post as well as this. Insisting that "critics cannot be sure" is a remarkable thing for a Chief Statistician to say; statistics is not in the business of providing absolute certainty.

Mr Smith has been dealt a very bad hand to play, and his only real hope is for a miracle: a voluntary survey that somehow produces a random sample. Worse, he has to do it without much of an advertising budget; his ever-thoughtful masters have taken away the extra $30m that had been earmarked for a campaign to persuade Canadians to answer the NHS.

I understand that as Chief Statistician, there are certain truths that Mr Smith cannot speak in public. But that's no excuse for making statements that are contrary to available evidence. Statistics Canada has a reputation to maintain; Mr Sheikh resigned in order to protect it. It would be a very bad thing indeed if Statistics Canada is itself the next victim in the continuing assault on evidence-based policy.

Comments

[Kevin Milligan]

Hear hear. The job of civil servants is to carry out the orders of the political representatives. When told to conduct a voluntary NHS, they should do the best they possibly can given the resources provided.

However, I don't think that this obligation extends to carrying the 'communication strategy' water for the government. They ought not to be forced to say things that are simply untrue.

I don't know whether to hope that Mr. Smith actually does know these statements he is making are untrue, or that he thinks his job is to say untrue things to make his Minister more comfortable. Neither is terribly palatable.

[Stephen Gordon]

Yeah. If (when?) the government decides that it will conduct policy under the assumption that the world is flat, civil servants can't publicly say otherwise. But that doesn't mean that a civil servant can go on to claim that there's no evidence that the world is round.

[Borror0]

Mr. Smith has no place at StatsCan if he really thinks that "there’s no scientific basis for saying this is going to be fundamentally flawed." He is either incompetent, or corrupt. I'm not sure which is worse.

I hope that academics will organize and make a lot of noise, as a response. It's not gonna happen, but it should.

[Ben Spigel]

I guess based on probability theory, there *is* a non-zero chance that by pure randomness, the results of the voluntary long-form survey will accurately reflect the population of Canada. He's right that we can't rule that chance out.

[Kevin Milligan]

Ben: And Stephen is right that anyone hinging their argument on certainty has no business being within miles of a position that has 'statistician' in its title.

[Chris Auld]

Even if by some miracle it should happen to be the case that the response pattern to the voluntary census mimics, in every relevant statistical sense, the response pattern to a mandatory census, we would have no way of determining as much. We will have to treat the respondents to subsequent censii as unobservably different than respondents to previous censii.

Smith is right that the new versions will not be "fundamentally flawed"---we work with self-selected samples all the time. But that's also beside the point: the argument against the voluntary census is not that it will be utterly worthless, but rather that it will be inferior and incomparable to previous versions.

Smith's subsequent remarks suggest he isn't aware that the census is used for anything other than estimating frequencies of various ethnic groups, and he is wrong even in that limited and straightforward context to argue that the mandatory census was not a superior instrument.

[Pangloss]

I don't think it's just StatCan that has been infected with a does of political careerism amongst civil servants and as so well demonstrated by the Smith character. Check the meteoric rise of some officials in other key departments. Bright people but junior or middle grade and not Ass DM caliber. Good thing there is no really serious federal policy initiatives, other than destroying StatCan, being undertaken by what's left of the federal gov't.

[Stephen Gordon]

Chris: That's a very good point. Even if everything does work out, there would be no way of knowing.

[Frances Woolley]

The Canadian Financial Capabilities Survey, that I'm working with right now, is a voluntary survey. The response rate is 56%. But there's fairly large interprovincial differences in response rates - e.g. Alberta, Sask have 61% response rates, BC has a 49% rate.

The Labour Force Survey, which seriously pursues its interviewees (and which people have some kind of obligation to answer), has a response rate of 90% (random self-referencing link thrown in to boost our google search ranking).

Sigh.

[Guillaume]

It appears to be a concerted attack: http://www.winnipegfreepress.com/breakingnews/study-claiming-crime-on-the-rise-gets-hung-out-to-dry-by-experts-in-the-field-116112184.html

[Appalled but not confused]

Thanks for sounding the alarm, but, um, "'I' before 'E' except after 'C'" (seige)

[Stephen Gordon]

Good lord. Fixed.

[Nspector4]

Please address his argument:

“There is no scientific reason why you would say before it even starts, before I see results, that there’s going to necessarily be a significant problem with the count of Inuit or Métis or immigrants beyond the levels we’ve seen in the 2006 census.”

[Stephen Gordon]

I thought I did. All of the available evidence shows that there *will* be a significant problem.

There's no scientific reason why we would say even before it starts, that the sun will necessarily rise tomorrow. But it's still a terrible idea to conduct ourselves as though it won't.

[Nspector4]

He says there will be no greater problem with the count of Inuit or Métis or immigrants beyond the levels we’ve seen in the 2006 census. Please address that argument.

[Kevin Milligan]

Hi Norman,

I did address that argument in a LTTE I submitted to the Globe on Sunday morning. May have missed their Monday paper deadline, or they may just have rejected it. In any case, here is what I wrote, for posterity:

---

Dear Editor,

Chief Statistician Wayne Smith is quoted in your article (Chief Statistician asked to rethink Census for 2016) saying there is “no scientific basis” for concerns about the reliability of the coming voluntary National Household Survey. Mr. Smith is wrong. There is ample evidence, from Canada and abroad, that voluntary surveys produce biased results when not corrected against a population benchmark such as a census. From this gathered evidence, we can induce that there is a strong likelihood of problems arising with the National Household Survey. Inductive reasoning has been a core tenet of scientific and statistical practice since at least the time of David Hume.

When given a task by our elected representatives, civil servants have an obligation to carry out the task even if it may be poorly conceived. However, whether this obligation extends to acceding to scientifically dubious assertions of the elected representatives seems to be an open question.

[Stephen Gordon]

Actually, the choice of the Aboriginal/Métis example is a bit disingenuous. Previous censuses *already* had problems with those groups.

But for the other groups, we'll be introducing problems where they didn't exist before.

[Jacques René Giguère]

Pangloss:
Good thing there is no really serious federal policy initiatives, other than destroying StatCan, being undertaken by what's left of the federal gov't.
Apart from the destruction of climate research? Apart from the destruction of efficient crime fighting strategies? Apart from...

[Nspector4]

In other words, while you don't like the examples he chose, you do not dispute his statement as quoted.

[Stephen Gordon]

If the journalist misrepresented him here

"Mr. Smith is adamant, however, that critics cannot know for sure that the results from the optional long form, even for smaller sub-groups of the population, will be inferior to what was collected via the mandatory approach in 2006."

Then he's being disingenuous at best and deliberately obfuscating at worst. It the journalist correctly represented his views, then he's just as wrong as I said in the OP.

[Kevin Milligan]

The statement "no scientific basis" is false. Induction is a scientific principle.

The statement "critics cannot know for sure" is literally true, but is not scientific. As I tweeted, these are weasel words befitting of a politician or lawyer, not a statistician.

[Alan]

Seems like the best way to save the census is to get the worst response rate possible.

If you love the census, don't fill it out.

[Patrick]

"There is no scientific reason why you would say before it even starts, before I see results, that there’s going to necessarily be a significant problem with the count of Inuit or Métis or immigrants beyond the levels we’ve seen in the 2006 census."

Be generous to Smith. Agree with him that there are potentially significant problems that will appear when we see the results. This raises the question: why in the world did we switch to a system that potentially has significant problems from one that was sound?

[Stephen Gordon]

Alan: The problem with that strategy is that it gives the government the perfect excuse: "It's not *our* fault; blame the boycott organized by our opponents."

Patrick: That's being *too* generous. At best, he's not being honest about the evidence on the sample selection problem. At worst, he's denying it exists.

[K]

"critics cannot know for sure that the results [...] will be inferior"

Imagine if that was the test of public policy: "We are going to do this unless you can *prove with 100% certainty* that it's going to make things worse."  Of all the ridiculous things to say...

[Appalled but not confused]

The full transcript of the interview is now up at:
www.theglobeandmail.com/news/politics/ottawa-notebook/is-census-data-usable-our-thinking-has-evolved-chief-statistician-says/article1906804/singlepage/#articlecontent

Is census data usable? ‘Our thinking has evolved,’ chief statistician says
STEVEN CHASE and TAVIA GRANT
Globe and Mail Update
Posted on Monday, February 14, 2011 4:57PM EST

[K]

Boycott or not: why would I fill it out? Takes time, makes no difference to me. In an election, at least my favorite party gets some extra funding, and even for parties who can't win, having a higher vote count makes a difference. I don't see what good it does for me to participate in a totally flawed survey. Seems like a waste of time.

[Appalled but not confused]

So here's a soundbite from the CPC's newly appointed chief statistician on one of the methods for how the country might achieve at least its now, er, conservatively projected 50% response rate for the NHS:

"In the past, support has been expressed in the form of census beer. It's been painted on the side of trucks: census messages."

So, in light of the out-of-context line from one the CPC's new attack ads quoting the Lib. leader re: the Cndn. flag, one might be sorely tempted draw a mock-up of the envisioned "NHS beer ad" campaign, & quip:

"The NHS: a failing imitation of a census label."

[Stephen Gordon]

Yikes. After reading the full transcript, I have to set aside the possibility that the Globe misrepresented Wayne Smith's views.

This is worse than I thought.

[Patrick]

Right off the bat, he's dreaming if he thinks he's going to get a 50% response rate. Marketing people jump for joy if they get 20% response rates from the general public.

[Mike Moffatt]

This might be the stupidest thing I've ever read:

"If the response falls to 50 per cent, there's one inevitable consequence of that. And that's exactly why the sample has been bumped to one in three from one in five. To compensate for that effect to the extent possible.

At a 50-per-cent response rate with a one in three sample, the sampling error would still be somewhat worse in the National Household Survey than it was in the 2006 census, the long form.

If the response is actually 60 per cent on the NHS it’s a wash. They are basically just as accurate, one as the other.

And if it goes to 70 pct actually the NHS sampling error is smaller than it was in 2006."

Just because the sample is larger, does *not* mean the sampling error is smaller. This was shown by Literary Digest back in the 1930s:

http://en.wikipedia.org/wiki/The_Literary_Digest#Presidential_poll

"In retrospect, the polling techniques employed by the magazine were to blame. Although it had polled 10 million individuals (only about 2.4 million of these individuals responded, an astronomical sum for any survey),[3] it had surveyed firstly its own readers, a group with disposable incomes well above the national average of the time (shown in part by their ability still to afford a magazine subscription during the depths of the Great Depression). The magazine also used two other readily available lists: that of registered automobile owners and that of telephone users. While such lists might come close to providing a statistically-accurate cross-section of Americans today, this assumption was manifestly untrue in the 1930s. Both groups had incomes well above the national average of the day, which resulted in lists of voters far more likely to support Republicans than a truly typical voter of the time."

The response rate was gigantic, but since the sample was biased, the results were garbage. Standard Stats 101 stuff.

[Stephen Gordon]

Yep. This is extremely discouraging. If the Chief Statistician doesn't seem to understand the problem of sample selection bias, then we are all in deep, deep trouble.

[Appalled but not confused]

Here's his bio. Like our illustrious PM, he's got a Master's in econ. He has been with StatCan for 30 years, but on the communications side of things. Which he doesn't seem that good at; towards the end of the interview, when his back got up over the crit. from the international community, he sounded just like a lot of Conservative bloggers, with his little judo move of twisting someone's words to deflect a legitmate q. ("So what you're saying is...")

http://www42.statcan.ca/smr09/smr09_024-eng.htm

[Patrick]

Just finished reading the rest. Oh man. The bit Mike quotes is a real whopper. Like an undergraduate who didn't do the readings and is trying to fake his way through the quiz with nonsense that sounds like statistics.

[manchil]

After looking at his bio, my only thought is: Wouldn't we want the chief statistician to actually be a statistician?!?

[Andrew F]

This is madness. One talking point is how a mandatory long form is an invasion of privacy. Another is that we should consider a register based system where the government maintains vast comprehensive databases of personal information, which would be a far greater blow to personal privacy than a long form census could ever be.

[Andrew F]

He apparently doesn't understand or have any concept of cost accounting. To suggest that you can't reasonably attribute any cost to the NHS is ridiculous.

[Simon C.]

His own expert don't share his optimism. A friend pointed me to this. Save it now before it's gone.

(http://www.statcan.gc.ca/survey-enquete/household-menages/nhs-enm-eng.htm)

Non-sampling error
[...]
However, it is believed that the most significant source of non-sampling error for the National Household Survey will be non-response bias. All surveys are subject to non-response bias, even a Census with a 98% response rate. The risk of non-response bias quickly increases as the response rate declines. This is because, in general, non-respondents tend to have characteristics that are different than those of the respondents and thus the results are not representative of the true population. Given that the National Household Survey is anticipated to achieve a response rate of only 50% there is a substantial risk of non-response bias.

Statistics Canada is very much aware of these risks and their associated adverse effects on data quality. The Agency is currently adapting its data collection and other procedures to mitigate as much as possible against these risks. In particular, we will be using data on response patterns from the 2006 Census and information generated during data collection in 2011 to guide our field follow-up effort to minimize non-response bias. As well, where possible, 2011 Census data will be used as auxiliary information in National Household Survey estimation procedures to partially offset some of the remaining biases. However there is certain to be some residual, significant bias that will be impossible to measure and correct.
[...]

Comparability of data over time
[...]
Any significant change in the methods of a survey can affect the comparability of data over time. There is a real risk that this will be the case for the National Household Survey. There will always and inevitably be an element of uncertainty as to whether and to what extent a change in a variable reflects real change or an artefact arising from the change in methodology from the mandatory long-form census to the voluntary National Household Survey.
[...]
The French version is even harsher:
Il est clair qu’on ne pourra jamais déterminer avec certitude si et dans quelle mesure la modification d’une variable est attribuable à un changement réel ou découle de la transition du recensement à formulaire détaillé obligatoire à l’Enquête nationale auprès des ménages à participation volontaire.

Conclusion
[...]
We have never previously conducted a survey on the scale of the voluntary National Household Survey, nor are we aware of any other country that has. The new methodology has been introduced relatively rapidly with limited testing. The effectiveness of our mitigation strategies to offset non-response bias and other quality limiting effects is largely unknown. For these reasons, it is difficult to anticipate the quality level of the final outcome.

The significance of any quality shortcomings depends, to some extent, on the intended use of the data. Given that, and our mitigation strategies, we are confident that the National Household Survey will produce usable and useful data that will meet the needs of many users. It will not, however, provide a level of quality that would have been achieved through a mandatory long-form census.

[Sina Motamedi]

It's disgraceful, really.

[dj]

The CRA has all the info the Gov. really needs...kill statcan...just think of all that money we could save!