Yup, the NHS did produce some weird data

While writing a short comment for today's Globe, I put together a table comparing the 2011 National Household Survey data on ethnicity with the 2006 data. As the raw numbers didn't make it the Globe article, I'm reproducing them here:

Regular readers may recall that I predicted "There is, therefore, likely to be a correlation between a person's willingness to fill out the National Household Survey and the strength of their religious beliefs." 

Take a look at the table above. I don't know if it sugests a correlation between the willingness to fill out the NHS and the strength of a person's religious beliefs. But some of the more surprising results - the apparent shrinking of the West Asian population, for example - are suggestive of a correlation between a person's willingness to fill out the National Household Survey and that person's cultural heritage.

Honestly, I'm surprised that these discrepencies are so readily discernable in the published data. It would have been possible to adjust for differential response rates across ethnic groups using the 2006 census population counts as a baseline, but these numbers suggest that Statistics Canada did not do so.

Note: the spreadsheet containing the numbers above is here:  Download Download_10_43_54

Note: I have deleted the numbers on West Asians, Koreans and Japanese from the above table. The West Asian numbers, in particular, are very sensitive to whether or not one uses the answers to the ethnic origin question or the visible minorities question, how multiple responses are treated, and so on, making it extremely difficult to compare published numbers across categories. 

Comments

[Kevin]

Hi Frances,

I wonder what it would look like to add 2001 to the table so that we could see the 'normal' intercensal variation in these variables.

(I realize I'm whining about not getting a third scoop from your free ice cream dispenser, when I should be happy with the two scoops of free ice cream I've already received.)

[Frances Woolley]

Kevin - if you're the Kevin I think you are, I can send you the spreadsheet and so you can fix your own ice cream. Just let me know.

I agree, that would be a good check. The check I used was the annual immigrant #s on the Citizenship and Immigration web site (the link is in the Globe piece). Those suggest that the # for Chinese is definitely too low. And I don't think it really would take a lot of checking to figure that the West Asian # is too low also.

[Stephen Gordon]

Frances, if you send it to me, I can upload it here on WCI. I'm curious to see that as well.

But wow. I thought that there wouldn't be anything glaring at the national level, and that we'd stumble across things over time. But this is right out of the gate.

[bigcitylib]

Your surprised that discrepancies made it into the national-level data, or that Stats Canada could have corrected and didn't?

[bigcitylib]

Never mind. Your last comment answered question.

[Kevin Milligan]

Yes Frances, that is me! If you have it I'd be happy to look at that spreadsheet.

[Frances Woolley]

Stephen, the immigration Canada data is here: http://www.cic.gc.ca/english/resources/statistics/facts2011/permanent/10.asp.

bigcitylib - more that it could have been corrected but wasn't.

[Frances Woolley]

Kevin, it's now downloadable - see the last line of the post. I made some errors in the original post - for the West Asian, Korean and Japanese 2006 numbers I originally used 15 or older. I've now fixed that, but so it shows a less dramatic increase in the Korean population and a much more dramatic decline in the West Asian population. You have to go to this table http://www12.statcan.gc.ca/census-recensement/2006/dp-pd/tbt/Rp-eng.cfm?LANG=E&APATH=3&DETAIL=0&DIM=0&FL=A&FREE=0&GC=0&GID=0&GK=0&GRP=1&PID=92333&PRID=0&PTYPE=88971,97154&S=0&SHOWALL=0&SUB=0&Temporal=2006&THEME=80&VID=0&VNAMEE=&VNAMEF= for the 2006 West Asian, Korean and Japanese #s.

[Mark_dowling]

FW: "more that it could have been corrected but wasn't."

Perhaps initiative is no longer valued at Statscan.

[Zoe]

"Perhaps initiative is no longer valued at Statscan."

Bah. I think it's more that initiative at Statscan is no longer valued by the people who set Statscan's budget and direction...

[Neil]

I don't think Stats Canada would have any incentive to correct for responses by using 2006 census data. After all, the statisticians at StatsCan aren't exactly big fans of losing their reliable data either. Also, given that such methodology couldn't be maintained in the long run, they would want to have a consistent method for handling NHS data, even if it reduces the quality of the current NHS.

[Jciconsult]

Remember that crude corrections of some variables may add more noise than it clears up because of the issue of consistency of the data set. "Simple" guess-timates of bias based on a 5 year dataset and flow adjustments (where are the people now?) may work at the provincial level. It is not sensible at the CA level. We have to treat this as a "New" survey and be very careful of comparing to other datasets. I hope that Statistics Canada will attempt to provide some measure of accuracy greater than their combined non-response rate. The latter seems to be inadequate. We need to see sample size at the cell level to get a feeling (an inaccurate one).

[Frances Woolley]

Zoe, Neil, JCI - excellent points.

Mark - I think they're pretty stressed. There were way more typos than usual in the various releases.

[Jciconsult]

Note this quote from non-response bias
"Several data sources were used to evaluate the NHS estimates for ethnic origin such as: 2006 Census of Population, 2011 Census of Population results for mother tongue, the Longitudinal Immigrant Database (IMDB) and administrative data pertaining to permanent residents and non-permanent residents from Citizenship and Immigration Canada."
Source Link is
http://www12.statcan.gc.ca/nhs-enm/2011/ref/guides/99-010-x/99-010-x2011006-eng.cfm#a5

[Jacques René Giguère]

Why correct if you can expose the gunmint folly just by releasing the data they got? By not correcting, they refuse to be accomplices.

[Frances Woolley]

JCI (may I call you Paul? And how were things on The Current this morning? They bumped me for a "different voice")

Thanks for that link. This is what it says about correction of non-response bias:

"This is done by transferring the weights of non-respondent households to the nearest-neighbour respondent households. The latter are identified in a manner similar to the imputation process described in Section 4.2, using known variables for respondent and non-respondent households, including census variables and a few variables resulting from matches to administrative databases."

O.k., but which census variables were used? How is a nearest-neighbour respondent household defined? We don't know. The documentation is quite frank about the likely over-representation of people of Filipino origin, but it doesn't talk about some of the other issues.

[Jciconsult]

You can always call me Paul.
I think I will be a lot more comfortable when we get all the documentation about the correction for non-response bias. They publish a statistic but we don't know yet if the adjustment impact was bigger for one ethnic group or another or one religious group. We need to get detailed to comfortable. Since I would expect systematic biases in ethnicity, religion, possibly education and occupation, I am very concerned about the "multiple answer" questions. I am not sure how to interpret them yet. At this juncture, we don't have enough information.

[Frances Woolley]

JCI (Paul) - that's my feeling too. I wanted to get more info on the 'multiple visible minorities' because work by e.g. Krishna Pendakur has pointed to this group as one of the most rapidly growing v.m. populations, and also because it's hard to get an intuitive sense of the labour market barriers faced by people with mixed heritage - what factors contribute to a person of mixed heritage being considered a "visible minority" by potential employers?

[George Hariton]

Two things. First, we now have two new categories that we didn't have before, visible minorities n.e.s., and multiple visible minorities. The numbers showing up here are not negligible. They are big enough to potentially explain some of the movements.

Second, the first point strongly suggests that definitions have changed, or been refined, or otherwise may be understood differently by respondents. It may be that some respondents have reclassified themselves. For example, there used to be a strong stigma among Japanese regarding Korean origins. If that stigma has abated somewhat, some people who would have identified themselves as japanese might now identify as Korean. That might (or might not) explain the rise in Koreans and concurrent ebb in Japanese.

Similarly the rise Souteast Asians and drop in West Asians might be a change in self-identification.

I haven't seen the actual questions from 2006 and 2011 side by side. It would be instructive to do so.

[jen jones]

Thank you for compiling this.

[Sandwichman]

Oceania has always been at war with Westasia.

[Frances Woolley]

George -

Categories like South Asian are derived from other responses, for example, someone who says they're Sri Lankan would be categorized as South Asian. I don't think the categorization of countries has changed much, though please feel free to do your own digging.

The point about multiple ethnicities is a good one. The 2006 #s above do include people who report multiple ethnic origins. That may explain the apparent fall in the Japanese #s, because quite a large percentage of Japanese-Canadians report multiple origins. I don't think it can explain the West Asian numbers, however, which are 223,030 in 2006 for single origins. That's still higher than the 2011 number reported, despite a good level of immigration from these countries and also, presumably, some number of new births.

[Simonc]

I just had a quick look at 2006 data, and it shows 156 695 west asians in 2006, not 300k:

http://www12.statcan.gc.ca/census-recensement/2006/dp-pd/tbt/Rp-fra.cfm?LANG=F&APATH=3&DETAIL=0&DIM=0&FL=A&FREE=0&GC=0&GID=0&GK=0&GRP=1&PID=92338&PRID=0&PTYPE=88971,97154&S=0&SHOWALL=0&SUB=0&Temporal=2006&THEME=80&VID=0&VNAMEE=&VNAMEF=

Source : Statistique Canada, Recensement de la population de 2006, Produit no 97-562-XCB2006011 au catalogue de Statistique Canada (Canada, Code01)

[Simonc]

(note sure if last URL works) if not, use these beyond 20 20 links :
http://www12.statcan.gc.ca/census-recensement/2006/dp-pd/tbt/Download.cfm?PID=92338

http://www12.statcan.gc.ca/nhs-enm/2011/dp-pd/dt-td/Download.cfm?PID=105395

[Frances Woolley]

Simon - try this one: http://www12.statcan.gc.ca/census-recensement/2006/dp-pd/tbt/Rp-eng.cfm?LANG=E&APATH=3&DETAIL=0&DIM=0&FL=A&FREE=0&GC=0&GID=0&GK=0&GRP=1&PID=92333&PRID=0&PTYPE=88971,97154&S=0&SHOWALL=0&SUB=0&Temporal=2006&THEME=80&VID=0&VNAMEE=&VNAMEF=

Even if we take your number, given that this is a young community with good-ish birth rate, and looking at the immigration #s on the West Asian countries (Iran, Afghan, Israel, Turkey, Georgia, Azerbijan etc) from here http://www.cic.gc.ca/english/resources/statistics/facts2011/permanent/10.asp I still think the West Asian #s are low.

I would also say that it's *really* confusing to have several completely different #s for the # of people with West Asian ancestry - if there is no info on which tables are most comparable to which other tables, users of the numbers are going to make inappropriate comparisons.

[Simon van Norden]

Given the heightened language tensions here in Montreal since the election of the PQ, the NHS survey was splashed across the headlines here. My local paper led with stories like

"For the first time in modern history, the first official language of more than half of Quebec immigrants is French, according to Statistics Canada’s 2011 National Household Survey. (Read more: http://www.montrealgazette.com/news/French+gains+ground+among+newcomers/8357411/story.html#ixzz2T54EePmr)"

and

"How accurate a picture does the National Household Survey paint of Quebec’s population? Very accurate, says Jean-Pierre Corbeil, Statistics Canada’s chief language specialist and assistant director of social and aboriginal statistics. (Read more: http://www.montrealgazette.com/Montreal+boasts+high+response+rate/8357222/story.html#ixzz2T54zghfJ" The article goes on to argue that the high response rate in Montreal (73%) should allay fears that local results are unreliable"

Now I'm left with a puzzle.
- I'm told that a figure of 51.5% is both a modern high and the first time it has been over 50%
- this is based on a survey response rate of 73%
- the historical comparison is with census results
- we're looking at a provincial (rather than national) sample

Based on what you're seeing in the national results, how much stock would you put in these conclusions?

[Frances Woolley]

Simon - o.k., this is interesting. Language data was collected through the mandatory short-form census. According to this methodological document:

"The 2011 census questionnaire, distributed to all households in Canada, included two language questions from the 2006 census long questionnaire (knowledge of official languages and languages spoken at home) along with the mother tongue question, the only language question in the 2006 short questionnaire. As a result, the order and context in which the language questions appear were modified.....The results of this study clearly showed that the distribution of responses to the mother tongue question was almost identical to the distribution observed in the data from the long questionnaire, but different from the distribution observed in the data from the short questionnaire."

In other words, whether or not you can believe that conclusion depends on whether the 2006 #s are taken from the long-form or short-form questionnaire.

What I might try to do to put those #s in context is to compare this table:

http://www12.statcan.gc.ca/census-recensement/2011/dp-pd/hlt-fst/lang/Pages/highlight.cfm?TabID=1&Lang=E&Asc=1&PRCode=01&OrderBy=999&View=1&tableID=401&queryID=1&Age=1

to the comparable one for 2006 (if I could find the comparable one for 2006).

[Jacques René Giguère]

Another problem is how conversant with statistics is the writer, enabling him-or-her to ask the relevant questions and understand the answer.
And the political problem , no less or more acute since Sept. 4 but benefiting from The Gazette hyperventilation, is more about language used at work or language used for customer service, especially on first contact. Other things, important in the political context , are not in any census. Such as , in what language is the interface of the wi-fi service of your hotel? I am in Quebec City today and it is in englisg only...( I know from experience that other hotels of this chain have multlingual versions of this software.)

[Frances Woolley]

Jacques Rene - When I recently booked a hotel in Montreal for the CEA meetings, the rate on Expedia (in English) was $40 more than the rate booking through the hotel (in French). So language might be being used as part of a price discrimination strategy? (we've talked with Shangwen about this before in the context of restaurant specials advertised in Chinese only).

[Jacques René Giguère]

Apart from booking sites being weird, it is clear that hotels discriminate between the locals who know the alternatives and the foreigners ( beginning at the Ottawa river...). In fact,last July , I booked a hotel in Tucson, AZ, for $49 ( it is the low season there, known as "monsoon", one of the five AZ seasons (winter, spring, first growth, monsoon, second growth)). The season was so low that a Hilton in the Coronado Mountains nearby advertised a 3-room suite for $549 instead of the winter rate of $ 2500...
As a"Platinum member " of that chain, I am entitled to "always best price available" (and I am supposed to be greeted as "Welcome Mr.Giguere, Platinum member"...) When I arrived, a display in Spanish advertised a rate 10% lower, which they gave me. I presumed that "Platinum " trumped "less-wealthy Mexican...".
We could also view that as restoring PPP...

[Simon van Norden]

Seems the ASA is closely watching Canada's experience with the move from census to survey....and don't like the results.
http://community.amstat.org/Blogs/BlogViewer/?BlogKey=ec830d99-561e-4ce5-ae79-d73b77b4f2a7

[Frances Woolley]

Simon - I got a book out of the library the other day on the history of the US census (yes, there were cobwebs on the bookshelf). There has been a long long history of opposition to, and politicization of, the census in that country. I'm working on more about it.