When academic publishing goes wrong: the case of missing women and Hepatitis B

This is a story about how something that turns out to be wrong can be published in a top journal, and what happens next.

The Freakonomics team tell good stories, so I'll let them begin. In 2005, they wrote an article in Slate magazine lavishly praising the work of a young economist called Emily Oster. 

You can read the whole story here. I've condensed it a little. Dubner and Levitt start with Amartya Sen's "incendiary argument" that there were 100 million missing women in Asia:

While the ratio of men to women in the West was nearly even, in countries like China, India, and Pakistan, there were far more men than women. Sen charged these cultures with gravely mistreating their young girls—perhaps by starving their daughters at the expense of their sons or not taking the girls to doctors when they should have. Although Sen didn't say so, there were other sinister possibilities. Were the missing women a result of selective abortions? Female infanticide?

But now another economist has reached a startlingly different conclusion. Emily Oster is an economics graduate student at Harvard who discovered a strange fact. A pregnant woman with hepatitis B is far more likely to have a baby boy than a baby girl. Oster was suitably intrigued. She set out on a vast data mission to determine the magnitude of that relationship.

She found that:

The regions with the most hepatitis B were the regions with the most "missing" women. Except the women weren't really missing at all, for they had never been born. If you believe Oster's numbers—and as they are presented in a soon-to-be-published paper, they are extremely compelling— Hepatitis B can account for roughly 75 percent of the missing women in China. Oster's analysis does show that economics is particularly useful for challenging a received wisdom—in this case, one that was originally put forth by another economist.

Oster's work resonated with Dubner and Levitt because it supported two of economists' most cherished beliefs:

1. Individuals making rational choices will usually produce good outcomes. This is why economists generally believe in the efficiency of markets and a limited role for government.

If individuals making rational choices decide to, say, abort female fetuses, it's hard to believe that freedom of choice leads to good outcomes. If, instead, skewed sex ratios are caused by Hepatitis B, our faith in individual choice is unscathed.

2. Economists are smart. Smarter than other people who are easily swayed by incendiary arguments. Levitt and Dubner put it a little more tactfully in Freakonomics when they say "the conventional wisdom is often wrong."

There's just one problem with this whole heart-warming story. Oster's analysis, as published in the Journal of Political Economy (editor: Steven Levitt) [ungated], was wrong. Studies of thousands of people with and without Hepatitis B infections find that the virus causes only a very small change in the probability of having a boy. Lin and Luoh published a rebuttal in the American Economic Review. Earlier this year, Oster, Chen, Yu and Lin published a paper acknowledging   Hepatitis B does not explain male-biased sex ratios in China.

Emily Oster, after all of this, remains a respected economist. Her creativity and intelligence is evidenced by her other work, for example, her recent paper arguing that cable television has improved women's status in India (now that's a cool paper). Creative people tend to come up with lots of ideas - not every one works out. 

But Levitt's reaction to the whole affair was somewhat lacking in introspection. He praised Oster: "She also has done something incredibly rare for an academic economist: she has admitted she was wrong." 

Yet I have not read any admission from Levitt that he, as editor of the Journal of Political Economy, made a mistake when he accepted her work for publication.

Someone arguing in Levitt's defence might say "well, no one could have known that Oster's hypothesis would turn out to be wrong." Could they? In 2005, the year that Oster's paper appeared in the JPE, Monica Das Gupta published a rebuttal in the Population and Development Review. She describes the results of a 1993 paper by Zeng et al, one cited by Oster:

...the sex ratio at birth varies sharply by the sex composition of the living children the woman already has.... Zeng et al. show that the sex ratio at birth was normal (1.056) for first births. For second births, it was strikingly different depending on whether the first child was male or female: women whose first child was a son had a low sex ratio (1.014) for the second child, while those whose first child was a daughter had a very high sex ratio (1.494) for the second child.

To produce a pattern like that, Hep B has to be one heck of a smart virus. So the first point is: anyone with even a passing familiarity with the literature would know there was something suspicious about the Oster results.

 Second, Oster's evidence was, by generally accepted research standards, weak. Basically the paper took a bunch of different countries, and looked at the relationship between the ratio of boys to girls in the population and the prevalence of Hepatitis B. It just happens to be the case that countries with high rates of Hep B - China, for example - also have skewed gender ratios. If you had replicated Oster's methodology and substituted "percentage of calories coming from rice" for "Hep B prevalence" you might have found that rice consumption causes missing women. Cross-country studies are particularly subject to this kind of problem, so many researchers (rightly) view them with suspicion.

Sometimes mistakes don't matter. Earlier this year I speculated that income inequality might by correlated with adultery. I was probably wrong, but I doubt anyone signed up for AshleyMadison.com after reading the blog.

But Oster's mistake had - and still has - the potential to do real harm. Type "missing women" into http://scholar.google.com. When I did that just now, the number four hit was to Oster's 2005 JPE article attributing 75 percent of China's gender imbalance to Hepatitis B.

I don't believe some of the fear mongering about "Bare Branches and Danger in Asia" - the idea that nations faced with "surplus men", that is, men unable to find wives, inevitably resort to war and imperialism. But the current demographic numbers in China, combined with the fact that some younger women would prefer to date a successful older man than a struggling young one, mean that a substantial proportion of young men will be unable to find partners.

And that will lead to profound social changes. Some studies have suggested that competition for scarce brides causes parents of sons to increase savings. Others suggest surplus men increase the demand for sex workers, leading to more prostitution and an increase in HIV infection rates. I've spoken to colleagues from India and asked "what does this gender imbalance mean for your daughter?" and heard stories about rape and sexual assault.

It's hard to predict what will happen - the gender imbalances currently emerging in parts of Asia have few historical parallels.

But I can say one thing for sure: providing an easy scapegoat in the form of Hepatitis B infection makes it easier for people to avoid taking a long hard look at the cultural and social factors that cause parents to abort, abandon and neglect girls. And that is a serious harm.

Professor Levitt, if you are reading this: I would like to see an Errata added to Oster's 2005 JPE piece - on the front page, where no one can miss it - saying "the research in this article has since been advanced in subsequent work, published in Economics Letters."

 

Comments

[Nick Rowe]

Yep. Newspapers give you a very distorted picture of the world, because they publish "man bites dog", but not "dog bites man". Top economics journals give you a very distorted picture of economics, for the same reason. Gradute programs in economics do the same. They don't teach stuff we already know, but where no exciting new work is being done. They teach stuff that is exciting and new, and therefore likely to be wrong ;-)

[Wendy Waters]

Only vaguely on topic, but I wonder if we'll start to see more single, educated, 30-something men from China immigrate to places like Canada, Australia, etc. in part because they'll have fewer ties holding them back, in part hoping that in a country with more balanced sex ratios they'll have a higher liklihood of finding a spouse, and in part for the opportunity generally.

Is there any evidence of this so far?

[Frances Woolley]

Newspapers publish corrections when they get things wrong. Bloggers go in and do a quick edit and correct the mistake.

If academic journals are going to get into the business that we're in - exciting, new, and not infrequently totally bogus claims - they need to work out some way of taking down/correcting research findings when they are (a) wrong and (b) it matters - by which I mean that real people could be harmed by the continued circulation of incorrect information.

You might find this hard to believe, but I recently spoke to someone who didn't even believe that there was a gender imbalance in China. The truth is out there, but so are a heck of a lot of misconceptions.

[Frances Woolley]

Wendy, good question.

If you believe that women "fall in love where money lies" then the best strategy for a man looking to find a wife is to search some place where the incomes are lower than where he is right now. So guys in Beijing look for brides in the smaller cities, men in smaller cities look for brides in the countryside, poor farmers look for brides in N. Korea, etc. When it all shakes down, the single guys will be the poorest, least attractive ones - they're the ones who will hide in containers and take crazy risks in the hope of making a better lives for themselves somewhere else - and getting lucky

It would be very hard to test your hypothesis by looking at #s on Chinese immigration to Canada, for a somewhat complicated reason. A non-trivial number of Chinese-Canadian women are more likely to marry non-Chinese men. Relatively fewer Chinese-Canadian men marry non-Chinese women. Which means that there are "surplus men" in the Chinese-Canadian marriage market. So what's a guy to do? The simple answer is: find a bride in China and bring her here. So that would tend to make immigration flows from China to Canada more pro-female.

[Just visiting from Macleans]

Isn't the lesson to be learned from this experience that cultural bias matters? Or that a method of research dealing solely with data mining is perilous?

Not quite to the same level as searching the grad photos of UCC for changing demographics, but if you look at the surnames of the author/editor etc you don't see anything that remotely resembles a Chinese surname until you get into the rebuttal/refutation stage.

China's one child policy, no doubt, was mentioned somewhere in the linked references. "Little Emperors" with four doting and increasingly wealthy grandparents tends to make single male Chinese less attractive partners, wherever they locate. A policy that is being relaxed/revisited (and was not universally applied in the past, it should be noted)

[Curmudgeon]

Something that has been elided here is that the concept of taking research by economists seriously when they are working in areas far removed from their field is deeply dubious in its own right. Research by economists into matters of biology and epidemiology should be given no more credence than research into economics by biologists.

[Frances Woolley]

Just visiting - "Little Emperors" with four doting and increasingly wealthy grandparents tends to make single male Chinese less attractive partners, wherever they locate.

I've heard that - despite the gender imbalance - in big cities like Shanghai and Beijing a lot of career minded, educated women are staying single (there are tales of the desperate ends to which match-making parents are going...).

Think about a 20- or 30-something Chinese woman's choices (the first one child babies turned 30 this year).

On the one hand - travel, career, minimal responsibilities.

On the other hand - think about the expectations of caring a one child woman potentially faces - her 2 parents, her 2 parents in-law, her husband, her children ... Given that choice, I would be tempted to stay single, go shopping, and watch Sex in the City with subtitles in Friday night.

[Just visiting from Macleans]

I doubt if you lived in Shanghai or Beijing you'd need the subtitles. Maybe at the karaoke bar, however.

[Mike]

What I find disturbing about this story is that Oster was able to obtain a PhD from Harvard, get published in a top economics journal, and become a member of a prestigious economics faculty, all based on work that was discredited almost immediately. This is certainly not Oster's fault, but I think it leads to serious questions about the processes involved in publication and teaching positions.

[Frances Woolley]

Mike: "I think it leads to serious questions about the processes involved in publication"

Yup.

[Adam P]

but is it Oster's fault? Frances said "anyone with even a passing familiarity with the literature would know there was something suspicious about the Oster results".

Surely Oster should be more familliar with a literature that she's contributing to than Levit? Shouldn't that statement of Frances' be a stronger indictment of Oster?

[Frances Woolley]

Adam P - I've thought about this point. Three observations:

1. Lots of us do stupid things in grad school.

2. When one is totally obsessed with a topic - which you have to be in order to have the momentum to do a PhD thesis - it can be hard to get a sense of perspective. One falls in love with one's theories and is blind to their faults. Which is as it should be, because otherwise you'd never have the drive to get them written up and out there.

3. Academic success requires one to suppress one's own doubts. I have a paper that I'm about to go and work on right now - do I worry that my results are just being driven by a small sub-sample of the population? yes. Do I express those doubts in my paper? No. Did the journal editor express those doubts? - you better believe he did.

I'm reluctant to be too hard on people for responding rationally to the incentive structure within academia.

[Jack]

Absolutely: a retraction note should be uploaded to the electronic version of the journal--why not remove the article entirely if it's factually wrong?

Note: Oster was originally on the economics faculty website at U Chicago, but is now only on the Booth GSB faculty website at U Chicago... coincidence?