On the significance of statistically significant results

Andrew Gelman notes that

there are suprisingly many papers with results that are just barely statistically significant (t=1.96 to 2.06) and surprisingly few that are just barely not significant (t=1.85 to 1.95).

in a selection of empirical studies in political science; Mark Thoma wonders if this extends to economics as well. I'm pretty sure it does, although perhaps not to the same extent.

There is a case for excluding superfluous explanatory variables: if a right-hand-side variable really does have a 'true' coefficient of zero, there are efficiency gains to be had by imposing a correct restriction. But careful researchers are also aware of the perils of interpreting the results of 'pre-test estimators' that first test for significance and then re-estimate using only the RHS variables that pass a significance test. The standard errors for this estimator are not what the regression package might report.

As a Bayesian, I'm not ashamed to report results in which the posterior mean is (say) one posterior standard deviation greater than zero. Although a frequentist might think that the coefficient was not significantly greater than zero, I'd note that there's about an 85% posterior probability that the coefficient is positive.

And in any case, a good model isn't one with tightly-estimated parameters; any model with a sufficiently large sample size will give you arbitrarily large t-statistics. The true test of a model is out-of-sample prediction.

Comments

[Paul Mason]

An interesting study would investigate the dodginess of published results with the proximity of the date for renewal of the researcher's funding.

[EclectEcon]

Is it possible that articles without statistically significant results tend to be rejected, whereas those with significant results do get published?

[optimuscrime]

Great timing for this post -- our graduate methods class just read Cohen's 'The World Is Round (p<.05)' and had a great discussion about the problem of these useful-but-artificial distinctions between significance and insignificance. Thanks for the links.

[optimuscrime]

Here's the ref. for anyone interested:
Cohen, J. (1994). "The world is round (p < .05)," American Psychologist, 49, 997–1003.