How the hell can we tell (if fiscal policy worked)?

I feel guilty about not wading into the recent debate about whether fiscal policy helped the Canadian economy recover from the recession. It's one of the things I am supposed to know something about -- basic macro. But there's a reason I couldn't be bothered to take sides and join in.

The patient is ill. The doctor gives him the medicine. He recovers. Did the medicine work? From a sample of one, you can't tell. He might have recovered anyway. Maybe he would have recovered more quickly without the medicine. Or maybe his wife's chicken soup is what cured him.

Canada is a sample of one. We can't tell whether fiscal policy helped or hindered recovery by looking at one data point.

If we had two identical Canadas, gave them each a different fiscal policy, and made absolutely sure that everything else stayed identical, then we could tell what difference fiscal policy made to the recovery. But we don't.

If we had 100 different countries, divided them into two groups of 50 by tossing a coin, and gave the first group a different fiscal policy from the second group, then we would have a good chance of knowing if fiscal policy worked. (Or maybe have 150 countries, and sacrifice a goat in the third group of 50 to see if the placebo-effect of confidence-building is what really matters.) That's how drug companies see if medicines work.

Now, we do have lots of different countries in the world. But unfortunately, none of them is willing to toss a coin to decide what fiscal policy to use. Maybe the sickest countries were the ones desperate enough to resort to fiscal policy medicine, so they would have recovered more slowly anyway. Or maybe the healthier countries were the ones that had the ability to take a fiscal medicine, so they would have recovered more quickly anyway. The bias could go either way.

What would have happened if Canada had had a tighter fiscal policy, instead of the fiscal policy we actually did have? That's what we need to know. By comparing the actual outcome to the counterfactual, we can ascribe any difference to the effects of fiscal policy. What economists usually do is build a structural econometric model to answer this question. But a structural model already contains the answer to the question of whether fiscal policy works. We build that answer into the very structure of the model. It's implicit in the model's assumptions.

I can build a structural model in which fiscal policy works, fit it to the data, and it will tell me that the fiscal medicine worked. Or I can build a structural model in which fiscal policy doesn't work, fit it to the data, and it will tell me that the fiscal medicine didn't work. Big deal. Sure, you need a model to interpret the data. But if the data is just one spoken sentence, the first model will translate it as "fiscal policy worked", and the second model will translate it as "fiscal policy didn't work". The model is not interpreting the data; it's a ventriloquist for the data's dummy.

Arnold Kling said something similar in a very important but somewhat nihilistic post recently. (As far as I can tell, his post was ignored; probably because it's too depressing.)

So anyway, Stephen's sensible post raised my guilt level past the threshold needed to read (well, skim) some of the recent debate on whether fiscal policy helped Canada recover from the recession.

It turns out they aren't in fact debating whether or not the fiscal medicine worked. They aren't at that stage yet. They are still debating whether or not the patient actually received any fiscal medicine. And to answer that question you need to figure out how much medicine the patient would have taken in any case, even if the doctor hadn't tried to administer any. You need to know the counterfactual....You need a model of how much medicine a patient normally takes...

Comments

[Jon]

hahaha.

[Jon]

Now to be serious: I think one of the most depressing moments in my economic studies came upon being presented an overlapping generations model of the business cycle that used ... 'genetic algorithms' to model the agents.

Our professor (an author on the two papers in question) carefully explained to us how the agents 'mated': agent preferences were encoded in some binary number in a funny way, and then the offspring adopted a set of preferences formed by taking the first a bits from parent 1 and n - a bits form parent 2. Plus some other variables.

Yes, indeed sounds practical. My father has a propensity to save of 56, my mother has a propensity to save of 38, so yes, obviously I should adopt a propensity to save of 36...

God she spent half-an-hour discussing binary math.

Finally, she ended it with some graphs showing how well this model hindcast some 'real data'. Her paper also derived all sorts of convergence criteria, etc. She was really proud.

Whereas I was contemplating how could it be possible the reviewers hadn't heard that genetic algorithms are a general optimization technique that's been well studied (since the 1950s) in much greater detail and rigor.

[Don the libertarian Democrat]

It seems to me that the idea of a Reinforcing Stimulus to Monetary Policy makes a lot of sense, and is being vindicated as we speak. It certainly seemed a wise move in terms of Political Economy. Isn't that the best we can do? Why the hand-wringing?

[Nick Rowe]

Jon: sounds a bit weird at first sight. But, if you want to wean yourself off full-bore rational expectations, and build in some sort of learning mechanism, it might be a way to go. If you interpreted the genetic stuff metaphorically, rather than literally. Because I don't think the Darwinian process works much at business cycle frequencies.

Don: I think it makes a fair bit of sense too. But I don't see how I could use data from the past 2 years to convince anyone who thought otherwise. How do we know it is being vindicated as we speak? The voters probably wanted fiscal policy; but they wanted it because they think it works. And they think it works because most economists have thought it works. Cue "academic scribblers" quote from Keynes. In other words, I really wish I had better reasons for believing what I believe.

[Phil Koop]

I agree with you completely, but ... it seems a bit odd coming from an economics professor! I mean, this problem is endemic to the whole field. Economics is not "the dismal science" because it is not science at all, it is philosophy: intellectually stimulating, fun to debate, it occasionally grants an insight into the human condition. But economics practically never makes falsifiable predictions*, and something that cannot be false empirically cannot be true in that sense either. The best you can do is correlational studies, and there you are lucky if you have more data points than confounding factors.

* Yes, there is "behavioral economics", but the testable bits of that are arguably more accurately described as cognitive psychology than economics.

[Nick Rowe]

Phil: I don't think it's *quite* that bad.

With just one data point, it's trivially easy to fit any model to the data. As you add more and more data points, it gets harder. You can always torture any model to fit the data, but it gets less and less plausible/credible the more data you have. And sometimes we do get what seem to be "natural experiments". And this is more of a problem with macro than with micro.

And compared to climate science, for example, macro has it easy. We have lots of countries to look at; they only have one world. And we have lots of business cycles and monetary/fiscal policy experiments to consider; they only have one.

[Just visiting from Macleans]

...and the Rosetta Stone proves that stimulus spending has a rich history:

The text on the stone is a decree from Ptolemy V, describing the repeal of various taxes and instructions to erect statues in temples.

http://en.wikipedia.org/wiki/Rosetta_Stone

p.s. Not sure if this contributed to the decline of the most advanced ancient civilization. The market for pyramids never did fully develop.

[Patrick]

Climate science at least has a solid foundation of physics to build on. Modelling climate is a problem of scale and complexity, but in principle we know what to do. A lot of modern macro starts off with a whopping big spherical cow: "assume rational expectation", and then proceeds to ignore finance.

More on topic: Maybe I'm being too simple minded, but once fiscal policy has run it's course can we estimate it's effect and then back it out of GDP? That should give us some idea of what would have happened in the counterfactual, given that monetary policy couldn't have got much looser sans QE - and in any case the evidence from the US and Japan seems to be that QE doesn't do much good.

Anyway, let's not forget how scary things where between fall 2008 and spring 2009. When the rabid grizzly bear is charging I'd rather let him have both barrels and worry about the cost of ammunition later.

[Nick Rowe]

JVFM: I'm not sure how Ptolemy financed the resulting deficits though. By borrowing? Or did he have cash in the vaults, so it was a money-financed deficit?

Patrick: "Climate science at least has a solid foundation of physics to build on."
And macro modelling has solid foundations of, err, microeconomics to build on. But if we do build directly on those foundations, we get real business cycle theory. And real business cycle theory says either that fiscal policy won't work, or isn't needed anyway (unless you really rig the model to generate different conclusions).

"Maybe I'm being too simple minded, but once fiscal policy has run it's course can we estimate it's effect and then back it out of GDP?"

Nope, we can't. C+I+G+NX=Y. If we *assume* that C+I+NX stayed the same, then changes in G must have a unit multiplier effect on Y. But that's just accounting; no economics there at all. We know that that assumption is generally unsupported. Otherwise let's increase G without limit, all the time. And if we don't make that assumption, how can we estimate the effect of fiscal policy? With one data point, we simply can't estimate its effect.

[Patrick]

Ah, I see. So if we want to 'back out' G, we need some way to figure out what G did to C, I, and NX, and that's where the problem lies; nobody can agree on how to figure that out.

"And macro modelling has solid foundations of, err, microeconomics ... we get real business cycle theory "

In principle, and at the scale of the Earth's climate, all the physics scales-up without producing weird results. Physicists only run into problem in black holes or the nucleus of atoms, whereas it seems economists fall into the equivalent of a black hole almost as soon as they try to do anything interesting.

[Just visiting from Macleans]

He simply harvested more papyrus and paid off the Trojans.

[Nick Rowe]

Patrick: Yep. Or I could frame the question this way: MV=PY. Assume M and V stay the same, then an increase in G cannot change PY. Equally question-begging, just a different way of framing it, so it looks less plausible that fiscal policy could have done anything.

In principle, we too can scale up micro into general equilibrium theory. But experience has taught (most of) us that it doesn't really work for business cycles. We've seen enough business cycles and other crises to know that something weird happens when we aggregate into a complex system. Probably there are parallels in physics.

[Winslow R.]

You must have a hard to sleeping with all that professional guilt.

Your boss will probably want you to see someone in the medical/psychology/sociology profession for advice on how to suppress the symptoms :)

[Jon]

Nick: Et tu brute?

I don't doubt this, "But, if you want to wean yourself off full-bore rational expectations, and build in some sort of learning mechanism, it might be a way to go."

The trouble is that she didn't prescribe a learning mechanism. The agents swapped bits from a vector describing their preferences without regard to which preference variable any given bit belonged. Its an extremely stochastic process, and contributed nothing to the convergence beyond that.

The problem with the paper was not the underlying idea. Look, suppose, I find the roots of a function. Then you write a paper, describing a root finding method--lets say you invent newton's method--then you observe that you and I both found the roots of the function. Finally you conclude, newton's method must be a good model for what 'Jon' did.

But within the four-corners of the data, all that we actually know is that we both found the roots. I think that's the rub.

I think it gets worse when you start talking about optimization. All economics models are about optimization processes... and there is a rich field of optimization algorithms. Simply reproducing time-path of the equilibrium points is not meaningful.