Dumb questions about predictability of stock market returns

Something I always wondered about, but was too scared to ask. Noah Smith's (quite reasonable) post nudges me into asking it.

Even if everyone is perfectly rational, where is it written that stock market returns cannot be predictable? The stock market rate of return is a rate of interest. Where is it written that changes in interest rates cannot be predictable? Where is it written that changes in uncertainty of the market portfolio cannot be predictable? Where is it written that changes in liquidity premia cannot be predictable? Where is it written that apples cannot be predictably cheaper than bananas?

1. Suppose that every decade Nature tosses a coin. If the coin lands heads, she does something that causes real interest rates to be low. If the coin lands tails, she does something that causes real interest rates to be high. Maybe she messes with preferences, demographics, growth rates, the rate of return on investment, or whatever. There are any number of things she could mess with to do that.

If the coin lands heads, interest rates will be lower than normal, and will be expected to be higher in future. So stock prices will be higher than normal, and will be expected to be lower in future. Stock market returns will be lower than normal, just like all other interest rates will be lower than normal. Can you make a "profit" by selling stocks now and buying stocks in future? No. Not unless you, personally, are unaffected by whatever Nature did to lower interest rates.

2. Suppose that every decade Nature tosses a coin. If the coin lands heads, she does something that causes the uncertainty of the market portfolio of stocks to be low. If the coin lands tails, she does something that causes the uncertainty of the market portfolio of stocks to be high. If the coin lands heads, stock prices will be higher than normal, and will be expected to fall in future. Can you make a "profit" by selling stocks now and buying stocks in future? No. Not unless you, personally, are unaffected by whatever Nature did to lower uncertainty.

3. Suppose that every decade Nature tosses a coin. If the coin lands heads, she does something that causes the liquidity premium of the market portfolio of stocks to be low. If the coin lands tails, she does something that causes the liquidity premium of the market portfolio of stocks to be high. If the coin lands heads, stock prices will be higher than normal, and will be expected to fall in future. Can you make a "profit" by selling stocks now and buying stocks in future? No. Not unless you, personally, are unaffected by whatever Nature did to lower the liquidity premium.

4. If apples are cheaper than bananas, can you make a "profit" by selling bananas and buying apples? No. Not unless you, personally, are unaffected by whatever Nature did to make apples cheaper than bananas.

Yep, if apples and bananas taste exactly the same to you, you can indeed make a "profit" by selling all your bananas and buying apples, if apples are cheaper than bananas. So what?

Are finance people just reasoning from a price change?

Comments

[louis]

Tend to agree with this. Anyone who had the liquidity to buy assets in early 2009 could scoop up abnormally high expected returns, because not many investors had access to liquidity and many were forced sellers of assets. Predicting relative performance of individual stocks (i.e. noticing mispricings) seems to be harder than estimating total mkt returns over long periods.
The only thing I'd point out is that the actual performance of stock markets seems to feed into risk tolerances in a procyclical way. When markets have been rising, investors seem more comfortable investing and expecting higher prospective returns. They extrapolate recent results instead of seeing higher prices as indicative of lower future returns.

[Nick Rowe]

louis: plus, anyone who had high risk-tolerance in early 2009. If I remember back then, nobody knew whether things would get much worse.

[rsj]

This is the notion that there are time varying risk premiums. In 2009, most were willing to bear risk less than in 2007. Only those unaffected -- those with lots of wealth parked in safe government assets and nerves of steel -- were able to profit.

But this observation isn't very satisfying, because we are interested in the reason for the swing. But part of it has to be that we primarily buy assets by selling other assets, and so everyone is vulnerable to a type of margin call. Endogeneity is at the root of this, I think.

[Matt]

I'm no asset pricing theorist (not even close, thank god), but my understanding is that the theory asserts that 'm_t*p_t' is the martingale (i.e. "unpredictable" in a loose, popular sense of the term), not the price 'p_t' alone.

'm_t' here is the "stochastic discount factor" (SDF), which is both state and time-contingent, and is defined as the price we're willing to pay in equilibrium to receive money at that given state and time, in terms of some numeraire (usually money at date 0).

The notion that 'p_t' is itself unpredictable is a shaky simplification of the theory that arises when it's presented to popular audiences. Sometimes the actual theory is alluded to by mentioning that individual assets have varying "risk premia" when compared to, say, a riskless bond; these arise from the influence of 'm_t' and its possible covariance with 'p_t'.

Anyway, this theory is more than capable of capturing the possibility that changes in uncertainty are predictable. It's all wrapped up in the process for the SDF. Indeed, as a side note, generally stochastic volatility models in finance *do* display predictable movements in volatility. (They almost have to, because the variance of shocks is bounded from below by zero, so if it was a martingale -- i.e. lacking predictable movements -- it would eventually end up converging to something in every realization of the world due to the Martingale Convergence Theorem, and it sounds pretty crazy that volatility would ever converge to some definite amount and stay there forever.)

[Nick Rowe]

rsj: not just time-varying, but (to some extent) *predictably* time-varying. And not just risk premia, but liquidity premia, and the rate of interest itself.

Matt: OK. But what is it that really is "unpredictable"? Is it just forecast errors that are unforecastable?

[Matt]

"Matt: OK. But what is it that really is "unpredictable"? Is it just forecast errors that are unforecastable?"

At some level, I think so, yes. At least, my copy of asset pricing lecture notes literally proves that m_t*p_t is a martingale as an application of the fact that expectations are a martingale, i.e. that forecast errors are unforecastable.

Does the theory make any operational predictions beyond this? Not so much, I think. The existence of a stochastic discount factor 'm_t', such that p_s = E_s[(m_t/m_s)*p_t] for any time s prior to time t, can be derived from a no-arbitrage assumption under certain conditions. But precisely for this reason, the theory doesn't really have implications beyond the implications of no arbitrage.

The fact that we can think about the current price of a security as the expectation of future price times a state-dependent SDF (an expectation that is, naturally, a martingale, because forecast errors are unforecastable) is, in this view, just a nifty way to think about what no arbitrage really means. It's a little like deriving the existence of a utility function under certain assumptions about preferences, and then proving that preference optima are characterized in a certain way by the utility function.

[Nick Rowe]

Thanks Matt. I *think* I understand that, and agree.

[Avon Barksdale]

"...but my understanding is that the theory asserts that 'm_t*p_t' is the martingale."

Not quite. p(t) = E[m(t+1)x(t+1)] where x(t+1) is the time t+1 payoff. Price is the inner product of the stochastic discount factor with the payoff.

"But precisely for this reason, the theory doesn't really have implications beyond the implications of no arbitrage." Not true. No arbitrage is a condition on the stochastic discount, it has to point in the positive quadrant. p(t) = E[m(t+1)x(t+1)] is really ALL of finance. All factor models, All beta models, all consumption based models can be written this way. You can write the CAPM this way, and any factor model you like. In fact you can derive the efficient mean-variance frontier this way and that is the most natural way to think about it. In incomplete models, m(t) is not unique - multiple choices for m(t) have projections into different subspaces that leave the pricing implications for traded assets the same.

[Avon Barksdale]

Here’s the broad facts: low prices relative to dividends (and earnings) in the 1950s preceded the boom of the early 60s and that the high price/dividend ratios of the mid-1960s preceded the poor returns of the 1970s, and that the low prices ratios of the mid 1970s preceded the boom that went throughout the 1990s. As John Cochrane pointed out in the early 2000s “This rise [of stock prices in 1990s] has cut the postwar return forecasting regression in half. On the other hand, another crash or even just a decade of poor returns will restore the regression. Data going back to the 1600s show the same pattern, but we are often uncomfortable making inferences from centuries-old data. ” He couldn’t have been more right.

It turns out that the stock market’s incredible volatility is EXACTLY the same thing as return predictability: a statement that prices are too high or too low necessarily imply that future returns must correct the levels. So, when prices are high relative to dividends (or any other sensible divisor) one of three things must be true, as an identity: 1) Investors expect dividends to rise 2) Investors expect returns to be low 3) Investors expect prices to rise forever. We find that almost all variation in price/dividend ratios reflects varying expected excess returns. In fact, that the price/dividend ratio varies at all means that either dividend growth or returns must be forecastable. It turns out that it’s returns that are forecastable, not dividend growth.

Fama contributed greatly to this research, and don’t worry, this is completely consistent with informationally efficient markets. Predictable returns reflect time-varying risk premia coming from changing economic conditions. Again as John Cochrane points out “This is a testable view, and Gene takes great pain to document empirically that the high returns come at times of great macroeconomic stress. This does not prove that return forecastability is not due to “fads,” any more than science can prove that lightning is really not caused by the anger of the Gods. But had it come out the other way; had times of predictably high returns not been closely associated with macroeconomic difficulties, Gene’s view would have been proven wrong. Again, this is scientific work in the best sense of the word.”

If people want to learn more about this phenomenon, please read John Cochrane's Asset Pricing.

[Avon Barksdale]

rsj: "In 2009, most were willing to bear risk less than in 2007. Only those unaffected -- those with lots of wealth parked in safe government assets and nerves of steel -- were able to profit."

And those who actually understand the science of asset pricing. I levered up hard in the first half of 2009, so much so that it took me 5 years to pay back the margin loan. The theory of asset pricing told you that expected returns for the decade after 2009 would be high. This stuff is not just some academic exercise, it has real life implications, if we care to follow the logic.

[Matt]

Nick: one afterthought on the comparison to deriving utility functions from preferences. It occurs to me that this comparison is particularly apt if we think about deriving vNM utility from preferences over lotteries. Indeed, there is a close formal similarity between the two.

With vNM, you start with a ranking of lotteries. With the help of the crucial independence assumption -- which implies that you're indifferent between a lottery over lotteries and the corresponding single lottery over the suitably averaged odds -- you can break this down into a function that gives the utility of each individual outcome, such that your utility over a lottery is the corresponding expected value of the outcomes' utilities.

Similarly, in finance, you start with a bunch of prices for assets with state-dependent payouts (which is a little nicer than vNM, because it gives you the cardinality from the get-go). Then with a nearly identical linearity assumption (the law of one price for assets) you can back out a set of implied prices for each state that collectively generate the prices you see. If the assets together span all states, then the implied state price is unique.

I guess this is obvious to someone that's familiar with both (again, I'm a bit of an ignoramus when it comes to asset pricing), but I find the comparison useful. In both cases it turns out that "make some kind of linearity assumption and use it to back out state-by-state values from the data that you have" is a very productive idea. And in both cases the theory doesn't in principle give you anything that you didn't already have from the axioms, but it does give you a very nice way to think through the implications of those axioms.

[Matt]

Avon: "Not quite. p(t) = E[m(t+1)x(t+1)] where x(t+1) is the time t+1 payoff. Price is the inner product of the stochastic discount factor with the payoff."

Yes, of course, I was stating the simple version, and if there are dividends from the asset at t+1 you have to adjust for those (you can define a dividend-augmented version of the price for which the martingale property still holds). When people talk about the predictability of "prices" or lack thereof, they're taking for granted the predictable fall in price after a dividend payment. : )

"Not true. No arbitrage is a condition on the stochastic discount, it has to point in the positive quadrant"

I was thinking (a little sloppily perhaps - admittedly, I don't do asset pricing) about the stochastic discount factor as something that's defined to be positive. If we don't include that restriction in the definition, then it's true that no arbitrage provides an additional condition on the SDF.

If I remember correctly, finance textbooks usually refer to the linearity assumption on asset prices as the "law of one price" (the price of an asset that combines the state-by-state payoffs of two assets is the same as the combined price of the two assets); this, admittedly, isn't enough to ensure that the SDF is positive in every state. To get that, you need to assume something like "no arbitrage", i.e. that not only does the law of one price hold, but there is no zero-price portfolio you can acquire that guarantees you weakly positive payouts in all states and strictly positive in some.

I've never been a huge fan of this progression (to me, "no arbitrage" feels like just a slight additional proviso on top of the key thing, which is the "law of one price"), but that's how it is.

[Matt]

Avon: "It turns out that the stock market’s incredible volatility is EXACTLY the same thing as return predictability: a statement that prices are too high or too low necessarily imply that future returns must correct the levels."

Yes, this is a very good point. I'm continually surprised that people don't realize its implications at shorter to medium frequencies. A lot of commentators have a vague idea that the stock market is way too volatile on a day-to-day basis. But given its volatility at a yearly frequency, it's roughly as volatile on a day-to-day basis as it should be, since there isn't much return predictability at that short of a horizon. (If the variance of a sum is much less than the sum of variances, then the pairwise covariances have to be really negative, which they aren't in this case.)

Of course, the stock market is much more volatile at the yearly frequency than it should be from dividends alone; this is reflected in the longer-term return predictability that you mention. I think the jury is still out on question of whether asset pricing theory can explain this...

"Fama contributed greatly to this research, and don’t worry, this is completely consistent with informationally efficient markets. Predictable returns reflect time-varying risk premia coming from changing economic conditions..."

The question is whether you can justify the time-varying risk premia quantitatively. (Maybe you can statistically explain these premia in terms of variables representing economic conditions, but can you document the economic mechanism behind this statistical relationship in a way that works quantitatively?)

Since asset pricing has trouble even nailing a good explanation of the general risk premium puzzle, it seems like a stretch to say that this has been done. My impression is that consumption-based asset pricing is a bit of a disaster zone quantitatively. You have trouble explaining anything using a reasonable level of risk aversion and reasonable-looking preferences.

[Avon Barksdale]

As a cheat sheet for those who are interested:

law of one price => linearity;
The existence of a discount factor => law of one price; and it goes the other way too;
law of one price => existence of a discount factor (not trivial - this has a nice proof);
There is a strictly positive discount factor m such that p =E[mx] if and only if there are no arbitrage opportunities.

No one said that the world had to be iid-hence the possibility of return predictability.

[Nick Rowe]

Avon and Matt: assume people had state-*in*dependent preferences, that were linear (no declining MU), with constant rate of time-preference. Then stock market returns would be unforecastable. Right? And I can't think of any other sufficient conditions.

Which is a lot like saying that if people couldn't taste the difference between apples and bananas, then apples and bananas would always have the same price.

And the no-arbitrage condition is like saying that the price of a basket with 3 apples and 3 bananas must equal the price of a basket of 2 apples and 1 banana plus the price of a basket of 1 apple and 2 bananas. Which is true, but doesn't get you very far.

And even that no-arbitrage condition won't hold where transactions costs matter, so liquidity is an issue. Like on the run vs off the run bonds, which violate the no-arbitrage condition by having different prices.

[Avon Barksdale]

Nick, yes iid would mean an unforcastable stock market, but the world is not iid.

"And even that no-arbitrage condition won't hold where transactions costs matter, so liquidity is an issue. Like on the run vs off the run bonds, which violate the no-arbitrage condition by having different prices." This not true. The absence of arbitrage means that all portfolios that have positive probabilities of paying off positive, and zero probability off losing, those portfolios must come with a positive price. The difference between on the run and off the run bonds does not qualify because it IS possible to lose money trading between them - LTCM found this out the hard way.

[jonathan]

Okay, I know less about asset pricing than Matt, but I *think* that Shiller's original finding was that the simple p = NPV(dividends) failed. That is, he took the path of actual dividends and prices, and found that high dividends in the past predicted high dividends in the future, *after* taking into account the price. In other words, a high price-dividend ratio predicts the return on stocks will be low, and a low price-dividend ratio predicts the return will be high, *relative* to holding bonds.

Now this doesn't reject E[mR] = 1, but it *does* imply large predictable swings in the risk premium on stocks that many people find implausible.

(This is related to the finding that stock prices are a lot more volatile than you would expect based on the volatility of underlying information).

[Nick Rowe]

Avon: " The difference between on the run and off the run bonds does not qualify because it IS possible to lose money trading between them - LTCM found this out the hard way."

Ah. Because it was short on-the-run bonds, and ran out of margin, presumably?

"Nick, yes iid would mean an unforcastable stock market, but the world is not iid."

OK, that would be one sufficient condition. A second (different) sufficient condition would be the restrictions on preferences that I listed.

jonathan: right. Just like if we see apple prices rise relative to bananas, we look to see if the rise can plausibly be explained by the late frost that killed 20% of the apple blossoms.

It's still a puzzle though, why we see economists saying that EMH implies stock market returns should be unforecastable.

I think it's reasoning from a (forecasted) price change. "Starting in equilibrium, if people suddenly forecasted a fall in the returns on stocks, they would want to sell stocks now, so the price would fall now, eliminating that forecasted fall in the return on stocks." It's similar to saying: "Starting in equilibrium, if the price of apples increased, people would demand fewer apples, so the price would go back down to the original equilibrium."

It depends what causes the (forecasted) price to change.

[Avon Barksdale]

“It's still a puzzle though, why we see economists saying that EMH implies stock market returns should be unforecastable.”

Nick, the EMH tells us that stock market returns are unpredictable at short horizons, but that a stochastic discount factor whose properties slowly vary with time can lead to long term predictability. The stochastic discount factor is a measure change on the probability space – it tells us how to price assets under society's appetite for risk. How society changes the measure can be macroeconomic state dependent. Even Fama recognized this possibility in his original work going back to the early 70s. If the discount factor was iid (that the measure change was immutable), then we have no long term predictability, but since the statistical properties of the discount factor changes only slowly, over short horizons the discount factor does look iid. Think of it this way: In Ottawa, in the middle of January forecasting the temperature 7 days ahead is really hard – the variance is huge. But we all know that as we move from January to spring that temperature increases. The hypothesis is that if the discount factor changes slowly, you should only find predictability showing up on the time frames that drive the slow variation. That's a testable statement and the results – from lots of hard work – show that it comes out the right way. We get just the right amount of long term predictability to explain the high variance of the stock market.

Over short horizons, the market looks like a complete random walk, over long horizons, we see how changes in the macroeconomic environment imprint themselves on the stochastic discount factor (measure change).

This stuff has been seriously worked out in great detail. These same observations are at the heart of factor models, like the Fama-French three factor model. Value stocks have predictably higher returns than growth stocks, even when you correct for their CAPM betas. Again, the stochastic discount factor contains other information about sources of non-diversifiable risk which the three factor model captures and the CAPM doesn't.

Lots of “economists” say silly things like that the EMH says that returns are unforecastable. That's not true, and the ones that say stuff like that don't understand their discipline. If they understood a bit more measure theory, a bit more about Hilbert spaces, understood how most of asset pricing is the Riesz representation theorem in drag, they wouldn't throw such stuff out there. I see this all the time in the news, economists claiming all sorts of expertise when I can tell that haven't calculated anything.

[nivedita]

I think one of the sources of confusion, especially when talking about stock prices/returns, is the confusion between spot and forward prices.

To take Avon's example of temperature -- while it is true that you can predict that the spot temperature will rise between January and April, the forward temperature in April is unpredictable. i.e. in January, our best guess about April temperature is that it will be what it is expected to be. Come April, it may be higher or lower than what we predicted, and there is no way to really improve upon this.

Similarly, if the price of apples is low or high depending on the quality of the harvest, and the harvest is essentially unpredictable, then if this year was a good year, you can reliably forecast that the spot price of apples will rise next year. However, you cannot predict whether the price of apples next year will be higher or lower than what it is expected to be.

In terms of finance, you cannot really make money from predictable spot moves -- to make money, you need predictable forward price moves, and what the EMH is saying is that predictable forward price moves don't exist.

[Matt]

Avon: "the EMH tells us that stock market returns are unpredictable at short horizons, but that a stochastic discount factor whose properties slowly vary with time can lead to long term predictability".

I think this is where things become tricky. We can derive the existence of an SDF 'm' from the law of one price, and then m_t*p_t is a martingale - providing us with some form of the "EMH", which is just a restatement of the fact that rational expectations are a random walk.

But this is not operational unless we have some restrictions on the SDF, and obviously we can't get those restrictions from the law of one price. Instead, we have to build the SDF from the ground up, modeling where it comes from. One simple way to do this is to say that it's the MU of investors in each state. In principle, it seems like this should "slowly vary with time", so then we can say that the price itself is close to being a martingale at high frequencies - which is certainly a nice, operational result.

The problem is that the entire field of consumption-based asset pricing, which is built around this idea of getting the MU from the level of consumption and plugging it in as the SDF, is to my understanding an empirical disaster zone. The first disaster (one of many) was the equity premium puzzle itself: the volatility in aggregate consumption is far too small to justify much of any risk premium on equities. The wild attempts to fix this -- trying to justify unrealistically high risk aversion parameters, or baking in extreme preferences for early resolution of uncertainty with Epstein-Zin -- seem, from my outsider's perspective, to merely underline the problem.

---

Nick: "Avon and Matt: assume people had state-*in*dependent preferences, that were linear (no declining MU), with constant rate of time-preference. Then stock market returns would be unforecastable. Right? And I can't think of any other sufficient conditions."

Right. (Though note that this is assuming that the SDF comes from preferences in the first place, which as noted above does not follow merely from the formal derivation of the SDF from the law of one price. As a macro guy, I'm happy to make this leap, but sad that it doesn't seem to work well in explaining actual asset prices.)

[Nathan W]

Where is it written that people are rational, even imagining that they have good information and actually know their own preferences without contradiction from one day/year to the next?

[Mike Freimuth]

OK, coming late to the party here and not a finance guy, so much (but not all) of the technical details here are eluding my casual perusal of this thread. But I have a possibly boneheaded question. What is the technical definition of "forecastable?" For instance: I can "forecast" that a 1 year treasury bond with a 2% yield will rise in price by 2% in a year but obviously, I can't profit on that (I can "profit" by 2% but that is, in a sense, the going rate of profit). Likewise if I can "forecast" that the stock market is more likely to rise by 5% one year and only 2% another year, in each year I can "profit" by 5% or 2% respectively but that's the same profit everyone else can get. If the "forecast" changes from one year to another, it doesn't seem to violate the EMH, it just means something like the discount rate or liquidity premium or risk premium changed. These things are all subjective and only observable by observing market prices. So what, exactly is the hypothesis regarding forecastability that could be either confirmed or refuted somehow?

For the record, I believe "no arbitrage" as discussed above can be observed and refuted but seems to hold up pretty well. But this doesn't seem to be the same as what Noah has in mind when he talks about forecasting. In short, I don't see the " deep property of stock markets that we don’t understand -- some force as profound as the concept of efficient markets, but which also makes markets inefficient." Unless, of course that property is just some combination of subjective risk tolerance, uncertainty, time preference and liquidity preference, but I don't see the conflict with efficient markets.

[Mike Freimuth]

Nathan,

It's written either implicitly or explicitly at the beginning of every econ book or paper. If you start a theory with irrational people, the whole thing is just arbitrary rambling. Anything can happen if you impose no notion of rationality on actors. And to say people "don't know their own preferences" is confusing the definition of preference. And nobody ever wrote that they can't change from one day/year to the next, it just doesn't make for a very interesting theory to say "hey, peoples' preferences might change."

[Nick Rowe]

Mike: my dumb question was the same as your boneheaded question.

[Mike Freimuth]

yeah I kind of thought as much. And I kind of feel like we both know the answer. And I kind of feel like the technical points made here agree with our answer. I'm just not totally sure whether or not finance people agree with our answer when you ask it in a dumb/boneheaded way or not.

[Nick Rowe]

Mike: nor am I. But some of the commenters above know a lot of finance, and they don't seem to me to be disagreeing with us. Just saying the same thing in a more technical way.

[Mike Freimuth]

Yeah that seems to me like what they are saying too. It seems like Noah is on a different page from us though. And for some reason they game Schiller a Nobel prize. I get the sense that what he said had been misrepresented by many since then but I'm not really sure what it was. Probably the best way to figure it out would be to just read his paper which I may do one of these days.

[Jack PQ]

There are some great technical comments, but let me try to put it simply (and less accurately).
1. Forecasting is a statistical question, and the answer depends on your metric (calibration, sharpness).
2. In finance, we start with no-arbitrage. No one can make a riskless profit requiring zero investment.
3. The Fama-Hansen-Shiller Nobel prize for in part for bridging points 1 and 2.
4. Fama says (very loosely) you cannot make profitable short-run forecasts. Yes, expected returns can be forecast in the sense that they are driven by different priced risks, and the more an asset loads on priced risk, the higher the expected return. But that's not arbitrage because it's not riskless.
5. Shiller says (loosely) yes you can make profitable long-run forecasts. The trick is you have to be willing and able to hold a long run strategy, and you cannot be scared off by downturns. Here, expected returns are not entirely explained by priced risk, and you can make a pretty sure profit (still not arbitrage) through certain strategies.

So, are returns predictable? Yes, but that doesn't mean you can make easy money, unless you are indifferent to risk.