When is the stock market right, and when does it reflect mass confusion?

by Alan Cohen

I am somewhat interested in ideas of crowdsourcing and collective wisdom. There are clearly times when the average opinion of lots of normal people is a more reliable predictor than experts. (In fact, there are numerous studies debunking the expertise of experts. But that’s an aside.) There are many examples of this, and the betting market Intrade is one of the principle forums for such collective wisdom, with good histories predicting election results among other things.

But even better known than Intrade are stock markets, which – though designed for other purposes – effectively pool collective wisdom about the values of publicly traded companies, as well as about the state of the economy in general. When a company (or the market in general) is undervalued, savvy individuals will buy in. When it is overvalued, they will sell. We don’t need to know a whole lot about the individual buyers and sellers, since the market effectively averages their opinions into the price of a stock or the level of market indices. But, as we saw during the 2008 US financial crisis, this collective wisdom can be wrong – very wrong – and with grave consequences for the rest of society. So how can we tell these two apart?

The principle idea of this post is that there are some times when collective wisdom works well, but that there are other times when it works poorly because of collective psychology. And I think we can distinguish these ideas statistically. This is probably old news in the world of finance and economics, but because I am neither an economist nor a financier, and you probably aren’t either, here is my take on it:

Most of the time investors on Wall Street are looking at specific stocks and evaluating their value relative to their price. Sometimes, such as when the Federal Reserve makes an announcement about interest rates or when economic data are released, collective knowledge will affect the calculations of many people at once, causing the entire market to go up or down a bit in response. But this is still a relatively rational process. However, once in a while economic conditions become quite uncertain, and individual investors become very confused and have a tendency to follow the actions of other investors rather than their own judgement. During the US financial crisis, we saw this often: the market would drop a bit, and within a few hours everyone would pile on board and the market would drop 5% in one day. Then the next day the reverse would happen and it would regain the lost ground.

It seems pretty clear that what happened with the 2008 market fluctuations was due to mass hysteria rather than some reliable collective wisdom. My contention here is that we can detect mass hysteria statistically. The math is pretty simple and intuitive, so I hope that the non-statisticians among you will stick with me while I explain.

When mass hysteria takes over the markets, we see wide swings day-to-day. What this means is that, relative to the average value of the market over a week, there is a lot more variation day-to-day when no one knows what they are doing. We can measure this by smoothing the data using moving averages. A moving average is where we take the average of days before and after a given day to calculate its value. For example, a three-day moving average averages a given day, the day before it, and the day after it. A five-day average takes two days on either side, a seven-day average takes three days on either side, and so forth. Once we have a moving average we can see how much each day’s raw estimate differs from its moving average estimate. Here, for example, is raw data for the January 2008 Dow Jones Industrial average (in black) with three different moving averages and, on bottom, their differences from the raw data:

Notice how the red line (the 7-day moving average) is much smoother than the black line, and as a consequence, the red dotted line, reflecting the difference between the daily value and the 7-day moving average, goes up and down more than the others. This data is just for one month to show the principle, but we can easily take this 7-day average difference since January 2000 and compare it to the Dow Jones. It’s much easier to do this if we smooth it as well using a 31-day moving average: in this case, it just helps with the visualization:

And here we see something very interesting: the peaks in black on the bottom (when there’s a lot of uncertainty and day to day variation in the market) seem to come at about the same time as the troughs in red, the actual value of the market. And this suggests that there’s a lot of uncertainty in the markets right around the time they drop sharply.

More interesting still, we can ask if these two things are perfectly synchronized, or if one predicts what’s about to happen in the other: it would be very useful to predict a coming drop in the market, if we could do so base on volatility today. To run this analysis, we can calculate a correlation between the two. A perfect correlation has a value of 1, a perfect negative correlation has a value of -1, and 0 indicates no correlation. In order to see whether we can make predictions, we calculate correlations of lagged data. For example, rather than correlate the moving average for today with the value of the DJI today, we correlate the value of the moving average for today with the DJI for tomorrow (lag of +1) or yesterday (lag of -1). In fact, we do this for lots of different lags, from -100 to +100, to see which predicts the best, in both directions:

So here we see that there is a negative correlation for most lags, and that it becomes stronger as the lag becomes more positive. A correlation of -0.4 is not particularly strong in general, but it is quite strong in terms of being able to predict the behavior of the stock market. And we find this correlation with a positive lag: in other words, a lot fluctuation in the markets this week means that there’s a good chance the markets will go down substantially over the next few months. And you can imagine that this sort of information could be very useful to those seeking to make some money.

Pure-minded academic that I am (hee hee), I present this more out of interest than as a way to profit. And there are substantial caveats: This data was chosen because I knew it contained the 2008 financial crisis. The results would need to be replicated for other time periods and for other markets to before we could say they are truly robust. But it certainly seems that we can detect irrational behavior in the markets, and that this irrational behavior often presages a drop!