After many years of running back tests and analyzing the results, I have developed a check list of some of the most important things to look for in evaluating them. Some of these things are technical in nature and are often not disclosed in the end result you see, but all of them are very important in determining the degree of trust that should be given to the output of the tests.
Here are a few key things to look for.
1 – Does the Test Account For Human Emotions?
This is obviously a trick question because back tests can’t account for the human element of investing. This human element comes into play in two ways. First, the manager of a strategy is likely to deviate from it at some point in the real world. Even if it is a quantitative strategy, the temptation to alter it when it isn’t working will be strong. Second, the investor in it is also likely to panic and make poor decisions somewhere along the way. These are both likely to cause real world results to be significantly worse than the back test. It is difficult to quantify how much worse, but the more the strategy loses during the down periods and the more it deviates from its benchmark, the bigger the gap is likely to be.
2 – Does the Strategy Have an Economic Reasoning Behind It?
When you test something, it is very important that the input used for the test relates to the output in some intuitive way. In other words, if you test a strategy that relates valuations to stock prices, that makes sense. If you test phases of the lunar cycle against stock prices, that does not.
3 – Does the Test Cover a Very Long Period of Time?
Cycles in the market can last a lot longer than you think. Investing styles can go in and out of favor for long periods of time. It is important that any investment strategy shows that it can perform well across multiple market cycles to ensure its performance isn’t just the result of a trend that won’t persist long-term. If you were to run back tests on the past decade, you would likely conclude that value investing is a waste of time and growth investing produces significant outperformance. Looking at 50 years of data, however, would yield the opposite conclusion.
Even periods of 20+ years can be problematic. For instance, consider the current belief many have that bonds provide a hedge for stocks during market declines. If you look at the past 30 years, you would find strong evidence to support that statement. The major exception to it, though, is that both stocks and bonds can struggle during periods of high inflation. A test would need to go back into the 1970s to see the most recent example of high inflation, and any test that didn’t include that period may lead to an incorrect conclusion.
4 – Does the Test Show Periods where the Investment Strategy Struggles?
If something seems too good to be true, it almost always is. Every investment strategy will go through periods where it doesn’t work. If you see a back test that does not include these periods, you should run, not walk, away from the investment strategy it is being used to advertise.
5 – Does it Assume Knowledge that wasn’t Available until After the Fact?
This is a common problem in back testing. It is easy to know what happened after the fact. It is easy to know now that value investing didn’t work from 2007 on. It was impossible to know that then. If armed with what I know now I go back and look for indicators that might have told me to avoid value starting in 2007, I could easily find them. But it wouldn’t have been so obvious at the time. Any good back test should only include what could have been known at the time and not information that only became obvious after the fact.
This is by no means an all-inclusive list and there are many other more technical things (like testing a strategy out of sample and avoiding survivorship bias) that are key to successful back testing, but the most important point here is that all back tests should be viewed with a degree of skepticism. When used properly, back tests can be a useful tool for evaluating the pros and cons of an investment strategy, but more often than not they are instead used as a marketing tool and don’t accurately depict how a strategy will perform in the real world.
Like everything else in investing, a common-sense approach can help to avoid relying on test results that are clearly flawed. If something seems to good to be true, it likely is. There is perhaps no area of investing that applies more to than back testing.
For more investment strategies, visit the Leveraged Inverse Channel.