Backtests have become the weapon of choice for rationalizing various forms of tactical asset allocation, which has become increasingly popular as a risk-management tool since the 2008 crash. The hazards of backtesting—studying how a strategy performed in the past–are well known, which leads some folks to shun the concept entirely. But that’s going too far.
In some respects, every investment plan owes a debt to some type of backtesting—even for a buy-and-hold strategy, which assumes that the future will deliver gains on par with what was earned in the past. The proper lesson is that designing robust backtests, which demands close attention to detail. Easier said than done, of course, in part because the pitfalls can be subtle. Here are three that routinely trip up the novice and perhaps even some experienced investors:
1) the use of total-return prices for technical signals
2) failing to correct for look-ahead bias by not using lagged signals
3) overlooking the importance of neutral signals for computing backtest results
The good news is that these traps are easily avoided. But there’s a catch: you have to be aware of the hazards. With that in mind, let’s briefly review these backtesting snares with some simple examples.
Total return data. Imagine that you’ve created what you think of as a winning investment strategy that’s based on two signals: a) the ratio for a set of short and long moving averages; b) the trailing return for a rolling x-day window. The results look encouraging, but the upbeat outcome may be an illusion if the calculations use total return prices.
Why? Consider a mutual fund that’s unchanged on the day but dispenses a hefty distribution at the close of trading. Imagine that this fund is priced at $10 a share and it spits out a 50-cent-per-share payout. Although the underlying portfolio value was unchanged on the day the mutual fund’s price falls by 50 cents to $9.50 to compensate for the distribution. The net result for shareholders: their holdings in the fund remain unchanged on the day. The 50-cent-per-share drop is offset by a 50-cent distribution. In short, a net wash.
It’s a routine affair in day-to-day market activity but it’s a trap if you’re looking at a fund’s technical profile without adjusting for distributions. Let’s say that the 50-cent price decline pushes the fund into negative territory in terms of the short/long moving-average ratio and trailing x-day return. On the surface, this looks like a sell signal when in fact it’s nothing of the sort since the fund’s portfolio value hasn’t changed.
The solution is to use price data that’s strips out distributions. If you don’t make that adjustment, your backtests using technical signals are probably faulty. Keep in mind too that the total return price histories aren’t real in the sense that the prices have been retroactively adjusted down to compensate for dividends, capital gains, etc. In other words, total return prices weren’t available in real time through history. Ignoring this issue runs the risk that your backtests are telling lies.
Lagged signals & avoid look-ahead bias. This is another common mistake that can turn a sow’s ear into pearls, if only on paper. There are many variations to this trap, depending on the complexity of the strategy, but the basic form can be illustrated with a simple example.
Take a strategy that issues a “sell” signal when price falls below an x-day moving average and a “buy” when price rises above that average. Let’s also assume that we’re using end-of-day closing prices. You test the strategy and discover that it delivers a strong performance through time. But you forget one small item: the end-of-day signals aren’t available until after the market closes. In other words, calculating returns for a real-world version of the strategy requires using lagged “buy” and “sell” signals.