Listen Up, Backtesting is NOT Real-Time
I’ll take a month of real-time trading over a 10-year backtest any day of the week.
That may be a surprise coming from someone whose blog almost wholly consists of backtests, but at the end of the day I know (and so should you) that NONE of this really matters. The only thing that matters is what you do in real-time, audited.
That’s why I don’t release backtests for our own proprietary strategies and I raise a wary eyebrow anytime I see a sexy backtest bandied about.
It’s not that I don’t trust the backtester. There’s a whole universe of good folks whose workmanship I respect very much, but at the end of the day ‘the best laid schemes of mice and men often go awry’.
The table above lists the evil little demons that lead our backtests astray. None of these are new…just a reminder of what we already know (but sometimes forget).
There are things we should be able to control…the hard-and-fast, black-and-white “math” of backtesting.
Have we accurately modeled the trading environment including transaction costs, slippage, realistic quotes, and survivorship bias? Small mistakes here compounded = hugely inaccurate results.
Have we built a mathematically-sound model? You would be shocked to know how often I test published strategies only to find that the results rubbish (reader beware).
And there are things we can try to control but never totally will…the far more fuzzy “art” of backtesting.
We have to cope with curve-fitting and other biases (read more from CXO), markets that are constantly evolving at the most fundamental level (an example) and markets that inevitably go through abrupt “abnormal” periods where nothing works the way it’s supposed to (another example).
. . . . .
The only way – the ONLY WAY – to judge how well a trader has responded to this myriad of challenges is real-time audited results.
It’s easy to churn out fancy charts showing what has worked in the past. It’s a very different thing to put yourself on the line each and every day.
Done right (independently-audited without cherry-picking) there are no mulligans. Your moments of glory and defeat, of brilliance and stupidity, are laid bare.
We the investing community get way too excited about sexy backtests and make way too half-hearted a demand for the real-time.
In my mind, in your mind, in all of our minds, 1-month of real-time audited trading should mean more than 10-years of backtesting any day of the week.
. . . . .
Filed under: Random Stuff | 18 Comments