Backtests Ain’t Worth the Pixels They’re Written On
The most common feedback (by far) I’ve received about our proprietary strategies over the last 6+ years have been requests for backtested results.
My response has always been that I don’t release backtested results for our own strategies…ever.
Backtests are perfectly fine for testing, sharing, and discussing ideas; that’s more or less the only thing we do here on this humble blog.
But backtests should never be used to judge the efficacy of a black box you’re considering committing hard earned funds to.
I’m cynical. I’ve seen way too many examples of folks who sold based on backtests, only to fall woefully short in real-time trading. I’m sure you could recount your own share of horror stories.
The only thing that really matters in judging a strategy is actual, real-time, verifiable trading results. Everything else is window dressing.
If I could somehow guarantee that folks would see backtests for what they’re really worth: best guesstimates, guilty of the same cognitive biases that we ALL face and inherently curve-fit, I would release them in a heartbeat.
But I can’t guarantee that.
I know that readers’ eyes get wide when they see a sexy equity curve because even though I know I shouldn’t, mine do too.
So sorry, but I can’t in good conscience share backtests. Rest assured that they would be sexy, and your eyes would go wide, but that, like all backtests, they wouldn’t be worth the pixels they were written on.
Happy (actual) trading,
ms
P.S. Tangentially-related link on this subject (h/t Abnormal Returns):
http://www.institutionalinvestor.com/Article/3081318/Study-Finds-Many-ETF-Indexes-Misleading.html
. . . . .
To stay up to date with what’s happening at the MarketSci Blog, we recommend subscribing to our RSS Feed or Email Feed.
Filed under: Random Stuff | 14 Comments
I am not protecting backtest, but in your opinion, if a quantitative based strategy doesn’t even pass backtest (e.g. big drawdown, low to negative return, high volatility etc.) does it worthwhile to pursue further even if it “make sense” (whatever that means)?
Hello EEE – of course not, but who sells a (quantitative) black box with an ugly backtest? That would be a first for me…
I agree about the dubious value of backtested results. I do, however, want to suggest that bactests are inversely useful to the number of moving parts they contain. In other words, the more stuff in the backtest, the less likely that it will continue to be robust in the futures. Conversely, exceedingly simple backtests have, I think, a greater chance of matching future returns with the fiction of past performance.
Hello Alex – I totally agree, but that’s a whole different discussion.
This post is about folks selling black boxes. With a black box, by definition you don’t get to know what’s under the hood. My point is that THOSE backtests are meaningless (or at least any worth they might have is outweighed by the fact that they tend to harm investors by leading them astray).
Tangentially this post could also be about say, a dynamic ETF where the rule set is disclosed (and backtested). But as the link at the end of my post shows, these too tend to fall painfully short (because they are also, like ALL backtests, curve fit). In that case, to your point, yes, less moving parts could be a good thing, but IMHO that good isn’t enough to justify using them as a marketing tool.
michael
Excellent points, Michael. If you don’t mind, could I ask you a question?
In your opinion, how long should one (either a seller of a system or simply an individual who has developed one) trade live before he can be reasonably assured that it works? I’ve heard everything from one year to several market cycles, so I’d love your views on the subject.
Hello Alex – great question without a single simple answer. IMHO it depends on the nature of the strategy.
A longer-term strategy (ex. trend-following or TAA), because of how infrequently it trades, requires a longer track record over different market regimes and multiple entry/exits, That could be years.
A hyperactive system trading many times per day might need just a few months or less to prove out the basic concept (but that’s a SWAG b/c this is definitely not my cup o’ tea).
An active swing trading strategy (like what I usually trade) would fall somewhere in between. In a perfect world you would want to see many contrasting market regimes, but sometimes that’s not feasible. Inefficiencies always get shutdown by the market eventually. Sit frozen on the sidelines too long and you’ll never reap the rewards.
Just my $0.02.
michael
Michael,
I agree wholeheartedly with your point, with the following clarification: backtests *developed by somebody else* are useless, unless you have access to all the info on the backtest, and can duplicate it yourself. Even then, they may still be useless, depending on how the backtest was structured.
But, I will also say that backtests are essential to developing a new system. After all, if it doesn’t even backtest well, what are the chances it will trade fine going forward? (maybe this point is so obvious it didn’t need to be mentioned, but I didn’t want someone to walk away thinking there’s no sense in backtesting)
One technique I like (because it’s fast and easy, not necessarily rigorous), is to develop the system using in-sample data, and then to test it by looking at out-of-sample data *on both ends* of the curve. If a system it works well with data that it has never seen that is both before AND after the in-sample data, then I think your odds of success going forward are improved. (not guaranteed, just improved)
Of course, all of this is somewhat off subject, and does not in any way diminish your main point.
=Carlos=
The reason the financial industry shows backtests is as a marketing tool. They use backtests in much the same way companies who sell products use advertisements. How else do you convince somebody who’s never seen or heard of what you’re doing that it’s going to work?
The best way is of course to show live trading results. That is basically your point and I agree with it 100%. Nothing but live trading matters, and in a perfect world that’s all we would ever see. The problem with that is, most people don’t have several years of live trading results to show people. They usually only have a few months with a new system before they feel ready to release it to the public.
We can never know the future of the trading system, obviously. What we can know is how well it is performing in the present, as well as how well it would have performed in the past. For me personally, i’d like to see both. Maybe i’m not representative of the average guy I don’t know, but I just feel that live results coupled with a backtest is a more powerful combination than just the short term live trading results on their own.
I think it comes down to how you are using the backtest. If like most of the financial industry, the backtest is being used to sell a sub-par system off to the public and make money off their ignorance, then I think it’s despicable. However if the backtest is being used in an honest way, taken with a grain of salt, to increase peoples understanding of the product then I think it’s actually a good thing to show it.
I agree with all the above. Here’s a follow-up on what I meant by a simple system.
Basically, I think that as the system gets simpler, the distinction between backtesting and live trading diminishes. With a very simple system, I think that backtesting is equivalent to live trading.
For example, let’s say that, for some reason, you believe that a good system is to buy the S&P 500 when it’s above its 3 month moving average and to go to bonds when it’s below the average. You don’t curve fit, you don’t adjust, you simply believe in this system. Now, you backtest since 1990 and you get X results. I think that it’s not unreasonable to believe that, in the future, your average return for every five or so years (I just picked that number out of thin air, btw) will be pretty close to X. Of course, there may be anomalies along the way, but my point is that live trading is also subject to anomalies so, I think, is no more trustworthy than my backtesting thought experiment.
Of course, all bets are off if the system keeps getting adjusted (i.e., curve fit) to improve results along the way.
That is utter nonsense. Saying back tests do not have a place in quantitative algorithmic trading is like saying that we can let lose some wild swinging cowboys to trade large prop positions. Nobody on the algorithmic hedge fund side would ever give you shot if you could not at the very least show back-tested results that underlie realistic assumptions.
Now do not get me wrong, of course live trading results trump hypothetical results at any time, but categorically ruling out the usefulness of back tests makes no sense and much rather makes me wonder whether you just got duped on a too-good-to-be-true back tested strategy as soon as you turned on the live switch.
Of course I would dismiss 95%+ back tests of not being rigorous enough and of not making more realistic assumptions, however, this does not dismiss the necessity to run such tests.
Matt – if you’re going to flame someone, I find it useful to first read the post.
The point was not that backtests are unnecessary to what you do internally – to your own design process. That’s basically all I do and the whole function of this blog.
The point is related to SELLING based on a backtest (as opposed to actual results) b/c most readers don’t understand the biases that are inherent in all backtests.
michael
Michael, well them maybe you want to make adjustments to your title and first 1-2 paragraphs because it sounds very “flamy”. No, I find your blog post misleading in that you first blindly resort all back testing of being useless and then later you pick up on details (most of which I agree with, actually), in that you almost make a 180 throughout your post. I see how you want to set yourself apart from the pack and a little self-advertising never hurt but I just did not get your generalizations in the first half of your blog post. By the way you claim you are audited, do you mind sharing which auditor you use? Thanks.
Matt – I have no tolerance for snarky back and forth – I’ll let your comment stand and let readers judge for themselves the point of the post.
In the early years I used Theta Research. The YK Strategy and earlier programs are likely still available there (or at least they were last I checked). All of my programs except the most recent two using ETFs were also tracked by Timer Trac (links available at MarketSci.com). Since moving on to actively trading ETFs, something TT doesn’t support (as opposed to leveraged mutual funds, my old vehicle of choice), I provide actual account statements (also available at MarketSci.com). For managed accounts and/or subadvisory relationships of significant size, we also provide signed auditing statements by outside accounting firms. But that is of course expensive, so not something I do except when justified.
michael