[jackb]

One of the most remarkable aspects about traders who pound the table about back-testing (particularly system traders), is that few do the requisite dirty work to compile a decent data set to test off of. I can't tell you how many times I have spoken to vendors at trade shows about the processes they used to arrive at the set of data upon which they have now generated output metrics of their system. Most have no idea what I'm talking about. Occasionally, I'll get someone who tells me they use 2 sources from different data providers to cross-reference output metrics. Sadly, most are quick to rationalize why they have no intention of undertaking any effort to address the shortcoming. It's hard to say if it's laziness or stubbornness.

Friday on the SPX is a perfect example of why any serious back-tester needs to take the time and come up with a scrubbed data set. My broker's opening price on this index is way off and I'm finding the same error with most other data providers (probably because the problem lies with the exchange). If you're not maintaining a contemporaneous scrubbed data set, then the only way to somewhat get around that is to have multiple data sets (a minimum of 3, imo). Even then, they may all be reporting the same invalid number. How big of difference can it make? Well, a couple of days being off a couple of points could be enough to materially skew results. In the case of last Friday, it's not off a couple of points...rather it's about 10 points. And this happens more than just a few times during any given year.

So hopefully the next time someone comes running up to you with a printout of favorable metrics for a system, you'll now know the first question to ask. For those of you running free and loose with only one data set (likely the one from your broker), you should be seriously skeptical about any of the back-testing results.

In summary, forward-testing rules.

[SIUYA]

Great point and 100% correct, and that is why backtesting can give you an idea of how something might work, but is in no way a predictor of exactly what to expect....it is just another indicator.
Data management is a major part of back testing, and the ongoing management of data when auto trading is another aspect that many do not take into consideration - especially if trading a variety of instruments. If you think you can set and forget then you can probably kiss your money bye bye
If you trade longer term models, slippage can make a difference but having poor data makes an even bigger difference, and while many system vendors trade shorter term they still underestimate the importance of having accurate data that can cope with the real world. (Today I missed a stop as my slippage was a few extra points and it also meant I did not get a reversing order completely done - no biggie, but it turned a theoretically flat day into a looser - a model would not show this)

I never challenge system sellers as I dont care for them but its definitely a point to make for those that do.