Backtesting As a Multi-Step Process Pt. 2

[RichardCox 10]

In part one of this article, we looked at key elements of the backtesting process and discussed some of the differences between “in-sample” and “out-of-sample” data. We also looked at some of the complexities and procedures involved when optimizing strategies so that traders can use these methods to settle on an approach to technical analysis that is likely to result in long-term profitability. It is important to read the first part of the article, as there are some basic steps in the process that must be first understood before moving on to the final steps. In the next sections, we will look at correlation as a critical component of the total process.

 

Defining Correlation

 

In this context, correlation refers to the performance similarities that can be found when comparing a technical analysis strategy that is applied to more than one data set. The results from correlation measurements will allow traders to spot broad trends and evaluate the performance of a given strategy over your chosen test period. When strong correlations can be found between results generated from all data sets (in-sample data and out-of-sample data), there is an enhanced probability the system will generate positive results when used in the following steps (forward testing and in live trading).

 

There will be cases where a system is found to be curve-fit to perform strongly with one of your data sets but these situations fail to meet the requirement threshold to make it to the next phase of the testing process. When these situations are seen, it is likely that the system is over-optimized and much less likely to generate long-term gains when applied to live trading conditions. High levels of correlation mean that the system is ready for the next step, which requires further out-of-sample data testing. This data will form the basis for the next part of the process, which is forward testing.

 

System Rules when Forward Testing

 

Forward testing might sound overly complicated for traders new to the process. But forward testing is simple paper trading (trades are placed with a demo account using “virtual” money). With an additional set of out-of-sample data, forward testing gives traders new information with which to test a strategy that has already performed well (and shown strong correlation) in historical back tests. Forward testing simulates the live trading environment and gives us a better idea of how a strategy will work when real trades are placed.

 

This is an important step of the process because there has been no optimization in the same way there was with historical price data. Optimization at this phase of the process would be impossible because there is no way to know where prices will travel next. This is also why it is important to place these trades with a demo account first, before moving on to live trading with real funds. The results (profits and losses) of each forward testing trade will be recorded, so it is vital that the rules of your system are rigorously followed. Any changes made (with respect to the rules of the system) will cloud the data and make it difficult to accurately assess the viability of the system in achieving consistent profitability.

 

Making Valid Assessments

 

When forward testing, it might become tempting to cherry-pick some trades that might look especially attractive and neglect others. There might be instances where your system send a trading signal but you look at the situation and avoid the trade because you believe it is unlikely you would actually take the position with a real account. But what is important to remember here is that you are not testing your skills as a trader, you are testing the viability of your proposed system. If the rules of your system send a signal to execute a trade, that is the trade that should be placed. This is the only way to properly evaluate your strategy.

 

So, while the term “forward testing” might seem as though we are attempting to predict the future, the fact is that we are really only backtesting with live data, rather than historical price data. Because of this, the same standard apply when we look to assess the performance of a proposed system. We are looking for trading results (profits vs. losses) to show consistent profitability through all phases of the testing. This includes in-sample data, out-of-sample data, and forward tested data (or live data). You will also need to see strong correlations in all phases of your testing. When this is found, you will know you have a trading system that holds up even in situations that were not optimized. This means you have found a system that is likely to perform well in active markets (with real funds).

 

Conclusion

 

Backtesting a strategy can give traders some highly valuable information when looking for new ways to approach the markets. Most of the latest trading platforms enable traders to conduct these tests, and here we looked at some of the key elements required when we evaluate regular trading programs. There are many critics of backtesting, as people will often argue that historical price data does little to tell us how markets will behave in the future. While there is some validity to these arguments, we do have ways of dividing up historical data to increase the probability that our assessments are accurate.

 

When we make our determinations using in-sample and out-of-sample data, traders have an efficient (and relatively easy) way of evaluating the potential success rate of a technical analysis strategy. These strategies can also be optimized (in the hope of improving on potential profit performance) but it is important to retest your approach on multiple data sets in order to prevent your trading profits from becoming a “self-fulfilling prophecy” that might not work under live trading conditions.

 

As a final step, traders should implement out-of-sample forward testing, as this puts your strategy up against a new set of price activity and provides another layer of protection against potential losses. Once these steps are completed, you are ready to start actively trading with real money. So, while the general process cannot guarantee your next trades will be profitable, it does increase your chances of success when compared to systems that have received no backtesting. Ideally, you will be looking for strong profit performance and high levels of correlation between all data samples, as this creates the best scenario for systems later used in live markets.

[gianno 10]

Backtesting is one of the most dangerous practices known to man.