by Bill Brower on September 17, 2016 | Categories: Uncategorized |
Curve fitting in strategy performance is when the historic back-test looks much better than the real-time performance. Here are some basic ideas to keep in mind when developing a trading strategy.
1) The ultimate objective is to have a smooth equity curve.
2) Find a way to avoid curve fitting.
3) Measuring return without measuring risk is meaningless.
4) Simple is usually better than complex.
While there are many other issues that could be discussed in regards to strategy development, these 4 are very basic. Let’s take them one at a time.
Smooth Equity Curve
When it comes to an equity curve, there are many tools that have been developed to assist in this process. For instance, the Tradestation strategy performance report includes the RINA index which does a fair job of accounting for things that could degrade an equity curve. The Pessimistic Return-to-Risk Ratio that I developed and have published in the LBE book is even better. Perhaps the best tool is to visually inspect the equity curve. Some traders believe that you should test your strategy on an in-sample dataset and when you have developed it to your satisfaction, you test it on an out-of-sample data set. If the out-of-sample data set results are satisfactory then you can be sure you have a robust strategy. The flaw with this concept is that the trader continues to modify the concept and test it again and again and again until eventually the out-of-sample dataset is satisfactory. I would argue that the trader should have used the entire dataset to begin with and looked at the equity curve to be sure that the behavior across the entire timeframe was uniform and satisfactory because that is essentially the result he has attained by throwing out all of the bad results. Either way, there is a likelihood that the result is a curve fit.
It is very difficult to avoid curve fitting because we do not have sufficient quantities of daily data to generate a lot of trades. An example of a way to avoid curve fitting would be to require that the same strategy with the same parameter settings be used on all symbols traded. In fact this was how I built and developed trading strategies for a portfolio of 36 futures symbols between 2008 and 2010. This resulted in over 2000 trades in the historic back-test. It is difficult to curve fit a strategy with that many trades. If you plan to trade stocks, you probably have far more opportunities to take trades and if you carefully select trading opportunities you might even be able to run your strategy with the same parameter set and generate more than 2000 trades. However there is a bit of difficulty with stocks because they are so highly correlated so there may be 2000 trades but in reality you may have only 200 unrelated trading opportunities. Sorting that out could be very difficult. You could drop the timeframe down to a lower level and look to day-trade but, you might be stepping into a difficult arena where professional traders with access to high-frequency data lines and vast sums of money can outgun you. Still there are places that you can trade where the high-frequency traders and the big firms don’t like to play. For instance the Russell 2000 future is generally considered too illiquid for those types of firms.
Any measure of a strategy performance must include both return and risk. To any seasoned trader this seems totally obvious but there have been countless clients of mine that have been enamored by a measure of a strategy’s return. That could be net profit, percentage of winning trades, average winning trade, or a number of other measures. A successful trader needs to understand that trading is more about managing risk than measuring profit. It is like a poker game where your goal is to find a way to keep your seat at the table. In my experience a very good trading strategy is likely to have the same annual return as risk. So if you plan to make 15% annually you are likely to have a 15% drawdown during the year. Usually drawdowns are greater than expected by any testing measure you can perform so, when you’re drawdown exceeds the 15% you may be knocked out of the game.
Building a trading strategy should always start with the simplest algorithms and build-up to more complex. A common error is to combine more than one concept in the 1st go. This means there is no way to measure to what extent one concept is positively influencing the performance. You should always be able to quantify the extent to which each element is positively contributing to the performance and this can only be done by testing the components one it time or at least testing 1 component and then testing that component with another to see if the 2nd component improves the result. This is true even for the simplest concepts like profit targets, stop losses and trailing stops. The more components that are added to a trading strategy the easier it is to end-up with a result that is totally curve fitted. For instance, when you add a profit target and optimize it, you may find that only a small percentage of the trades are impacted. Let’s say you had a strategy that took 200 trades and the profit target affects 15 trades. By any measure, 15 trades is not a significant data sample to generate reliable statistics. Therefore it is a curve fit. I suspect that most traders don’t understand the risk of this kind of curve fitting.
While the main focus of the EasyLanguage Learning-by-Example training program is to teach you the programming language, a good deal of the material will be presented in a way to teach you good practices in strategy development. Some of these exercises are specifically directed at strategy development and teach you how to measure performance of individual algorithms in an objective way. You will also be learning techniques that will help you avoid curve fitting and allow you to build more robust trading strategies. Some of these techniques include learning how to use the Pessimistic Return-to-Risk Ratio, using random entry/random exit routines for bench testing algorithms and even measuring performance using Portfolio MCS.