{"id":70,"date":"2019-07-31T22:07:11","date_gmt":"2019-07-31T22:07:11","guid":{"rendered":"http:\/\/algotrading101.com\/learn\/?p=70"},"modified":"2020-06-24T17:16:41","modified_gmt":"2020-06-24T17:16:41","slug":"what-is-overfitting-in-trading","status":"publish","type":"post","link":"https:\/\/algotrading101.com\/learn\/what-is-overfitting-in-trading\/","title":{"rendered":"What is Overfitting in Trading?"},"content":{"rendered":"
<\/div>

<\/path><\/svg><\/i> \"Loading\"<\/p>

<\/div>\n
\"\"
Source<\/a><\/figcaption><\/figure><\/div>\n\n\n\n

What is overfitting in trading? Overfitting in trading is the process of designing a trading system that adapts so closely to historical data that it becomes ineffective in the future. <\/strong><\/p>\n\n\n\n

Overfitting (AKA curve fitting) your strategy gives you false confidence that your strategy will be profitable. In fact, if you overfit your backtests well enough, you might produce strategies that seemingly make thousands of percent per year.<\/p>\n\n\n\n

How do we overfit?<\/h2>\n\n\n\n

We overfit by adapting our strategies to noise instead of signals. <\/p>\n\n\n\n

Signals are useful fundamental information. Noise are distractions that don’t offer useful qualities.<\/p>\n\n\n\n

\"Overfitting
Figure 1: Overfitting data points on a chart<\/figcaption><\/figure>\n\n\n\n

In figure 1, we have 3 charts with the same data. We are trying to create a model that fits the shape of the data. This model will be used to predict future data points. <\/p>\n\n\n\n

We can clearly see that the data forms a U shape. This U shape is our signal. <\/p>\n\n\n\n

In the leftmost chart, our model is a straight line. This is not representative of the data at all. It will have poor predictive abilities. We describe this model as an underfitted model.<\/p>\n\n\n\n

In the rightmost chart, our model intercepts every data point. This model fits the data perfectly. On paper, this seems like the perfect model. <\/p>\n\n\n\n

And it is, if we are not using it to predict the future. <\/p>\n\n\n\n

You see, unless future data points follow the past perfectly, this model will have very poor predictive value. This is an overfitted model.<\/p>\n\n\n\n

In the middle chart, our model describes the general shape of the data points. This model does have some level of error \u2013 it does not intercept all the data points. <\/p>\n\n\n\n

However, this is fine. We need our models to have a certain degree of error. This means that the model does not rigidly follow the past. <\/p>\n\n\n\n

This model captures the signal in the data (signal refers to the U-shaped data points). Thus, it should be able to adapt to minor changes to the data structure in future. <\/p>\n\n\n\n

This model is a good fit aka it is robust. <\/p>\n\n\n\n

Is overfitting bad?<\/h3>\n\n\n\n

The short answer is, yes.<\/p>\n\n\n\n

The past does not predict the future perfectly, especially in financial markets.<\/p>\n\n\n\n

Adapting strategies too closely to past data will result in an inflexibility to adapt to the future. Hence, it leads to poor performance in the future.<\/p>\n\n\n\n

How do we reduce overfitting?<\/h3>\n\n\n\n

Design strategies that exploit fundamental inefficiencies that make sense from a market or economic point of view. <\/p>\n\n\n\n

Examples of fundamental inefficiencies:<\/p>\n\n\n\n