{"id":464,"date":"2019-05-15T16:22:44","date_gmt":"2019-05-15T16:22:44","guid":{"rendered":"http:\/\/algotrading101.com\/wiki\/?p=464"},"modified":"2021-01-08T21:13:59","modified_gmt":"2021-01-08T21:13:59","slug":"backtesting-biases-and-risks","status":"publish","type":"post","link":"https:\/\/algotrading101.com\/wiki\/backtesting-biases-and-risks\/","title":{"rendered":"Backtesting Biases and Risks"},"content":{"rendered":"\n

Definition<\/strong><\/p>\n\n\n\n

\n\n\n\n

Backtesting biases refer to how the results of a trading strategy backtest can be misleading. <\/p>\n\n\n\n

Description<\/strong><\/p>\n\n\n\n

\n\n\n\n

Here are the 8 common biases:<\/p>\n\n\n\n

  1. Black Swan Reconciliation<\/li>
  2. Survivorship Bias<\/li>
  3. Spreads<\/li>
  4. Cost of carry\/Holding costs<\/li>
  5. Inaccurate Price Simulation<\/li>
  6. Change in Contract Specifications<\/li>
  7. Look-ahead Bias<\/li>
  8. Curve-Fitting and Optimization Bias<\/li><\/ol>\n\n\n\n
    \n\n\n\n
    \"black
    Black swans in real life<\/figcaption><\/figure><\/div>\n\n\n\n

    Bias 1 – Black Swan Reconciliation<\/strong><\/p>\n\n\n\n

    Black swan events refer to events that come as a surprise and have a huge impact.<\/p>\n\n\n\n

    Brokers and exchanges may alter the prices of assets after a volatile price moves (black swan events). There are 2 types of alteration.<\/p>\n\n\n\n

    Type 1 – Changing the fill price<\/u><\/p>\n\n\n\n

    After an unexpected large price move, brokers and exchanges might change the prices that you got filled on your trades.<\/p>\n\n\n\n

    Example<\/em> <\/p>\n\n\n\n

    EURUSD is trading at 1.1300. A black swan event occurs and EURUSD spikes up 2000 pips (to 1.3300) (1 pip = $0.0001). <\/p>\n\n\n\n

    You long EURUSD 1000 pips into the 2000 pips move. You are long EURUSD at 1.2300. It is now trading at 1.3300. You close the trade at a 1000 pips profit.<\/p>\n\n\n\n

    A few hours after the trade, you receive an email saying that \u201cIn view of this unexpected event, all trades will be cleared at 1.1800 price\u201d. <\/p>\n\n\n\n

    Your 2000 pips profit becomes a 500 pips loss. Your account gets wiped out.<\/p>\n\n\n\n

    Real example: Saxo Trades Lawsuits With Clients After Swiss Currency Turmoil<\/a><\/p>\n\n\n\n

    Type 2 – Changing their historical price<\/u><\/p>\n\n\n\n

    After an unexpected large price move, brokers and exchanges might not change the prices that you got filled on your trades. <\/p>\n\n\n\n

    However, they alter the price on the historical charts and data. Thus, the prices you see in your charts are different (almost always worse) than the prices you get in live trading.<\/p>\n\n\n\n

    In your backtests, you might have bought Apple shares at $180, but in real life, you would have gotten those shares at $250.<\/p>\n\n\n\n

    \n\n\n\n

    Bias 2 – Survivorship Bias<\/strong><\/p>\n\n\n\n

    Survivorship bias, or survival bias, refers to the fact that people overlook entities\/processes that failed because they only see successful entities\/processes. <\/p>\n\n\n\n

    Example<\/u><\/p>\n\n\n\n

    We are selecting a bunch of stocks to trade. We create a list of criteria to identify potentially successful stocks.<\/p>\n\n\n\n

    Next, we filter the universe of stocks listed in the US based on these criteria.<\/p>\n\n\n\n

    And with that, survivorship bias just got to us. This universe of stocks only includes stocks that survive. There may be stocks that are delisted but fit our criteria. <\/p>\n\n\n\n

    We need to consider those stocks as well to give us an idea of how sound our strategy is.<\/p>\n\n\n\n

    \n\n\n\n

    Bias 3 – Spreads<\/strong><\/p>\n\n\n\n

    The difference between the price we can buy at (bid price) and the price we can sell at (ask price) is called the spread.<\/p>\n\n\n\n

    Spreads change in real time. It depends on the buyers and sellers on exchanges, or brokers.<\/p>\n\n\n\n

    During volatile events, spreads usually widen, sometimes by a 100 times. <\/p>\n\n\n\n

    Without accurate bid and ask data, these spread widening events will make our backtests inaccurate.<\/p>\n\n\n\n

    \n\n\n\n

    Bias 4 – Cost of carry\/Holding costs<\/strong><\/p>\n\n\n\n

    If you are leveraged (you trade a size larger your capital by borrowing from the broker), shorting or trading a derivative, you might need to pay interest to hold your positions.<\/p>\n\n\n\n

    This interest represents the fees needed to cover the capital loaned to you, or the costs to hold any underlying assets.<\/p>\n\n\n\n

    These holding costs might vary without warning during the lifetime of a trade. Hence, it is difficult to estimate these costs in your backtest.<\/p>\n\n\n\n

    Example<\/u><\/p>\n\n\n\n

    The usual interest cost to short a stock is less than 2% a year. <\/p>\n\n\n\n

    However, for a period in early 2019, the cost to short Tilray, a cannabis stock shot up to over 800% a year. <\/p>\n\n\n\n

    \n\n\n\n

    Bias 5 – Inaccurate Price Simulation<\/strong><\/p>\n\n\n\n

    Not all backtesters replicate the exact historical price movement, some use simulated fake price movements. <\/p>\n\n\n\n

    This might not be significant if you make a few trades a year and analyze the market using end-of-day data.<\/p>\n\n\n\n

    However, your backtest results will be greatly skewed if your strategy is related to scalping (price action and movement) and fires many trade per day on lower timeframe data. <\/p>\n\n\n\n

    \n\n\n\n

    Bias 6 – Change in Contract Specifications<\/strong><\/p>\n\n\n\n

    An exchange or broker may change the contract specifications (i.e. details) of their products. <\/p>\n\n\n\n

    For instance, they may increase margin requirements, change the settlement specifications or contract size of their products. These may lead to jumps in market prices.<\/p>\n\n\n\n

    The main takeaway here is \u2013 in such cases, do not take a price change at face value. Your P&L may not change proportionally to a price change. <\/p>\n\n\n\n

    For instance, increasing the margin requirement for silver may cause silver prices to fall. In your backtest, your short silver position may look like it is doing well. However, if you had traded that move in real-life, you may get a margin call and be forced to close the position.<\/p>\n\n\n\n

    Real-life examples<\/u><\/p>\n\n\n\n