Backtesting Biases and Risks

Definition

Backtesting biases refer to how the results of a trading strategy backtest can be misleading.

Description

Here are the 8 common biases:

1. Black Swan Reconciliation
2. Survivorship Bias
3. Spreads
4. Cost of carry/Holding costs
5. Inaccurate Price Simulation
6. Change in Contract Specifications
7. Look-ahead Bias
8. Curve-Fitting and Optimization Bias

Bias 1 – Black Swan Reconciliation

Black swan events refer to events that come as a surprise and have a huge impact.

Brokers and exchanges may alter the prices of assets after a volatile price moves (black swan events). There are 2 types of alteration.

Type 1 – Changing the fill price

After an unexpected large price move, brokers and exchanges might change the prices that you got filled on your trades.

Example

EURUSD is trading at 1.1300. A black swan event occurs and EURUSD spikes up 2000 pips (to 1.3300) (1 pip = $0.0001).

You long EURUSD 1000 pips into the 2000 pips move. You are long EURUSD at 1.2300. It is now trading at 1.3300. You close the trade at a 1000 pips profit.

A few hours after the trade, you receive an email saying that “In view of this unexpected event, all trades will be cleared at 1.1800 price”.

Your 2000 pips profit becomes a 500 pips loss. Your account gets wiped out.

Real example: Saxo Trades Lawsuits With Clients After Swiss Currency Turmoil

Type 2 – Changing their historical price

After an unexpected large price move, brokers and exchanges might not change the prices that you got filled on your trades.

However, they alter the price on the historical charts and data. Thus, the prices you see in your charts are different (almost always worse) than the prices you get in live trading.

In your backtests, you might have bought Apple shares at $180, but in real life, you would have gotten those shares at $250.

Bias 2 – Survivorship Bias

Survivorship bias, or survival bias, refers to the fact that people overlook entities/processes that failed because they only see successful entities/processes.

Example

We are selecting a bunch of stocks to trade. We create a list of criteria to identify potentially successful stocks.

Next, we filter the universe of stocks listed in the US based on these criteria.

And with that, survivorship bias just got to us. This universe of stocks only includes stocks that survive. There may be stocks that are delisted but fit our criteria.

We need to consider those stocks as well to give us an idea of how sound our strategy is.

Bias 3 – Spreads

The difference between the price we can buy at (bid price) and the price we can sell at (ask price) is called the spread.

Spreads change in real time. It depends on the buyers and sellers on exchanges, or brokers.

During volatile events, spreads usually widen, sometimes by a 100 times.

Without accurate bid and ask data, these spread widening events will make our backtests inaccurate.

Bias 4 – Cost of carry/Holding costs

If you are leveraged (you trade a size larger your capital by borrowing from the broker), shorting or trading a derivative, you might need to pay interest to hold your positions.

This interest represents the fees needed to cover the capital loaned to you, or the costs to hold any underlying assets.

These holding costs might vary without warning during the lifetime of a trade. Hence, it is difficult to estimate these costs in your backtest.

Example

The usual interest cost to short a stock is less than 2% a year.

However, for a period in early 2019, the cost to short Tilray, a cannabis stock shot up to over 800% a year.

Bias 5 – Inaccurate Price Simulation

Not all backtesters replicate the exact historical price movement, some use simulated fake price movements.

This might not be significant if you make a few trades a year and analyze the market using end-of-day data.

However, your backtest results will be greatly skewed if your strategy is related to scalping (price action and movement) and fires many trade per day on lower timeframe data.

Bias 6 – Change in Contract Specifications

An exchange or broker may change the contract specifications (i.e. details) of their products.

For instance, they may increase margin requirements, change the settlement specifications or contract size of their products. These may lead to jumps in market prices.

The main takeaway here is – in such cases, do not take a price change at face value. Your P&L may not change proportionally to a price change.

For instance, increasing the margin requirement for silver may cause silver prices to fall. In your backtest, your short silver position may look like it is doing well. However, if you had traded that move in real-life, you may get a margin call and be forced to close the position.

Real-life examples

• Margin change didn’t cause silver slide – CME
• ICE Changes Gold Contracts Specifications, Opts for Physical Delivery of Bullion

Bias 7 – Look-ahead Bias

Look-ahead bias involves having prior knowledge of how the market behaves before running a backtest.

Example

You want to run a strategy that takes advantage of trends. You look for assets that trend and discard those that don’t trend.

You then run a backtest on these assets using a trending strategy. Unsurprisingly, your strategy does well.

These tests are not useful as you have only chosen assets that you know would have done well in your backtests.

Bias 8 – Curve-Fitting and Optimization Bias

Curve fitting is the process of adapting a trading system so closely to the past that it becomes ineffective in the future.

Optimizing strategies too closely to past data will result in inflexibility to adapt to the future. Hence, it leads to poor performance in the future.

We need to adapt our trading strategies to signals in historical data, not noise.

Links to Other Explanations

• Backtesting Biases and How To avoid them – Auquan
• Backtesting bias and how we avoid it – Keepstock
• 9 Mistakes Quants Make that Cause Backtest to Lie – Dr Balch

Related Terms

• Algorithmic Trading Strategies
• Backtesting (Upcoming)
• Market Inefficiencies