HomeKnowledge Base

Why do backtest results change?

From time to time we receive questions about why you can get different results back-testing the same code.

There are five reasons for differences in backtest results:

  1. Different data (for example if past history is updated/changed due to splits for example or backfill)
  2. Different settings / parameters (if your formula uses Param() functions that output values that may be changed from the Parameter window). Note that Parameters in Analysis window use shared ChartID=0, which means that if two formulas use same parameter names they would share parameter values as well.
  3. Different formula (sometimes even slight change to the formula causes big change in the results, for example if your formula uses #include and included code has changed)
  4. The formula that self-references its previously generated results. Such code produces some data that is later used to produce next run output (for example your code produces composites that are later used – if those composites change – the input data change so the results change, or if your formula uses previous backtest equity or if your formula uses STATIC variables without deleting/clearing them at the beginning)
  5. The formula directly or indirectly calls any function that produces random numbers such as Random or mtRandom, or functions that read external data that have changed