Walk-forward testing

AmiBroker 5.10 features the automatic Walk-Forward test mode.

The automatic Walk forward test is a system design and validation technique in which you optimize the parameter values on a past segment of market data (”in-sample”), then verify the performance of the system by testing it forward in time on data following the optimization segment (”out-of-sample”). You evaluate the system based on how well it performs on the test data (”out-of-sample”), not the data it was optimized on. The process can be repeated over subsequent time segments. The following illustration shows how the process works.

The purpose of walk-forward test is to determine whenever the performance of optimized trading system is the realistic or the result of curve-fitting. The performance of the system can be considered realistic if it has predicitive value and performs good on unseen (out-of-sample) market data. When the system is properly designed, the real-time trading performance should be in relation to that uncovered during optimization. If the system is going to work in real trading, it must first pass a walk-forward test. In other words, we don't really care about in-sample results as they are (or should be) always good. What matters is out-of-sample system performance. It is the realistic estimate of how the system would work in real trading and will quickly reveal any curve-fitting issues. If out-of-sample performance is poor then you should not trade such a system.

The premise of performing several optimization/tests steps over time is that the recent past is a better foundation for selecting system parameter values than the distant past. We hope is that the parameter values chosen on the optimization segment will be well suited to the market conditions that immediately follow. This may or may not be the case as markets goes through bear/bull cycle, so care should be taken when choosing the length of in-sample period. For more information about system design and verification using walk-forward procedure and all issues involved, we can recommend Howard Bandy's book: "Quantitative Trading Systems" (see links on AmiBroker page).

To use Walk-Forward optimization please follow these steps:

1. Goto Tools->Automatic Analysis
   
2. Click Settings button, then switch to Walk-Forward tab
   
   
   
3. Here you can see Walk forward settings for In-sample optimization, out-of-sample backtest
   
   Start and End dates mark initial period begin / end
   This period will be moved forward by Step until the End reaches the Last date.
   
   The Start date can move forward by step too, or can be anchored (constant) if Anchored check is on.
   
   If you mark Use today then Last date entered will be ignored and TODAY (current date) will be used instead.
   By default an “EASY MODE” is selected which simplifies the process of setting up WF parameters.
   
   It assumes that:
   a) Out-of-sample segment immediatelly follows in-sample segment
   b) the length of out-of-sample segment equals to the walk-forward step
   
   Based on these two assumptions the “EASY” mode takes in-sample END date and sets out-of-sample START date to the following day. Then adds in-sample STEP and this becomes out-of-sample END date.
   
   In-sample and Out-of-sample step values are set to the same values. The “EASY” mode guarantees correctness of WF procedure settings.
   
   You should use Easy mode (EOD) when testing on end-of-day data or Easy mode (Intraday) when testing on intraday data. The difference is that in EOD mode the END date of previous period and START date of next period are the same - thus avoiding gap
   between periods. Intraday mode set START date of the next period as NEXT DAY after END of previous period. That guarantees
   that boundary day is not counted twice when testing on intraday data.
   
   In the Advanced mode, the user has complete control over all values, to the extent that they may not constitute valid WF procedure.
   The interface allows to selectivelly disable in-sample and out-of-sample phases using checkboxes at top (for special things like running sequential backtests without optimization).
   All settings are immediatelly reflected in the PREVIEW list that shows all generated IS/OOS segments and their dates.
   
4. The “Optimization target” field defines the optimization raport COLUMN NAME that
   will be used for sorting results and finding the BEST one. Any built-in column can be used
   (as appears in the optimization output), or you can use any custom metric that you define
   in custom backtester. The default is CAR/MDD, you can however select any other built-in metric from the combo.
   You can also TYPE-IN any custom metric that you have added via custom backtester interface.
   
5. Once you defined Walk-Forward settings, please go to Automatic Analysis and
   
6. press the dropdown ARROW on the Optimize button and select “Walk Forward Optimization”
   
   This will run sequence of optimizaitons and backtest and the results will be displayed in the “Walk Forward” document that is open in the main application frame. When optimization is running you can click “MINIMIZE” button on the Progress dialog to minimize it - this allows to see the Walk Forward output during the optimization steps.

IN-SAMPLE and OUT-OF-SAMPLE combined equity

Combined in-sample and out-sample equities are available by ~~~ISEQUITY and ~~~OSEQUITY composite tickers (consecutive periods of IS and OOS are concatenated and scaled to maintain continuity of equity line - this approach assumes that you generally speaking are compounding profits).

To display IS and OOS equity you may use for example this:

PlotForeign("~~~ISEQUITY","In-Sample Equity", colorRed, styleLine);
PlotForeign("~~~OSEQUITY","Out-Of-Sample Equity", colorGreen, styleLine);
Title = "{{NAME}} - {{INTERVAL}} {{DATE}} {{VALUES}}";