What is backtesting?

What a backtest actually does

A backtest engine walks through historical bars in order. At each bar it evaluates your rules using only the information that existed at that moment, decides whether the position should change, and records the outcome. At the end you have a record of simulated decisions. That is the whole mechanism. Everything a backtest can tell you comes from that record, and everything it cannot tell you comes from the fact that the data is already in the past.

The single most common way to get this wrong is look-ahead: accidentally letting a decision use information that would not have been available yet. A normalisation step computed over the whole series, a high or low that includes the current bar, a resampled indicator that peeks forward, or a survivorship-filtered list of instruments are all ways for future knowledge to leak into a past decision. A backtest with look-ahead is not optimistic, it is fictional.

Why a good-looking backtest is weak evidence

• You chose the rules after seeing the data. Even with honest intentions, the idea usually arrives because something in the history suggested it.
• You tried more than one version. Ten variations produce ten results, and the best of ten is not the same claim as the best of one.
• The market regime in your sample may not repeat. A rule tuned to one period of low volatility can look durable and behave differently when conditions change.
• Costs were assumed rather than modelled. Spread, commission, slippage and financing decide whether a high-frequency rule survives contact with execution.
• The sample is short relative to the number of decisions. Twenty trades can look convincing and still be indistinguishable from luck.

The checks that turn a replay into evidence

Each of the following addresses a specific way a backtest misleads. None of them is sufficient alone, and each one can be reported separately so you can see which claims rest on which test.

• Out-of-sample: hold back a period, develop on the rest, then test once on the held-back data. The result is more honest because the rules never saw it.
• Walk-forward: repeatedly develop on one window and test on the next, rolling forward. A chain of consistent out-of-sample segments is much harder to fake than one curve.
• Monte Carlo resampling: shuffle or resample the sequence of outcomes to see whether the result depends on a lucky ordering of events.
• Trial counting and selection-bias correction: record how many candidates were searched, then adjust the significance of the winner for that search. Without a trial count this correction cannot be computed at all.
• Cost re-testing: re-run the surviving candidates with wider spreads and realistic slippage, then see which ones still hold up.
• Parameter sensitivity: perturb the parameters slightly. A result that collapses under a small change is usually fitting noise rather than structure.

What backtesting is not

A backtest is not a forecast, and it is not a promise. It cannot tell you what will happen next, because it only knows what already happened. What it can do is eliminate ideas cheaply and show you the conditions under which an idea stopped working. That is a genuinely useful function: the point of a test is to let you discard things before they cost you anything.

This is also why a high failure rate inside a search process is normal rather than alarming. If you generate many candidate structures and most of them do not survive validation, the process is doing its job. The number worth paying attention to is not how many candidates you produced, but how much of that list remained genuinely independent after screening.