LEARN · EN · easyquanttrading.com
Why trying many strategies breaks the one you keep
Search hard enough and you will find something that looks like an edge. This is not a comment on your skill or your data — it is arithmetic. The problem is that the act of searching changes what your best result means, and the change is invisible unless you deliberately account for it.
- Selecting the best of many attempts inflates the winner's apparent performance
- The effect grows with the number of attempts and with their similarity
- The correction is called the Deflated Sharpe Ratio
- The cheapest defence is to record how many things you tried
The coin-flip demonstration
Give a thousand people a fair coin and ask each to flip it twenty times, recording the longest run of heads. Somebody will get ten in a row. That person has not discovered a biased coin; they have discovered that a thousand attempts is a lot of attempts.
Strategy search works the same way. Test enough rule combinations and the best one's performance includes the luck of the draw, not only the quality of the idea.
Why it is worse than it sounds
Two factors make the problem more severe than the coin example suggests. The first is that strategy variations are highly correlated: testing twenty moving-average periods is much closer to testing one idea twenty times than to testing twenty independent ideas, and the best of a correlated set still benefits from selection.
The second is that the selection is invisible in the output. A backtest report shows the winner's statistics with no indication of how many alternatives were discarded, so the number looks like a measurement when it is closer to a maximum.
The correction, in outline
The Deflated Sharpe Ratio adjusts the observed Sharpe for the number of independent trials and for the non-normality of returns. The mechanics involve the expected maximum of a set of random variables, which is why the adjustment can be large.
| Trials attempted | Rough effect on a Sharpe of 1.0 |
|---|---|
| 1 (a pre-specified idea) | Essentially unchanged |
| 10 | Modest discount |
| 100 | Substantial discount |
| 1,000 | The result may no longer be distinguishable from noise |
| 4,500 | A measured Sharpe of 1.0 can deflate to near zero |
Illustrative magnitudes, not exact values. The direction is what matters: more trials means a larger discount, and the effect compounds quickly.
What this means for how you work
- Write down the number of configurations you tried, before you look at the results. That number is part of the result.
- Prefer pre-specified ideas over searched ones. A hypothesis stated in advance needs no correction.
- If you must search, hold back data and search only on the rest. The held-out portion has not been contaminated by the search.
- Be suspicious of results that are only marginally better than the alternatives. The selection effect is largest where the field is crowded.
- Report the trial count whenever you report a result. A performance number without it is incomplete.
The uncomfortable part
Most published strategy results do not report how many attempts preceded them, which means most published results cannot be interpreted. This is not a criticism of any particular source; it follows from the structure of the problem.
The practical consequence for you is that a strategy found by someone else's big search deserves more scepticism than one you specified yourself, even if the former has better numbers. The numbers are not comparable, precisely because the searches were not.
FAQ
- Does this mean optimisation is always wrong?
- No. Optimisation is a legitimate tool for finding robust regions of parameter space, which is different from finding the single best point. Fitting a parameter and then testing on held-out data is fine; choosing the parameter by its performance on the data you then report is not.
- How do I count trials when I did not keep track?
- Estimate generously. The count only needs to be the right order of magnitude to be useful, and undercounting defeats the purpose. If you genuinely do not know whether it was fifty or five hundred, assume the larger figure.
- Is a strategy from a provider automatically suspect?
- It is unverifiable rather than automatically suspect. You cannot know the search that produced it, so you should apply your own out-of-sample test and your own cost model before drawing conclusions.
More guides
- How EasyQuant validates strategies — evidence you can filter
- Honest backtesting, not pretty curves
- Gold strategy research that stays honest
- Overfitting detection: catch it before you deploy
- System Forge: design, then prove
- Walk-forward analysis: the only backtest that fights overfitting
- MT5 export without custody
- Glass box, not black box AI signals
Not investment advice. Historical results do not guarantee future performance. EasyQuant is a research factory — you execute on accounts you control.