Download the audit counts (CSV)中文报告 · Aggregate audit snapshot; raw price history is not included.
LEARN · EN · easyquanttrading.com
When a larger backtest holdout leaks training data
A longer test period is not independent evidence if it reuses data already seen during strategy discovery. In our October 4, 2026 NZDUSD daily audit, a terminal-history test contained 1,001 bars, but 620 overlapped the discovery training or validation period. The corrected frozen holdout contains 110 bars and remains pending under our configured 1,000-bar minimum.
By the EasyQuant Research Team·Published 2026-10-04·We publish the tests our own strategies fail. Nothing here is a return promise.
- Observed dataset: MT5_NZDUSD_D1, 730 bars from December 12, 2023 through October 2, 2026.
- Chronological split: 510 training bars, 110 validation bars and 110 final holdout bars.
- Two LinearGP candidates were replayed against the same frozen boundary.
- Pending evidence blocks deployment certification; this audit does not establish profitability.
What we measured
This is a diagnostic case study of one dataset and two candidates, not a benchmark of the entire strategy library. The audit compared chronological discovery boundaries with a separately loaded terminal history. Counts describe price bars, not trades or independent statistical observations.
| Check | Before correction | Frozen holdout |
|---|---|---|
| Evaluation bars | 1,001 terminal-history bars | 110 reserved discovery bars |
| Training / validation overlap | 620 bars (61.94% of 1,001) | 0 bars by split boundary |
| Configured minimum | 1,000 bars | 1,000 bars |
| Current evidence status | Historical window was not independent | Pending: 110 is below 1,000 |
Audit snapshot, October 4, 2026. The denominator for 61.94% is the old 1,001-bar terminal test.
How to reproduce the boundary check
The implementation uses evaluation identity version frozen_holdout_20261004_v1. Dataset content, signal identity and validation configuration participate in cache identity so evidence from another split is not silently reused.
- Sort the 730-bar discovery dataset chronologically and retain its timestamps and content identity.
- Reserve rows 0–509 for training, 510–619 for validation, and 620–729 for the final holdout.
- Keep the final 110 rows out of candidate ranking and fitness evaluation.
- Replay final evidence only on the reserved timestamps, May 4 through October 2, 2026.
- Compare test timestamps with all training and validation timestamps. Any intersection invalidates a claim of independent testing.
- Keep status pending when the configured evidence minimum is unmet; adding older history does not repair independence.
What this result does and does not show
The audit demonstrates an independence defect and verifies the corrected evaluation boundary. It does not demonstrate positive expected returns, successful broker execution, or future live performance. A 110-bar daily window is short, and adjacent bars can be dependent. A minimum expressed in bars is a configuration rule, not a substitute for trade-count and statistical-power analysis.
Two candidates were checked in this case study. The same rule is enforced in the evaluation code, but this report does not claim that every historical library record has been revalidated. Reproducible research should publish the dataset boundary and evidence status before discussing performance.
Current platform facts
Read live from the strategy library when this page was generated. These are the same counts published on our transparency page, and they change as strategies are added and rejected.
| Strategies in the audited library | 3794 |
|---|---|
| Flagged by the audit | 2128 |
| Flag rate | 56.1% |
| Checks still pending | 1656 |
| Passed the DSR overfitting check | 1 |
| Passed the significance check | 513 |
| DSR threshold used | 0.90 |
FAQ
- Does a larger out-of-sample period make a backtest more reliable?
- Only if the additional observations remain independent of discovery, selection and tuning. Reusing training or validation history increases the sample count without adding clean held-out evidence.
- Can a strategy with pending validation be called deployment certified?
- In the corrected pipeline, pending, missing or errored required evidence blocks deployment certification. Research candidates and completed certification are separate states.
More guides
- How EasyQuant validates strategies — evidence you can filter
- Overfitting detection: catch it before you deploy
- Walk-forward analysis: the only backtest that fights overfitting
- Walk-forward analysis: testing a strategy the way you would actually trade it
- Expectancy vs profit factor: when to use which
- Look-ahead bias in trading: examples and an audit checklist
Not investment advice. Historical results do not guarantee future performance. EasyQuant is a research factory — you execute on accounts you control.