Why 100 strategies are usually 3 ideas

Convergence is the whole point — and the whole problem

A search method that did not converge would be useless. Convergence is what makes a genetic algorithm, a tree search or a template sweep find anything at all: the population or the grid settles on regions of the space that fit the data. But settling has a second face. Once a search finds a structure that fits, the cheapest way to improve the fit is to keep returning to that structure with slightly different settings.

The result is a candidate list where a small number of structural ideas appear many times under different parameters. Nothing has gone wrong. The search did what it was asked to do. But the list now misrepresents itself: it looks like twenty findings and is actually two.

How correlation de-duplication works

• Generate the return or signal series for each candidate on the same dataset and window, so the comparison is between behaviours rather than between parameter values.
• Compute pairwise correlation between candidates. Two rules with different parameter values and near-identical signal series correlate very highly — that is the signal we are looking for.
• Cluster above a correlation threshold. Each cluster nominates a representative, and every other member is marked as redundant with respect to that representative.
• Record the correlation to the representative, so a marginal case is visible rather than silently classified.
• Re-count. The number of candidates is unchanged; the number of independent ideas is what falls, and that is the number that matters.

Why this is not a defect in any particular engine

It is tempting to read redundancy as a sign that an engine is weak. It is the opposite. An engine that produces highly diverse output on a single dataset is usually an engine that is not converging — which means it is not really searching, it is sampling.

The redundancy rate is largely a property of the dataset and the hypothesis space, not of the optimiser. A narrow space explored thoroughly will produce a great deal of internally correlated output. That is what thorough exploration of a narrow space looks like.

Which is why the honest response is to measure and disclose it rather than to engineer it away. Suppressing near-duplicates at generation time would be easy to fake and would hide the real number.

What we found when we measured our own shelf

We applied correlation clustering to our own candidate shelf and published the result, because a de-duplication method is only credible if its author is willing to be de-duplicated. Figures below are from our shelf as of September 2026 and will move as the shelf changes.

• Across the whole shelf, roughly 48% of candidates cluster as redundant with another candidate. So the effective number of independent ideas is about half the visible count.
• On XAUUSD D1 with the CTA template sweep, 33 candidates were generated and, after correlation grouping, zero were independent — every one was a variation on a single structural idea.
• On XAUUSD M15 with Tree-GP, 56 candidates collapsed to 2.
• The pattern is not uniform across markets. Gold cells clustered at 91–100% redundancy, while crypto cells sat at 60–71%. The same engines, the same procedure, a materially different amount of independent structure found.

The gold-versus-crypto contrast is the interesting part

We did not expect the market split to be that wide, and we do not have a definitive explanation. The plausible reading is that the search space is simply less concentrated on the crypto datasets: fewer candidates fit the quality bar at all, and the ones that do are less alike.

We are flagging it as an observation rather than a conclusion, because a two-market comparison with a single procedure is a hypothesis-generating result, not a finding. It is on this page because a platform that only publishes results it can explain is a platform that is curating. We would rather show the number we cannot fully account for.

What to do with this as a user

• Before counting your strategies, cluster them. A portfolio of fifty correlated strategies has less diversification than its position count suggests, and its risk is concentrated in whatever idea they share.
• Before correcting for multiple testing, de-duplicate. One hundred correlated candidates are not one hundred independent trials. Counting them as such overstates the correction; ignoring the correlation understates it. Neither approximation is good.
• Look at the fitness landscape, not just the optimum. If a rule works at one parameter set and fails on either side of it, you are looking at a fitted point rather than a stable region.
• Treat breadth claims sceptically everywhere, including here. If a product advertises a large strategy library without publishing a redundancy measure, the number you are being shown is the gross count, not the independent one.

How this is handled in EasyQuant

Redundancy is tracked as structured data, not as a footnote. Candidates carry a redundancy status, a cluster identifier, and the correlation to their cluster representative, all of which are visible rather than collapsed into a single quality score.

This connects to two other things on the platform. First, the trial-count correction used for overfitting statistics depends on knowing how many independent tests were really run, and redundancy clustering is how that number is estimated. Second, portfolio construction depends on it for the same reason: a basket of highly correlated strategies is a concentrated position with a diversified appearance.

We treat this as the same category of disclosure as everything else here — the failure modes belong on the page, not in a footnote. If the shelf is mostly one idea, the shelf should say so.