Overfitting

Also seen as: Curve-fitting, curve fitting

Why does an overfitted strategy fail out of sample?

An overfitted strategy fails out of sample because every historical price series contains two components — a repeatable one and an unrepeatable one — and fitting cannot tell them apart. As you add rules, thresholds, and exceptions, the strategy gains the flexibility to describe finer and finer detail in the sample. Some of that detail is a real effect that will show up again; the rest is coincidence specific to those particular dates. Past a certain point, each extra degree of freedom buys mostly coincidence, so in-sample performance keeps improving while out-of-sample performance flattens and then deteriorates.

This is the standard bias–variance trade-off applied to trading rules: a very simple rule may systematically miss part of the real effect, while a very flexible one reproduces the sample almost exactly and generalises badly. The complexity that performs best on the fitted sample is therefore almost never the complexity that performs best afterwards.

How many parameter combinations does a search really cover?

The number of distinct strategies a parameter search covers is the product of the values tried for each parameter, so it grows multiplicatively, not additively.

where: $k$ is the number of tunable parameters, $v_i$ is the number of values tried for parameter $i$, and $N$ is the number of distinct strategy variants the search covers.

Four parameters with ten candidate values each is not forty tests — it is $10^4 = 10{,}000$ distinct strategies, of which the best-looking one will look very good on that sample whether or not any real effect exists. This is the mechanical link between overfitting and data-snooping bias: the more variants a search covers, the more the winner's apparent quality is explained by the size of the search.

Worked example: a 12-parameter strategy, perfect in sample and dead out of sample

Consider two versions of the same idea, both fitted on 2000–2012 and then run unchanged on 2013–2025.

On the fitted window the tuned version is clearly the better strategy on every figure, and that is the result a single backtest would have shown. On the unseen window it collapses, while the simple version behaves roughly as it did before. The 12 parameters did not discover a better strategy; they described 2000–2012 more precisely — including the parts of it that were accidents. Note also which figure moved most: the tuned version's drawdown nearly tripled, because tuning had quietly removed the specific historical declines it was fitted to avoid rather than teaching the strategy to avoid declines in general.

What is an in-sample / out-of-sample split?

An in-sample / out-of-sample split is the standard defence against overfitting: the history is divided in two, the strategy is designed and tuned on the first part only (in sample), and the second part (out of sample) is then run once, unchanged, as a test of whether the result survives on data that played no part in shaping it. The out-of-sample result is the one that carries information, because it is the only one the rules were not fitted to.

Two conditions make the split meaningful. The out-of-sample period must be genuinely untouched — each time you look at it, adjust the rules, and look again, it becomes part of the fitting sample and stops being a test. And the split must be chronological rather than random, so the earlier data trains and the later data tests, which also keeps the exercise free of look-ahead bias. Repeating the split as a series of rolling train-then-test windows is walk-forward validation, the sequential form of the same idea.

What counts as a robust result?

Robustness is about stability rather than a level: a result is described as robust when it does not depend on one exact set of parameter values, one start date, or one sample of instruments. Practically, that shows up as a broad plateau instead of a sharp peak — small changes to a threshold move the outcome slightly rather than destroying it — plus a narrow spread of outcomes across many start dates, which is what the start-date sensitivity view reports. A strategy whose result exists only at one precise parameter setting is the textbook profile of an overfitted one. One perturbation is easy to forget: a result that only survives with costs left out is carrying cost-ignoring bias on top of everything else, so a robustness check is worth repeating on the net figures. Robustness is a statement about how a result behaves under perturbation, and no level of it makes a strategy safe or a future return likely.

What does Fincanva do about overfitting?

• You define the rules. Fincanva has no automatic optimizer: it does not search parameter values or tune your thresholds against history for you, so a backtest reports what your exact stated rules would have done and nothing else.
• The start-date sensitivity view re-runs one strategy across many entry dates and holding windows and reports the spread, which is a direct robustness check on a single flattering run.
• A backtest always runs to the latest available market close, so the most recent history is never held back automatically. A chronological in-sample / out-of-sample split is something you construct yourself from the periods you compare, primarily via the simulation start year.

See the nine biases Fincanva helps you avoid for the product behavior in context.

How is overfitting different from data-snooping bias?

Overfitting and data-snooping bias are the same statistical problem seen from two ends. Overfitting describes the strategy: it has too many degrees of freedom for the data, so it has absorbed noise. Data-snooping describes the search: many candidates were tried and only the winner is being reported, so the winner's apparent quality partly reflects the number of attempts. A single strategy with twelve hand-tuned parameters can be overfitted with no search at all, and a single-parameter strategy chosen as the best of 500 tests can be data-snooped without being complex. Most real cases are both.

Backtests show what would have happened — not what will. Fincanva provides no financial advice — see Is this financial advice?.