Cherry-picking bias is the practice of quoting only the periods, instruments, or runs that flattered a strategy while leaving the rest out, so every number reported is true and the picture they add up to is not. Nothing has to be falsified — a good year really was a good year — but the reader is shown a slice of the evidence that was chosen because it was favorable, and has no way to see what was left out. Cherry-picking bias is an error of reporting: the underlying test can be entirely correct, and the distortion enters only when its results are described.
Also seen as: Cherry picking, selective reporting
What gets cherry-picked in a backtest?
Four things are usually picked, and each is picked the same way — after the results are already known:
- The window. A start and end date chosen because the stretch between them went well. This is the most common form, because a backtest makes any window one setting away.
- The instruments. The names in the report are the ones that contributed; the ones that dragged get described as "not really part of the idea".
- The metric. The return is quoted and the worst drawdown is not, or the gross curve is shown while the net-of-costs one — the number a real account would have kept — is not (see cost-ignoring bias).
- The run. One configuration out of many tried is presented as the strategy, with no mention of the others.
One test catches all four: could someone reproduce the claim without knowing which slice you chose? If the claim only holds on your slice, the slice is doing the work, not the strategy.
Worked example: reporting 2019 and omitting 2022
A strategy is backtested over the ten calendar years 2015–2024. Two of those years stand out: 2019 returned +31% and 2022 returned −29%.
A report built around 2019 — the year label, the rising curve, the +31% — states a fact. Here is what the same run also says:
| Figure from the same run | Value |
|---|---|
| 2019 calendar year | +31% |
| 2022 calendar year | −29% |
| Full period, total return | +72% |
| Full period, annualized (CAGR) | 5.6% a year |
| Worst peak-to-trough fall in the period | −34% |
Quoting 2019 alone invites the reader to treat +31% as what the strategy does in a year; the run's own annualized figure is about a fifth of that. Put the omitted year back and the pair alone leaves the strategy below where it started: 1.31 × 0.71 = 0.93, or −7% across the two years together. None of those five numbers contradicts the others — they all come from one run. The distortion is entirely in which of them got quoted.
How is cherry-picking bias different from selection bias and data-snooping bias?
The three differ by which step goes wrong: cherry-picking bias is about what you report, selection bias is about what you tested, and data-snooping bias is about how many things you tested before something looked good.
They also stack, in that order. A researcher who tries fifty variants (data-snooping), keeps the one with the most flattering instrument list (selection), and then presents its best three years (cherry-picking) has committed all three, and the final report shows no trace of the first two.
What does Fincanva do about cherry-picking bias?
A Fincanva backtest reports the whole period it ran rather than a chosen stretch of it, and reports the falls alongside the gains: the same run that produces the return also produces its worst peak-to-trough drawdown, so the bad part of the history arrives attached to the good part. Because the run is a walk-forward replay of the full span, there is no version of the result that covers only the years that worked.
Start-date sensitivity is the direct answer to the window form of cherry-picking: it re-runs the same strategy across many entry dates and several holding windows and reports the range of outcomes — the best start against the worst start, and the share of start dates that ended positive. A claim that survives only one entry month shows up immediately as a wide range.
What the product cannot do is decide what you tell other people, or yourself. Choosing one favorable screenshot out of a run that offers the full picture is still available to anyone, which is why cherry-picking is usually discussed next to confirmation bias — the same selection applied inwards. See the nine biases Fincanva helps you avoid for the product behavior in context, and data-quality bias for the case where the numbers themselves, not their selection, are the problem.
Backtests show what would have happened — not what will. Fincanva provides no financial advice — see Is this financial advice?.
Where this term is used
Generated · 1 pagesThe pages that reference this term — so a term page is somewhere you pass through, not somewhere you land and stop.
Fincanva provides no financial advice. Backtests show what would have happened — not what will.
GLOSSARY · 193 TERMS