Confirmation bias

Also seen as: Confirmatory bias, my-side bias

How does confirmation bias show up when you read a backtest?

It rarely feels like bias from the inside. It feels like judgment. The recognizable patterns are:

• Stopping when it agrees. The first run that supports the idea ends the investigation; a run that contradicted it would have prompted three more.
• Asymmetric scrutiny. A confirming result is accepted as-is. A disagreeing one gets audited — wrong universe, wrong period, "something's off with the data" — and the audit stops as soon as a reason is found.
• Reasons produced after the result. The objection to a run is invented once its number is known. If a disagreeing run had come out well, the same objection would never have been raised.
• Reading only the flattering metric. The return is read and the worst drawdown is skipped, or the strategy's own figure is read without the benchmark beside it.
• Remembering the run that agreed. Weeks later the memory is "it worked" — the version of the test that agreed, not the version that didn't.

A useful check is a single question asked before a run finishes: what result would make me abandon this idea? An idea with no such result is not being tested.

Worked example: the run you kept and the run you explained away

You believe a twelve-month momentum rule works, and you test it over 2015–2024.

• Run 1 applies the rule to twenty large, familiar companies. It returns 9.1% a year. This matches what you expected, so you save it.
• Run 2 applies the identical rule to a broad four-hundred-name universe. It returns 3.4% a year. You conclude the wider universe has too many low-quality names in it, and you set the run aside.

Now put the benchmark next to both. Over the same period it returned 7.8% a year, so run 1's excess return is +1.3pp and run 2's is −4.4pp. Two things follow. First, even the run you kept beat its benchmark by a much thinner margin than "9.1%" suggested on its own. Second, and more important: the reason you gave for discarding run 2 was produced after you saw its number. Had run 2 returned 12%, the four-hundred-name universe would not have been "too junky" — it would have been "a broader, fairer test". That asymmetry, not the numbers, is the bias.

What separates confirmation bias from cherry-picking bias?

The difference is the audience. Cherry-picking bias is selective reporting outwards — you have the full result and you show someone else a favorable part of it. Confirmation bias is the same selection turned inwards — you show it to yourself, usually without noticing, and there is no moment of deciding to omit anything.

The two feed each other. Confirmation bias decides which run you believe; cherry-picking decides which run you present. And a disagreeing result blamed on "bad data" without anyone checking the data is confirmation bias borrowing the language of data-quality bias — as is switching costs off because the net curve "looks wrong" (see cost-ignoring bias).

What does Fincanva do about confirmation bias?

A backtest applies your rules mechanically across the whole history regardless of what you hoped would happen, and reports the result whole — the return, the worst peak-to-trough drawdown, the losing months, and the benchmark run over the identical period beside it. That gives the rules a standing chance to disagree with you, and it puts the disagreeing evidence on the same screen as the agreeing evidence rather than one search away.

It cannot make you read it. Nothing in a backtest stops you from dismissing an inconvenient run, and no tool can supply the intent to be proved wrong. See the nine biases Fincanva helps you avoid for the product behavior in context.

Backtests show what would have happened — not what will. Fincanva provides no financial advice — see Is this financial advice?.