Five Ways Backtests Deceive You — A Collection of Actual Cases We Experienced

There are many articles about backtest traps. This article is different in one aspect — these are all actual cases we experienced, and we present the incorrect numbers from that time alongside the corrected numbers.

1. Look-ahead — Backtests that know the future

While verifying spike stock data, a backtest that "enters at the closing price the day before a spike" showed a 20-day return of +9.4%p. The problem: "spike" was included in the entry conditions. To enter the day before a spike, one must know that it will spike tomorrow. When the spike condition was removed and the test was re-run for all periods where the same precursors (price +10% accompanied by a 2x volume increase) were observed, the result became -3.7%p. The difference between +9.4%p and -3.7%p is the magnitude of one look-ahead bias. A significant portion of backtests for foolproof spike stock strategies take this form.

2. Incomplete samples — When data is already biased

We used a brokerage ranking API to collect spike stocks, but this API had a fixed 30 items + poor sorting. On some days, only 16 out of 26 spike stocks were returned, and due to the omitted stocks, the determination of that day's "leading theme" itself was incorrect. The sample only became complete after switching to a full scan of the entire price database. Before a backtest, one must first verify whether the universe is complete.

3. Unit traps — Silently being off by 1,000 times

We once missed that the listed shares field was in units of a thousand shares and calculated the floating supply, resulting in all stocks being classified as "scarce stocks". Unit errors do not throw exceptions — they silently produce plausibly incorrect numbers. Whether market capitalization is in hundred millions of won or won, or whether investor flows are in millions of won or won varies by table, and this can only be caught through cross-validation with a single known stock, not through documentation (Samsung Electronics has 5.9 billion listed shares. If 5.91 million appears, it is in units of a thousand shares).

4. Post-selection — Bias created by the researcher's memory

A case where we built a strategy using sectors that had risen in May after observing them, and "confirmed" a +272%p improvement by backtesting over 12 years including May — we performed a full autopsy on this in a separate part. When decomposed by year, the improvement was effectively just the single year of 2023. The total return of a long-term backtest must be viewed alongside the time distribution of its contributions.

5. Multiple testing — If you throw enough, something will stick

If you test dozens of parameter combinations, some of them are bound to look good (at a 5% significance level, 5 out of 100 will pass purely by chance). We use two defensive rules: Preserve the entire list of tested variations along with their results (do not keep only the good ones), and record 50% of every backtest's reported value as the expected value alongside it — this is a combined discount for selection bias, survivorship bias, costs, and regime dependence. It is rough, but we only deploy strategies that withstand this discount.

Summary

The purpose of a backtest is not to prove that it "works," but to exclude what "definitely does not work." This is also why the rejection verdicts in this series are more reliable than the adoption verdicts — because the traps above mostly operate in a direction that makes strategies look good.

Data Sources

• Actual accident and correction records from our internal validation pipeline (2026-08)