How We Fooled Ourselves With Our Own Backtest — Autopsy of a Post-Hoc Theme Sleeve

This series does not only dissect other people's conventional wisdom. This installment is a record of how the system we run was fooled by our own backtest, and how we found it and cut it out.

What happened

In May 2026, the Korean equity monthly strategy we were running missed every one of that month's leading names (the semiconductor and display surge). The failure review produced a design for "a sleeve allocating 30% of the portfolio to 10 specific theme sectors," and a 12-year (2014–2026) backtest showed this improvement:

+272%p. It went live starting in June.

What the autopsy found three months later

① The sector list was chosen after the fact. The basis was still there in the review document: we first analyzed the names that rose in May, then said "this sleeve would have caught May's leaders," built the list out of those sectors — and backtested it over a period that included that same May.

② The 12-year improvement was effectively one year. Decomposing the sleeve's contribution by year:

What the "12-year validation" actually was: the 2023 semiconductor boom, one year, plus early 2026. The tech sectors we picked in hindsight simply met their own boom inside the sample. On top of that sat multiple testing — adopting the best performer among several parameter variants.

③ The only genuine out-of-sample test = 3 months live. The theme sleeve's performance versus KOSPI: June +14.1%p, July -16.3%p, August -5.2%p — roughly -9%p cumulative.

The sleeve was removed

We decided to remove it the day we found it. The basis was not the weak three months live (three months proves nothing) but the discovery that the basis for adopting it was itself invalid. The same decision would have been correct even if performance had been good.

What this case says

• The most dangerous error in a backtest is not a code bug but the researcher's memory. Design a strategy to catch what you recently saw, and the backtest will always "validate" it
• The longer the backtest, the more you have to look at the time distribution of the contribution. If total return is concentrated in one period, that is evidence about one year, not twelve
• If you leave a record of "why this design" in the strategy adoption document, you can perform an autopsy on yourself later. That is why this discovery was possible

Data sources

• Backtest artifacts and live rebalancing records from our own trading system (2026-05 – 2026-09)