When Forward Results Flip Your Backtest — OOS vs Backtest Explained
When Forward Results Flip Your Backtest — OOS vs Backtest Explained
> Signal Syndicate Research Library · Model Validation · Educational intelligence · Not betting advice
Your backtest says +27%. Your forward month says −14%. Both can be true at once — and that gap is one of the most important ideas in serious betting research.
This report explains out-of-sample (OOS) testing vs in-sample backtest in plain English, using a real Signal autopsy on K-props shadow data where June flipped year-over-year.
Backtest vs forward test — 30-second version
| **Backtest (in-sample history)** | **Forward / OOS test** | |
|---|---|---|
| **Question** | "How would this have done on past data we designed around?" | "How does it do on slates we held back?" |
| **Risk** | Overfitting, look-ahead stats, cherry-picked windows | Smaller samples, real market drift |
| **Signal use** | Hypothesis generation | Promotion evidence — shadow first |
A strategy is not "validated" until forward windows behave — and even then, sample size gates apply.
What we saw (K-props shadow · historical estimates)
June 2025 (tiny forward window):
- ~3 graded shadow rows
- ~+27% ROI (illustrative — not promotion-grade sample)
June 2026 (denser shadow window):
- 20 graded rows
- ~−14% ROI
- Under segment weaker than over segment (~−20% vs ~−5% on splits shown in autopsy)
Same lane label. Same research intent. Opposite story when the calendar rolled forward and sample grew.
That is not a spreadsheet bug — it is regime and sample reality.
Why the flip matters (even if you never bet K props)
1. Small forward wins lie. Three green rows in June 2025 would tempt a tout to "go live." Our shadow gate blocked that.
2. Market years differ. Books tighten, player usage shifts, strike density changes — especially in prop markets.
3. Honest shops publish the flip. We retired the May–Jul K lane after expanded review (~−10% on 34 rows) — see Why We Retired K Props.
What Signal did (process, not panic)
- Kept lane in shadow (`do_not_apply=true`) while evidence accumulated
- Ran multi-AI courtroom — majority RETIRE
- Stopped paid credit pulls when forward ROI stayed negative (credit gate report)
- Published this autopsy for Ask Signal and the Research Library
No user-facing "AI K props" product shipped on the June 2025 hot streak.
How to read this as a learner
When evaluating any model vendor — or your own sheets — ask:
1. What is OOS sample size? (not just backtest rows)
2. Did forward windows include bad months? or only kind ones?
3. What happened when they flipped? silence = red flag
Ask Signal: "What's the difference between backtest and out-of-sample?"
Research only · estimates only · not betting advice · past results do not predict future performance.