We Caught a Look-Ahead Bug Before It Shipped
We Caught a Look-Ahead Bug Before It Shipped
> Signal Syndicate Research Library · NBA Model Validation · Educational intelligence · Not betting advice
Most sports models quietly cheat without meaning to. They score past games using today's team stats — pace, offensive rating, defensive rating — as if those numbers existed on game day. Backtests look better. Holdouts lie. The same quiet cheat recurs across calendars too — when a June regime flips mid-month — which is exactly why our holdouts are rebuilt as-of game day. Users lose trust when the gap shows up in forward testing.
We caught that pattern in our NBA totals lane before anything shipped to users. This report explains what shifted, how we fixed it, and what we did not do (promote, spend odds credits, or hide the failure).
Executive Summary
| Finding | Detail |
|---|---|
| **The bug** | Full-season stat refresh on past holdout games |
| **Impact (est.)** | Average projection moved **+11.7 points**; 3 grade flips on locked holdouts |
| **The fix** | **As-of stats** — team numbers pulled only through each game's date |
| **After fix** | Projection error vs "cheating" refresh ≈ **0 points** on locked holdouts |
| **Promotion** | **NO** — model still fails gates on honest stats |
What Went Wrong
In Phase 7 we refreshed team stats the standard way: pull current season numbers and re-score historical holdout games. Policy C (our edge-selection path) went from 8 graded bets at −4.6% ROI (PASS on small n) to 13 graded at −11.9% ROI (FAIL).
The average total projection moved +11.7 points upward. Example: Philadelphia @ Detroit shifted +28 points — enough to flip a loss to a win in the grading ledger.
That is not a "bad model day." That is a look-ahead data bug: the model saw future performance when judging past games.
What We Did
Phase 8–9: Pull as-of team stats from the NBA stats API with `date_to` set through each holdout game date. Cache snapshots per pull date under controlled holdout windows.
Result: When we compare as-of projections to full-season cheat refresh on the same locked games, average error collapses to approximately zero points. The integrity path is validated.
What we did not do:
- Promote the NBA totals lane to users
- Spend Odds API credits on new pulls for this fix
- Quietly ship the old refresh path
What Still Fails (On Purpose)
Honest as-of stats do not magically make the model promotable. Policy C on as-of holdouts still shows negative ROI at meaningful sample sizes on expanded REGULAR windows. The product win here is trust in the testing pipeline — not a new pick product.
Multi-AI courtroom and founder gates remain BLOCK on Step 9b global promotion. This document is transparency, not a launch announcement.
Why This Matters If You Are Not a Quant
If a platform shows you pretty backtests but never explains how stats were frozen in time, you are reading marketing — not research.
Signal's answer: document the bug, fix the pipeline, re-run holdouts, publish the failure modes. Users can Ask Signal how we validate without cherry-picking — this page is part of that answer.
Method Notes (Plain English)
- Holdout: games we lock before testing — model never "saw" them during tuning
- Policy C: our edge-selection shadow path — not a user-facing pick feed
- As-of stats: team pace and efficiency computed using only games played before the slate date
- All ROI figures are historical estimates from backtests — not promises
Research only · estimates only · not betting advice · past results do not predict future performance.