When the Filter Passes and the Model Doesn't
When the Filter Passes and the Model Doesn't
> Signal Syndicate Research Library · NBA Model Validation · Educational intelligence · Not betting advice
Sometimes the edge filter looks fine while the base projection model does not. That split is easy to miss — and dangerous to productize. The stats-integrity failure behind the same NBA story is documented in the as-of holdout report. This report documents our B_tight shadow policy (15-point edge cap) on NBA totals holdouts after we fixed look-ahead team stats.
Verdict today: hypothesis only. No JSON policy change. No user-facing picks.
Executive Summary
| Layer | Locked holdout (small n) | Expanded REGULAR holdout |
|---|---|---|
| **Base model + Policy C** | FAIL | FAIL (42 graded, est. −27% ROI) |
| **B_tight shadow (15-pt cap)** | PASS on tiny sample | FAIL at scale |
| **Adoption** | **NO** | Paper-test only |
Expanded testing used 57 games across 7 pull dates at 0 Odds API credits.
Two Layers, Two Stories
Model layer: NBA totals projections on honest as-of team stats still fail our holdout promotion gates. Negative ROI persists when we grade enough edges.
Policy layer: B_tight caps graded edges at 15 points. On a small locked holdout, that shadow path once looked acceptable — a classic small-sample trap.
When we expanded to a broader REGULAR holdout window (Phase 9), Policy C graded 42 bets at roughly −27% ROI (historical estimate). Stable pass at n≥15: false.
What We Did Not Do
- Change production policy JSON
- Promote B_tight to users
- Spend credits on new odds pulls for this audit
- Conflate "shadow PASS on 6 bets" with product readiness
Court requirement remains 2/3 AI panel + founder before any adoption. Sample must grow.
Plain-English Lesson
A filter can "work" while the engine is still broken. Good research separates those layers and publishes both truths.
If you only ship the filter because it backtested well on six games, you are selling a highlight reel — not validation.
Related Reading
- We Caught a Look-Ahead Bug Before It Shipped — why honest stats matter first
- Why We Publish the Process, Not Just the Picks — workflow thesis
Research only · estimates only · not betting advice · monitor-only lane — not a pick product.