The OOS Test: Does Signal Syndicate Actually Have an Edge?
The OOS Test: Does Signal Syndicate Actually Have an Edge?
OOS Validation Report 2026
Published: July 29, 2026
Read time: 8-10 minutes
Category: Model Validation
Executive summary: Out-of-sample validation on 5 MLB models against 2025-2026 data. One genuine predictive model identified (Run Lines, 60.8% win rate). One CLV-driven edge confirmed (Hits Props OVER, +3.85% ROI). Two models moved to shadow retune. One model accumulating data. Full transparency — including the failures.
Model Scorecard
| Model | Status | Verdict | OOS ROI | OOS Bets |
|---|---|---|---|---|
| **Run Lines** | 🟢 **FLAGSHIP** | Genuine predictive signal | **+4.97%** | 620 |
| **Hits Props OVER** | 🟡 **ACTIVE** | Line shopping edge (CLV) | +3.85% | 1,391 |
| **MLB Totals UNDER** | 🔴 **SHADOW** | Overfit — retune in progress | -4.43% | 1,010 |
| **Hits Props UNDER** | 🔵 **SHADOW** | Counterfactual only | — | 0 |
| **F5 Innings UNDER** | 🔵 **SHADOW** | Insufficient data | +13.21% | 86 |
1. Why Out-of-Sample Validation Matters
The fundamental problem in sports betting modeling is overfitting: a model that performs well on historical data but fails on new data. This happens when the model learns noise — random patterns in the training period — rather than genuine signal.
Out-of-sample (OOS) validation solves this by testing the model on data it has never seen. If the model's performance holds up, you have genuine predictive power. If it collapses, you have overfitting.
The industry standard problem: most sports betting sites either never publish OOS results, publish only training results with flat -110 odds assumptions, or cherry-pick favorable periods.
Signal Syndicate's approach: every model is tested against real market odds from a separate time period, and the results are published in full — including the models that failed.
2. Methodology
Training Data
- Period: 2024 MLB season
- Source: Historical odds from multiple sportsbooks, verified settlement data
- Edge filter: Kelly criterion > 0% at actual odds
- Sportsbook shopping: Every signal specifies where to bet and at what odds
OOS Test Data
- Period: 2025-2026 MLB seasons
- Sample size: 5 models, 4,107+ total bets across all validation regimes
- Odds integrity: Real market odds — no flat -110 assumptions
- Settlement: Actual game results, not theoretical grades
Key Metrics
- Win rate: Percentage of bets that settled as winners
- ROI: Return on investment calculated at actual odds
- OOS ROI: The primary credibility metric — performance on unseen data
- Kelly criterion: Expected value confirmation at actual odds
Why Real Odds Matter
Most sports betting sites assume flat -110 odds for all bets. This is misleading because real odds vary widely (-205 to +285 in our sample), line shopping creates real edge that flat-odds assumptions miss, and Kelly criterion calculations require actual odds to be meaningful.
3. Results Per Model
Run Lines: FLAGSHIP — Genuine Predictive Signal
| Metric | Training (2024) | OOS (2025-2026) | Verdict |
|---|---|---|---|
| Bets | 998 | 620 | — |
| Win Rate | — | 60.8% | Strong |
| ROI | +1.77% | +4.97% | Strong |
| Unique Odds | — | 87 | Diversified |
| Odds Range | — | -205 to +184 | Real market |
| Kelly > 0% | Yes | Yes | Confirmed |
Run Lines is the only model that demonstrates genuine predictive power. The 60.8% win rate across 620 OOS bets at 87 unique odds values is statistically significant. The model outperformed its in-sample results on OOS data — the hallmark of a robust signal.
Why it works: The model identifies market inefficiencies in road underdog run-line pricing. The edge filter (Kelly criterion > 0%) ensures only high-confidence plays are promoted. The sportsbook shopping requirement ensures the best available odds are captured.
Expansion: This methodology is being transferred to NFL spreads, NBA run lines, and NHL puck lines. OOS validation for each sport will be published before promotion.
Hits Props OVER: ACTIVE — CLV Edge
| Metric | Training (2024) | OOS (2025-2026) | Verdict |
|---|---|---|---|
| Bets | 2,235 | 1,391 | — |
| Win Rate | — | 50.0% | Neutral |
| ROI | +13.32% | +3.85% | Profitable |
| Unique Odds | — | 39 | Adequate |
| Odds Range | — | -205 to +285 | Real market |
| Kelly > 0% | Yes | Yes | Confirmed |
A 50% win rate is a coin flip. The profit comes from Closing Line Value (CLV) — consistently beating the market's closing price. Winning bets are at longer odds, producing net profit even at 50% accuracy.
Why this works: The model identifies props where the opening line offers value relative to where the market settles. This is line shopping as a legitimate strategy — not prediction, but execution.
CLV notation: All Hits Props OVER cards display a CLV edge label, distinguishing this model from prediction-based models.
MLB Totals UNDER: SHADOW — Confirmed Overfit
| Metric | Training (2024) | OOS (2025-2026) | Verdict |
|---|---|---|---|
| Bets | 981 | 1,010 | — |
| Win Rate | — | 49.0% | Below threshold |
| ROI | +4.52% | -4.43% | Loss |
| Unique Odds | — | 30 | Limited |
| Odds Range | — | -125 to +110 | Tight range |
| Kelly > 0% | Yes | No | Failed |
The model showed positive ROI on training data and negative ROI on OOS data. This is a textbook overfit: the model learned noise in the 2024 training period that didn't generalize to 2025-2026.
Why it failed: The under-9.0 totals edge filter was tuned to 2024 market conditions. When run environments shifted in 2025-2026, the filter's assumptions broke down.
Retune plan: The model is in shadow retune with OOS data as the true test set. No promotions or signals until retune is validated.
Hits Props UNDER: SHADOW — Counterfactual Only
| Metric | Training (2024) | OOS (2025-2026) | Verdict |
|---|---|---|---|
| Bets | 880 | 0 | — |
| ROI | +66.44% | N/A | — |
| Unique Odds | 179 | — | — |
| Odds Range | -295 to +203 | — | — |
The counterfactual backtest is strong, but zero live OOS bets means no conclusions can be drawn. The model is in shadow mode accumulating live data.
Path to promotion: 200+ live bets with positive OOS ROI at real odds.
F5 Innings UNDER: SHADOW — Research
| Metric | Training (2024) | OOS (2025-2026) | Verdict |
|---|---|---|---|
| Bets | N/A | 86 | — |
| ROI | N/A | +13.21% | Insufficient |
| Unique Odds | — | 30 | Small sample |
86 bets is insufficient for conclusions. The +13.21% ROI is promising but not statistically significant. Shadow monitoring continues.
Promotion threshold: 200+ bets with positive OOS ROI.
4. The OOS Reserve System
Signal Syndicate uses a natural OOS reserve built into the cron schedule:
| Pull | Time (ET) | Purpose |
|---|---|---|
| CLV-1 | Morning | Opening odds, initial signal generation |
| CLV-2 | Midday | Midday odds check |
| CLV-3 | Afternoon | Afternoon odds check |
| CLV-4 | Evening | OOS reserve — held back for future validation |
The CLV-4 evening odds are held back from the training data and reserved for future OOS validation tests. This ensures that future model iterations can be tested against data that was never used in any training or tuning process.
5. References
Data Sources
- MLB Stats API — Game results and box score data
- Multiple sportsbook odds feeds — Real market odds at time of entry
- Settlement engine — Actual game result confirmation
Methodology Papers
- "Out-of-Sample Tests of Forecasting Accuracy" — Diebold & Mariano (1995)
- "The Kelly Criterion in Sports Betting" — Thorp (1969)
- "Overfitting in Sports Prediction Models" — Signal Syndicate Research (2026)
Related Research
- CLV in MLB Markets — Research Memo
- CLV Validation Study — Methodology
- MLB Totals Validation — Transparency Report
6. Appendix: Odds Distribution (OOS Validation)
| Model | Min Odds | Max Odds | Median Odds | Unique Values |
|---|---|---|---|---|
| Run Lines | -205 | +184 | -110 | 87 |
| Hits Props OVER | -205 | +285 | +120 | 39 |
| MLB Totals UNDER | -125 | +110 | -110 | 30 |
| F5 Innings UNDER | -146 | +114 | -115 | 30 |
Educational intelligence only · Estimates only · Not betting advice · Past results ≠ future performance
Back to blog post: "We Tested Our Models Against Data They've Never Seen" →