The Very Low Tier Post-Mortem: Why a 46.8% Win Rate Wasn't a Calibration Failure

Executive Summary

Every live-deployed model eventually produces a number that looks wrong. The question is never "does this look wrong" — it is whether the wrongness survives contact with a null hypothesis. This post walks through the full audit we ran on our XGBoost_TOP17_v1.4_Platt model's worst-performing confidence band after a full NBA season of live predictions, and why — after nine sections of statistical testing — we changed nothing about the model itself.

The short version: n=126 is not enough evidence to convict a classifier, and the specific way it underperformed (a two-week cluster, near-zero Murphy resolution, zero correlation with any of 17 input features) points at sampling variance and an inherently low-signal decision region, not a broken pipeline.

The Symptom

Our production tiering logic buckets every prediction by winner_probability into four bands: Very Low (50.0% to 55.0%), Low, Medium, and High. The Very Low band is, by construction, the coin-flip-adjacent edge of the model's output — games where the classifier believes it has some edge over a naive 50/50 split, but not much.

At season close, the active Platt-scaled model (XGBoost_TOP17_v1.4_Platt, 742 resolved games) showed this for the Very Low tier:

A confidence band with a sub-parity win rate is the kind of number that triggers an immediate instinct: re-fit Platt's A and B parameters, audit the feature pipeline for drift, maybe retrain. Before touching any of that, we ran the number through a standard significance workflow.

Calibration vs. Variance

Statistical power, first

Before testing significance, we checked whether the test even had the power to detect a real effect at this sample size. Given n = 126 and alpha = 0.05 (one-sided), the power to detect a genuine drop from 52.5% to 46.8% was only 35.6% — meaning even if the true effect were real, we would have missed it roughly two-thirds of the time. Reaching conventional 80% power for a gap this size would require n ≈ 479 resolved games — nearly four full seasons of Very Low tier volume at current rates.

That does not prove the gap is noise. It does mean a single-season read should never have been treated as dispositive in either direction.

Exact binomial and likelihood ratio tests

We ran an exact two-sided binomial test comparing the observed record (59W–67L) against both the model's calibrated target and theoretical parity:

H₀: p = 0.5248   (calibrated target)      p = 0.2125
H₀: p = 0.5000   (parity)                 p = 0.5331

Neither clears alpha = 0.05. We cross-checked with a likelihood ratio test comparing the empirical MLE (p̂ = 0.4683) to the calibrated target:

LLR = 2 · [ℓ(p̂) - ℓ(p₀)] = 1.6153,   χ²(df=1),   p = 0.2038

Same verdict: fail to reject H₀. The Clopper-Pearson exact 95% confidence interval on the observed record is [37.88%, 55.92%] — an 18-point-wide band that comfortably encloses both the 52.48% target and 50.00% parity.

Reconciling four interval methods

Because interval width matters more than any single point estimate here, we did not stop at one method. We ran the observed 59/126 record through four independent 95% interval constructions plus a 10,000-iteration non-parametric bootstrap:

The four methods agree to within 0.46 percentage points at both bounds — the interval is not an artifact of a particular approximation. All four cover both the calibrated target and parity. At the 99% level, the bootstrap envelope widens to [35.71%, 57.94%], and the bootstrap's own resampled distribution shows that 26.3% of resamples land at or above parity and 9.0% land at or above the 52.48% target — an ordinary tail event, not an outlier.

Signal Dissection: Reliability vs. Resolution

Statistical insignificance tells us the gap could be noise. It does not tell us why the tier behaves the way it does. For that we turned to Murphy's (1973) decomposition of the Brier score:

Brier Score = Reliability (calibration error)
            - Resolution (discrimination)
            + Uncertainty (base-rate entropy)

Reliability measures whether stated probabilities match observed frequencies (a calibration defect). Resolution measures whether the model can rank-order outcomes at all within a band (a discrimination defect). These are different failure modes with different fixes — the first says "adjust the sigmoid," the second says "there is no more signal to extract here".

For the Very Low tier specifically:

Resolution sits at roughly 0.9% of the uncertainty ceiling. For context, across the full active-model deployment (N = 742), resolution runs at 7.5% of uncertainty with a positive Brier Skill Score of +0.0674. Within the Very Low band alone, the ROC AUC collapses to 0.4698 — statistically indistinguishable from a coin flip, against 0.6684 overall.

We also tested internal homogeneity — whether the deficit was concentrated in one sub-pocket of the band or spread evenly across it. A chi-square test across three internal sub-bins ([0.500, 0.517) at 50.0%, [0.517, 0.533) at 44.9%, [0.533, 0.550) at 46.1%) returned p = 0.8896: the sub-bins are statistically homogeneous. Splitting the band in half at 52.5% tells a sharper storym though - the lower half ([50.0%, 52.5%), n = 62) hit 51.61%, right on target, while the upper half ([52.5%, 55.0%), n = 64) hit 42.19%, carrying essentially the entire deficit. The mean sigmoid gradient across the band, ∂P/∂z ≈ 0.2492, confirms the model is operating at maximum sensitivity to raw-score perturbations in this region — exactly where a small amount of noise produces the largest probability swings.

Temporal & Covariate Autopsy

An acute drawdown, not a chronic drift

A 14-day rolling-window scan of the season immediately isolated where the deficit lived. Between January 20 and February 2, 2026 — the pre-trade-deadline window, when NBA rotations are at their most volatile — the Very Low tier went 5W–16L (23.81%) across 21 games, a net deficit of −11 games in two weeks. A Fisher exact test comparing that window against the remaining 105 games of the season returned p = 0.0298, a genuinely significant divergence.

Crucially, this was not tier-specific noise: control tiers (Low through High confidence) simultaneously recorded their lowest accuracy of the entire season (58.5%) during the same window — market-wide volatility around the deadline, not a Very Low tier defect.

Excluding that 14-day window entirely, the tier's ex-slump record is 54W–51L (51.43%) across the remaining 105 games — squarely inside the designed [50.0%, 55.0%] band, and statistically indistinguishable from the 52.48% target (p = 0.8456).

A cumulative sum (CUSUM) of prediction residuals (ε_i = y_i − p̂_i) bottomed out at −9.81 games below expectation on March 7, 2026, then recovered steadily — peaking at 61.5% (8W–5L) in early April and stabilizing at or above 50% through the remainder of spring.

No feature signature, no covariate drift

Two independent lines of evidence rule out a pipeline or feature defect:

Residual regression. We regressed the outcome residual ε_i = y_i − p̂_i against all 17 standardized model features using OLS, restricted to the Very Low tier. The model-level F-statistic came back at p = 0.8216 with an adjusted R-squared of −0.0463. Zero of 17 coefficients reached significance (best candidate, away_tov_pct_l10, at p = 0.2138). If a specific feature — rest asymmetry, recent efficiency, head-to-head history — were driving the miss, it would show up here as a significant predictor of error direction. None do.

Population stability. Comparing feature distributions between the first and second half of the season via Population Stability Index (PSI) and two-sample KS tests, core inputs were stationary: b2b_diff (PSI = 0.0107), away_optimal_rest (PSI = 0.0017), pf_l10_diff (PSI = 0.0396), pie_diff (PSI = 0.0591) — all well under the 0.10 "stable" threshold, and tail outliers (|z|>3) stayed under 2% across every feature. The handful of features that did shift significantly (home_l, PSI = 1.7666; h2h_dominance, PSI = 3.4364; home_e_net_rating, PSI = 0.3631) are exactly what you'd expect from natural season progression — cumulative game counts rising, early-season head-to-head noise compressing as rivalry matchups repeat, standings differentials widening as contenders separate from tanking rosters. None of it is a leakage or pipeline symptom.

One structural pattern did emerge, though it's descriptive rather than a defect: Very Low tier games disproportionately feature a visiting team that is higher-quality on paper than the home team (w_diff: −1.42 vs. +1.04 in control, KS p < 0.0001). Because the model's learned home-court prior works against that matchup type, these games land directly on the 50–55% margin by construction — and when the model backs the away side in this band specifically, accuracy drops to 43.4% (23W–30L) versus 49.3% (36W–37L) when it backs home. That is a real, explainable population effect, not a bug — narrow-margin, quality-inverted matchups are inherently the hardest games on the slate to call, for any forecaster.

Diagnostic Synthesis

Pulling all four investigative threads together:

Four independent lines of inquiry — significance testing, signal decomposition, temporal isolation and feature/population analysis — converge on the same conclusion from four different directions. That convergence is what turns "probably noise" into an actual engineering decision.

Production Directives

This is the part of a post-mortem that's easy to get backwards. The instinct after seeing a red number is to do something — retune Platt's parameters, add features, retrain on a shifted window. Every one of those actions would have been the wrong move here, and here's the accounting for why:

Concretely, we shipped three changes:

1. Tier reclassification. The [50.0%, 55.0%) band is now labeled "Neutral / No-Action" rather than an active prediction tier. Production publication requires a minimum probability of P ≥ 0.5500.
2. Uncertainty cushioning. Because the empirical standard error (4.44 pp) nearly matches the entire tier width, we now require a safety margin of at least 1.28 × SE ≈ 5.7 pp above 50.0% — roughly a 90% one-sided confidence buffer — before a prediction is tagged as actionable edge.
3. Volatility circuit breaker. A monitoring flag now watches for late-January-style schedule congestion: if rolling 14-day control-tier accuracy drops below 60%, confidence thresholds escalate automatically across all lower tiers until the window clears.

None of this required touching the model weights. The classifier was doing exactly what a well-calibrated model should do at the edge of its own confidence: telling us, correctly, that it did not know much — and we were the ones who built a product decision on top of that honesty without asking it to be more certain than the math allows.