Benchmark

How the model does on games it had not seen, scored with proper scores and checked for honest uncertainty. Every ± is a 95% interval clustered on both teams' seasons — games that share a team are not independent. The walk and series follow Settings unless this page names others.

scoredwalkforward-v5-mens-min10 · smoothed protocolPostseason holdout seasonsall games680

Each season's postseason is priced from one snapshot: the walk's last run that trained strictly before the first postseason tip — after the conference tournaments, before the play-in games (the Kaggle cut). The walk kept fitting through the tournament; those later runs are not used. Headline: the NCAA bracket (63 games, play-in excluded) and the NIT; other postseason is shown but kept out of the headline.

Snapshots: 2021 at 2021-03-14 04:30 (89 priced) · 2022 at 2022-03-13 19:30 (130 priced) · 2023 at 2023-03-12 19:30 (113 priced) · 2024 at 2024-03-17 19:30 (120 priced) · 2025 at 2025-03-16 03:30 (123 priced) · 2026 at 2026-03-15 03:59 (105 priced)

Headline — NCAA bracket and NIT

Slice	Games	Log loss	Brier	Accuracy	Calib. slope	Spread RMSE	Spread CRPS	rms(z)	Total RMSE
NCAA + NIT	548	0.5809 ±0.0408	0.1994 ±0.0161	68.6%	0.80	11.52 ±0.62	6.55 ±0.36	0.939 ±0.051	16.47 ±1.11
ncaa	377	0.5512 ±0.0530	0.1868 ±0.0203	71.4%	0.81	11.98 ±0.77	6.83 ±0.46	0.974 ±0.064	16.99 ±1.32
nit	171	0.6464 ±0.0551	0.2273 ±0.0241	62.6%	0.71	10.44 ±0.95	5.95 ±0.52	0.857 ±0.076	15.25 ±1.98

Slice

Games

Log loss

Brier

Accuracy

Calib. slope

Spread RMSE

Spread CRPS

rms(z)

Total RMSE

NCAA + NIT

548

0.5809 ±0.0408

0.1994 ±0.0161

68.6%

0.80

11.52 ±0.62

6.55 ±0.36

0.939 ±0.051

16.47 ±1.11

ncaa

377

0.5512 ±0.0530

0.1868 ±0.0203

71.4%

0.81

11.98 ±0.77

6.83 ±0.46

0.974 ±0.064

16.99 ±1.32

nit

171

0.6464 ±0.0551

0.2273 ±0.0241

62.6%

0.71

10.44 ±0.95

5.95 ±0.52

0.857 ±0.076

15.25 ±1.98

Every postseason slice

Slice	Games	Log loss	Brier	Accuracy	Calib. slope	Spread RMSE	Spread CRPS	rms(z)	Total RMSE
ncaa	377	0.5512 ±0.0530	0.1868 ±0.0203	71.4%	0.81	11.98 ±0.77	6.83 ±0.46	0.974 ±0.064	16.99 ±1.32
ncaa_play_in	24	0.7157 ±0.0877	0.2595 ±0.0411	58.3%	0.96	12.48 ±3.57	6.95 ±2.19	1.014 ±0.289	24.50 ±4.94
nit	171	0.6464 ±0.0551	0.2273 ±0.0241	62.6%	0.71	10.44 ±0.95	5.95 ±0.52	0.857 ±0.076	15.25 ±1.98
other_postseason	108	0.6135 ±0.0656	0.2130 ±0.0293	65.7%	0.98	10.45 ±0.94	6.04 ±0.54	0.869 ±0.079	19.63 ±3.24
all postseason	680	0.5909 ±0.0347	0.2037 ±0.0138	67.8%	0.87	11.39 ±0.56	6.49 ±0.32	0.931 ±0.045	17.35 ±1.08

Slice

Games

Log loss

Brier

Accuracy

Calib. slope

Spread RMSE

Spread CRPS

rms(z)

Total RMSE

ncaa

377

0.5512 ±0.0530

0.1868 ±0.0203

71.4%

0.81

11.98 ±0.77

6.83 ±0.46

0.974 ±0.064

16.99 ±1.32

ncaa_play_in

0.7157 ±0.0877

0.2595 ±0.0411

58.3%

0.96

12.48 ±3.57

6.95 ±2.19

1.014 ±0.289

24.50 ±4.94

nit

171

0.6464 ±0.0551

0.2273 ±0.0241

62.6%

0.71

10.44 ±0.95

5.95 ±0.52

0.857 ±0.076

15.25 ±1.98

other_postseason

108

0.6135 ±0.0656

0.2130 ±0.0293

65.7%

0.98

10.45 ±0.94

6.04 ±0.54

0.869 ±0.079

19.63 ±3.24

all postseason

680

0.5909 ±0.0347

0.2037 ±0.0138

67.8%

0.87

11.39 ±0.56

6.49 ±0.32

0.931 ±0.045

17.35 ±1.08

NCAA bracket by season

Slice	Games	Log loss	Brier	Accuracy	Calib. slope	Spread RMSE	Spread CRPS	rms(z)	Total RMSE
2021	62	0.6323 ±0.1590	0.2187 ±0.0620	69.4%	0.48	13.22 ±1.64	7.67 ±1.04	1.082 ±0.138	17.56 ±2.34
2022	63	0.6298 ±0.1521	0.2203 ±0.0585	65.1%	0.53	12.74 ±2.11	7.24 ±1.29	1.022 ±0.168	18.69 ±2.30
2023	63	0.6310 ±0.1289	0.2083 ±0.0389	66.7%	0.33	11.91 ±2.11	6.75 ±1.22	0.970 ±0.178	17.46 ±2.88
2024	63	0.5541 ±0.1152	0.1911 ±0.0480	71.4%	0.88	12.56 ±1.76	7.23 ±1.13	1.037 ±0.147	19.48 ±4.68
2025	63	0.4278 ±0.0668	0.1379 ±0.0287	81.0%	3.02	9.81 ±1.48	5.64 ±0.77	0.781 ±0.119	13.89 ±2.21
2026	63	0.4335 ±0.0842	0.1452 ±0.0361	74.6%	1.63	11.31 ±1.57	6.44 ±0.91	0.925 ±0.126	14.03 ±2.25

NIT by season

Slice	Games	Log loss	Brier	Accuracy	Calib. slope	Spread RMSE	Spread CRPS	rms(z)	Total RMSE
2021	16	0.7067 ±0.0533	0.2564 ±0.0261	56.2%	—	10.25 ±1.63	5.89 ±0.87	0.829 ±0.140	12.76 ±3.22
2022	31	0.5702 ±0.1194	0.1967 ±0.0546	67.7%	1.59	10.36 ±2.33	5.97 ±1.37	0.861 ±0.187	16.58 ±4.32
2023	31	0.7091 ±0.1433	0.2479 ±0.0549	64.5%	0.01	11.33 ±1.71	6.44 ±1.00	0.924 ±0.122	15.68 ±4.39
2024	31	0.6490 ±0.1193	0.2276 ±0.0547	61.3%	-0.59	9.11 ±1.82	5.36 ±0.94	0.763 ±0.164	15.00 ±4.32
2025	31	0.7113 ±0.1106	0.2568 ±0.0505	54.8%	0.84	10.96 ±3.05	6.17 ±1.74	0.912 ±0.254	15.84 ±6.97
2026	31	0.5616 ±0.1415	0.1923 ±0.0635	67.7%	3.17	10.39 ±2.38	5.85 ±1.12	0.831 ±0.171	14.22 ±2.51

Uncertainty — NCAA + NIT

Market	Bias	CRPS	rms(z)	50% covers	80% covers	95% covers
spread	0.81 ±1.00	6.55 ±0.36	0.939 ±0.051	49.5%	81.8%	96.7%
total	2.55 ±1.49	9.20 ±0.60	1.061 ±0.074	48.2%	78.3%	93.2%

An honest sigma has rms(z) near 1 and covers each interval at its nominal rate. Brier splits into reliability 0.0012 (lower is better calibrated) and resolution 0.0167 (higher separates outcomes better); a calibration slope below 1 means the probabilities are too extreme.

NCAA bracket vs the Kaggle boards

No board placement: the cached boards live in cache/kaggle_leaderboards, which is gitignored and exists only on the build host.

mens 2021: board not cached on this machine
mens 2022: board not cached on this machine
combined 2023: board not cached on this machine
mens 2024: Launched Brier Bracket/portfolio construct — not computable here
combined 2025: board not cached on this machine
combined 2026: board not cached on this machine