White's Reality Check and SPA Test
White’s Reality Check asks whether the best strategy in a searched family is genuinely better than a benchmark once the search itself is acknowledged. Selecting the maximum observed Sharpe from dozens of rules changes its null distribution; ordinary one-strategy p-values are no longer valid. Reality Check uses a bootstrap distribution of the maximum relative performance under a no-superiority null.
Let d[t, k] be the period-by-period performance differential of rule k versus a benchmark: net P&L, utility, or loss difference. The observed statistic is the maximum mean differential across rules. A stationary bootstrap resamples time blocks to preserve dependence, recenters differentials under the null, and recalculates that maximum. The p-value is the proportion of bootstrap maxima exceeding the observed maximum.
| Test | Null | Strength | Limitation |
|---|---|---|---|
| Single t-test | One rule has no edge | Simple | ignores search |
| Reality Check | No candidate beats benchmark | accounts for search | conservative with poor rules |
| SPA | No candidate is superior | more power | tuning/bootstrap choices matter |
Hansen’s Superior Predictive Ability (SPA) test improves Reality Check by downweighting or excluding extremely poor candidates. A huge collection of hopeless rules can inflate bootstrap variability and make White’s test excessively conservative. SPA uses standardized differentials and truncation to concentrate inference on plausible competitors.
import numpy as np
def max_stat(differentials):
# rows: time; columns: strategies; precompute net differentials
return np.sqrt(len(differentials)) * differentials.mean(0).max()
def moving_block_bootstrap(x, block, rng):
n = len(x); starts = rng.integers(0, n - block + 1, size=int(np.ceil(n/block)))
return np.concatenate([x[s:s + block] for s in starts])[:n]
The omitted details are the important ones in production: center the null correctly, use enough bootstrap repetitions, select a block length consistent with serial dependence, and avoid regenerating candidate rules inside each resample. Mature implementations should be checked against a statistical library or published reference.
Define a credible benchmark
The benchmark is not necessarily zero return. It may be a passive index, a simple momentum rule, a current production model, or a cost-adjusted cash alternative. The differential must include all realistic trading costs and be aligned at the same timestamps. Testing gross strategy returns against a net benchmark answers a biased question.
| Choice | Bad practice | Better practice |
|---|---|---|
| Candidate list | retain only finalists | include every searched rule |
| Dependence | IID bootstrap daily P&L | block bootstrap |
| Metric | gross return | net, aligned differential |
| Decision | p-value as a trading signal | combine with economics and OOS |
Reality Check and SPA address data snooping within a supplied universe. They do not repair look-ahead bias, survivorship bias, or a researcher who tested many unrecorded families. Before inference, validate the data and workflow with avoiding overfitting in trading. For supervised models with overlapping outcomes, establish valid folds via purged cross-validation.
Reading results
A large p-value says the evidence is insufficient to reject the hypothesis that no candidate beats the benchmark after the search. It does not prove that all candidates are worthless. A small p-value supports relative predictive superiority in the tested history; it does not guarantee future alpha, capacity, or stable execution. Report the candidate count, dependence treatment, bootstrap seed/replications, benchmark, performance metric, and effect size.
SPA, Reality Check, deflated Sharpe and multiple-testing, and PBO answer related but distinct questions. Agreement across them is stronger than any one test, especially when a sealed walk-forward period remains unexamined.
Sensitivity analysis is part of the result. Re-run the test over defensible block lengths, benchmark definitions, and cost assumptions, then explain changes rather than selecting the most favorable specification. A test that reverses under a modest increase in trading cost is describing a fragile economic effect even if its nominal p-value is small.
Key takeaways
- Reality Check bootstraps the maximum performance over the full searched family.
- SPA increases power by reducing the influence of clearly inferior rules.
- Use net benchmark-relative differentials and dependence-preserving resampling.
- Include discarded candidates and disclose every design choice.
- Statistical significance supports an inference claim; it is not a deployment decision.