False Discovery Rate Control in Trading Research
When a research platform tests thousands of signals, asking whether each p-value is below 5% is a machine for manufacturing false alphas. False discovery rate (FDR) control changes the target: rather than trying to prevent every false positive, it limits the expected fraction of false positives among the signals declared significant. That is often the useful trade-off when the objective is a diversified research pipeline rather than a single hypothesis.
The Benjamini-Hochberg (BH) procedure sorts m p-values and finds the largest rank i satisfying p(i) <= i*q/m, where q is the desired FDR. It rejects all hypotheses through that rank.
import numpy as np
def benjamini_hochberg(pvalues, q=0.10):
p = np.asarray(pvalues); order = np.argsort(p)
threshold = q * np.arange(1, len(p) + 1) / len(p)
passed = p[order] <= threshold
cutoff = p[order][np.where(passed)[0].max()] if passed.any() else -np.inf
return p <= cutoff, cutoff
| Error criterion | Controls | Research consequence |
|---|---|---|
| Per-test alpha | Error for each isolated test | many false discoveries overall |
| Family-wise error | Probability of any false positive | often very conservative |
| FDR | Expected false share among discoveries | retains more power |
BH is valid under independence and certain positive-dependence conditions. Trading signals are usually correlated: variants share returns, factors, universes, and data vendors. The Benjamini-Yekutieli correction is valid under arbitrary dependence but conservative; permutation or bootstrap calibrations and hierarchical grouping can be more informative. Do not claim a nominal 10% guarantee without stating the dependence assumption.
What is the hypothesis?
A p-value has meaning only after defining the signal, score, sample, and null. A cross-sectional factor might test whether its net, sector-neutral information coefficient has positive mean; a strategy might test benchmark-relative net return. If researchers altered the horizon, filters, or metric after seeing results, those choices belong in the tested family. FDR cannot recover experiments that were never logged.
| Good input | Weak input |
|---|---|
| preregistered net IC test | best backtest Sharpe after tuning |
| point-in-time universe | revised membership history |
| p-values for all trials | p-values for survivors only |
| block-bootstrap inference | IID t-tests on autocorrelated P&L |
Apply FDR at a meaningful family boundary: all signals proposed for one asset class and decision horizon, for example. Combining unrelated tests can obscure interpretation; splitting one large search into many “families” to avoid correction is equally misleading. A hierarchy is useful: screen broad economic themes, then test controlled variants within the selected themes while accounting for both stages.
FDR should coexist with effect-size thresholds. A statistically detectable micro-alpha can be too small to survive costs, capacity, and model error. Require economic significance, stability across regimes, and a post-selection walk-forward optimization pass. Compare final claims with deflated Sharpe and multiple-testing when the output is a selected strategy rather than a collection of discoveries.
Maintain a discovery ledger containing the hypothesis, code and data version, p-value method, economic effect size, family assignment, and later live outcome. This turns FDR from a one-time statistical filter into a calibration process: if the supposedly controlled discoveries repeatedly fail after selection, revisit the null model, dependence treatment, and definition of the research family.
Key takeaways
- FDR limits the expected false proportion among reported discoveries, not every error.
- BH is simple and powerful but requires careful treatment of correlated tests.
- Define the complete hypothesis family before looking at results.
- Feed it valid, point-in-time, cost-aware inference—not optimized backtest artifacts.
- Combine FDR with economic hurdles, temporal validation, and full experiment logs.