Experiment Design for Trading Research
Most failed trading research does not fail because the researcher cannot fit a model. It fails because the experiment quietly answers a weaker question than the one required for capital allocation. “Does this feature correlate with returns in my notebook?” is not “would a portfolio formed with information available then, after costs, survive selection and deployment?” Experiment design closes that gap before a result becomes persuasive.
Start with a falsifiable hypothesis. Write the mechanism, instrument universe, timestamp convention, forecast horizon, execution rule, risk controls, and failure criterion before inspecting the final performance chart. This makes later changes visible rather than invisible degrees of freedom.
A research protocol
| Stage | Precommitment | Typical failure prevented |
|---|---|---|
| Hypothesis | economic mechanism and sign | feature fishing |
| Dataset | source, PIT cutoff, universe | look-ahead and survivorship |
| Split | train, validation, final test dates | repeated test-set tuning |
| Portfolio | rebalance, constraints, costs | paper-only alpha |
| Decision | promotion threshold | narrative-driven selection |
The protocol should name the benchmark. A stock signal may need to beat sector-neutral momentum, a risk model, or a simple equal-weight portfolio—not zero return. It should also define the unit of observation: daily cross-section, event, trade, or portfolio month. Treat dependent observations as such; 10,000 overlapping five-day labels are not 10,000 independent trials.
Time-respecting splits
Use an early period for model development, a middle validation period for a limited set of predeclared choices, and a final untouched test period for one evaluation. For overlapping labels, use purged cross-validation: remove training observations whose label windows overlap the test fold and add an embargo after it.
# Sketch: decision at t, holding horizon h business days
train = data[data.date < fold_start - embargo]
test = data[(data.date >= fold_start) & (data.date <= fold_end)]
# additionally remove train rows with t + h >= fold_start
train = train[train.label_end < fold_start]
Walk-forward evaluation is valuable when parameters must be periodically refit. Freeze each training window, make predictions for the next test segment, then roll forward. Aggregate only the genuinely out-of-sample predictions. Do not tune window lengths after examining the full walk-forward equity curve unless that tuning is nested inside an earlier validation layer.
Choose metrics that reflect the decision
Forecast metrics such as rank IC, AUC, or calibration diagnose a model; they do not establish a strategy. Translate predictions into a portfolio with realistic position limits, borrow availability, delay, spread, impact, fees, and turnover. Report gross and net Sharpe, drawdown, hit rate, capacity, tail exposure, and factor exposures.
| Result pattern | Likely interpretation | Next action |
|---|---|---|
| strong IC, weak net PnL | costly or unmonetizable signal | optimize trading or discard |
| weak IC, strong PnL | nonlinear sizing or coincidence | inspect attribution |
| one strong year | regime dependence | test conditional hypothesis |
| high gross, high turnover | cost omission | rerun cost sensitivity |
Stress costs at multiples of your estimate and delay signals by one additional bar or day. A signal that survives only an ideal fill assumption is not robust. Review avoiding overfitting in trading when the search involved many features, thresholds, and universe definitions.
Control the research search
Every experiment has multiple testing, even if no p-value is printed. Track each attempted specification: data version, code commit, features, hyperparameters, and decision. A research registry can be a simple append-only table. It lets you estimate the breadth of the search and prevents “forgotten” failures from becoming an uncounted selection mechanism.
Use negative controls. Try the same pipeline on a future-shifted feature, a randomized cross-section, or an asset class without the proposed mechanism. Positive results in a negative control suggest leakage or a generic exposure. Also define a placebo horizon where the causal effect should not yet exist.
Promotion and replication
A result should be reproduced from raw data by a second script or researcher, with a fresh snapshot and a clean environment. Then run a constrained paper-trading or shadow experiment, comparing expected versus realized turnover, fills, and forecast calibration. The production handoff belongs with live deployment monitoring, including explicit rollback criteria.
A negative result is valuable if its protocol was sound. Record it. A desk that preserves failed tests learns faster than one that repeatedly rediscovers them.
Key takeaways
- Precommit the hypothesis, data clock, splits, portfolio rule, and promotion threshold.
- Keep every label, fold, and implementation assumption chronologically valid.
- Evaluate net portfolio economics and stress realistic delays and costs.
- Maintain a research registry, negative controls, and independent replication before capital.