Model Monitoring and Drift Detection

A trading model fails gradually more often than it fails loudly. A vendor changes a field definition, volatility doubles, a sector disappears from the universe, or execution latency widens. None necessarily causes an exception, yet each can invalidate the relationship that justified a position. Monitoring is therefore a four-layer process: inputs, predictions, realized outcomes, and trading implementation.

Layer	Question	Example alert
Data quality	Is the feed usable?	stale timestamp, null-rate jump
Covariate drift	Do features resemble training?	PSI or KS statistic exceeds limit
Prediction drift	Is the model behaving differently?	score mean or confidence collapses
Outcome/execution	Is the decision monetizing?	IC, slippage, fills, drawdown degrade

Detect the right change

Feature drift does not prove model failure. Markets can legitimately change distribution while a rank-based signal remains useful. Conversely, a stable feature distribution can conceal target drift: the same value predicts a different return because liquidity, policy, or participant behavior changed. Alerting should distinguish diagnosis from action and require persistence, magnitude, and economic relevance.

Population stability index (PSI) compares binned baseline and current feature frequencies:

import numpy as np

def psi(reference, current, bins=10):
    edges = np.unique(np.quantile(reference, np.linspace(0, 1, bins + 1)))
    p, _ = np.histogram(reference, edges)
    q, _ = np.histogram(current, edges)
    p = np.clip(p / p.sum(), 1e-6, None)
    q = np.clip(q / q.sum(), 1e-6, None)
    return np.sum((q - p) * np.log(q / p))

PSI is convenient, but arbitrary bins and autocorrelated observations make textbook thresholds unreliable. Use it alongside distribution plots, missingness, freshness, and segment-level checks. Calculate baselines by market regime and asset class; comparing a crisis month with a calm training average will generate alerts that are true but unhelpful.

Delayed labels need proxy metrics

Trading labels arrive after a holding period, so immediate accuracy is often unavailable. Monitor score distribution, rank turnover, exposure, confidence, feature coverage, and order behavior in real time. When labels mature, calculate information coefficient, calibration, residual return, net P&L, and cost attribution by cohort. Evaluate them with uncertainty: a week of negative alpha is not decisive evidence in a noisy process.

Signal	Likely cause	First response
Input null spike	vendor/pipeline failure	quarantine feature or halt model
Score collapse	feature scale/schema change	compare serving and training transforms
IC deterioration	target/concept drift	reduce risk; diagnose by regime
Slippage increase	liquidity/execution change	tighten capacity and re-estimate costs
Exposure breach	optimizer/config issue	enforce hard limit and rollback

Drift actions must be precommitted

An alert without a response rule becomes dashboard theater. Define severity and owners before deployment. A critical freshness failure can halt new orders immediately. A sustained calibration shift may cut gross exposure, freeze retraining, and open an investigation. A statistically weak performance decline may only increase review frequency. The action should be proportional to capital at risk and reversible where possible.

Avoid automatically retraining on every alarm. Retraining during a transient shock can fit the shock and worsen future behavior. Candidate retrains should pass the same point-in-time validation, including purged cross-validation, as the incumbent. Compare them in shadow mode; a model that looks better on recent data may be another instance of avoiding overfitting in trading.

Monitoring must also cover the execution path: rejected orders, fill ratio, queue delay, borrow availability, position reconciliation, and realized transaction costs. The forecast may remain valid while its implementation becomes untradeable. This operational layer belongs with live trading deployment and monitoring, not only in a machine-learning dashboard.

Key takeaways

Monitor data, features, predictions, realized outcomes, and execution separately.
Drift statistics are diagnostic signals, not automatic evidence that a model is broken.
Use regime-aware baselines and uncertainty-aware performance reviews.
Predefine owners, severity levels, safe fallbacks, and rollback paths.
Retraining must be validated as rigorously as an initial model release.