MLOps for Trading Models

Trading MLOps is a risk-control discipline, not an automated path from a good notebook to production. A model can be statistically valid and still lose money because the live feature cutoff differs from training, its order assumptions are impossible, or a routine retrain promotes a fragile parameter set. The operating system must preserve the evidence supporting each model and require deliberate gates between research, shadow trading, and capital allocation.

The unit of production is a model package, not a serialized estimator. It includes the estimator, feature-set version, training-data manifest, label definition, code commit, hyperparameters, prediction schema, risk limits, and evaluation report. A registry makes that package immutable and gives every live prediction a traceable provenance.

Stage	Goal	Required evidence
Research	Find plausible hypotheses	Logged experiments and point-in-time data
Validation	Estimate generalization	Purged CV and untouched holdout
Shadow	Test operational behavior	Live inputs, no capital at risk
Canary	Limit economic exposure	Risk limits and rollback trigger
Production	Execute governed model	Monitoring and periodic review

Make experiments auditable

Tracking only the best run is selection bias. Log every feature family, parameter grid, failed trial, random seed, and data snapshot. This is essential when interpreting a headline Sharpe with deflated Sharpe and multiple-testing. A useful experiment record has metrics by fold and regime, transaction-cost assumptions, turnover, prediction calibration, and an artifact reference—not just a scalar score.

Training must consume a manifest rather than a mutable warehouse table. The manifest identifies the universe at each date, corporate-action adjustments, raw vendor snapshot, and feature versions. Pair it with a deterministic environment:

from dataclasses import dataclass
from hashlib import sha256
import json

@dataclass(frozen=True)
class ModelManifest:
    git_commit: str
    feature_set: str
    label_version: str
    data_snapshot: str
    universe_version: str

    def model_id(self):
        return sha256(json.dumps(self.__dict__, sort_keys=True).encode()).hexdigest()[:12]

The identifier is a starting point, not proof of reproducibility: pin package versions, random-number seeds, and hardware-sensitive libraries where they affect fitted results.

Validation is not deployment

Cross-validation chooses among candidates, but it cannot recreate the sequence of a live investment process. Use purged cross-validation for overlapping labels, then use walk-forward optimization or a sealed period to mimic repeated retraining and rebalance decisions. Evaluate realized P&L after spreads, fees, impact, borrow, rejects, and capacity—not prediction accuracy alone.

Metric	Why it matters	Common trap
Information coefficient	Prediction ranking	Ignores monetization
Net Sharpe	Risk-adjusted return	Hides tails and turnover
Calibration	Position-size confidence	Assumes stationary mapping
Fill/reject rate	Execution feasibility	Backtest assumes fills
Feature coverage	Input health	Missing values silently imputed

Promotion gates should be declarative. For example: no leakage test failures; walk-forward net Sharpe above a hurdle with uncertainty; exposure and turnover below limits; no material degradation in stress periods; and a signed model review. A CI pipeline can execute schema, temporal, and unit tests automatically, but a human should approve new risk-bearing behavior.

Deployment patterns

Batch models can publish target weights after the close. Event-driven models need feature freshness guarantees and a bounded inference latency. In both cases, separate prediction from execution: prediction service returns a timestamped forecast and confidence; a portfolio/risk service applies holdings, limits, borrow, liquidity, and kill-switch rules. This separation prevents a model release from bypassing trading controls.

Shadow deployment is especially valuable. Feed live features to the candidate, record counterfactual orders, and compare its inputs and decisions with the backtest assumptions. Canary release then allocates a small, capped sleeve. Rollback must restore a known package and configuration, not merely “the previous code,” because data and feature definitions may have changed too.

Retraining and model decay

Scheduled retraining is not automatically adaptive. It can perpetually select recent noise. Specify the retraining cadence, lookback, candidate set, and promotion test in advance; log the decision to keep the incumbent. In many low-signal strategies, an incumbent is safer unless the challenger clears a meaningful hurdle.

Finally, MLOps continues after release. Prediction distributions, data freshness, fills, realized slippage, exposures, and realized-versus-expected outcomes should feed the live trading deployment and monitoring process. An alert is useful only when it has an owner and a documented action: investigate, reduce risk, freeze retraining, or roll back.

Key takeaways

• Promote immutable model packages, not isolated estimator files.
• Log the complete research search so model-selection claims remain auditable.
• Use temporal validation, shadow runs, and canaries before material capital.
• Separate forecasts from portfolio construction and execution risk controls.
• Treat retraining and rollback as governed investment decisions, supported by live monitoring.