State-Space Models for Trading

Market data is an imperfect measurement of an evolving economic state. A quoted price contains a latent efficient price, bid-ask bounce, asynchronous trading, and occasional bad prints. A factor return mixes persistent premium, temporary flow, and sampling noise. State-space models make that distinction explicit: a hidden state evolves through time and observations reveal it noisily.

This is more than smoothing. The model supplies a likelihood, uncertainty estimates, missing-observation handling, and a disciplined route from a forecast to a position size. The Kalman filter is the linear-Gaussian workhorse, but the modeling choices around it determine whether it is useful in trading.

The two equations

In the basic discrete model, a latent vector xt follows a transition equation and the observed vector yt follows a measurement equation:

x_t = A_t x_(t-1) + B_t u_t + w_t,    w_t ~ N(0, Q_t)
y_t = H_t x_t + v_t,                  v_t ~ N(0, R_t)

A controls persistence, Q is state innovation variance, H maps states to observations, and R is measurement noise. A local linear trend for log price can use state [level, slope]; a dynamic hedge ratio can use [alpha, beta]; a multivariate model can treat factor exposures as states.

Trading question	Hidden state	Observation	Useful output
Is a move informational?	efficient log price	trades or midquotes	filtered return
Is a spread reverting?	intercept and hedge ratio	two asset prices	residual z-score
Has a factor changed?	time-varying beta	asset and factor returns	exposure forecast
Is volatility elevated?	latent variance	squared returns	risk scale

The filtered estimate uses observations through time t; the smoothed estimate uses future data too. Only the filter may drive a historical live-trading backtest. Using a smoother to generate signals is a subtle, common form of look-ahead bias.

Kalman recursion and uncertainty

Given prior mean m^-t and covariance P^-t, prediction and update are:

m^-_t = A_t m_(t-1)
P^-_t = A_t P_(t-1) A_t' + Q_t
K_t = P^-_t H_t' (H_t P^-_t H_t' + R_t)^-1
m_t = m^-_t + K_t (y_t - H_t m^-_t)

The innovation yt - Ht m^-_t is as important as the state. Standardized innovations should be approximately uncorrelated and unit variance if the model is adequate. Clusters of large innovations flag a volatility shift, stale measurement variance, a corporate action, or a broken feed.

Avoid explicitly inverting the innovation covariance in production. Cholesky solves are more stable, and the Joseph covariance update preserves positive semidefiniteness. At intraday frequency, numerical hygiene matters more than an elegant symbolic formula.

A dynamic pairs model in Python

For two related log prices, let yt = alphat + betat xt + noise, with a random walk for [alpha, beta]. The following sketch highlights the clock discipline:

import numpy as np

state = np.array([0.0, 1.0])
P = np.diag([1.0, 0.1])
Q = np.diag([1e-6, 1e-5])       # beta turnover prior
R = 2e-4                        # residual measurement variance

for x, y in synchronized_log_prices:
    P = P + Q                   # predict: random-walk state
    H = np.array([[1.0, x]])
    innovation = y - H @ state
    S = H @ P @ H.T + R
    K = (P @ H.T / S).ravel()
    state = state + K * innovation.item()
    P = (np.eye(2) - np.outer(K, H.ravel())) @ P
    z = innovation.item() / np.sqrt(S.item())

The z score is not a trade by itself. Require synchronized, tradable prices; account for hedge-ratio estimation turnover; and use a rolling or filtered volatility estimate. A beta that adapts quickly reduces residual variance but can convert a genuine convergence trade into parameter churn.

Estimation is the real model

Maximum likelihood estimates Q, R, and transition parameters by running the filter and summing innovation log likelihoods. Expect weak identification: increasing both Q and R can make similar in-sample fits while implying very different responsiveness. Put economically motivated bounds on half-lives and update rates, validate in chronological folds, and inspect parameter stability.

Symptom	Likely cause	Response
State tracks every tick	`Q` too large or `R` too small	increase measurement noise
State never adapts	`Q` too small	permit faster transitions
Innovation autocorrelation	omitted dynamics	add slope, seasonality, or lags
Fat innovation tails	Gaussian misspecification	robust filter or jump flag

For returns, conditional heteroskedasticity makes constant R unrealistic. Couple the measurement variance to a volatility forecast, or use a stochastic-volatility state. GARCH volatility modeling is often a defensible variance layer when a full nonlinear filter is unnecessary.

From state estimate to trading decision

The posterior covariance prevents false precision. For a forecasted excess return mu with predictive variance sigma2, a constrained position might scale with mu / sigma2, clipped for leverage and liquidity. Widen entry bands when posterior uncertainty is high; a residual of two standard deviations means less when beta is poorly identified.

Backtest with as-of data, realistic rebalance timing, and transaction costs. Re-estimating hyperparameters on the entire sample leaks future regimes even if each filtering update is chronological. Treat model diagnostics, forecast calibration, and trading P&L as separate reports.

Key takeaways

State-space models distinguish a changing latent process from noisy market observations.
Use filtered, never smoothed, states for executable historical signals.
Q and R govern responsiveness and are economically consequential, not cosmetic settings.
Size trades with posterior uncertainty and validate innovations as well as P&L.