Kalman Filters in Trading: Dynamic Hedge Ratios and Smoothing

The Kalman filter is the minimum-variance recursive estimator for a hidden state that drifts over time and is observed only through noise — which describes most estimation problems in statistical arbitrage. In Kalman filter trading, the hidden state is typically a time-varying hedge ratio between cointegrated assets, the fair value of a spread, or a smoothed trend level. Its appeal over rolling windows is that it is online (O(1) per update, no refit), it produces calibrated uncertainty for free, and it replaces an arbitrary lookback length with an interpretable signal-to-noise ratio. Its danger is that, misconfigured, it will happily track a relationship that has actually broken.

Why not just use a rolling window?

Rolling estimators (a 60-day regression beta, a 60-day vol) have two chronic defects: they lag, because every observation in the window is weighted equally, and the window length is arbitrary. The Kalman filter treats the quantity you want as a hidden state that evolves, and updates its belief optimally as each observation arrives — weighting new data against prior belief by their relative uncertainties. There is no window to pick; the filter adapts as fast as the data justify.

State-space model

Two equations define the system:

state_t        = F @ state_{t-1} + process_noise     # cov(process_noise) = Q
observation_t  = H_t @ state_t   + measurement_noise # var(measurement_noise) = R

F describes the state dynamics (often identity: "tomorrow's state is roughly today's"), H_t maps the state into observation space, and the two noise covariances encode the core trade-off:

Q (process noise) — how fast you believe the true state moves. Large Q ⇒ trust

new data, adapt quickly.

R (measurement noise) — how noisy observations are. Large R ⇒ lean on the prior,

stay smooth.

The ratio Q/R is the single dial that replaces window length, and unlike a lookback it has a probabilistic meaning you can reason about.

The predict-update recursion

For each new point the filter tracks the state estimate and its error covariance P:

# Predict
state_pred = F @ state
P_pred     = F @ P @ F.T + Q          # uncertainty grows with process noise

# Update
innovation = obs - H @ state_pred     # the surprise
S          = H @ P_pred @ H.T + R     # innovation variance
K          = P_pred @ H.T / S         # Kalman gain
state      = state_pred + K @ innovation
P          = (I - K @ H) @ P_pred     # uncertainty shrinks after seeing data

The Kalman gain K is the heart of it: large when prediction uncertainty is high (lean on the observation), small when measurement noise is high (trust the prior). The innovation and its variance S are a bonus output — standardized, the innovation is a tradeable z-score.

Dynamic hedge ratio for pairs trading

The canonical application. In pairs trading, you trade the spread y - beta*x between two cointegrated assets, but beta is not constant — a static OLS fit gives one number for the whole sample while the true relationship drifts. Model [alpha, beta] as the hidden state (random walk) and y as the observation with Ht = [1, xt]. Each step yields an updated hedge ratio and an innovation whose standardized value is the spread z-score you trade.

import numpy as np

def kalman_hedge(x, y, delta=1e-4, R=1e-3):
    """Time-varying [alpha, beta] linking y to x (Chan-style), fully online.
    delta sets process noise Q; R is observation noise. Returns betas + z-scores."""
    n = len(y)
    Q = delta / (1 - delta) * np.eye(2)
    state = np.zeros(2)            # [alpha, beta]
    P = np.eye(2) * 1e-3
    betas = np.zeros((n, 2))
    z = np.zeros(n)
    for t in range(n):
        H = np.array([1.0, x[t]])
        # predict (F = I)
        P = P + Q
        # innovation
        y_pred = H @ state
        e = y[t] - y_pred
        S = H @ P @ H + R
        # update
        K = (P @ H) / S
        state = state + K * e
        P = P - np.outer(K, H) @ P
        betas[t] = state
        z[t] = e / np.sqrt(S)      # standardized spread, the trading signal
    return betas, z

You enter when |z| exceeds a threshold and exit as it reverts toward zero — the same playbook as cointegration-based pairs trading, but with a hedge ratio that adapts in real time. Note this is genuinely leak-free: every estimate at t uses only data through t.

Tuning Q and R by maximum likelihood

Hand-picking delta and R invites a backtest bias if you optimize them on the full sample. The principled approach maximizes the Gaussian log-likelihood of the innovations on a training window only, then freezes the parameters for out-of-sample use:

import numpy as np

def innovation_loglik(x, y, delta, R):
    """Gaussian log-likelihood of one-step innovations; maximize on TRAIN only."""
    n = len(y)
    Q = delta / (1 - delta) * np.eye(2)
    state, P, ll = np.zeros(2), np.eye(2) * 1e-3, 0.0
    for t in range(n):
        H = np.array([1.0, x[t]])
        P = P + Q
        e = y[t] - H @ state
        S = H @ P @ H + R
        ll += -0.5 * (np.log(2*np.pi*S) + e*e / S)   # per-step innovation likelihood
        K = (P @ H) / S
        state = state + K * e
        P = P - np.outer(K, H) @ P
    return ll
# Grid- or optimizer-search delta, R over the training window; keep the argmax,
# then evaluate frozen parameters out-of-sample.

This turns "what window length?" into "what Q/R maximizes the likelihood of the data I actually had" — a defensible, look-ahead-free choice.

Local-level smoother (adaptive moving average)

The simplest case: model a price's hidden "true level" as a drifting scalar observed through noise (F = 1, H = 1). The result is a low-lag adaptive moving average — a higher Q/R gives a faster, twitchier line; a lower ratio a smoother, laggier one.

def kalman_level(prices, Q=1e-5, R=1e-2):
    """1D local-level filter: an adaptive moving average with no fixed window."""
    n = len(prices)
    est, P, out = prices[0], 1.0, [0.0]*n
    for t in range(n):
        P = P + Q                              # predict
        K = P / (P + R)                        # gain
        est = est + K * (prices[t] - est)      # update
        P = (1 - K) * P
        out[t] = est
    return out

The pykalman library offers a full multivariate implementation with EM-based parameter estimation; the hand-rolled versions above expose exactly what happens under the hood.

Diagnosing a healthy filter through its innovations

A Kalman filter that is correctly specified produces innovations that are white (serially uncorrelated) and have variance matching the filter's own predicted S. This gives you a free, principled diagnostic that does not exist for a rolling window. If the standardized innovations are autocorrelated, the filter is mis-tuned (usually Q too low, so it lags and the errors trend) or the linear-Gaussian model is wrong for the data. If their realized variance is persistently larger than the predicted S, your R is too small and the z-scores you trade on are overstated. Monitoring these two statistics live is the cheapest early warning that a pair's relationship is breaking.

import numpy as np
from statsmodels.stats.diagnostic import acorr_ljungbox

def innovation_diagnostics(z):
    """z: standardized innovations. Healthy filter -> white, unit-variance."""
    z = np.asarray(z)
    lb = acorr_ljungbox(z, lags=[10], return_df=True)   # autocorrelation test
    return {
        "realized_var": float(np.var(z)),     # should be ~1.0 if R is right
        "ljung_box_p": float(lb["lb_pvalue"].iloc[0]),  # >0.05 -> innovations white
    }

A realized variance far from 1.0 or a Ljung-Box p-value near zero tells you to re-tune before trusting the spread z-score — a discipline that distinguishes a maintained filter from one quietly tracking a dead relationship.

Strengths, limits, and realistic expectations

Property	Implication
Online, O(1) per step	Ideal for live trading; never refit on full history
Provides uncertainty (`P`, `S`)	Free confidence bands and a natural z-score
Optimal under linear-Gaussian assumptions	Minimum-variance estimator if assumptions hold
Assumes linearity, Gaussian noise	Fat tails and nonlinearity need EKF/UKF/particle filters
`Q/R` still chosen	More principled than a window, not parameter-free
Random-walk drift assumption	Tracks a broken relationship instead of warning you

Two honest caveats define expectations. First, the filter is a state estimator, not a forecaster — it tells you the current hedge ratio or level well, but predicts the future no better than the underlying dynamics allow. Second, a constantly drifting beta implies constant rebalancing; set Q too high and the resulting turnover and transaction costs can erase the edge while the spread never looks stationary. Set Q too low and you are back to a laggy rolling estimate. And crucially, a dynamic hedge ratio does not rescue a pair that is not genuinely mean-reverting — confirm cointegration first, and monitor P: if state uncertainty stops shrinking, the fit is poor or the relationship has broken.

Conclusion

The Kalman filter replaces arbitrary rolling windows with a principled, online estimate of a hidden, drifting quantity, updating optimally as each observation arrives. Its two most valuable trading uses are a dynamic hedge ratio for pairs trading — where it conveniently yields both the adaptive beta and the spread z-score — and low-lag adaptive smoothing. It is not a forecasting oracle, it still asks you to specify the Q/R trade-off (tune it by maximum likelihood on training data, never the full sample), and it will track a broken regime if you let it. Pair it with rigorous cointegration testing and honest out-of-sample validation, and it earns its place in any quant's toolkit.