Cointegration Explained: The Foundation of Pairs and Stat Arb Trading

Cointegration is the property that lets you manufacture a stationary, tradeable spread out of two or more non-stationary price series — the statistical engine behind pairs trading and statistical arbitrage. The textbook definition is easy; the parts that lose money are the test assumptions, the power problems, and the look-ahead traps in estimating the hedge ratio. This article is about those parts. If you only remember one thing: a cointegration p-value is a statement about a finite sample under maintained assumptions, not a guarantee that the spread will revert tomorrow.

The Granger representation, precisely

Two I(1) series A and B are cointegrated if there exists a vector (1, -beta) such that At - beta*Bt is I(0) — integrated of order zero, i.e. stationary. The Granger Representation Theorem makes the trading-relevant claim: if a cointegrating relationship exists, the system has an error-correction representation,

dA_t = c_a + gamma_a * (A_{t-1} - beta*B_{t-1}) + (lagged diffs) + e_t
dB_t = c_b + gamma_b * (A_{t-1} - beta*B_{t-1}) + (lagged diffs) + u_t

The term in parentheses is last period's spread, and gammaa, gammab are the speeds of adjustment. At least one must be non-zero and signed so the spread is pulled back toward equilibrium. This is the formal content of "mean reversion": the spread does not just happen to be stationary, there is an economic force (gamma) correcting deviations. The half-life of that correction is what determines holding period and capacity — estimate it, do not assume it.

Why correlation is a different object

Correlation is a contemporaneous statement about returns; cointegration is a statement about the levels sharing a common stochastic trend. Two independent random walks have zero expected return correlation but can drift together for years; a cointegrated pair can have unremarkable daily return correlation while its spread snaps back reliably. You can construct examples with any combination of high/low correlation and present/absent cointegration. For a reversion trade you need the cointegration; correlation tells you nothing about whether the spread is anchored.

The hedge ratio is an estimate with bias

Engle-Granger step one regresses A on B by OLS to get beta. Two issues that the textbook glosses over and that matter at the trading-money level:

Asymmetry. Regressing A on B and B on A give different betas unless

the fit is perfect. The OLS slope is attenuated by the variance of the regressor, so the choice of dependent variable is not innocent. With cointegrated I(1) data OLS is super-consistent (converges at rate T rather than sqrt(T)), so the betas agree asymptotically — but in finite samples they differ, and the difference is a directional bet you did not mean to make.

Endogeneity and second-order bias. OLS residuals are correlated with the

regressor in finite samples, biasing beta. This is why dynamic OLS (adding leads and lags of dB) or the Johansen MLE are preferred when you care about the exact ratio rather than a quick screen.

import numpy as np
import statsmodels.api as sm
from statsmodels.tsa.stattools import adfuller, coint

def engle_granger(a, b):
    # Super-consistent but finite-sample-biased hedge ratio
    x = sm.add_constant(b)
    ols = sm.OLS(a, x).fit()
    beta = ols.params.iloc[1]
    spread = a - beta * b
    adf_stat, adf_p = adfuller(spread, maxlag=1, regression="c")[:2]
    eg_stat, eg_p, _ = coint(a, b)   # uses correct EG critical values
    return dict(beta=beta, spread=spread, adf_p=adf_p, eg_p=eg_p)

Critically, the ADF p-value from adfuller applied to estimated residuals is wrong — the residuals were chosen to minimize variance, so the test is biased toward rejecting the unit root (toward declaring cointegration). Use statsmodels.coint, which applies the correct Engle-Granger / MacKinnon critical values that account for the pre-estimated beta. This single mistake manufactures spurious pairs at scale.

Test power is the real enemy

The ADF and Engle-Granger tests have a null of "no cointegration / unit root in the spread." Their power — the probability of detecting cointegration when it truly exists — is low precisely in the cases you care about. If the spread reverts slowly (AR(1) coefficient near 1, long half-life), the test cannot distinguish it from a random walk in a short sample. The rule of thumb: detecting a spread with half-life h reliably needs a sample many multiples of h long. A 40-day half-life reversion needs years of data to clear the test, which is the same data over which the relationship may have structurally changed.

Spread AR(1) phi	Half-life (days)	Detectability with ~1yr daily
0.90	~6.6	high power
0.97	~22.8	moderate
0.99	~69	low power, many false negatives
0.995	~138	effectively undetectable

The flip side is the multiple-testing problem. Scanning k pairs at a 5% level yields about 0.05*k false positives even if nothing is cointegrated. A 500-name universe has ~125,000 pairs and ~6,000 spurious "cointegrated" results. Any pair screen must be paired with a multiple-testing correction, an economic prior, and out-of-sample confirmation, or it is a false-discovery generator.

Johansen for baskets and symmetric estimation

For more than two assets, or when you want to avoid the dependent-variable asymmetry, the Johansen procedure tests the rank of the long-run impact matrix via a VECM and returns cointegrating vectors symmetrically. The trace and max-eigenvalue statistics tell you how many independent cointegrating relationships exist — useful when building a basket rather than a single pair.

from statsmodels.tsa.vector_ar.vecm import coint_johansen

def johansen_rank(prices, det_order=0, k_ar_diff=1, sig=1):
    # sig: 0=90%, 1=95%, 2=99%
    res = coint_johansen(prices, det_order, k_ar_diff)
    rank = int((res.lr1 > res.cvt[:, sig]).sum())   # trace test
    vec = res.evec[:, 0]                             # leading cointegrating vector
    return rank, vec / vec[0]                        # normalized weights

See the Johansen test and VECM for the full treatment. Johansen is more powerful and avoids EG's normalization choice, at the cost of more assumptions about deterministic terms and lag length — get kardiff wrong and the rank inference shifts.

From spread to signal without look-ahead

The most common backtest inflation in this whole field is estimating beta and the spread's mean/variance on the full sample, then "trading" that sample. The spread is constructed using future information; the z-score is centered on a mean it could not have known. Everything must be causal: fit beta on a training window, standardize with trailing statistics only, and re-estimate on a rolling basis (or via a Kalman filter so the hedge ratio adapts).

def rolling_zscore(spread, window=30):
    mu = spread.rolling(window).mean()
    sd = spread.rolling(window).std()
    return (spread - mu) / sd      # uses only past data at each point

The OU framework gives the principled version: model the spread as an Ornstein-Uhlenbeck process, estimate the mean-reversion speed kappa and the equilibrium, and set entry bands from the stationary distribution rather than ad hoc |z| > 2. The optimal band is a function of kappa, the spread volatility, and transaction costs — wider bands when costs are high relative to the reversion amplitude.

Calibrating the reversion: half-life and OU

The decision-relevant statistic is not the p-value but the reversion speed, which sets holding period and capacity. Fit an AR(1) to the spread, st = c + phi*s{t-1} + eps, and the half-life of mean reversion is ln(0.5)/ln(phi). Equivalently, calibrate the continuous-time Ornstein-Uhlenbeck parameters by mapping the discrete AR(1): kappa = -ln(phi) (per period), the equilibrium is c/(1-phi), and the stationary standard deviation is sigma_eps / sqrt(1 - phi^2).

import numpy as np
import statsmodels.api as sm

def reversion_stats(spread):
    s = spread.dropna()
    lag = s.shift(1).dropna()
    s = s.loc[lag.index]
    fit = sm.OLS(s, sm.add_constant(lag)).fit()
    phi = fit.params.iloc[1]
    if not (0 < phi < 1):
        return dict(phi=phi, half_life=np.inf, tradeable=False)
    half_life = np.log(0.5) / np.log(phi)
    sigma_eq = fit.resid.std() / np.sqrt(1 - phi**2)
    return dict(phi=phi, half_life=half_life, sigma_eq=sigma_eq, tradeable=True)

The half-life feeds two decisions directly. First, the z-score window should be on the order of a few half-lives — too short and you standardize against transient noise, too long and the band lags the relationship. Second, capacity: a spread with a 2-day half- life turns over far more than one with a 40-day half-life, so the shorter reverter is more cost- and capacity-constrained even if its statistical edge looks larger. A collapsing half-life over time is the cleanest early warning that the relationship is deteriorating.

Trading the spread: sizing and breakage

A "unit" of the spread is one unit of A against beta units of B. The position is dollar-hedged to first order, capturing the convergence regardless of market direction — the market-neutrality is the appeal. But cointegration breaks: index reconstitutions, mergers, recapitalizations, and regime shifts permanently change beta or destroy the relationship. A "reverting" spread that has actually broken is an unbounded loss while you wait. Therefore:

Stop on spread divergence (|z| > 3-4) and on time (a max holding equal to a

few half-lives), because a structurally broken spread will not trigger a z-based stop before it ruins you.

Monitor the adjustment speed gamma/kappa live; a collapsing reversion speed is

an early warning the relationship is dying.

Trade a portfolio of independent spreads so idiosyncratic breakages diversify, and

validate with realistic costs since spread trading is high-turnover and cost-sensitive — a proper backtest with slippage will kill many statistically "valid" pairs.

Honest limits

Cointegration is a linear, constant-parameter model of a relationship that is neither linear nor constant. The tests have a clean null but low power against the slow-reverting spreads that are most attractive, and high false-positive rates under mass scanning. Super-consistency saves the asymptotics but not your finite sample. The economic anchor — shared inputs, ETF-versus-basket arbitrage, same-sector exposure — matters more than the p-value, because it is what makes the relationship likely to persist out of sample. Test honestly on training data with the correct critical values, build causal signals, size to the estimated reversion speed, stop on both divergence and time, and treat every cointegrated pair as a relationship that will eventually break rather than one that is guaranteed to revert.