Market Regime Detection with Hidden Markov Models

Market regime detection is the problem of inferring an unobserved, persistent market state — calm-bull, volatile-bear, crisis — from noisy returns, so you can run the right strategy and size risk for the environment you are actually in. Hidden Markov Models (HMMs) give a principled likelihood-based framework for this: they model the latent state, its persistence, and the conditional return distribution it emits. The single most important discipline, and the one most often violated, is using forward-only filtered state estimates in any backtest — never the full-sample smoothed path, which peeks at the future.

What a regime is, statistically

A regime is a period during which the return distribution's parameters (drift, volatility, tail behavior, correlation) are roughly stable. You never observe the label; you infer it from the statistical fingerprint of returns. Common states:

Bull / low-vol — small positive drift, tight dispersion.
Bear / high-vol — negative drift, large swings, correlation spikes.
Crisis — extreme negative returns, volatility explosion, liquidity withdrawal.

The three HMM ingredients

Hidden states. One of K states at each time; you see only the data it generates.
Transition matrix A. A[i,j] = P(state j next | state i now). Diagonal entries

are persistence; daily equity regimes typically show 0.90-0.98 on the diagonal.

Emission distributions. Each state emits returns by its own law (Gaussian as a

baseline; Student-t is far better for fat tails).

Persistence has a clean interpretation: expected regime duration is 1 / (1 - A[i,i]). A state with 0.95 daily persistence lasts ~20 trading days on average — useful for sizing how reactive a regime filter should be.

Filtered vs. smoothed: the look-ahead trap

This distinction decides whether your backtest is honest.

Estimate	Uses	Valid for live?
Filtered P(state_t \	data up to t)	Past + present only
Smoothed P(state_t \	all data)	Past + future
Viterbi path (full sample)	Past + future	No — global decode peeks ahead

model.predict (Viterbi) and full-sample predict_proba smooth over the entire series, so a state at time t is informed by data after t. Backtesting a filter built from them is cheating. In simulation you must recompute the filtered probability using only data through each point.

Fitting and labeling in Python

import numpy as np
import pandas as pd
from hmmlearn.hmm import GaussianHMM

# observations: 2D array, e.g. [log_return, realized_vol] per day (stationary inputs)
X = np.column_stack([returns.values, returns.rolling(20).std().bfill().values])

model = GaussianHMM(n_components=3, covariance_type="full",
                    n_iter=2000, random_state=0).fit(X)

# Resolve label-switching: sort states by mean return so labels are stable
order = np.argsort(model.means_[:, 0])           # 0 = most bearish ... K-1 = most bullish
remap = {old: new for new, old in enumerate(order)}

for i in range(model.n_components):
    mu, sd = model.means_[i, 0], np.sqrt(model.covars_[i, 0, 0])
    dur = 1 / (1 - model.transmat_[i, i])
    print(f"state {remap[i]}: mean={mu:.4f} std={sd:.4f} exp_duration={dur:.0f}d")

EM (Baum-Welch) assigns arbitrary integer labels, and they can permute between refits — the label-switching problem. Always remap states by a stable property (mean or variance) before using them, or your "bull filter" silently inverts after a refit.

Forward-only filtering for an honest backtest

The correct way to use an HMM live is to expand the window, fit (or update) on the past, and read the current filtered probability. A pragmatic, leak-free pattern refits periodically and filters one step at a time:

import numpy as np
from hmmlearn.hmm import GaussianHMM

def online_regime_prob(X, train_len=750, refit_every=21, k=3):
    """Filtered P(bull) using only past data at each step. No look-ahead."""
    n = len(X)
    p_bull = np.full(n, np.nan)
    model = None
    for t in range(train_len, n):
        if model is None or (t - train_len) % refit_every == 0:
            model = GaussianHMM(n_components=k, covariance_type="full",
                                n_iter=500, random_state=0).fit(X[:t])
            bull = int(np.argmax(model.means_[:, 0]))   # highest-mean state
        # filtered posterior over states using data through t-1 only
        post = model.predict_proba(X[:t])               # last row = filtered at t-1
        p_bull[t] = post[-1, bull]
    return p_bull

Because each step fits on X[:t] and reads the last filtered row, the estimate at t never sees t or later — the discipline that separates a real edge from a backtest bias.

What to feed the emissions

A univariate HMM on returns alone tends to separate states almost entirely by volatility, because variance differences dominate the likelihood far more than the tiny drift differences between bull and bear. That is often fine — risk-on/risk-off is mostly a volatility distinction — but if you want regimes that capture more than dispersion, give the emissions multivariate, stationary observations: returns paired with realized volatility, a trend measure, a credit or term spread, or cross-asset correlation. Each added dimension sharpens the separation but costs parameters (a full covariance grows as the square of the dimension), so keep the observation vector small and economically chosen, exactly as in feature engineering. Standardize the inputs on the training window only, and prefer Student-t or volatility- augmented emissions so a single fat-tailed day does not spawn a spurious "crisis" state.

It is also worth being explicit that the Markov assumption — that the next state depends only on the current one — is an approximation. Real regime durations are not memoryless; crises, for instance, have a characteristic length that a geometric duration model fits poorly. Hidden semi-Markov models relax this by modeling duration explicitly, at the cost of more parameters and estimation fragility. For most desks the plain HMM is the right trade-off, but know that the persistence figure it reports is a simplification of richer duration dynamics.

Choosing K without overfitting

More states always fit better in-sample. Penalize complexity with BIC, or compare out-of-sample log-likelihood, and bias toward fewer states.

def select_k(X, ks=(2, 3, 4)):
    rows = []
    for k in ks:
        m = GaussianHMM(n_components=k, covariance_type="full",
                        n_iter=1000, random_state=0).fit(X)
        ll = m.score(X)
        n_params = k*k + 2*k*X.shape[1]            # transitions + means + (diag) covs
        bic = -2*ll + n_params*np.log(len(X))
        rows.append((k, ll, bic))
    return rows                                     # pick the lowest BIC

Two states (risk-on / risk-off) is the most robust default; add a third only when two clearly fails to isolate crisis periods.

Using regimes as a filter, not a forecaster

The durable value of regime detection is conditioning, not prediction. Knowing you are likely in a bear state does not tell you when it ends, but it does tell you which edge to run and how much risk to carry.

Strategy	Best regime	Worst regime
Trend following	Steady trend, low-vol	Choppy range
Mean reversion	Low-vol range	Trending / crisis
Volatility selling	Calm, low-vol	Crisis, vol spike
Breakout	Range-to-trend transition	Late-trend exhaustion

def regime_overlay(strategy_returns, p_bull, lo=0.4, hi=0.6):
    """Size by filtered probability instead of a hard on/off switch."""
    import numpy as np
    weight = np.clip((p_bull - lo) / (hi - lo), 0.0, 1.0)
    return strategy_returns * weight     # scale exposure with conviction

The gain usually shows up as smaller drawdowns rather than higher returns — sitting out the periods where your edge is negative. Combine the HMM with a clean volatility-targeting overlay and it becomes a risk-management layer rather than a prediction engine.

Alternatives and sanity checks

HMMs are not the only way to detect regimes, and a simpler method often validates whether the HMM is finding something real or just labeling volatility. A plain two-state split on trailing realized volatility (above/below its rolling median) captures most of the risk-on/risk-off distinction that a Gaussian HMM recovers, and it has zero estimation risk and no label-switching. If your HMM-filtered overlay does not beat this trivial volatility filter out-of-sample, the HMM is adding complexity without value. Other principled alternatives include Markov-switching regression (regimes in the relationship between variables, not just the marginal of returns) and structural-break tests for a one-off regime change rather than a recurring cycle. The discipline is the same as elsewhere in machine learning for trading: the fancy model must earn its keep against a dumb baseline on data it has not seen.

import numpy as np, pandas as pd

def vol_regime(returns: pd.Series, lookback=20, window=252) -> pd.Series:
    """Trivial risk-on/off baseline to benchmark any HMM against."""
    rv = returns.rolling(lookback).std()
    calm = (rv < rv.rolling(window).median()).astype(int)   # 1 = low-vol regime
    return calm.shift(1)        # forward-only: decide at t using data through t-1

Run this baseline first; only adopt the HMM if it improves out-of-sample drawdown or Sharpe net of the extra turnover that regime switching creates.

Pitfalls and realistic expectations

Detection lag. Any filter needs several observations in a new regime before it

switches; you will always be slightly late, and crisis onsets are the costliest place to be late.

Non-stationarity of regimes themselves. "Bull" in 2010 need not match 2020; refit

on rolling windows.

Gaussian under-modeling tails. Use Student-t emissions or volatility features so a

fat-tailed day is not misclassified.

Probabilities are not certainties. A 60% bull probability is not "bull" — size

proportionally.

Regime detection is not alpha. It improves Sharpe mainly by drawdown avoidance;

expect a modest, regime-dependent uplift, validated by whether the filtered overlay improves out-of-sample risk-adjusted performance — not by how clean the chart looks.

Conclusion

Hidden Markov Models give a likelihood-based way to infer latent market regimes from returns, using a transition matrix for persistence and per-state emissions for each regime's fingerprint. Their greatest practical value is as a filter that turns off strategies losing in the current environment. Fit with care, resolve label-switching, choose K by BIC with a bias toward two states, model fat tails, and — above all — drive every backtest with forward-only filtered probabilities rather than the smoothed or Viterbi path. Combined with sound trend-following or mean-reversion strategies and the broader ML toolkit, regime detection is a risk- management layer that keeps you trading the right strategy in the right environment.