Multi-Armed Bandits for Strategy Allocation

Multi-armed bandits formalize the explore-vs-exploit problem: you have K strategies (arms), each with unknown reward distributions, and you must allocate capital over time to maximize cumulative reward. Unlike a static portfolio optimizer that assumes known means and covariances, bandits learn while trading — useful when edges decay, regimes shift, and you cannot wait for a perfect backtest. This article maps bandit algorithms to strategy allocation and shows where they fail.

The allocation problem as a bandit

Each day (or week) you choose a weight vector over strategies. Simplest reduction: each period pick one strategy to overweight (discrete arm). Richer: continuous allocation with a contextual bandit.

reward_t(arm_a) = strategy_a return_t - cost_t(a) - risk_penalty_t(a)

The reward must include transaction costs and risk — raw return encourages the noisiest high-vol arm.

Classic algorithms

Epsilon-greedy

With probability ε explore (random arm); otherwise pick the empirical best.

import numpy as np

def epsilon_greedy(means: np.ndarray, epsilon: float = 0.1) -> int:
    if np.random.rand() < epsilon:
        return int(np.random.randint(len(means)))
    return int(np.argmax(means))

Simple, tunable, often hard to beat as a baseline. Decay ε over time as estimates stabilize.

UCB (Upper Confidence Bound)

Pick the arm with highest mean + exploration bonus:

score_a = μ̂_a + c × sqrt(ln(t) / n_a)

Arms with few pulls get a boost — systematic exploration.

def ucb_select(means, counts, t, c=2.0) -> int:
    scores = []
    for mu, n in zip(means, counts):
        if n == 0:
            return scores.__len__()  # force unvisited
        scores.append(mu + c * np.sqrt(np.log(t + 1) / n))
    return int(np.argmax(scores))

Thompson Sampling

Maintain a posterior on each arm's mean; sample from posteriors; pick the max sample. Natural Bayesian fit for Bayesian methods.

def thompson_gaussian(means, precisions) -> int:
    samples = [np.random.normal(m, 1 / np.sqrt(p + 1e-9)) for m, p in zip(means, precisions)]
    return int(np.argmax(samples))

Thompson often outperforms UCB in non-stationary finance settings when you discount old observations.

Non-stationarity: the finance problem

Markets violate the stationary-arm assumption. A strategy that worked last year may be crowded or decayed. Fixes:

Fix	Mechanism
Sliding window	Estimate means only on last W days
Exponential discounting	Weight recent rewards more
Change-point detection	Reset counts when regime breaks
Contextual bandits	Condition on regime features

Without non-stationary handling, bandits lock onto yesterday's winner and ride it into a drawdown.

Contextual bandits

Context x_t (volatility, funding, seasonality, regime) changes which arm is best:

π(a | x) = policy that maps context → allocation

LinUCB / contextual Thompson use linear reward models:

E[r | a, x] = θ_a · x

This connects to reinforcement learning for trading but with a simpler one-step horizon — often more stable than full RL.

def linucb_score(x, theta, A_inv, alpha=0.5) -> float:
    """UCB score for linear contextual arm."""
    mean = float(theta @ x)
    bonus = alpha * float(np.sqrt(x @ A_inv @ x))
    return mean + bonus

From discrete arms to portfolio weights

Practical pattern used by quant desks:

1. Run N strategy sleeves with fixed internal logic
2. Bandit (or meta-model) chooses relative risk budgets each week
3. Enforce max weight, max turnover, and correlation constraints
4. Log every decision for offline evaluation (IPS / Doubly Robust)

Do not let the bandit freely rebalance daily at high turnover — exploration costs are real market impact.

Evaluation without fooling yourself

Online algorithms need offline validation:

• Replay with inverse propensity scoring if you logged propensities
• Walk-forward bandit on historical returns (walk-forward)
• Compare to equal-weight and static Markowitz baselines
• Penalize for the number of arms tested (multiple testing)

A bandit that beats equal-weight only by taking more equity beta is not alpha.

When not to use bandits

• You have one strategy and no alternative sleeves
• Horizon is so short that exploration is pure cost
• Rewards are too noisy for means to converge (need more aggregation)
• Regulatory / risk limits require fixed allocations

Bandits shine when you have several plausible edges and the ranking changes over time — not as a substitute for having an edge.

Key takeaways

• Bandits allocate under uncertainty while learning — explore vs exploit
• UCB and Thompson Sampling are strong defaults; discount for non-stationarity
• Contextual bandits condition on regime features
• Constrain turnover and risk — exploration is not free
• Evaluate against equal-weight and static optimizers with honest costs