Combinatorial Purged Cross-Validation (CSCV)

Combinatorially symmetric cross-validation (CSCV) asks a deliberately uncomfortable question: when a strategy wins on one half of history, does it still rank well on the other half? Unlike an ordinary train/test split, it repeats that experiment across every symmetric partition of time. This transforms one favorable historical path into a distribution of selection outcomes.

CSCV is often discussed with the Probability of Backtest Overfitting (PBO). It is not the same as purged cross-validation: purging protects supervised learning samples whose labels overlap. CSCV evaluates a matrix of strategy returns and the selection rule that chooses a winner. When returns contain model predictions built from overlapping labels, use both controls in the proper stage.

Method	Primary question	Output
Walk-forward	Would deployment sequence work?	One chronological OOS path
Purged CV	Did labels leak across folds?	Honest ML validation folds
CSCV	Does selection survive resampling?	Distribution of OOS ranks

Constructing the splits

Arrange T return observations for N candidate strategies in a T × N matrix. Divide time into S contiguous, equal blocks, typically an even number. For every choice of S/2 blocks as in-sample, the complementary blocks become out-of-sample. On each split:

1. score every strategy in-sample, using net Sharpe or another predeclared measure;
2. select the in-sample winner;
3. rank that winner out-of-sample among all candidates;
4. map the fractional OOS rank r to log(r / (1-r)).

The fraction of logits below zero is PBO: the share of splits in which the selected strategy falls below the OOS median. A high PBO means the selection process repeatedly chooses configurations that fail to generalize.

from itertools import combinations
import numpy as np

def cscv_pbo(returns, blocks=8):
    groups = np.array_split(np.arange(len(returns)), blocks)
    logits = []
    for chosen in combinations(range(blocks), blocks // 2):
        ins = np.concatenate([groups[i] for i in chosen])
        oos = np.concatenate([groups[i] for i in range(blocks) if i not in chosen])
        winner = returns[ins].mean(0).argmax()
        rank = (returns[oos].mean(0) <= returns[oos, winner].mean()).mean()
        rank = np.clip(rank, 1e-4, 1 - 1e-4)
        logits.append(np.log(rank / (1 - rank)))
    return (np.array(logits) < 0).mean(), np.array(logits)

For real research replace the mean with a correctly annualized, cost-adjusted metric and account for different strategy volatilities. The code is intentionally schematic: shared observations and serial dependence mean its numerical output is an estimate, not a universal significance level.

What CSCV reveals

The average OOS return can look adequate while the logit distribution is wide or left-skewed. That is a warning that the selected parameter is unstable. Conversely, a low PBO says the winner tends to retain relative rank across historical partitions; it does not establish causality, capacity, or future profitability.

Result	Interpretation	Next step
High PBO	Search is likely fitting noise	simplify family; hold out data
Wide logits	Selection is fragile	inspect regimes and costs
Low PBO, low return	Stable but uninteresting	reject on economics
Low PBO, high return	Promising evidence	confirm walk-forward/live

Choose blocks long enough to contain meaningful market states and avoid splitting a single event cluster into artificial independent samples. Very few blocks yield little resampling; too many produce noisy half-samples. Eight to sixteen blocks are a practical starting sensitivity range, not a magic setting.

CSCV is strongest when it is decided before the search and run on all candidates, including discarded ones. Reporting it only for the eventual winner understates selection risk, just as excluding failed variants distorts deflated Sharpe and multiple-testing.

Key takeaways

• CSCV repeatedly tests whether an in-sample winner stays above median out of sample.
• PBO is the fraction of symmetric splits where that winner ranks below the OOS median.
• It complements, rather than replaces, purging and strict walk-forward validation.
• Use net, economically feasible strategy returns and report the full logit distribution.
• A low PBO is evidence against one form of overfitting, not proof of tradable alpha.