Market Data for Quants: Sources, Cleaning and Pitfalls

Market data is the substrate every strategy rests on, and it is where most backtests quietly die. A correct model fed dirty, biased, or look-ahead-contaminated data produces a beautiful equity curve that evaporates live. Professional desks spend more effort sourcing, cleaning, and validating market data than on modeling, because the data layer is where edges are preserved or destroyed. This guide covers data types and bar construction, the discipline of point-in-time correctness, cleaning the common defects, quantifying survivorship bias, and storage and ingestion patterns that keep a research stack honest.

Data types and choosing granularity

• OHLCV bars — Open/High/Low/Close/Volume over an interval. Compact and ubiquitous,

but time bars sample fixed clock intervals regardless of activity.

• Tick data — every trade and/or quote, microsecond-stamped. Essential for

microstructure and execution research; large.

• L1/L2 order book — best bid/ask or full depth, for realistic execution and

market making modeling.

• Fundamentals — earnings, balance sheets, estimates; basis of

factor investing.

• Reference data — symbols, corporate actions, exchange calendars, identifier maps.
• Alternative data — see alternative data trading.

Match granularity to holding period: a daily-rebalanced model does not need ticks, while an execution algorithm cannot be validated without them.

Beyond time bars

Time bars over-sample quiet periods and under-sample bursts, producing returns with poor statistical properties (heavy autocorrelation in variance, non-normality). Volume bars and dollar bars — sampling a new bar every fixed quantity of volume or notional — yield returns far closer to i.i.d. and are better inputs for ML features.

import numpy as np, pandas as pd

def dollar_bars(ticks: pd.DataFrame, threshold: float) -> pd.DataFrame:
    """Aggregate trade ticks into bars of ~constant traded notional."""
    ticks = ticks.sort_values("ts")
    dollar = (ticks["price"] * ticks["size"]).to_numpy()
    group = (dollar.cumsum() // threshold).astype(int)
    g = ticks.assign(grp=group).groupby("grp")
    bars = g.agg(ts=("ts", "last"), open=("price", "first"),
                 high=("price", "max"), low=("price", "min"),
                 close=("price", "last"), volume=("size", "sum"))
    return bars.set_index("ts")

Point-in-time correctness and look-ahead

The most underappreciated concept in quant data is point-in-time (PIT): the dataset must reflect only what was known at each historical moment, with no future leaking backward. Two classic violations:

• Restated fundamentals. Companies restate earnings months later. A non-PIT store

overwrites the original print with the corrected one, so the backtest "knows" the restated figure before publication — look-ahead that inflates any fundamental strategy.

• Index membership. Backtesting on today's index constituents over 20 years trades

only the names that survived and thrived enough to be in the index now — severe survivorship bias.

A PIT store records, for every datum, when it became known. If a vendor cannot supply the as-of date of a fundamental, treat any fundamental backtest with deep suspicion. This ties directly to backtesting biases.

Cleaning market data

Splits and dividends

A 2-for-1 split halves the price with no economic change; unadjusted, your strategy sees a -50% return that never happened. Keep raw prices plus a corporate-actions table (more transparent and PIT-friendly) or use back-adjusted series for return calculation. Mind total-return vs. price-return: price-return data understates dividend-heavy strategies.

import pandas as pd

def back_adjust(prices: pd.Series, split_ratio: pd.Series) -> pd.Series:
    """split_ratio: 2.0 on a 2:1 ex-date, else 1.0. Produces a continuous series."""
    factor = split_ratio[::-1].cumprod()[::-1].shift(-1).fillna(1.0)
    return prices / factor

Bad ticks and outliers

Feeds carry erroneous prints: a 0.01 or 10,000 trade on a 100 stock, decimal errors, fat-fingers. A robust filter flags prints far from a rolling median (median absolute deviation is more outlier-resistant than mean/std):

import numpy as np, pandas as pd

def flag_bad_ticks(prices: pd.Series, window=50, k=8.0) -> pd.Series:
    med = prices.rolling(window, center=True, min_periods=5).median()
    mad = (prices - med).abs().rolling(window, center=True, min_periods=5).median()
    robust_z = 0.6745 * (prices - med).abs() / (mad + 1e-12)
    return robust_z > k     # True where the print is implausibly far from local median

A real flash crash also looks like a bad tick — distinguish single-instrument data errors from genuine extreme moves corroborated across the market before deleting anything.

Gaps, missing data, and timestamps

Decide deliberately: forward-fill low-frequency series where a gap means "no change"; drop rather than interpolate for return calculations so you do not invent trades on closed days; align to an exchange calendar so a holiday is not treated as zero-volume. Never linearly interpolate prices through a gap and then compute returns — that fabricates smooth moves. Standardize all timestamps to UTC, record the exchange, and state whether a bar is stamped at its open or close; mixing timezones is a quiet source of look-ahead.

Quantifying survivorship bias

Survivorship bias is the most pervasive trap: a universe of only currently existing securities silently excludes every bankruptcy, delisting, and failed acquisition. The inflation is not small.

Backtest universe	What it omits	Typical effect
Current index members	Past members that dropped out	Overstated returns, understated drawdowns
Surviving tickers only	Delisted/bankrupt names	Phantom edge on "buy the dip" strategies
Live exchanges only	Defunct exchanges, halted pairs (crypto)	Inflated crypto backtests

To avoid it, you need delisted securities with full history to their delisting date, plus historical index membership at each point in time. This is exactly what separates a serious (usually paid) provider from a free one.

Reconciliation and data-quality monitoring

Cleaning is not a one-time batch job; it is a continuous control problem. Feeds degrade, vendors change methodology, and a corporate action handled wrong on one symbol can poison a whole backtest. Two practices catch most of this. First, reconcile two independent sources for anything you trade — compute the daily return from each and alert on any divergence beyond a tolerance, because disagreement almost always reveals a real defect (a missed split, a stale quote, a timezone slip). Second, run automated data-quality checks on every ingest and track their pass rate over time, so a slow rot in feed quality surfaces before it reaches your model.

import numpy as np, pandas as pd

def quality_report(df: pd.DataFrame) -> dict:
    """Cheap sanity checks to run on every ingest; alert on regressions."""
    ret = np.log(df["close"]).diff()
    return {
        "dup_timestamps": int(df.index.duplicated().sum()),
        "non_monotonic": int((df.index.to_series().diff() < pd.Timedelta(0)).sum()),
        "neg_or_zero_px": int((df["close"] <= 0).sum()),
        "zero_volume": int((df["volume"] == 0).sum()),
        "extreme_moves": int((ret.abs() > 0.5).sum()),     # >50% single-bar moves
        "max_gap_days": float(df.index.to_series().diff().dt.days.max()),
    }

The cost of these checks is trivial; the cost of discovering a data bug after it has shaped a live allocation is not.

Free vs. paid sources

Free data is fine for learning and prototyping but commonly carries gaps, silent adjustments, no PIT guarantees, and survivorship bias. For crypto, exchange APIs via CCXT give generous, high-quality history. Paid vendors provide cleaned, adjusted, point-in-time equity/futures/options/fundamental data with documented methodology, and tick specialists sell full order-book history. The rule of thumb: the cost of data is trivial next to the cost of trading a strategy built on bad data — if quality determines whether the edge is real, pay for the good version.

Storage and ingestion

Format	Best for	Notes
CSV	Small datasets, sharing	Slow, untyped — prototyping only
Parquet	Columnar analytics	Compressed, typed, fast partial reads — the default
HDF5	Large numeric arrays	Fast but fiddly tooling
SQLite / Postgres	Reference data, corporate actions	Great for relational metadata
Time-series DB	Tick workloads	Built for high-volume ingestion

For most research, Parquet partitioned by symbol and date is the sweet spot: columnar compression keeps storage small, predicate pushdown loads only the slice you query, and typed columns prevent the silent string-vs-float corruption that plagues CSV. Tick-level workloads, where a single liquid name can produce hundreds of millions of rows per year, are the exception — there a purpose-built time-series database earns its complexity.

import pandas as pd

df.to_parquet("data/equities", partition_cols=["symbol", "year"])
aapl = pd.read_parquet("data/equities", filters=[("symbol", "==", "AAPL")])

Ingestion habits that prevent silent corruption:

• Idempotent pipelines — key records by (symbol, timestamp) and upsert; re-running

never duplicates.

• Validate on ingest — reject duplicate timestamps, non-monotonic time, negative

prices, and zero volume before data enters the store.

• Immutable raw layer — keep the original download untouched and derive cleaned and

adjusted versions from it, so you can re-clean when you find a bug.

• Log corporate actions explicitly rather than relying solely on pre-adjusted prices,

so PIT views are reconstructable.

• Version caches so a cleaning-logic change invalidates stale derived data.

These feed directly into a trustworthy backtest in Python.

The mistakes that recur

• Trusting free data blindly — verify silent adjustments and gaps against a second

source.

• Ignoring corporate actions — unadjusted splits create fake returns that dominate.
• Current index membership used historically — bakes in survivorship bias.
• Interpolating through gaps — fabricates trades and smooths volatility.
• Mixing timezones — subtle look-ahead.
• Treating restated fundamentals as PIT — restatement look-ahead inflates fundamental

strategies.

Conclusion

Market data is the foundation everything rests on, and the discipline of sourcing, cleaning, and validating it separates backtests that survive live contact from those that do not. Match granularity to your horizon (and prefer volume or dollar bars for ML), insist on point-in-time correctness for anything fundamental, ruthlessly handle splits, dividends, bad ticks, and gaps, and quantify survivorship bias rather than assuming it away. Store in fast columnar formats, build idempotent and validated ingestion, and keep raw and derived layers cleanly separated. Do this well and your backtests tell the truth; skip it and even the best model will lie. The same principles carry over to alternative data and crypto exchange APIs.