Tick Data Architecture for Quant Research

Tick data is the atomic record of market activity — every trade, quote change, and order book update with microsecond timestamps. Microstructure signals (order flow, lead-lag, market making) require tick resolution; bar data smears the information away. But tick datasets are terabytes, messy, and full of exchange-specific quirks. This article covers storage formats, normalization, replay infrastructure, and the design choices that separate research you can trust from backtests that hallucinate fills.

What tick data contains

Event type	Fields (typical)	Use case
Trade	price, size, side, timestamp, trade_id	Volume, aggressor flow, VWAP
Quote (BBO)	bid, ask, bidsize, asksize, timestamp	Spread, mid, quote imbalance
Order book (L2)	price levels, sizes per side, timestamp	Depth, OFI
Order book (L3)	individual orders, queue position	Queue modeling, HFT
Auction	indicative price, imbalance	Open/close auction strategies

Crypto adds: funding ticks, liquidation events, on-chain settlement timestamps. Futures add: open interest changes, settlement prices.

Storage formats

Columnar (Parquet / Arrow)

Best for batch research — scan millions of rows, filter by symbol/date, compute features.

/data/BTCUSDT/trades/year=2025/month=06/day=15/*.parquet

Partition by symbol and date. ZSTD compression typically achieves 5-10x on tick data. Use with DuckDB or Polars for SQL-on-files without a database server.

Time-series databases (QuestDB, TimescaleDB, kdb+)

Best for time-range queries and live ingestion:

QuestDB: fast SQL, good for mid-size shops, open source
TimescaleDB: PostgreSQL extension, familiar SQL, hypertables
kdb+: industry standard at HFT firms, steep learning curve, expensive

Choose based on query pattern: random access by time range → TSDB; bulk feature computation → Parquet.

Binary replay formats (custom)

For backtest replay at full tick speed, convert to a compact binary layout:

struct Trade { int64_t ts_ns; double price; double qty; int8_t side; }

Memory-mapped files let the backtest engine read sequentially without deserialization overhead. Event-driven backtesting engines often use this internally.

Normalization: the hard part

Raw tick data from exchanges is not research-ready:

Problem	Example	Fix
Timestamp ambiguity	Exchange time vs receive time	Store both; research on exchange time, latency on difference
Sequence gaps	Missing packets overnight	Flag gaps; do not forward-fill trades
Duplicate events	Reconnect replays	Dedup on trade_id
Price scaling	Contract multipliers	Normalize to dollars per unit
Corporate actions	Stock splits	Adjust historical prices (trades) or restart chain (crypto: N/A)
Session boundaries	RTH vs ETH equities	Tag session; filter consistently
Crypto 24/7	No close	Define artificial day boundaries for daily features

import pandas as pd

def normalize_trades(raw: pd.DataFrame) -> pd.DataFrame:
    """Minimal trade normalization pipeline."""
    df = raw.drop_duplicates(subset=['trade_id']).copy()
    df['ts'] = pd.to_datetime(df['exchange_ts'], utc=True)
    df = df.sort_values('ts')
    # Flag gaps > 60s (adjust per asset)
    df['gap'] = df['ts'].diff().dt.total_seconds() > 60
    df['side'] = df['side'].map({'buy': 1, 'sell': -1, 'BUY': 1, 'SELL': -1})
    return df[['ts', 'price', 'qty', 'side', 'gap']]

See market data cleaning for vendor-specific pitfalls.

Building a replay engine

A tick replay engine feeds events to a strategy in chronological order across symbols:

events = merge_sorted(trade_streams + quote_streams)
for event in events:
    book.update(event)
    signal = strategy.on_event(event, book)
    if signal:
        sim.execute(signal, book)

Requirements:

Multi-symbol merge — heap-based k-way merge on timestamp
Book state — maintain L2 from incremental updates (add, modify, delete)
Latency model — optional delay between signal and fill
Fill model — conservative: fill at ask for buys, bid for sells, plus queue position estimate

Compare to vectorized backtesting — vectorized is 100x faster but cannot model queue position or adverse selection.

Feature computation at tick scale

Common microstructure features from tick data:

def ofi_from_bbo(bbo: pd.DataFrame) -> pd.Series:
    """Order flow imbalance from BBO changes (Cont-Kukanov-Stoikov)."""
    db = bbo['bid'].diff()
    da = bbo['ask'].diff()
    ofi = pd.Series(0.0, index=bbo.index)
    ofi[db > 0] += bbo.loc[db > 0, 'bid_size']
    ofi[db < 0] -= bbo.loc[db < 0, 'bid_size'].shift(1)
    ofi[da < 0] -= bbo.loc[da < 0, 'ask_size']
    ofi[da > 0] += bbo.loc[da > 0, 'ask_size'].shift(1)
    return ofi

def realized_vol_from_trades(trades: pd.DataFrame, window='5min') -> pd.Series:
    """Realized variance from trade prices."""
    trades = trades.set_index('ts')
    log_p = np.log(trades['price'])
    return log_p.diff().pow(2).resample(window).sum()

Aggregate tick features to decision frequency (1s, 1min, 5min) before combining with daily signals — different decay profiles.

Data volume and cost planning

Rough sizing (single symbol, one year):

Asset	Trades/day	Raw size/day	Parquet/day
BTCUSDT (Binance)	2-5M	200-500 MB	30-80 MB
ES futures	500K-2M	50-200 MB	10-40 MB
SPY	200K-500K	20-50 MB	5-15 MB
Small-cap equity	1K-10K	<1 MB	<0.5 MB

100 symbols × 5 years × 50 MB/day ≈ 9 TB compressed. Budget storage, backup, and compute accordingly. Cloud object storage (S3/GCS) + local SSD cache for hot windows is the standard pattern.

Vendor vs exchange-native

Source	Pros	Cons
Exchange API (websocket)	Free, real-time	Gaps, no history, rate limits
Exchange historical API	Free/cheap	Incomplete, inconsistent format
Databento, Polygon, TickData	Clean, normalized	$$$
Crypto: Tardis, Amberdata	Good crypto coverage	Cost scales with history depth

For crypto, CCXT handles live feeds; historical tick usually requires a dedicated vendor or self-archived websocket recordings.

Research hygiene

Never backtest on mid-price fills for tick strategies — use bid/ask
Store raw + normalized — reprocess when you find a bug
Version your normalization — git-tag the pipeline that produced each dataset
Test on one day manually — replay a known event (FOMC, liquidation cascade) and

verify book state by eye

Separate research and production feeds — production may have lower latency but

same schema; test that research features match live

Key takeaways

Tick data enables microstructure alpha; bar data cannot replicate it
Parquet for batch research, TSDB for time-range queries, binary for fast replay
Normalization (timestamps, dedup, sessions) is where most bugs hide
Event-driven replay with conservative fill models is mandatory for honest backtests
Plan terabytes upfront — 100 symbols × years adds up fast