Order Book Reconstruction from Tick Data

Order-book reconstruction turns an exchange’s event stream into the state that a trader could have observed at each instant. It is the foundation for queue-aware execution research, short-horizon alpha, and credible transaction-cost simulation. A bar database cannot answer whether a bid was cancelled before a marketable sell arrived. For that, the backtest needs the book.

The relevant raw inputs are typically add, cancel, delete, replace, trade, and trading status messages. Their meaning varies by feed: some publish order IDs and full depth, while others provide only price-level deltas. Start with the feed specification, rather than inferring semantics from sample data. The storage and clock choices described in tick-data architecture determine whether this replay remains reproducible at scale.

Define the state and event contract

At event \(t\), maintain an order map and aggregated bid/ask ladders. An order-ID feed lets the book retain time priority; an aggregated feed does not. This distinction is material for passive fills.

Event	Order-ID book action	Level book action	Invariant
Add	Insert ID, price, size	Increase level size	Size is positive
Cancel	Reduce/remove ID	Decrease level size	Never below zero
Replace	Delete then add	Remove then add	Preserve feed ordering
Execute	Reduce resting order	Reduce displayed level	Match known liquidity
Trade print	Often informational	Often informational	Do not double count

Do not assume every trade print consumes displayed liquidity. Feeds can report an execution and a separate consolidated trade message; dark or hidden executions may have no displayed predecessor. Treat the exchange sequence number as authoritative, then retain receive time as a separate field for latency research.

Replay in sequence, not timestamp order

Nanosecond timestamps are not an ordering guarantee: multiple messages may share a timestamp, clocks can be corrected, and vendor normalization can reorder packets. Replay by (tradingdate, venue, channel, sequencenumber). Detect gaps before applying later messages; a locally consistent book after a gap is still economically wrong.

from collections import defaultdict

class Book:
    def __init__(self):
        self.orders = {}
        self.levels = {"B": defaultdict(int), "S": defaultdict(int)}

    def add(self, order_id, side, price, qty):
        self.orders[order_id] = [side, price, qty]
        self.levels[side][price] += qty

    def reduce(self, order_id, qty):
        side, price, open_qty = self.orders[order_id]
        removed = min(qty, open_qty)
        self.levels[side][price] -= removed
        if removed == open_qty:
            del self.orders[order_id]
        else:
            self.orders[order_id][2] -= removed

Production code additionally rejects duplicate adds, unknown cancels, crossed books outside special states, and invalid price increments. Log each violation with the raw message; silently clipping negatives creates a cleaner but fictional dataset.

Snapshots, recovery, and corporate actions

Most feeds provide periodic snapshots. Use one to seed a session, replay incrementals after its sequence number, and compare subsequent snapshots against the locally reconstructed state. On a sequence gap, discard the affected channel until a valid snapshot arrives. “Repairing” by assuming the missing messages were harmless biases depth and queue estimates exactly when markets are stressed.

Normalize prices to integer ticks, quantities to exchange lots, and symbols to a point-in-time security master. Split adjustments, symbol changes, halts, and auction states need explicit events. A continuous-book model should not manufacture depth during an opening or closing cross; see auction mechanics.

Validation is an empirical process

Validate at both message and market levels:

Test	Expected result	Failure usually means
Sequence continuity	No unexplained gaps	Packet loss or bad partition
Best bid < best ask	Except locked/cross rules	Event ordering error
Snapshot reconciliation	Exact depth match	Incorrect event semantics
Executed size	Does not exceed visible queue, where applicable	Double-counted prints
Top-of-book quote count	Matches vendor reference	Wrong session/status handling

For each instrument-day, store gap counts, reconciliation differences, and the fraction of events rejected. These are data-quality features, not merely pipeline telemetry: abnormal rates cluster around halts, upgrades, and volatile periods.

From book state to research features

Once replay is deterministic, derive microprice, depth-weighted imbalance, cancellation intensity, and order-flow imbalance. Keep feature timestamps causal: a feature at event \(t\) may use state after the event only if the strategy could receive that event before submitting its order. The distinction matters for market microstructure signals and for estimating adverse selection.

Persist compact snapshots only when needed, but retain immutable normalized events. Snapshots accelerate queries; events are the audit trail that lets a researcher repair a bug and reproduce a historical decision.

Key takeaways

• Reconstruct from documented message semantics and sequence numbers, not timestamps alone.
• Separate executions from informational trade prints to avoid consuming depth twice.
• A sequence gap requires snapshot recovery; plausible-looking interpolation is not valid.
• Validate against exchange snapshots and retain quality metrics per instrument-day.
• Queue-sensitive research requires order-ID state, causal clocks, and immutable events.