Live Trading: Deployment, Monitoring and Ops

The gap between a working backtest and a robust live system is where most retail quant projects quietly die. A profitable strategy that crashes at 3 a.m., double-submits an order after a reconnect, or silently stops trading is not a profitable strategy — it is a liability. Going live is an operations problem as much as a trading one: infrastructure, reconciliation, monitoring, and failure handling. This article is a practical field guide to deploying a strategy from paper trading into production and keeping it alive, drawing on hard-won devops discipline.

From paper to live

Paper trading validates that your trading bot reacts to live data and produces sensible orders, but it hides the costs that only appear when real money hits the book: slippage, partial fills, queue position, and rejected orders. Make the transition gradually:

• Shadow mode. Run the live system end to end but route orders to a paper endpoint, comparing intended vs simulated fills.
• Minimum size. Go live at the smallest tradeable size to surface real fill behavior with negligible risk.
• Scale on evidence. Increase size only once live fills match expectations and the ops layer has proven stable for weeks.

Track the divergence between expected (backtested) and realized PnL obsessively. A persistent gap usually means an unmodeled cost or a subtle bug in the bot that paper trading let slide.

Infrastructure choices

You do not need a colocated rack for most strategies, but you do need reliability.

• Compute. A small cloud VPS near your broker/exchange's region minimizes latency and avoids your home internet as a single point of failure. For non-HFT strategies, geographic proximity matters less than uptime.
• Scheduling. Use a supervised process manager (systemd, supervisor, or a container orchestrator) that restarts the bot on crash — but only restart safely (see reconciliation below). A naive auto-restart that re-enters positions is dangerous.
• Time synchronization. Sync the clock with NTP. Many exchange APIs (including those reached via ccxt) reject requests whose timestamp drifts beyond a tolerance, and inconsistent clocks corrupt your event logs.
• Secrets. Keep API keys out of source control; use environment variables or a secrets manager, and create keys with the minimum permissions (trade but not withdraw).

Concern	Minimum viable choice	Why it matters
Hosting	Cloud VPS in broker region	Uptime + lower latency
Process mgmt	systemd / supervisor with safe restart	Recovers from crashes
Time	NTP-synced clock	Avoids rejected/duplicated orders
Secrets	Env vars / secrets manager, scoped keys	Limits blast radius

State and order reconciliation

The single most important production discipline: on startup and periodically, reconcile your internal state against the broker's truth. Your bot's idea of positions and open orders can diverge from reality after a crash, a missed message, or a network blip. Never assume; always query.

def reconcile(broker, local_state):
    """Make local state match the broker before trading resumes."""
    broker_positions = broker.fetch_positions()
    broker_orders = broker.fetch_open_orders()

    # 1. Adopt broker positions as ground truth
    local_state.positions = {p['symbol']: p['qty'] for p in broker_positions}

    # 2. Cancel or adopt orders the bot does not recognize
    known = set(local_state.open_order_ids)
    for o in broker_orders:
        if o['id'] not in known:
            broker.cancel_order(o['id'])     # orphan -> cancel (conservative)

    # 3. Flag any drift for a human
    if local_state.drift_detected():
        local_state.halt("position mismatch on reconcile")
    return local_state

Use idempotent client order IDs so that retrying a submission after an ambiguous timeout cannot create a duplicate order — the exchange rejects the second use of the same ID. This is the antidote to the classic "did my order go through?" double-submit bug.

Monitoring and alerting

If you cannot see it, you cannot trust it. Monitor at three layers:

• System health — process alive, CPU/memory, disk, network, websocket connected.
• Market data — heartbeat freshness; alert if no tick/bar arrives within an expected window (a stalled feed is silent and deadly).
• Trading health — realized vs expected PnL, fill rates, rejected orders, current exposure vs limits, and drawdown against your risk budget.

Push alerts to a channel you actually watch (SMS/phone for critical, chat for warnings). The cardinal rule: alert on the absence of expected events, not just on errors. A bot that stops trading throws no exception — you find out from a missing heartbeat.

Kill switches and risk limits

Hard, automated limits are non-negotiable. Encode them so the system disables itself before a bug or a market dislocation does real damage:

• Max daily loss — halt and flatten if cumulative loss breaches a threshold.
• Max position / exposure — reject orders that would exceed size limits.
• Order rate limit — cap orders per minute to catch runaway loops.
• Sanity checks — refuse orders priced far from the last trade (fat-finger / stale-data guard).
• Manual kill switch — a one-command flatten-and-stop that any operator can trigger.

These limits are the operational expression of position sizing and risk management: the strategy decides what it wants to do, the risk layer decides what it is allowed to do.

Logging, disconnects, and partial fills

• Structured logging. Log every decision, order, fill, and state change with timestamps and a correlation ID. When something goes wrong at 3 a.m., the log is your only witness — make it greppable and complete.
• Disconnects. Websocket feeds drop routinely. On reconnect, re-subscribe, re-fetch state, and reconcile before resuming — never blindly continue from stale in-memory state.
• Partial fills. Real orders fill in pieces. Track filled vs remaining quantity per order, and decide a policy: re-quote the remainder, cancel-replace, or wait. Backtests that assume instant full fills mislead here.
• Postmortems. After every incident, write a blameless postmortem: timeline, root cause, and a concrete fix. Treat each near-miss as a free lesson that prevents the expensive version later.

Common mistakes

• Trusting in-memory state after a restart. Always reconcile against the broker before trading.
• Non-idempotent order submission. Retries without a client order ID create duplicate fills.
• Alerting only on errors. A stalled feed or stopped bot is silent; monitor for missing heartbeats.
• No kill switch. Without automated loss and rate limits, a single bug can drain the account.
• Skipping shadow/min-size rollout. Jumping straight to full size turns ordinary teething bugs into large losses.
• Sparse logging. You cannot debug an overnight incident you did not record.

Key takeaways

• Treat going live as an operations problem: infrastructure, reconciliation, monitoring, and failure handling decide success.
• Roll out gradually — shadow mode, then minimum size — and track expected-vs-realized PnL relentlessly.
• Reconcile state against the broker on startup and reconnect, and use idempotent client order IDs to prevent duplicates.
• Monitor system, data, and trading health; alert on missing events, not just errors.
• Enforce kill switches and hard risk limits, log everything, handle partial fills explicitly, and write postmortems. Start from a solid trading bot and graduate it through paper trading.