Migrating Off Fragmented Market Data Vendors: Dual-Run, Field Mapping, Cutover

Market data vendor migration is rarely a hard technical problem; it is a reallocation of cost and risk. Once a product spans equities, FX, crypto, and commodities, the ingestion layer is usually stitched from three or four vendors — each with its own auth, symbols, field semantics, and throttling. Consolidating onto one unified API narrows the operational surface, but you maintain two feeds while you migrate. This is an executable path: inventory the call surface, normalise field semantics, dual-run the diff, cut over consumer by consumer, and verify against a checklist.

The cost is in the dual-run window, not the code

Most teams estimate only engineering hours. Real cost has three layers, rising in magnitude.

Window length drives magnitude. Size it to one full business cycle: at least one month-end close, one large move, one upstream incident. A quiet trading day proves nothing — the failure paths are what expose mapping defects.

Quota during the dual run is easy to miss. Shadow traffic is real usage: per the docs, REST quote and kline calls plus the WebSocket handshake count toward daily quota and per-minute limits, while pushed ticks are not re-counted. So shadow-compare over WebSocket subscriptions rather than polling the same symbols. Size it with the quota calculator before fixing the window.

Inventory the call surface before writing an adapter

The common failure is writing the new client first and discovering gaps as you go. Instead, produce an inventory with four columns: endpoint, symbol, field, consumer.

The fourth column carries the decision. Grouping by endpoint systematically understates the work, because one endpoint can be read under conflicting assumptions: a terminal treats last as a live trade, a risk service assumes bid and ask always exist, a report assumes volume accumulates across sources. Group by consumer and staged cutover becomes possible.

The inventory also surfaces hidden dependencies: hard-coded field names, timezone conversion done downstream, silent acceptance of delayed data. None of that is in the API docs — it is only in the code. The site's multi-market integration guide covers building on a unified API from scratch; this covers moving an existing pipeline onto one.

Field mapping is semantics, not renaming

Renaming is one line. Normalising semantics needs a table plus contract tests.

Two rules. Missing is not zero: 0 is a value, null is the absence of one, and backfilling with 0 makes a downstream service read a trading halt as a crash. And convert semantics once, in the adapter layer — every extra downstream conversion is a long-term drift point.

The unified quote schema is a ready-made target: envelope succ / data / msg / code, with data carrying code / symbol / market / category, last / bid / ask / volume, change / changePercent / high / low, timestampMs, and the delayed and unavailable booleans. Align the adapter to it and downstream barely changes. One call takes up to 50 codes, so FX and commodity codes such as EURUSD and XAUUSD batch together; see FX market coverage.

Dual run: shadow traffic and tolerance-aware diffing

Dual run means requesting the same symbols from both feeds and comparing field by field. The comparison must be tolerance-aware or the output is pure noise.

Three tolerances are all required: relative deviation for prices (a few basis points), a lag ceiling for timestamps (seconds), and a magnitude band for volume (percent). Treat "value on one side, empty on the other" as a mapping defect, not volatility — count it separately.

# parity_check.py — compare two vendor snapshots field by field
# Prices move between calls, so exact equality is the wrong test.
TOL = {"last": 0.005, "bid": 0.005, "ask": 0.005, "volume": 0.05}
MAX_LAG_MS = 5_000
FIELDS = ("last", "bid", "ask", "volume", "timestampMs")


def _rel(a, b):
    return abs(a - b) / (max(abs(a), abs(b)) or 1.0)


def compare(code, old, new):
    out = []
    if bool(old.get("delayed")) != bool(new.get("delayed")):
        out.append((code, "delayed", "flag", old.get("delayed"), new.get("delayed")))
    for f in FIELDS:
        a, b = old.get(f), new.get(f)
        if f == "timestampMs":
            if a and b and abs(a - b) > MAX_LAG_MS:
                out.append((code, f, "lag", a, b))
            continue
        if a is None or b is None:
            if a != b:  # one side missing is a mapping bug, not volatility
                out.append((code, f, "missing", a, b))
            continue
        if _rel(a, b) > TOL.get(f, 0.005):
            out.append((code, f, "value", a, b))
    return out


def report(pairs):
    diffs = [d for code, old, new in pairs for d in compare(code, old, new)]
    for code, field, kind, a, b in diffs:
        print(f"{code:<10} {field:<12} {kind:<8} {a} -> {b}")
    return diffs


if __name__ == "__main__":
    report([
        ("EURUSD", {"last": 1.0850, "bid": 1.0849, "ask": 1.0851, "volume": 1200,
                    "timestampMs": 1_757_000_000_000, "delayed": False},
                   {"last": 1.0851, "bid": 1.0850, "ask": 1.0852, "volume": 1210,
                    "timestampMs": 1_757_000_002_000, "delayed": False}),
        ("XAUUSD", {"last": 2410.5, "bid": None, "ask": None, "volume": 300,
                    "timestampMs": 1_757_000_000_000, "delayed": True},
                   {"last": 2410.9, "bid": 2410.4, "ask": 2410.6, "volume": 302,
                    "timestampMs": 1_757_000_000_500, "delayed": False}),
    ])

Aggregate by symbol and field rather than printing a stream. A one-off deviation on one symbol is normal; the same field drifting across many symbols almost always means the mapping rule is wrong. Track the share of each diff class and use it as a gate: no cutover until the rate falls under your threshold.

Cut over by consumer, not by endpoint

Wrong granularity turns controlled risk into uncontrolled risk. Cutting by endpoint changes every consumer at once and leaves you unable to name the blast radius. Cutting by consumer lets you move one non-critical consumer entirely, watch a full business cycle, then continue.

Routing must be deterministic: one consumer and symbol stays on one side for the whole window, or a single dashboard mixes two vendors and becomes harder to debug than a data error. Hash-bucketing consumer plus symbol beats a random draw.

# cutover.py — shift consumers, not endpoints
import hashlib

CONSUMERS = {
    "web-terminal": {"pct": 100, "fallback": "legacy"},
    "risk-monitor": {"pct": 25, "fallback": "legacy"},
    "batch-backfill": {"pct": 0, "fallback": "legacy"},
}


def _bucket(consumer, symbol):
    # Deterministic: one consumer and symbol must stay on one side
    # for the whole window, or a dashboard mixes two vendors.
    key = f"{consumer}:{symbol}".encode()
    return int(hashlib.sha256(key).hexdigest()[:8], 16) % 100


def route(consumer, symbol):
    rule = CONSUMERS.get(consumer)
    if rule is None:
        return "legacy"  # an unknown consumer never gets the new feed
    return "unified" if _bucket(consumer, symbol) < rule["pct"] else rule["fallback"]


def rollback(consumer):
    """One line puts a consumer back on the old feed, no redeploy."""
    CONSUMERS[consumer]["pct"] = 0


if __name__ == "__main__":
    for c in CONSUMERS:
        hits = sum(route(c, f"SYM{i}") == "unified" for i in range(200))
        print(f"{c:<16} pct={CONSUMERS[c]['pct']:>3} routed_new={hits}/200")
    rollback("risk-monitor")
    print("after rollback:", route("risk-monitor", "SYM7"))

Rollback must be a config change, not a redeploy. rollback() above zeroes a consumer's share and the next call returns to the old feed. Keep the old feed and key valid throughout; do not revoke before the observation period ends. Re-derive limits too: alert on 401, 403, and 429 separately and per consumer — see the rate limits and error codes glossary — and persist X-Request-Id for reconciliation.

Acceptance checklist

Migration is done when you can show evidence for each item below, not when the new feed returns data.

1. Every consumer and endpoint in the inventory has a dual-run comparison record under threshold
2. The mapping table covers every field, with explicit rules for missing versus zero
3. All timestamps are millisecond integers, spot-checked on at least one cross-timezone market
4. delayed and unavailable are genuinely read downstream, not ignored
5. Limits and quota are re-derived for the new feed, shadow traffic included
6. 401, 403, and 429 alert per consumer, with X-Request-Id persisted
7. Rollback was rehearsed for real and needs no redeploy
8. The old feed and key stay live until the observation period closes

FAQ

How long should the dual-run window last?

Measure it in business cycles, not days. Cover at least one month-end close, one significant move, and one upstream incident. A window without failure samples cannot validate degradation or alerting.

Does shadow traffic consume quota?

Yes. REST quote and kline calls and the WebSocket handshake count toward daily quota and RPM; pushed ticks are not metered per message. Shadow-compare by subscribing over WebSocket instead of polling, which is cheaper and closer to production shape.

Do downstream services need code changes?

The goal is none. That holds if the adapter normalises semantics first: names, units, timestamp precision, and missing-value rules aligned before traffic moves. If downstream must change, the mapping table has a gap — close it before cutting over.

Can you cut over every consumer at once?

Not advisable. Staged cutover preserves a control group: if the new feed misbehaves, unconverted consumers keep working, business continuity holds, and comparison data keeps flowing. A single big-bang switch throws that away.

The order is fixed: inventory first, then field mapping, then the dual-run window, then consumer-by-consumer cutover. The first two cost effort rather than time, and they set the risk ceiling for the last two.

See the quota, RPM, and SLA on each plan →

Endpoint paths, field definitions, error codes, and plan tiers come from the official tiqex.io documentation (updated 2026-09-12) and pricing page, accessed 2026-09-24. Sample code illustrates engineering practice only; nothing here is investment advice.