Present but Already Wrong: Six Silent Market Data Failures and How to Detect Them

Market data quality validation is rarely about whether data arrives; it is about whether the arriving data is correct. A broken connection raises an error. A wrong value does not: a price spike, a timestamp regression, or a crossed book passes silently through collection, storage, and consumption, surfacing later inside a false alert or a polluted backtest. This article defines six silent failure classes on documented TIQEX fields, with a dependency-free Python validator you can run today.

A failure that never alerts

Consider a pipeline already in production. The WebSocket is connected, the type=subscribed acknowledgement arrived, tick frames keep coming, and connection count, frame rate, and HTTP status are all green. The logs say otherwise: one symbol's tick.at stopped advancing twelve minutes ago. The upstream feed died; the client now receives repeats of the same stale value.

The pattern is always the same: transport is healthy while content is already wrong, and most teams monitor only transport. Even an HTTP 2xx response can carry a business failure — TIQEX's error documentation states that the envelope's succ field may be false on a 2xx response, so both must be checked. Treating HTTP 200 as a pass delegates content-layer decisions downstream.

Why silent failures cost more than outages

An outage is explicit: it triggers retries and alerts, and its cost is visible and bounded. A silent failure triggers nothing and compounds. The cost has three parts:

1. False-signal cost. A spike reaches the alerting system and triggers a manual review. What erodes is signal-to-noise ratio: as false positives accumulate, real signals are missed more often.
2. Recomputation cost. Once a bad value enters a backtest dataset or report, the fix is re-running the pipeline and re-issuing conclusions.
3. Trust cost. The most expensive part. Teams start discounting all data and fall back to manual judgement.

As an order-of-magnitude reference, assume a false alert costs fifteen minutes of review and a backtest re-run four hours — illustrative parameters, not measured figures. The important asymmetry is that detection is nearly free: all six assertions run locally against data that already arrived, issuing no API calls and consuming no quota. Trading zero marginal cost for downstream credibility is the core reason to validate at the collection edge.

Six silent failure classes

These six cover most "present but already wrong" cases, and each is detectable with a cheap assertion.

Three notes. Class 1 depends on session context: crypto trades continuously (see the crypto market page), while equities and futures have closed windows where the same threshold produces heavy false positives — stale thresholds must be layered by session. Class 3 uses median absolute deviation rather than standard deviation, because the standard deviation is itself contaminated by the spike. Class 4 depends on field semantics documented under OHLCV candlestick fields, where mixing adjustment conventions is the most common false-positive source.

A validator you can run

The implementation below collects all six rules in one class and depends only on the standard library. Input is a TIQEX WebSocket tick frame; output is a list of violation reasons, where an empty list means the tick passed. Frame formats are in the WebSocket live quotes guide.

# quality.py — 行情数据质量校验：六条廉价断言，仅依赖标准库
from collections import deque
from datetime import datetime, timezone


class QualityGate:
    """median_window: 尖峰检测窗口长度；spike_k: MAD 倍数阈值；stale_after: 陈旧阈值（秒）。"""

    def __init__(self, median_window=200, spike_k=8.0, stale_after=60.0):
        self.prices = deque(maxlen=median_window)
        self.spike_k = spike_k
        self.stale_after = stale_after
        self.last_at = {}
        self.last_vol = {}

    def check(self, tick, now=None):
        now = now or datetime.now(timezone.utc)
        sym = tick["symbol"]
        out = []
        at = datetime.fromisoformat(tick["at"].replace("Z", "+00:00"))

        # 1 陈旧值：帧仍在到达，时间戳不再前进
        lag = (now - at).total_seconds()
        if lag > self.stale_after:
            out.append("stale %s lag=%.0fs" % (sym, lag))

        # 2 时间戳回退：同一标的后一条早于前一条
        prev = self.last_at.get(sym)
        if prev is not None and at < prev:
            out.append("time_regression %s" % sym)
        self.last_at[sym] = at if prev is None else max(at, prev)

        last = tick.get("last")

        # 3 价格尖峰：相对滚动中位数的 MAD 偏离
        if isinstance(last, (int, float)):
            if len(self.prices) >= 30:
                med = sorted(self.prices)[len(self.prices) // 2]
                mad = sorted(abs(x - med) for x in self.prices)[len(self.prices) // 2]
                if mad and abs(last - med) > self.spike_k * 1.4826 * mad:
                    out.append("spike %s %s" % (sym, last))
            self.prices.append(last)

        # 4 有价无量的伪更新：volume 未变而 last 变化
        vol = tick.get("volume")
        if isinstance(vol, (int, float)) and isinstance(last, (int, float)):
            pv = self.last_vol.get(sym)
            if pv is not None and pv[0] == vol and pv[1] != last:
                out.append("phantom_update %s" % sym)
            self.last_vol[sym] = (vol, last)

        # 5 盘口交叉 / 非正报价：不得进入中间价计算
        bid, ask = tick.get("bid"), tick.get("ask")
        if isinstance(bid, (int, float)) and isinstance(ask, (int, float)):
            if bid <= 0 or ask <= 0 or bid > ask:
                out.append("crossed_book %s bid=%s ask=%s" % (sym, bid, ask))

        return out


def check_bar(bar):
    """第 6 类：K 线内部自洽性，落库前逐根断言。"""
    o, h, l, c = (bar[k] for k in ("open", "high", "low", "close"))
    if h < l or h < max(o, c) or l > min(o, c):
        return ["inconsistent_bar %s" % bar.get("openTime")]
    return []

Wire check() into the first line of the frame handler: an empty list means store, a non-empty list means store with the reasons attached rather than dropping the record.

Do not hard-code thresholds

Treating stale_after=60 as a global constant is the next common mistake: the same thresholds applied to one-minute and daily bars differ by an order of magnitude in false-positive rate. Derive them per symbol from its own history:

# thresholds.py — 从历史分布反推阈值，不要写死全局常量
import statistics as st


def suggest_thresholds(bars):
    """bars 取自 /api/v1/market/equity/kline 的 bars[] 数组。"""
    gaps = [bars[i]["closeTime"] - bars[i - 1]["closeTime"] for i in range(1, len(bars))]
    steps = [abs(bars[i]["close"] - bars[i - 1]["close"]) for i in range(1, len(bars))]
    med = st.median(steps)
    mad = st.median([abs(s - med) for s in steps]) or med * 0.01
    return {
        "stale_after_ms": int(st.median(gaps) * 5),     # 连续 5 个周期无更新即判陈旧
        "spike_abs": round(med + 8 * 1.4826 * mad, 6),  # 8 倍 MAD 以上判为尖峰
    }

The input is a response body from /api/v1/market/equity/kline, whose bars carry openTime and closeTime in milliseconds. interval defaults to 1D and accepts 1T, 5T, 30T, 1H, 2H, 4H, 1D, 1W, and 1M; limit defaults to 100, range 1 to 1000. Thresholds derived this way track a symbol's volatility structure better than any rule of thumb.

Three checkpoints

Validation placed in the wrong layer doubles the cost. Three checkpoints, each with one job:

• Collection edge (highest priority). Validate as each WebSocket frame arrives. Context is richest here, and pushed ticks are not metered per message, so the marginal cost is zero.
• Before storage. Apply the class 4 self-consistency assertions and timestamp monotonicity to klines. Rejecting before write prevents contaminating the historical dataset, the most expensive layer to repair.
• Before consumption. Validate again when alerts and backtests read the data. Not redundancy: upstream may revise history, and collection-time assertions do not cover the revised shape.

One rule: failed records should be flagged, not deleted. Deletion hides the failure rate.

What validation costs

The key benefit of validating at the collection edge is the cost structure: TIQEX counts WebSocket handshakes toward daily quota and RPM, while subsequent tick pushes are not metered per message. Per-tick validation therefore adds essentially no call cost.

What consumes quota is backfill after an incident. It goes over REST, and two limits matter: the quote endpoint accepts at most 50 symbols per codes parameter, and kline limit caps at 1000. Batch accordingly, and on 429 honour Retry-After with jittered exponential backoff.

Rate limits are tiered: Free and Trial at 30 RPM, Growth at 120, Pro at 600, Enterprise at 3000. WebSocket caps are 1 connection and 10 symbols on Free, 3 and 25 on Growth, 10 and 50 on Pro, 50 and 200 on Enterprise. Quotas run from 300 calls per day on Free (delayed crypto, US equities, metals) to 500,000 per month on the $99 Growth tier and 5,000,000 per month with real-time coverage of all markets on the $299 Pro tier; see the pricing page.

One last distinction is easy to miss: an SLA covers availability, not correctness. The 99.9% standard and 99.99% professional figures describe availability and latency tiers; content correctness is the job of the checks above, and the two need separate metrics.

Frequently asked questions

Will validation slow the stream?

It will not become the bottleneck. All six assertions are constant-time or logarithmic local computations. The real concern is per-symbol state growth beyond a few thousand symbols, which a ring buffer solves. Per-tick validation typically costs microseconds.

Fixed or adaptive thresholds?

Adaptive, but layered: session first, then per symbol from historical distribution, then split by bar interval. Global constants look fine on small samples and break the moment you cross markets.

Drop bad ticks or keep them?

Keep and flag them. Dropping makes the failure rate unmeasurable and hides whether upstream is degrading. Add a quality_flags field at write time and let consumers filter — the same principle as missing values: preserve the state first, decide the use second.

Can the free tier run this?

Yes, and it is the recommended starting point. Free provides delayed crypto, US equities, and metals with WebSocket streaming, and 300 calls per day is enough to build the validator and calibrate thresholds, since the checks themselves consume no calls.

Next step

The path to production is short. Copy the two scripts, run them against one symbol with a free key for 24 hours, and count how often each class fires. Adjust thresholds from that data, then wire the gate into the frame handler. No new dependencies, no data model changes — only a quality flag.

Pipeline reliability does not come from how high the upstream SLA is. It comes from how quickly you learn when the data goes bad. Moving that judgement to the collection edge is the cheapest step available.

Get a free API key and run the validator against live data →

API fields, error codes, rate limits, and plan coverage come from the official tiqex.io documentation (updated 2026-09-12) and pricing page, accessed 2026-09-22. Example parameters illustrate engineering practice only and constitute no investment advice.