Three Clocks in Every Tick: Timestamp Provenance and Clock Skew

At 02:10 your alert daemon's VM is suspended for patching. It resumes at 02:12 and the guest clock comes back 4.3 seconds behind, because the hypervisor restored it from the suspend image and NTP has not converged yet. The daemon carries a one-line guard you wrote two years ago:

age_ms = time.time() * 1000 - tick["timestampMs"]
if age_ms < 0:            # a tick from the future is corrupt
    continue

Every tick that arrives after 02:12 now has a negative age. The guard drops it. The daemon consumed zero ticks for 37 minutes, emitted zero alerts, and wrote zero errors, because the branch executed exactly as designed on 100 percent of messages. What surfaced the problem was not a log line but a dashboard panel, "ticks processed per minute", going flat while the socket stayed open and the subscription stayed confirmed.

The guard's intent was correct. A negative age really is impossible if both clocks agree. The error is in the inference you make when the impossible happens: you concluded the data was corrupt, when the evidence only said that two clocks disagreed. Two clocks, one subtraction, no way to tell which one moved.

Every tick you receive carries a clock you do not own and leaves your process with a third one you do. age = transit + offset has two unknowns and one equation, so a single sample cannot tell you whether the feed is late or your host is wrong. Only a window of samples, read as a floor rather than an average, can separate them.

Three questions, three instruments

The 02:10 incident happened because one number was asked to answer three different questions. Keep them apart and the instrumentation becomes small.

Here is the part that trips people up: timestampMs in a quote row is the upstream's stamp for the price, not the instant the HTTP response left the vendor's edge. For a liquid symbol inside its session the two are close enough to be interchangeable. For a thin symbol, or for any symbol whose market is closed, timestampMs can be hours old by design, and no amount of clock discipline changes that. So you measure per symbol, and only inside that symbol's session. If you skip the session gate, you will build an alert that fires every weekend and a clock dashboard that reports a 60-hour offset every Monday morning.

The second consequence is that age measures freshness, not latency. If you need a latency number for a service level objective, age cannot give it to you, because it includes the upstream's own caching and the upstream's clock. What you can attribute to yourself is the round-trip time measured on a monotonic clock around a single request, and even that is a client-side lower bound on path latency rather than one-way transit.

Solving for the offset bound instead of the offset

Write the equation once and the estimator falls out of it:

age = t_recv_wall − t_event
    = transit + (C_client − C_source)
    = transit + offset

Transit cannot be negative. Therefore age ≥ offset for every sample, and the minimum age across a window is offset + min_transit: an upper bound on your clock error, not an estimate of it. If your clock is correct and the transit floor is 8 ms, the minimum age is 8 ms and the bound reads "offset ≤ 8 ms", which is true and harmless. If your clock is 3,000 ms ahead, the minimum age can never fall below 3,000 ms, and the bound tells you exactly what you need to know: your host is wrong by seconds, and every age you compute is wrong by seconds in the same direction.

Averaging is the wrong operation. Mean age is offset plus mean transit, and mean transit contains retries, TLS handshakes, scheduler stalls, and the long tail of TCP. Averaging noise into an estimator of a floor gives you a number that moves with your load rather than with your clock. The minimum is the only order statistic that converges to the floor, which is why every serious clock-offset method starts with a minimum or a quantile near it.

Two failure modes deserve names.

Drift. An undisciplined VM clock moves slowly but continuously, at tens to hundreds of parts per million. One hundred ppm is 360 ms per hour, or 8.6 seconds per day. You measure it as the minimum age per window, plotted over hours, and report the slope in ms/hour. A slope near zero proves stability, not correctness, because a constant offset has a flat slope.

Step. A discontinuity: an NTP step correction, a resume from suspend, a manual date -s, a container migration onto a host with a different clock, or a leap-second smear handled inconsistently. Steps are the ones that break guards like the one at the top of this article, and you can detect them without any vendor data at all:

t0w, t0m = time.time(), time.monotonic()
# ... one request ...
t1w, t1m = time.time(), time.monotonic()
step_ms = (t1w - t0w - (t1m - t0m)) * 1000

The wall clock can jump; the monotonic clock cannot. Over a two-second interval, a magnitude above roughly 1,000 ms means the wall clock moved, not the network. That single comparison catches the 02:10 incident and costs one subtraction. Two caveats keep it honest. On Linux, CLOCK_MONOTONIC, which is what time.monotonic() reads, stops while the machine is suspended, so a resume produces a large Δwall − Δmono even when the wall clock is perfectly correct; use CLOCK_BOOTTIME or the boot id to separate "wall moved" from "monotonic paused". And chrony's default behaviour is to slew small offsets rather than step them, at up to 500 ppm, so a correction can cross your 200 ms alert threshold while the step detector stays silent. Detect both: threshold the bound, and watch the difference.

The HTTP Date header is the third instrument, and it is weaker than it looks. It has one-second granularity, so it carries plus or minus 500 ms of quantisation before you add transit. Use it as a cross-check with a two-second tolerance, not as a precise offset. It earns its place in one situation: when the payload timestamp is hours old because the market is closed, Date still advances, so it is the only vendor-side clock signal you have on a Sunday.

Reverse boundary: when this measurement is the wrong tool

Instrumentation has a cost, and this one costs a guard clause, a column, and a sampling budget. Six situations do not justify it.

1. Daily bars and end-of-day research. openTimeMs on a kline is a bucket label, not an event time. Age computed against a bucket start grows monotonically through the session and collapses at the boundary. If your decision cycle is daily, calendar alignment and adjustment rules decide your correctness; clock discipline does not move the answer.
2. Closed markets and thin symbols. Age is unbounded by design. A quote for a US name at 03:00 UTC is legitimately hours old. Gate on market state, or read the delayed and unavailable flags, and let the stale-age alarm apply only inside a session.
3. Hosts you cannot discipline. On an air-gapped or vendor-locked host, do not estimate an offset you cannot validate. Write recv_clock_untrusted = true next to the arrival time and use event time for every decision. An admitted unknown is cheaper than a confident wrong bound, and the wrong bound is the one that will page you at 02:12.
4. Freshness labels in a user interface. "Updated 3 seconds ago" computed from a browser clock turns a healthy feed into an incident on a laptop that resumed from sleep. Compute relative time on the server, or show the event timestamp and nothing else.
5. Deduplication and ordering keys. Arrival time must never be a key. A clock step reorders your stream, so your dedupe window either collapses and admits duplicates or widens and swallows legitimate prints. Key on (symbol, event_ts, price) and store arrival separately. The reconnect and aggregator failure modes are documented in Duplicate Ticks After WebSocket Reconnect and the K-line aggregator reconciliation post.
6. Sub-10 ms service level objectives. A client-side age cannot support a millisecond claim, because the transit floor of a single vantage point dominates the number. That measurement needs instrumentation at the edge, PTP, or both. Be explicit that your SLO covers what you can see, not what you wish you could.

The sampling budget deserves its own line, because it is the constraint that quietly kills this kind of probe. The Free plan allows 9,000 REST calls per month with a 300-call daily cap (pricing page, accessed 2026-10-20). Probing every two seconds for one day is 43,200 calls, which is 144 times the daily cap. So probe in windows: ten minutes at five-second intervals is 120 calls, run it weekly on a rotation of hosts, and you will still catch a step within a week. If you need continuous monitoring, move the probe to WebSocket, where the handshake counts once toward quota and later tick pushes are not metered, but note that streaming requires the Growth plan or above.

The probe

The script below runs twelve quote requests, prints an age table per symbol, bounds the offset, and flags wall-clock steps. It uses the real multi-market quote endpoint, keeps the wall clock and the monotonic clock separate at the point of capture, and records the HTTP Date offset as a cross-check.

#!/usr/bin/env python3
"""
clock_probe.py - bound your host clock offset against TIQEX quote timestamps.

    TIQEX_API_KEY=... python clock_probe.py

Cost: SAMPLES REST calls (12 by default). The Free plan allows 300 calls/day,
so this run consumes about 4 percent of one day's cap.
For a drift series, set SAMPLES=180 and INTERVAL_S=20 (one hour, 180 calls).
"""

from __future__ import annotations

import json
import os
import statistics
import time
from datetime import timezone
from email.utils import parsedate_to_datetime
from urllib.parse import quote as urlquote
from urllib.request import Request, urlopen

QUOTE_URL = "https://api.tiqex.io/api/v1/market/quote"
CODES = ["AAPL", "EURUSD", "BTCUSDT"]   # multi-market quote: 50 symbols per call
SAMPLES = 12
INTERVAL_S = 2.0
STEP_THRESHOLD_MS = 1000.0


def fetch_quote(codes: list[str], api_key: str) -> dict:
    """One request. Wall and monotonic clocks are captured around the same call."""
    url = f"{QUOTE_URL}?codes={urlquote(','.join(codes))}"
    req = Request(url, headers={"X-API-Key": api_key})
    t0_wall, t0_mono = time.time(), time.monotonic()
    with urlopen(req, timeout=10) as resp:
        payload = json.loads(resp.read().decode("utf-8"))
        date_header = resp.headers.get("Date")
    t1_wall, t1_mono = time.time(), time.monotonic()

    if not payload.get("succ"):
        raise RuntimeError(f"{payload.get('code')}: {payload.get('msg')}")

    return {
        "rows": payload.get("data") or [],
        "recv_ms": t1_wall * 1000.0,
        "rtt_ms": (t1_mono - t0_mono) * 1000.0,
        "step_ms": ((t1_wall - t0_wall) - (t1_mono - t0_mono)) * 1000.0,
        "date_header": date_header,
    }


def server_offset_ms(date_header: str | None) -> float | None:
    """Vendor edge clock minus local clock. Quantised to one second by the header."""
    if not date_header:
        return None
    try:
        server = parsedate_to_datetime(date_header)
    except (TypeError, ValueError):
        return None
    if server.tzinfo is None:
        server = server.replace(tzinfo=timezone.utc)
    return (server.timestamp() - time.time()) * 1000.0


def collect(api_key: str) -> list[dict]:
    samples: list[dict] = []
    for i in range(SAMPLES):
        got = fetch_quote(CODES, api_key)
        ages = {}
        for row in got["rows"]:
            ts = row.get("timestampMs")
            code = row.get("code") or row.get("symbol")
            if isinstance(ts, (int, float)) and ts > 0 and code:
                ages[code] = got["recv_ms"] - float(ts)
        got["ages_ms"] = ages
        got["server_offset_ms"] = server_offset_ms(got["date_header"])
        samples.append(got)
        if i < SAMPLES - 1:
            time.sleep(INTERVAL_S)
    return samples


def summarize(samples: list[dict]) -> tuple[list[dict], list[float], list[float]]:
    per_symbol: dict[str, list[float]] = {}
    for s in samples:
        for code, age in s["ages_ms"].items():
            per_symbol.setdefault(code, []).append(age)

    rows = []
    for code, ages in sorted(per_symbol.items()):
        ordered = sorted(ages)
        p95_index = max(0, min(len(ordered) - 1, int(round(0.95 * len(ordered))) - 1))
        rows.append({
            "symbol": code,
            "n": len(ages),
            "min_ms": ordered[0],
            "p50_ms": statistics.median(ordered),
            "p95_ms": ordered[p95_index],
            "offset_bound_ms": ordered[0],   # offset + transit floor: an upper bound
        })

    steps = [s["step_ms"] for s in samples if abs(s["step_ms"]) > STEP_THRESHOLD_MS]
    server_offsets = [s["server_offset_ms"] for s in samples if s["server_offset_ms"] is not None]
    return rows, steps, server_offsets


def main() -> None:
    api_key = os.environ.get("TIQEX_API_KEY")
    if not api_key:
        raise SystemExit("set TIQEX_API_KEY first")

    samples = collect(api_key)
    rows, steps, server_offsets = summarize(samples)

    print(f"{'symbol':<10}{'n':>4}{'min':>10}{'p50':>10}{'p95':>10}{'bound':>10}")
    for r in rows:
        print(f"{r['symbol']:<10}{r['n']:>4}{r['min_ms']:>10.0f}{r['p50_ms']:>10.0f}"
              f"{r['p95_ms']:>10.0f}{r['offset_bound_ms']:>10.0f}")

    if not rows:
        print("no timestamps returned; check symbol codes and market entitlement")
        return

    bound = max(r["offset_bound_ms"] for r in rows)
    spread = max(r["p50_ms"] - r["min_ms"] for r in rows)
    print(f"\noffset upper bound : {bound:,.0f} ms")
    print(f"queueing spread    : {spread:,.0f} ms (p50 minus min, per symbol)")

    if steps:
        print(f"clock steps        : {len(steps)} interval(s) moved more than "
              f"{STEP_THRESHOLD_MS:.0f} ms; treat absolute ages as unreliable")
        print(f"largest step       : {max(steps, key=abs):+,.0f} ms")
    else:
        print("clock steps        : none detected in this window")

    if server_offsets:
        median_server = statistics.median(server_offsets)
        print(f"Date header offset : {median_server:+,.0f} ms "
              f"(plus or minus 500 ms quantisation)")
        if abs(median_server - bound) > 3000:
            print("warning            : payload bound and Date offset disagree by more than 3 s;")
            print("                     a closed market or a cached row explains it before a clock fault does")

    if bound > 2000:
        print("\nverdict: your host clock is likely off by seconds. Fix NTP before")
        print("         trusting any age, any dedupe window, or any alert ordering.")
    elif spread > 500:
        print("\nverdict: clock looks usable; the spread points at transport or queueing,")
        print("         so raise the freshness threshold rather than the timeout.")
    else:
        print("\nverdict: age is small and stable; keep the session gate in place.")


if __name__ == "__main__":
    main()

Three details in that script matter more than the output format.

The clocks are captured inside the request function, at the same level as the socket call, so the wall and monotonic readings describe the same interval. Splitting them across functions is how people end up comparing a timestamp taken before DNS resolution with one taken after the TLS handshake, and then wondering why the step detector fires.

The offset bound is reported per symbol and then reduced with max, not min. You want the worst symbol, because a bound that holds for BTCUSDT and fails for a thinly traded name is not a bound on your clock, it is a bound on your luck. The reduced value is still an upper bound, just a conservative one.

The verdict distinguishes three states on purpose. A large bound means fix the clock. A large spread with a small bound means the age is dominated by something other than your clock, which is a queueing or cache problem and belongs in a different runbook. A small bound with a small spread means the instrument is working and the next step is automation, not more measurement.

Decision checklist

FAQ

Can I measure one-way latency exactly from my host? No. One equation with two unknowns cannot be solved from a single vantage point, and any vendor that reports one-way transit is either instrumenting the edge or estimating. You can bound the sum, which is what this article does, and you can attribute round-trip time to your own path, which is usually the actionable half.

Why did my age go negative? Because your clock is behind the source's clock by more than the transit time. It is not evidence of corrupt data. Log it as a clock event, increment a counter, and keep the tick.

Does the delayed flag replace this measurement? For freshness, largely yes: it is an authoritative statement from the vendor rather than an inference from two clocks, and it costs nothing to read. It does not tell you whether your host clock is wrong, which still matters for dedupe windows, alert ordering, and log correlation.

Is the HTTP Date header good enough on its own? Not for a precise offset, because of one-second granularity. It is good enough to answer "is my host off by more than ten seconds", and it is the only vendor clock that moves when the payload timestamp deliberately does not.

What tolerance should I set? Set it from the decision, not from the clock. If your alert needs to distinguish two prints 50 ms apart, a 200 ms offset makes the ordering meaningless. If your panel refreshes every five seconds, a two-second bound is irrelevant and you should not spend a probe on it.

Do I need this for a backtest? Only if you align live and historical data at sub-minute resolution. Otherwise the calendar and adjustment rules dominate, as covered in the timezone and calendar article. Measuring clock skew to improve a daily backtest is work that cannot change the answer.

Next step

Three steps, in order. First, add the one-line step detector to whatever process already calls the API, and log the result next to the request id. It costs a subtraction and it will tell you within a day whether your host moves. Second, run the probe above once on each host that produces timestamps you later trust, and record the bound in your runbook rather than in a chat message. Third, put the session gate in front of the age alarm, so the weekends stay quiet and the alarm keeps its meaning.

If the bound comes back large, the fix is platform work, not application work: point the host at a reachable NTP source, verify the daemon is not being blocked by egress rules, and check whether your container runtime is masking the host clock. If the bound is small but the spread is wide, the next article to read is the silent failure taxonomy, because a wide spread with a correct clock is a delivery problem wearing a clock problem's clothes.

Sources and scope. Field names, error codes, and limits in this article are taken from the TIQEX API v1 developer documentation at tiqex.io/docs/auth, tiqex.io/docs/errors, and tiqex.io/docs/market-coverage, together with the pricing page at tiqex.io/pricing; all were read on 2026-10-20. Quota figures, quote batch caps (US 20, multi-market 50, crypto 100 symbols per call), WebSocket metering, and service level commitments are documentation defaults and vary by plan; the console plan is authoritative, and latency tiers, history depth, and coverage depend on your entitlements, so verify them on the pricing page before you cite them. The code targets Python 3.11 or later and is written for demonstration only: it has no retry budget, no persistent storage, and no handling for redirects. This article demonstrates engineering practice and does not constitute investment advice. Market data is for reference only.