The Data Pipeline Behind Quant Backtesting: Collection, Adjustment, and Alignment

When a backtest is not trustworthy, the cause is rarely the strategy logic — it is the data pipeline. A backtest that runs cleanly can still produce a completely distorted conclusion through incorrect corporate action adjustment, time misalignment, or gap filling. And these errors do not raise exceptions. They just make the equity curve look better.

This article covers the four stages of a backtest data pipeline and the failure mode most likely to bite at each one.

The four stages

The order matters — collect, then adjust, then align, then store. Reversing it means recomputing everything repeatedly.

Stage one: granularity sets your ceiling

The test is simple: your signal's generation frequency must be higher than your data's granularity.

• Daily rebalancing — daily bars suffice, and they are small and fast to process.
• Intraday strategies — you need at least minute bars, and the exact resolution depends on the signal period.
• High-frequency and microstructure strategies — tick data is mandatory, along with attention to trade direction at tick level.

A common mistake is backtesting minute-level signals on minute bars. That introduces systematic bias: a minute bar preserves only four points (OHLC), so intra-minute entry and exit prices cannot be reconstructed, and results skew optimistic relative to reality.

The inverse mistake is equally real — using tick data to backtest a daily strategy. That does not improve accuracy; it raises compute and storage cost by an order of magnitude.

Stage two: adjustment, the most error-prone step

When a stock splits or pays a dividend, the price gaps. Without adjustment, a backtest reads that gap as a genuine crash or surge and generates a false signal.

There are two adjustment directions, and the choice depends on the backtest type:

For backtesting, backward adjustment is usually the right choice, because historical prices do not change when new data arrives — which keeps results reproducible. Forward-adjusted series shift with the latest price, so the same backtest can produce different results on different days. That is a subtle but serious reproducibility problem.

One more requirement: adjustment factors must come from a reliable source and must cover every corporate action — splits, reverse splits, dividends, and rights issues. Missing any one leaves a fake gap in history.

Stage three: alignment and gap handling

For multi-symbol backtests, the governing principle is: never let forward filling hide a gap.

Three kinds of absence need distinct treatment:

• Market closed — normal. Remove it from the timeline; it is not a gap.
• Halted — the symbol did not trade that day. Keep the position unchanged and mark it untradeable. Do not fill the price.
• Missing — a data source problem. Alert and backfill. Never fill silently.

Treating a halt as missing and forward-filling the price creates fake zero-volatility positions in the backtest, understating risk and overstating return. This is one of the most common sources of distorted results.

Stage four: storage and query semantics

The goal here is reproducibility. The same backtest, run at a different time on a different machine, must produce identical results. That requires:

1. Data versioning. Record the snapshot identifier or time range used, rather than "fetch the latest data".
2. Fixed query semantics. Pin the adjustment method, timezone, and gap rules — and encode them in code, not in documentation.
3. Columnar storage. Backtests typically read a handful of columns; columnar formats cut scan volume substantially.
4. Retain raw data. Aggregates support fast iteration, but raw data is the basis for recomputation and should not be discarded.

The three most common backtest data errors

1. Survivorship bias. Backtesting only symbols still trading today, excluding delisted and acquired names. Results skew systematically optimistic.
2. Look-ahead bias. Generating the signal from today's close, then filling at today's open. Impossible in live trading.
3. Filling that hides gaps. Forward-filling halts, which artificially suppresses volatility.

None of these raise errors. They just improve the curve. That is why the first priority of a backtest system is not speed — it is the ability to detect its own data problems.

Python: preparing backtest data

import httpx
import pandas as pd

API_KEY = "your_api_key"
BASE = "https://api.tiqex.io/api/v1/market"


def fetch_candles(symbol: str, market: str, interval: str = "1d") -> pd.DataFrame:
    """Fetch candles and normalise into a UTC-indexed DataFrame."""
    resp = httpx.get(
        f"{BASE}/{market}/kline",
        params={"symbols": symbol, "interval": interval},
        headers={"X-API-Key": API_KEY},
        timeout=30,
    )
    rows = resp.json()["data"]
    df = pd.DataFrame(rows)

    # 1) Normalise to UTC so cross-market series line up
    df["ts"] = pd.to_datetime(df["timestampMs"], unit="ms", utc=True)
    df = df.set_index("ts").sort_index()

    # 2) Assert monotonic timestamps; a regression means bad data
    if not df.index.is_monotonic_increasing:
        raise ValueError(f"{symbol} timestamps are not monotonic; inspect manually")

    # 3) Keep only backtest columns to reduce scan volume
    keep = [c for c in ("open", "high", "low", "close", "volume") if c in df.columns]
    return df[keep]


def apply_backward_adjustment(df: pd.DataFrame, factors: pd.Series) -> pd.DataFrame:
    """Backward adjustment from listing, keeping history reproducible."""
    aligned = factors.reindex(df.index, method="ffill")
    out = df.copy()
    for col in ("open", "high", "low", "close"):
        if col in out.columns:
            out[col] = out[col] * aligned
    return out


def mark_trading_gaps(df: pd.DataFrame, calendar) -> pd.DataFrame:
    """Mark closed / halted / missing separately instead of forward filling."""
    expected = pd.DatetimeIndex(calendar.sessions_in_range(df.index[0], df.index[-1]))
    missing = expected.difference(df.index)
    if len(missing) == 0:
        return df

    # Record an explicit flag; do not fill prices
    df = df.reindex(df.index.union(missing))
    df["no_trade"] = df["close"].isna()
    return df


candles = fetch_candles("AAPL", "equity")
print(f"rows {len(candles)}  range {candles.index[0].date()} -> {candles.index[-1].date()}")
print(candles.tail(3))

Three constraints are encoded here: UTC normalisation, monotonicity validation, and gaps represented by an explicit flag rather than a fill. apply_backward_adjustment uses backward adjustment so the historical series never shifts as new data lands — the precondition for a reproducible backtest.

FAQ

Should backtests use forward or backward adjustment?

Backward adjustment is usually the right choice. Its historical prices do not change as new data arrives, so results are reproducible. Forward-adjusted series shift with the latest price, meaning the same strategy can yield different results when rerun later.

How should halts be handled?

Hold the position and mark it untradeable. Never forward-fill the price — filling creates zero-volatility phantom positions that understate risk and overstate returns.

Do backtests need tick data?

It depends on the strategy horizon. Daily and minute-level strategies work fine on candle data; only high-frequency and microstructure strategies need ticks. Using ticks for a daily strategy adds cost without adding accuracy.

Why does the same backtest give different results on different runs?

Usually because query semantics are not pinned — you fetched "the latest data" rather than a fixed snapshot, or you used forward-adjusted data. Version your data and encode adjustment, timezone, and gap rules in code.

How do I avoid look-ahead bias?

Strictly separate the signal timestamp from the fill timestamp. A signal may only use data available at that moment, and the fill price must come after it. Enforce this in the backtest framework rather than by convention.

Next steps

A backtest data pipeline comes down to three words: reproducible, no filling, explicit marking. The full preparation workflow is in the backtest data setup guide; adjustment mechanics are in the stock split adjustment entry, and candle fields in OHLCV candle fields. See the backtesting solution for the wider pattern. Historical pull volume directly affects plan quotas, so size it before you start.