Sizing Tick Data Storage: What 10,000 Symbols for One Year Actually Costs

"Store every tick now, figure out what we need later" sounds prudent early in a project. It becomes an ever-growing bill. Tick data volume is a triple product — symbols × market activity × time — and none of those three factors shrinks on its own.

This article works through a concrete example: 10,000 symbols, one year of retention. How much disk, how much money, and how to bring that number down.

Assumptions first

To keep the conclusion meaningful, here are the assumptions stated explicitly. These are typical values; substitute your own:

Step one: annual tick volume

Get the order of magnitude on the table first:

• Ticks per second: 10,000 symbols × 2 = 20,000/sec
• Ticks per day: 20,000 × 86,400 = 1.728 billion/day
• Ticks per year: 1.728 billion × 365 = roughly 630.7 billion/year

630.7 billion. That number means nothing on its own — what matters is multiplying it by bytes per record.

Step two: how much the format changes the size

The same 630.7 billion records, stored three different ways, differ by more than 7×:

The gap is easy to explain. JSON writes field names and quotation marks 630.7 billion times; fixed-width binary does not. Columnar storage goes further, exploiting the monotonicity of timestamps and the low variance of prices through delta and dictionary encoding.

Step three: converting to money

Multiply volume by the object storage rate ($0.023/GB/month):

In other words, choosing JSON as the persistence format alone adds over $20,000 a year — and that is storage only, excluding write throughput, query compute, and backups.

One caveat: these are standard-tier rates. Infrequent-access and archive tiers are materially cheaper, at the cost of retrieval latency and retrieval fees. That trade-off is the next section.

Why JSON persistence is the expensive choice

JSON is entirely reasonable on the wire: readable, easy to debug, natively supported everywhere. As a storage format it costs you three ways:

1. Volume. Field names and structural characters are stored repeatedly — pure overhead.
2. Write throughput. Parsing JSON is an order of magnitude slower than fixed-width binary. At 20,000 records per second, CPU becomes the bottleneck.
3. Query compute. Columnar storage reads only the columns you need; JSON must parse whole rows, multiplying scan volume for backtest queries.

The sensible pattern: JSON in transit, columnar at rest. Adding a conversion stage to the write pipeline costs far less than paying the storage and compute premium indefinitely.

Tiered storage: not all data needs to be online

Another common waste is keeping every byte of history on the fastest storage. Real access patterns are predictable:

• Last 7 days — high frequency: live dashboards and recent backtests. Low-latency storage.
• Last 90 days — moderate frequency: strategy iteration. Standard object storage.
• Beyond 90 days — low frequency: mostly reproducing historical backtests. Archive tier.

Under this scheme only about 20% of the data sits on the most expensive tier, and the total bill drops noticeably. The cost is retrieval latency on archived data, so your tiering policy must match your backtest cadence. If the team reruns year-long backtests weekly, the archive tier is the wrong choice.

The more fundamental move: don't store it all

Finally, the most effective approach: aggregate first, then decide what raw data to keep.

• Pick granularity by need. Most strategies need minute bars, not ticks. Storing one-minute candles first reduces volume by two orders of magnitude.
• Store raw data selectively. Keep ticks for core symbols only; aggregate everything else.
• Set a raw retention window. For example, keep 90 days of raw ticks, then downgrade to aggregates.

Before provisioning more storage, ask a simple question: what fraction of those ticks has ever actually been queried? Many teams find the answer is under 5%.

Work out your own numbers in Python

TICKS_PER_SEC = 2
SECONDS_PER_DAY = 86_400
DAYS = 365
STORAGE_USD_PER_GB_MONTH = 0.023

# Approximate bytes per tick: JSON text / fixed binary / columnar compressed
BYTES_PER_TICK = {"json": 180, "binary": 48, "columnar": 24}


def annual_cost(symbols: int, ticks_per_sec: int = TICKS_PER_SEC) -> None:
    total_ticks = symbols * ticks_per_sec * SECONDS_PER_DAY * DAYS
    print(f"{symbols:,} symbols -> {total_ticks / 1e9:.1f} billion ticks/year")
    print(f"{'format':<10}{'size':>12}{'monthly':>12}{'yearly':>14}")
    for name, size in BYTES_PER_TICK.items():
        tb = total_ticks * size / 1e12                    # decimal TB
        monthly = tb * 1000 * STORAGE_USD_PER_GB_MONTH     # 1 TB = 1000 GB
        print(f"{name:<10}{tb:>10.1f} TB{monthly:>11,.0f} ${monthly * 12:>13,.0f} #34;)


annual_cost(10_000)
# 10,000 symbols -> 630.7 billion ticks/year
# format            size       monthly         yearly
# json          113.5 TB      $2,611      $31,330
# binary         30.3 TB        $696       $8,354
# columnar       15.1 TB        $348       $4,177

Swap in your real symbols and ticks_per_sec to get your own estimate. Run this during pipeline design — it costs a few minutes, while picking the wrong format costs years of billing.

FAQ

Should I retain all tick data?

Most teams should not. Identify which strategies genuinely depend on tick-level data, keep raw records only for those symbols, and store minute aggregates for the rest.

How does columnar storage reach 24 bytes per tick?

By exploiting structure: monotonically increasing timestamps support delta encoding, symbol codes and price levels suit dictionary encoding, and contiguous numeric columns compress well. None of these are available to row-wise JSON.

Can I just store ticks in a database?

It depends on scale. At 20,000 writes per second a general-purpose row store usually cannot keep up; you need a time-series database or columnar files backed by object storage. See the market data database guide for the overall pipeline design.

How much does archive storage save?

Unit cost is typically an order of magnitude lower, but retrieval is delayed and may incur fees. Suitable for reproducing historical backtests; unsuitable if you rerun full-history backtests weekly.

Does tick volume vary much between markets?

Substantially. Crypto trades around the clock and can produce very high tick rates, while a thinly traded equity may print only a handful of times per minute. Size each market separately rather than using one global average — the total is usually dominated by whichever segment is most active.

Does compression hurt backtest accuracy?

No. Columnar compression is lossless — decoded values are identical to the originals. Accuracy is affected by corporate action adjustment and time alignment, not by storage format.

Next steps

Work out your own numbers first, then decide format and tiering. The tick field definitions are in the tick data entry; for storage design see the market data database guide, and for backtest-specific preparation see the backtest data setup guide. If your plan is to pull history on demand from the API and retain only recent data locally, then plan quotas directly bound what is feasible — and note that streaming pushes are not metered per message, which suits continuous collection.