By Oleksii Tkachuk, Rajiv Shringi, Chris Lohfink, Abe Ratnofsky, Kartik Sathyanarayanan
Introduction
In an earlier submit, we launched the TimeSeries Abstraction, our high-throughput, low-latency platform designed to retailer and question temporal occasion knowledge at scale. It has since grow to be the operational spine for Netflix’s largest datasets, which now span a number of petabytes.
Nonetheless, managing such massive datasets in scorching storage, tuned for low-millisecond latency entry, just isn’t low-cost. At Netflix, cost-efficiency is a steady self-discipline somewhat than a brief initiative. This submit shares two options that considerably cut back storage prices by repeatedly transferring older, much less continuously accessed knowledge into cheaper object storage:
- Answer 1: Makes use of Apache Kafka and Flink to stream knowledge on to S3
- Answer 2: Leverages Cassandra backups in S3 and pluggable Memtables to learn knowledge straight from S3
We’ll begin with the Kafka/Flink-based design, then describe how we developed to a local Cassandra-based chilly tier.
Challenges
As Netflix’s TimeSeries datasets have expanded, we’ve encountered a number of compounding challenges:
- Rising retention calls for: Customers typically push for rising knowledge retention to serve a broader vary of use instances, which in flip will increase the storage prices.
- Escalating prices: Sustaining huge, dense clusters for occasionally accessed historic knowledge is prohibitively costly.
- Sparse historic entry: Most queries goal latest occasions, however older knowledge should stay available for infrequent audits and investigations.
- Operational pressure: Excessive storage density on scorching nodes complicates scaling, deployment, and cluster stability.
- Latency necessities: Even historic knowledge typically requires sub-second question efficiency for inside tooling and evaluation.
This creates a transparent alternative: maintain the newest knowledge on quick storage and migrate older, not often learn knowledge to cheaper storage whereas preserving a constant API. Finished nicely, this may meet consumer and operational wants whereas vastly decreasing our infrastructure spend.
TimeSeries Abstraction
Earlier than we dive into the chilly storage options, here’s a fast recap of how TimeSeries partitions its dataset into a number of tables, every masking a window of time we name Time Slices.

This permits a number of key optimizations, most notably permitting a desk to grow to be immutable as soon as it crosses the write boundary, which lets us safely apply additional optimizations. For extra particulars, confer with our earlier submit.
This slice info is important and is maintained in a devoted metadata desk:

Creating new Time Slices and sustaining their metadata is a major duty of the TimeSeries management airplane. The TimeSeries knowledge airplane makes use of this info to accurately route the learn and write site visitors to the provisioned tables.

With that background, let’s have a look at the totally different options we constructed for knowledge tiering.
Answer 1: Streaming with Apache Kafka and Flink
Our first answer adopted a well-known streaming sample utilizing Apache Kafka and Flink, the place we streamed knowledge to S3 in realtime whereas concurrently persisting it to Cassandra.

Getting Knowledge Into S3 Effectively
To optimize knowledge streaming to S3, the system hash-partitions Time Collection IDs into buckets. This partitions the stream and lets devoted Flink duties deal with every bucket, finally batching and decreasing the write throughput to S3. Flink staff batch occasions into Parquet information grouped by occasion time inside the hour. These information are nice for high-throughput streaming writes, however not for low-latency reads.
Enter the Compaction job.
Compaction Job to Optimize Reads
The compaction job runs day by day and merges small Parquet information right into a single, optimized day by day Parquet file per namespace.
The job additional optimizes the Parquet information within the following methods:
- Small Row Teams: Creates small (4–8 MB) row teams to attenuate lookup value. We are able to then use the Parquet Java library to carry out vary reads of row teams from S3. That is deliberately nicely under Parquet’s advisable row group flooring — smaller teams have been higher as a result of our reads are dominated by level/vary lookups somewhat than full scans.
- Occasion Sorting: Types occasions inside a row group by time, eliminating brute-force scans when querying for a time vary.
- Bloom Filters: Caches Bloom filters and file metadata in EVCache, permitting the TimeSeries question planner to effectively skip pointless reads from S3.

As soon as the compaction job completes, all compacted information per partition are recorded within the TimeSeries management airplane, and the new storage slices can now be truncated!
On a few of our largest datasets, this job sometimes takes about 12 hours, prioritizing decrease compute prices over execution velocity.

Streaming Chilly Tier Learn
As soon as compacted information are optimized in chilly storage, they’re able to be learn. The TimeSeries knowledge airplane followers out a given question throughout each the cold and hot tiers and merges their consequence. The customers nonetheless use the identical question interface:

The Chilly Tier Knowledge Airplane makes use of compaction metadata for aggressive planning and pruning, which dramatically reduces the S3 question fan-out. Because of this, retrieving a full day of knowledge requires at most three sequential S3 reads, even within the unusual case the place an ID spans two row teams. This yields a p50 latency of ~150 ms and a p99 latency of ~500 ms per learn.

On the time of writing, the streaming chilly tier implementation was processing as much as 1,250 reads per second from chilly storage globally:

Why not cease right here?
The streaming chilly tier has delivered actual financial savings, shrinking each Cassandra’s EBS footprint and the Cassandra backup footprint in S3. However the streaming infrastructure itself is dear and provides transferring items that widen the workforce’s on-call scope:

We needed to enhance this additional and requested ourselves:
- Can we keep away from streaming knowledge into the thing retailer and the related knowledge switch prices?
- Can we cut back the compute and operational overhead of recent parts?
That led to a extra radical query: Can we learn straight from present Cassandra backups in S3?
Answer 2: Cassandra Chilly Storage for Immutable Knowledge
At Netflix, we again up Cassandra knowledge into S3 at common intervals for catastrophe restoration. The concept for tiering is easy:
Learn straight from these S3 backups as a substitute of paying a separate knowledge switch value to S3.
As soon as TimeSeries marks a scorching Time Slice as prepared for chilly storage, we will customise Cassandra to learn straight from its backups, permitting us to delete the on-disk knowledge copy. This could additionally incur minimal modifications to the TimeSeries learn path!

To make this viable in manufacturing, we needed to reply a sequence of “How…?” questions.
How Can We Learn the Backups Effectively?
We launched a brand new compaction technique in Cassandra, which we name ReadOnlyCompactionStrategy (ROCS). Right here is the way it works:
As soon as a time slice crosses the user-configured cold-tier boundary, TimeSeries stops accepting writes for that point slice making it immutable. This immutability unlocks vital optimization alternatives:
- SSTable overlap: Commonplace methods (LCS, TWCS) typically produce SSTables that overlap within the token house, and so a single partition key might exist throughout a number of information.
- Learn fan‑out: As a result of a learn should contact each SSTable masking the important thing’s token vary, this overlap turns a single point-read into a number of lookups, rising latency throughout all tiers.
ROCS re-compacts all SSTables as soon as, aligning them exactly with token ranges:

ROCS applies extra broadly to all TimeSeries use instances, driving tail latency down on all storage tiers by decreasing the SSTable fan-out throughout learn operations:

As soon as the immutable Time Slices have been optimized, it’s time to set off backups.
How Would We Leverage Backups?
Cassandra backs up each node within the cluster, organizing every backup on S3 as a declarative manifest, together with the SSTables backing the keyspace’s tables:
{bucket}/
└── {prefix}/ ← cluster id (env_backup/hash_app)
└── {token}/ ← node id (ring token)
├── META_V2/ ← manifests, one per snapshot
│ ├── {snapshotInstantMs}.json
│ ├── {snapshotInstantMs}.json
│ └── ...
└── SST_V2/ ← precise SSTable information
└── {backupTimestamp}/ ← snapshot timestamp
└── {keyspace}/
└── {table-with-uuid}/
└── {compression}/
└── {encryption}/
├── {component-file}
└── ...The backup manifest orchestrates the backup, itemizing the paths to all its SSTables:
manifest.json
├── information
│ ├── appName "cass-cluster"
│ ├── area "us-east-1"
│ ├── snapshotInstantMs 1748908800000
│ ├── backupPathPrefix "prod_backup/1234_cass/token"
└── knowledge[]
└── [0]
├── keyspaceName "ks"
├── columnfamily "tbl-a1b2..."
└── sstables[]
└── [0]
└── parts[]
├── na-1-big-Knowledge.db
├── na-1-big-Index.db
├── na-1-big-Filter.db
├── na-1-big-Statistics.db
├── na-1-big-Abstract.db
├── na-1-big-CompressionInfo.db
Alongside the big Knowledge.db file, backups embody a compact set of index, metadata, and integrity information. This construction permits us to shortly find, validate, and restore particular partitions or knowledge ranges with out scanning your entire dataset. Moreover, for the reason that manifest catalogs each S3 path, we will obtain most of those parts on to the Cassandra node (excluding the Knowledge.db file) the place they occupy negligible house. These native parts allow environment friendly question planning earlier than any knowledge is learn from S3.
How Would Cassandra Learn From Backups?
Cassandra reads sometimes pull knowledge from two sources: the Memtable, which holds latest unflushed writes, and flushed SSTables on sturdy storage. As of Cassandra 4.1, we will leverage pluggable Memtables to redirect reads from “chilly” tables on to the backup storage in S3.
-- create desk for a given Time Slice
CREATE TABLE data_20260616_00_04 (
...
...,
PRIMARY KEY ((time_series_id, time_bucket, event_bucket), event_time, event_id, key)
) WITH CLUSTERING ORDER BY (event_time DESC, event_id DESC, key DESC)
...
AND compaction = ...
AND compression = ...
AND memtable = 'backupmemtable:timestamp=1781953200000' ← Customized memtable
...
The timestamp marks the purpose at which the backup is assured to cowl the total desk, and the desk is assured to be immutable.
Opening a backup Memtable kicks off a state-machine initialization that pulls the index knowledge from S3 onto the node as soon as:

This course of ensures all SSTable parts are downloaded reliably by means of a number of resilience mechanisms:
- Retries and backoff: All S3 operations and file-system cache writes use retries with exponential backoff.
- Useful resource isolation: Initialization runs on a devoted thread pool, remoted from regular Cassandra operations.
- Partial obtain safety: If a downloaded element’s measurement doesn’t match the manifest, it’s deleted and re-downloaded.
- DDL change safety: A .timestamp file marks accomplished initialization and ties it to the backup timestamp. If the desk’s backup timestamp modifications, this file is eliminated and initialization restarts for the brand new timestamp.
- Learn safety: Memtable initialization is asynchronous; reads to an initializing memtable block for at most half the learn RPC timeout.
As soon as initialization has pulled each SSTable element (besides Knowledge.db) from S3 onto the native disk, the Memtable is able to serve reads. Here’s what the learn course of seems like:

To reduce latency, the system parallelizes S3 reads throughout requests, SSTables, and particular person chunks. A shared cache deduplicates fetches and prefetches knowledge, so chunks are prepared earlier than they’re wanted.
We additional optimize cluster and desk settings to scale back the variety of S3 calls:
- Bigger compression chunks: Scale back S3 calls per partition by studying fewer, bigger chunks (accepting some over-fetch on small reads).
- Learn restore disabled: Keep away from costly, pointless restore site visitors towards immutable, pre‑repaired backup knowledge.
- Speculative retries disabled: Forestall S3’s greater latency from triggering redundant, overlapping reads.
How Would We Know When It Is Secure To Learn From Backups?
Nonetheless, earlier than the Memtable can level to a backup, we should affirm two issues throughout the entire ring, and we use Cassandra Digital Tables to do that.
First, that ROCS re-compaction succeeded:
cqlsh> SELECT * FROM netflix_views.cluster_view
WHERE keyspace_name = 'system_views' AND table_name = 'sstable_tasks';
keyspace_name | table_name | host | worth
--------------+---------------+----------+------------------------------------------------
system_views | sstable_tasks | 10.0.0.1 | [{"table_name":"<.>","kind":"compaction", ...}]
system_views | sstable_tasks | 10.0.0.2 | [] <---- no lively compactions
system_views | sstable_tasks | 10.0.0.3 | []
And second, that backups have been totally captured (denoted by uploaded=True):
cqlsh> SELECT * FROM netflix_views.complete_backups
WHERE keyspace_name = 'my_ks' AND table_name = 'my_table';
keyspace_name | table_name | timestamp | num_tokens | num_uploaded | uploaded
--------------+------------+---------------+------------+--------------+----------
my_ks | my_table | 1765211400000 | 3 | 3 | True
my_ks | my_table | 1765213200000 | 3 | 2 | False
How Would We Automate This Whole Circulate?
The TimeSeries management airplane automates your entire transition as slices cross the immutable boundary:

Every slice might be in any lifecycle stage at any time with out impacting system availability, and the system stays resilient to delayed backups or lagging compactions.
How Would Cassandra Deal with The Further Load?
Serving knowledge from S3 is way more CPU-intensive than studying from native storage. A sudden spike in learn requests can overwhelm the Cassandra cluster and have an effect on the new path, turning a cost-saving function right into a reliability danger.
To keep away from this, chilly reads can optionally run on a separate Cassandra cluster that factors its backup Memtables on the scorching cluster’s backups:

The chilly Cassandra cluster should match the unique cluster in topology however needn’t match it in {hardware}! We are able to run the chilly cluster on a lot smaller, cheaper, storage-light situations.
Actual World Efficiency
Cassandra Chilly Tier was rolled out to essentially the most storage‑intensive TimeSeries Abstraction clusters. It at the moment shops over 15 PB of compressed knowledge in Cassandra chilly storage and has allowed us to chop hooked up storage considerably on all affected clusters:

On the identical time, it serves 1000’s of reads per second from chilly tier storage in any given area:

Cassandra Chilly Tier improves p90 latency by ~30% over the earlier chilly storage implementation, because of extra environment friendly S3 entry patterns and smaller learn sizes, whereas retaining p99 latency below ~500 ms. Cassandra already handles bursts of chilly reads gracefully by way of its chunk cache, so this design preserves that very same resilience for the S3-backed chilly tier:

Acknowledgements: Particular because of our gorgeous colleagues who contributed to the TimeSeries Abstraction’s success and collaborated with us on the chilly storage options: Kaidan Fullerton, Matt Lehman, Sumanth Pasupuleti, Carlos Matias Herrero, Nick Pavlakis, Bryan Keller, Francesco Di Chiara
TimeSeries Tiered Storage Journey: Kafka/Flink Streams to Native Cassandra Chilly Reads was initially revealed in Netflix TechBlog on Medium, the place persons are persevering with the dialog by highlighting and responding to this story.