By Yogesh Nagarur
Introduction
In Half 1 of this sequence, we shared how Netflix constructed the Adverts Occasion pipeline and approached it as a system design drawback: changing fragmented pipelines with a centralized assortment, enrichment, and a normal knowledge contract. The end result was Adverts Occasion Writer, the system that collects advert telemetry from shopper units, enriches it with the context of the advert that was truly served, and publishes a single, unified stream of advert occasions to each downstream client: frequency capping, metrics, billing, measurement companions, and reporting. Any member viewing or account particulars that transfer via this pipeline end-to-end are encrypted and dealt with in accordance with Netflix’s strict privateness and safety practices to safeguard member PII.
If advert serving is the “mind” and advert occasions are the “heartbeats” that feed it, then enrichment is what provides every heartbeat its which means, turning a naked “[Somebody] watched an advert to completion” sign into “this marketing campaign, this artistic, at this worth, for this break.” Half 1 described that enrichment step as a single field within the structure. This submit is about what was inside that field, and why we rebuilt it.
Half 1 ended with a have a look at what was on the horizon: advertisements on dwell streams, de-duplication, and richer knowledge indicators. A single mission delivered all three by rebuilding the Adverts Occasion Writer round stateful stream processing. We changed a database lookup with a real-time streaming take part Apache Flink, which let the system scale for Reside, took a dependency off the essential path, and changed the engine of a high traffic pipeline with out downtime.
Let’s dive in.
A fast recap: enrichment and the Metadata Registry
When an advert is served, the advert server is aware of every part about it: the marketing campaign, the artistic, the worth of the impression, the concentrating on context, and the monitoring URLs that measurement companions count on us to fireplace. The shopper gadget, against this, is aware of nearly nothing: simply an opaque token and when to ship every occasion. Enrichment is the act of reuniting the two.
Within the structure from Half 1, that reunion occurred via the Adverts Metadata Registry, a Key-Worth retailer:
- When the advert server decided, it wrote the metadata for that impression into the registry.
- It encoded solely reference identifiers into the opaque token returned to the gadget, retaining the token small.
- Later, when the shopper fired an occasion carrying that token, the Adverts Occasion Writer decoded the token, appeared up the metadata within the registry, and stitched the 2 collectively into an enriched advert occasion.
This design served us properly. The registry was globally replicated, which gave us a helpful property nearly free of charge: an occasion arriving in any area might discover its metadata no matter the place the advert was determined. It future-proofed the token towards ever-growing metadata, and it carried us via the primary part of Netflix’s promoting development.

However as we appeared forward, particularly to advertisements on dwell streams, the bounds of this design turned clear.
Why we wanted to evolve
The forcing perform was Reside. Netflix’s dwell slate has grown quickly, from NFL video games to blockbuster struggle nights to the FIFA Ladies’s World Cup 2027, and viewership retains rising. That site visitors appears to be like nothing like Subscription Video on Demand (SVOD): a single occasion can surge towards tens of thousands and thousands of concurrent viewers inside minutes, then drain simply as quick when it ends. Constructing the advert pipeline for Reside meant constructing for that trajectory.
To see why the previous design couldn’t get there, let’s put some numbers on it. When Netflix streamed the Jake Paul vs. Mike Tyson struggle to greater than 100 million viewers, a single advert break at that scale would generate an unlimited quantity of advert occasions. Serving these advertisements within the previous design would have meant writing metadata for every of them to the Metadata Registry and ready for acknowledgment earlier than the advertisements might serve: thousands and thousands of must-succeed database writes per second on the ad-serving essential path. That sample created three core issues at Reside scale:
- It was a scaling bottleneck. Each advert and each advert break is a write. Multiply that by tens of thousands and thousands of concurrent viewers, preserve it non-lossy (a dropped write is a dropped advert, and misplaced income), add Reside’s spiky on/off sample, and the write load turns into monumental. However you can’t safely scale the registry up as soon as the occasion begins. Including nodes reshuffles knowledge throughout the cluster to rebalance it, and doing that in the course of a dwell occasion might gradual and even fail the very writes you’re including capability for. That leaves one protected choice: pre-scaling for the height prematurely and hoping the forecast is correct.
- Fewer dependencies on the essential path. Serving an advert meant a synchronous, must-succeed write to the registry, adopted by a anticipate its acknowledgment. Each system on the essential path is yet one more factor that may cease an advert from serving, and the metadata registry had no fallback, so an unhealthy registry meant serving stopped. Taking arduous dependencies off that path was the purpose, and the registry was the largest one we might take away.
- It put occasion freshness in danger. If writing slowed down below load, or lookups timed out, occasions spilled into retries and dead-letter queues, and enriched advert occasions had been delivered late. Some techniques is perhaps high-quality, however others, reminiscent of frequency capping, can’t. It wants occasions below a strict SLA to determine whether or not the following chosen advert can run with out breaking insurance policies.
So we requested a easy query: the advert server already is aware of and logs every part in regards to the advert it served. Does it actually need to jot down that to a database on the new path? Can we match that towards the downstream Kafka stream of shopper occasions, which is all the time there, after the reality?
The write leaves the essential path solely; no shared datastore outage can cease serving, and enrichment turns into one thing we scale elastically with Reside slightly than a cluster we babysit.
Matching two streams after the actual fact is a streaming be part of, and that’s the place Flink comes in.
Why Flink
We selected Apache Flink as a result of near-real-time, stateful stream processing is precisely what it’s constructed for, and since it has deep, production-tested platform assist at Netflix. Simply as necessary, we already knew the way to run it. As Half 1 famous, our Adverts Sessionizer is a big stateful Flink job, so working one at scale, tuning state, checkpoints, and restoration below manufacturing load, was acquainted floor for the group. Enrichment on Flink was constructed on that have slightly than ranging from scratch.
The brand new structure
At its core, the brand new Adverts Occasion Writer is constructed as an Apache Flink streaming job. Every occasion strikes via a handful of levels:
- Normalize. Consumer occasions arrive from a number of sources, together with SVOD, dwell, playback, and embedded advert breaks, every with a barely completely different form. We first normalize them into a standard type.
- Re-route. Consumer occasions and response occasions might seem in several areas; we examine and reroute them to the right job-processing area for a profitable be part of.
- Be part of. The center of the job: we match every shopper occasion to the ad-serving response for a similar impression, enriching it with the complete serving context utilizing a KeyedCoProcessFunction
- Deduplicate. Income-bearing occasions are de-duplicated, so nothing is counted twice.
- Publish. We extract measurement beacons for our companions and route the enriched advert occasion to its downstream customers.

The be part of is the fascinating half. Consumer occasions and advert serving responses arrive on separate streams at completely different occasions, so the job holds latest responses in reminiscence and matches occasions towards them over a 60-minute window. Three issues can occur:
- The response is already there when an occasion arrives → we enrich it on the spot.
- The occasion arrives first → we maintain it for a configuration length (presently set to 60 minutes) till the matching response reveals up, then enrich it.
- No response arrives throughout the window → we mark the occasion as unmatched for later restoration.
We constructed the be part of as a customized Flink operator (a KeyedCoProcessFunction) slightly than a built-in windowed be part of. We don’t want a windowed be part of ordering and time-window semantics, since downstream customers deal with ordering; we merely match every shopper occasion to its response by key. Proudly owning the operator at its lowest API type lets us management the matching state straight, together with how lengthy we maintain it and what we do with an occasion that by no means matches. This permits us to tune retention per stream or persist unmatched occasions to a sturdy retailer later with out rewriting the be part of. Our purpose is all the time to construct a powerful base to reap the advantages later as we scale.
To make that concrete, image a member watching a dwell sport. When the advert server decides which advert to indicate, it logs its response, and some seconds later, the member’s gadget fires a “first quartile” occasion carrying the opaque token. Each arrive on the job keyed to the identical impression, and since the response is already ready, the occasion is enriched on the spot with its marketing campaign, artistic, worth, and monitoring URLs, then revealed downstream in close to real-time.
Why 60 minutes? Nearly all occasions, roughly 99.999% of them, attain the job properly earlier than that, so extending the window additional would solely develop the state retailer and lengthen checkpoints for a vanishingly small achieve. The remaining 0.001% nonetheless issues for exact billing and reporting, and we get better these occasions by way of a separate path described later on.
Cross-region: matching occasions the place their knowledge lives
The Metadata Registry dealt with cross-region enrichment for us as a result of it was constructed on a globally replicated key-value retailer: a shopper occasion might land in any area and nonetheless discover its advert’s metadata. International replication was a characteristic the platform supplied (see Netflix’s Key-Worth Knowledge Abstraction Layer), so serving an advert in a single area and enriching its occasions in one other was merely not one thing we needed to suppose about.
A streaming be part of doesn’t include that comfort, and right here the 2 sides of the be part of dwell in other places. The advert serving response stays within the area the place the advert was served. The shopper occasion, although, is fired by the gadget, and the gadget may be anyplace: a member would possibly begin an advert on residence wifi and swap to cell knowledge partway via, and that community change can route the occasion to a special area than the one which served the advert. So a shopper occasion is routed to a area that doesn’t have the matching serving response.
The tempting repair is to stream each area’s serving responses into each different area so that every area can match any shopper occasion domestically. We rejected it: that will imply each area storing each serving response worldwide, the overwhelming majority of which can by no means be used within the be part of, which is wasteful in each value and compute.
As an alternative, we deal with this as a routing drawback and route the shopper occasion slightly than replicating the response. We encode the serving area into the advert token, so when a shopper occasion arrives within the unsuitable area, we ahead it to the area that owns its response. For instance, an advert is served in us-east-2, the member’s gadget switches from wifi to cell knowledge mid-ad, and the ensuing occasion arrives in us-east-1. That area reads the token, sees the advert belongs to eu-west-1, and forwards the occasion again there, the place its serving response is ready. Solely the small trickle of genuinely cross-region occasions pays for the additional hop.
Catching the lengthy tail of late-arriving occasions
Sixty minutes covers the overwhelming majority of occasions, however not all of them. A tool might have been offline, an advert paused, or an engagement occasion delayed by hours. Slightly than drop these stragglers, we set them apart and get better them via a companion path: an hourly Spark batch job, orchestrated by Maestro, Netflix’s workflow orchestrator, re-pairs every straggler with its serving response from our knowledge warehouse (which has a far longer retention than our in-stream window) and publishes it via the identical downstream sinks. This retains the streaming job’s reminiscence bounded whereas nonetheless catching the lengthy tail.
This additionally lets us assist occasion varieties that arrive properly after the 60-minute window, together with sure tracking-related occasions, whereas retaining the door open to extending the system to further occasion varieties down the road and bettering our match fee as wanted. We additionally designed this offline job with reuse in thoughts: it may replay dead-letter queues and backfill historic knowledge, giving us a single mechanism for each long-tail restoration and operational restoration.
What was arduous
Past the structure itself, three issues took actual work, and collectively they’re what made the pipeline steady, appropriate, and protected to change on.
Preserving the core lean.
Working a stateful streaming be part of at live-event scale took actual tuning, and most of our hard-won classes come again to 1 concept: the job’s solely job is matching occasions. Anything we had been tempted to bolt on, reminiscent of exterior lookups or incidental side-processing, was higher moved off the new path, or it will definitely confirmed up as instability below load. Getting this proper was the distinction between a job that comfortably stored tempo with dwell site visitors and one which fell behind. It’s unglamorous work, and it’s many of the work.
De-duplicating Occasions
Duplicates are a reality of life right here in an occasion processing pipeline. A shopper might retry an occasion, and our upstream supply is at-least-once in order that the identical occasion can arrive greater than as soon as. For revenue-bearing occasions, those that feed billing, nothing ought to be counted twice.
The answer right here is a straightforward and idempotent key. For every occasion, we compute a steady ID by hashing a number of fields that uniquely determine it, such because the advert ID, the occasion sort, and, for repeatable occasions, an offset or timestamp. The identical underlying occasion all the time hashes to the identical ID, so if now we have seen an ID earlier than, we drop the repeat.
What modified with Flink is the place we bear in mind these IDs. The earlier design stored them in an exterior key-value retailer and did a check-and-write over the community for each occasion; Flink lets us preserve that state native to the job as a substitute. We key the stream by occasion ID, which routes each copy of a given occasion to the identical place, and a neighborhood state retailer with a brief time-to-live remembers the IDs now we have not too long ago seen. Keying by ID additionally removes the race the exterior retailer needed to guard towards, since a given ID is dealt with in just one place at a time, with no community hop on the new path. A number of enhancements right here alone.
We’re deliberate in regards to the assure: at-least-once supply with best-effort de-duplication, not exactly-once. In-stream dedup removes the overwhelming majority of duplicates inside its window. Nevertheless, a number of can nonetheless slip previous, as an illustration, via a cross-region reroute, a late-arriving restoration, or a restart that replays latest occasions. So we additionally carry the occasion ID via to the enriched advert occasion itself, which lets customers who want actual counts, reminiscent of billing, run their very own idempotent de-duplication over their very own, longer home windows.
Manufacturing Rollout
This was essentially the most difficult a part of the mission. For a system that feeds billing and frequency capping, “belief me, the brand new one is appropriate” shouldn’t be adequate, so we ran the Flink pipeline alongside the legacy one and made it earn its site visitors.
Each pipelines constantly processed the complete occasion stream, permitting us to check them in actual time: we revealed their outputs aspect by aspect. We ran automated audits that alerted us when the 2 drifted aside. These audits earned their preserve, surfacing delicate discrepancies we’d by no means have caught from dashboards alone.
How do we all know the enrichment logic between the 2 techniques was equal for the audits to make sense? Early in improvement, we extracted that logic right into a shared module that each the legacy service and the Flink job rely on, so any change to how an occasion is enriched is utilized to each on the similar time. A failure within the audits, subsequently, pointed to an actual distinction within the pipeline mechanics.
Cutover was a separate query from comparability, and that is the half that warrants precision. We couldn’t merely ship a random share of occasions to every pipeline, as a result of the identical occasion touchdown in each could be double-counted downstream. As an alternative, the break up was deterministic: a steady key on every occasion determined which pipeline was chargeable for publishing it, so each occasion was revealed precisely as soon as, though each pipelines noticed all of them. Elevating the brink moved occasions from the legacy pipeline to Flink a slice at a time, and we might dial it again immediately if something appeared off.

By the point we totally lower over, the 2 pipelines agreed on properly over 99.99% of occasions, and we rolled it out area by area slightly than suddenly. Then got here the satisfying half: we turned off the advert server’s writes to the Metadata Registry, and decommissioned the legacy service and the registry itself. The one level of failure on the ad-serving essential path is solely gone. One of many oldest advert serving parts at Netflix was deleted.
Working the Flink platform
Working a big stateful streaming job in manufacturing is its personal job, and we didn’t construct that muscle alone. We leaned closely on Netflix’s Flink platform group. Early on, the job’s checkpoints had been gradual and heavy. The primary checkpoint specifically took a very long time, and we had been holding extra state in reminiscence than we wanted. Working with the platform group, we trimmed the state, lowered checkpoint durations, and validated the good points in load exams. That’s what retains the job from stalling throughout Reside bursts.
The identical group additionally gave us operational scaffolding we didn’t should spend money on, together with steady supply and canary deployments, so we might ship and roll out the Flink job with the identical guardrails and confidence as the remainder of our manufacturing techniques. Netflix operates greater than 25,000 Flink jobs (see Constructing a Scalable Flink Platform at Netflix), and that maturity is what now we have to construct on. We layered on customized canary metrics that catch points earlier than a launch, CI/CD that lets us stagger rollouts, and a deploy stream that retains the previous job operating till the brand new one has detected wholesome checkpoints, so we by no means destroy a working job earlier than the brand new job provides us the inexperienced mild.
The outcomes
- Enrichment is off the essential path for advert serving. The advert server not performs a synchronous, must-succeed database write to serve an advert. It logs and strikes on, and a complete class of “the lookup retailer is down, so advert serving is down” failures not exists.
- Fewer techniques, fewer schemas, fewer hops. We retired each the legacy service and the Metadata Registry.
- Constructed for Reside, and constantly examined for it. An advert break is what this service is designed to deal with, so throughout our journey, we additionally ran large-scale load exams to simulate real-life site visitors. In our newest load take a look at, a single AWS area dealt with one advert break for a big concurrent viewers, processing a whole lot of 1000’s of occasions per second, becoming a member of the massive majority of them towards their serving responses in actual time, and outputting enriched advert occasions at a comparable fee, all with negligible timeouts and errors. We take a look at this all through improvement and preserve doing in order dwell audiences develop, and we aren’t on the ceiling but.
- We delivered the Half 1 roadmap. Reside-stream advert occasions, in-stream de-duplication, and richer real-time enrichment all shipped on this basis.
Key takeaways
- Take the database off the new path when you possibly can. Our greatest win wasn’t uncooked efficiency — it was blast radius. Buying and selling a synchronous write-and-wait for a fire-and-forget log plus a downstream be part of eliminated a complete failure mode from advert serving.
- Match occasions the place their knowledge lives. Actual-time joins beat international replication by routing occasions to their knowledge slightly than copying the information in all places.
- Earn the cutover. Operating each techniques in parallel and diffing their output on dwell site visitors was slower than a flag-day swap — and it’s the one motive we belief the end result.
What’s subsequent
We have now an thrilling record of initiatives on the horizon: scaling the dwell path for the following tier of concurrent audiences, bringing the remaining delayed engagement occasions totally into the streaming world, decreasing the Flink state measurement we have to handle, engaged on bringing much more resilience to the eventing processing pipeline, and persevering with so as to add new occasion varieties and richer indicators for our downstream customers.
Re-architecting the engine of a operating, revenue-critical pipeline in place was an actual multi-team effort. If constructing real-time techniques at this sort of scale appears like your factor, Adverts Engineering is hiring.
Acknowledgements
This was a multi-team effort throughout Netflix. Thanks to everybody who designed, constructed, validated, and operated the brand new pipeline:
Adverts Engineering: Yogesh Nagarur, Kinesh Satiya, Jeet Mehta, Nitin Saini, Persis Shah, Guangyu Chen, Jimmy Liu, Riley Berton, Simon Spencer, Aadesh Patel
Adverts Knowledge Engineering: Eugene Yusim, Brandon Nicklas, Indu Narayan
Adverts SRE: Max Michaels, Kevin Chung, Carlos Avalos, Brian Christie
Knowledge Motion: Mark Cho, Herbert Wang, Shraddha Ladda, Nikhil Sulegaon, James Lubin, Daniel Trager
Behind the Scenes: Evolving Netflix’s Adverts Occasion Pipeline for Reside — Half II was initially revealed in Netflix TechBlog on Medium, the place persons are persevering with the dialog by highlighting and responding to this story.