By: Brett Axler, Casper Choffat, and Alo Lowry
Within the three years since our first Reside present, Chris Rock: Selective Outrage, we’ve witnessed an unimaginable growth of our reside content material slate and the reside operations that assist it. From modest beginnings of streaming only one present monthly, we at the moment are able to streaming over 9 reveals in a single day, reaching tens of thousands and thousands of concurrent members. This put up pulls again the curtain on the Reside Operations groups that allow this fast scale.
Humble Beginnings
In March 2023, the engineers who constructed Netflix’s first reside streaming pipeline additionally operated it. There was no devoted operations workforce or formal command heart. All of our incident response playbooks had been written for SVOD, and SLAs weren’t designed for the velocity of reside. For the primary reside reveals on the platform, the engineers who designed what’s described in earlier elements of this collection monitored dashboards on laptops, coordinated over Slack, and troubleshot in actual time whereas thousands and thousands of members watched.
The bodily setup matched the operational workflows: improvised. Short-term management rooms had been put collectively in convention rooms. For bigger occasions, Netflix rented third-party broadcast amenities, {hardware} management panels, multiviewers, and communication panels — the type of infrastructure that established broadcast networks had constructed over many years. Each present was a workforce effort. Engineers and management in any respect ranges had been concerned in each occasion. Every reside present, no matter measurement, was a large effort to launch.
Final month, in March 2026, Netflix streamed the World Baseball Traditional reside to members in Japan. 47 matches over two weeks, with peak concurrent viewership exceeding 9.6 million accounts for a single sport, operations working 24/7 from everlasting amenities in Los Gatos and Los Angeles, with worldwide protection extending to Tokyo. In March alone, Netflix launched roughly 70 reside occasions. That’s three occasions shy of the full quantity Netflix streamed reside in all of 2024. The technical techniques that make this doable have been coated intimately throughout this collection. What hasn’t been instructed is the operational story: the individuals, procedures, and amenities Netflix constructed to run these techniques in actual time, beneath strain, with no capability to pause or roll again.
The Structure of Reside Operations
The Structure of Reside Operations: Evolving the Broadcast Operations Heart
When a expertise firm transitions into reside broadcasting, it faces a singular problem: mixing conventional broadcast tv practices with massive-scale live-streaming engineering. On the coronary heart of this intersection is the Broadcast Operations Heart (BOC).
The BOC serves because the important “cockpit” for reside occasions. It’s the bodily command heart the place a completely produced video feed is acquired instantly from a stadium or venue after which handed off to the reside streaming infrastructure. Every little thing from sign ingest, inspection, and conditioning to closed-captioning, graphics insertion, and advert administration occurs inside these partitions. By using a hub-and-spoke mannequin with extremely redundant architectures, corresponding to twin web circuits and SMPTE 2022–7 seamless switching applied sciences, the BOC replaces direct, susceptible paths from the venue to the reside streaming pipeline, making every reside occasion extremely repeatable and much much less depending on the quirks of particular person occasion places.
Securing the Sign: Reliability from the Venue Earlier than the BOC can work its magic, we’ve to ensure the video and audio feeds really survive the journey from the manufacturing website to our facility. To make sure absolute reliability from the venue, Netflix enforces strict specs for reside sign contribution.
For any show-critical feed, which means the first feed our members will watch reside, we require three utterly discrete transmission paths. We make the most of a strict hierarchy of permitted transmission strategies, prioritizing devoted video fiber and single-feed satellite tv for pc hyperlinks, adopted by devoted enterprise-grade web and sturdy SRT contribution techniques.
We don’t simply depend on redundant transport strains; we require full {hardware} redundancy out of the manufacturing truck itself. This consists of utilizing separate router line playing cards and discrete transmission {hardware} to forestall any single level of failure. Moreover, each single piece of transmission {hardware} on the venue should be powered by two discrete energy sources, protected by uninterruptible energy provide (UPS) batteries, and surge-conditioned.
Lastly, earlier than we ever go reside to thousands and thousands of viewers, our operators execute exhaustive “FACS/FAX” (amenities checks) testing throughout rehearsals and earlier than each present. This entails working specialised Audio/Video sync checks, latency checks, and high quality checks to ensure excellent audio and video synchronization, validating closed captions, and touring the backup switcher inputs.
Constructing the Human Infrastructure: Constructing the human operational mannequin to run a facility just like the BOC didn’t occur in a single day. For a platform scaling from its very first reside comedy particular to streaming over 400 world occasions a 12 months, the operational technique needed to endure a large, multi-year evolution.
Part 1: The “All-Fingers” Engineering Period. Within the earliest days of reside streaming, there was no devoted operations workforce or formal broadcast operations heart. The software program engineers who wrote the code and constructed the live-streaming infrastructure had been the identical individuals manually working the occasions on launch evening. Each present was an “all-hands-on-deck” state of affairs. Whereas this uncooked, startup-style strategy labored for preliminary milestones, having core builders manually arrange and tear down software program configurations for each single broadcast was essentially incapable of scaling.
Part 2: The Shift to Specialised Engineering (SOEs and BOEs). To separate occasion execution from core software program improvement, the operational mannequin matured to introduce specialised engineering groups. First, the Streaming Operations Engineering (SOE) workforce was established. These are extremely expert streaming engineers whose sole focus is to configure the total occasion on the reside pipeline and assist it throughout the broadcast. By having SOEs act as the primary line of escalation, the core software program builders had been freed as much as give attention to constructing new live-streaming pipeline options.
Nonetheless, because the bodily broadcast amenities grew, it grew to become clear that supporting the streaming pipeline wasn’t sufficient; the bodily broadcast {hardware} and facility workflows wanted devoted oversight too. To resolve this, Broadcast Operations Engineers (BOEs) had been launched to work alongside the SOEs. The BOE acts as the first escalation level for all bodily broadcast facility and {hardware} points, overseeing the operation of all reveals throughout a given shift.
Part 3: The “Co-Pilot” Management Room Mannequin. With specialised engineers in place to deal with the deep technical infrastructure, the day-to-day operation of the particular video and audio feeds was handed over to devoted operators. Initially, the Broadcast Management Rooms had been structured very like an airplane cockpit.
This strategy utilized a “first and second captain” workflow, pairing two Broadcast Management Operators (BCOs) collectively to run a single occasion, functioning precisely like a pilot and co-pilot. This collaborative mannequin allowed for intense focus and high-quality execution, making it the best setup for working only one or two reside occasions per day. Nonetheless, because the ambition grew to stream as much as 10 concurrent occasions a day for enormous world tournaments, a 1:1 scale of pairing operators merely required an excessive amount of house and manpower. A brand new mannequin needed to be adopted.
Part 4: The Transmission Operations Heart (TOC) Fleet Mannequin. To handle high-density occasion days and steady match protection, the workflow was utterly reimagined with the launch of the Transmission Operations Heart (TOC) mannequin. Somewhat than treating each reside broadcast as an remoted launch in its personal room, the TOC treats reside occasions like a fleet. It centralizes operations and distinctly separates the normal broadcast capabilities from the streaming capabilities to maximise human effectivity.
The TOC mannequin divides the labor throughout three extremely specialised, tiered roles:
- Transmission Management Operator (TCO): The TCO is accountable for managing all inbound indicators arriving from the occasion venues, corresponding to fiber optic, SRT, and satellite tv for pc feeds. They guarantee these incoming feeds meet strict high quality, latency, and operational thresholds. Because of centralized dashboarding, a single TCO can handle as much as 5 occasions concurrently.
- Streaming Management Operator (SCO): Whereas the TCO handles what comes in, the SCO manages what goes out. They oversee all outbound feeds, together with the streams heading to the reside streaming pipeline and any syndication feeds despatched to 3rd events for business distribution. Just like the TCOs, SCOs can handle as much as 5 occasions concurrently.
- Broadcast Management Operator (BCO): With the inbound and outbound transmission mechanics dealt with by the broader TOC, the BCO is ready to focus completely on the artistic and qualitative execution of the occasion. Working on a strict 1:1 ratio (one operator per occasion), the BCO seamlessly switches between backup inbound feeds if a difficulty arises, ensures audio and video stay in excellent synchronization, and performs rigorous high quality management. In addition they monitor important metadata, corresponding to closed captions and digital ad-insertion messages (SCTE), proper earlier than the ultimate polished feed is handed into the reside streaming pipeline.
The Large Guess Exception. Whereas the fleet-style TOC mannequin permits immense concurrency for each day programming, probably the most important, high-visibility occasions, like main vacation soccer video games, make the most of a specialised Large Guess Mannequin. For these flagship broadcasts, a complete Broadcast Operations Heart is devoted solely to a single occasion. This hyper-focused atmosphere strips away the multi-event ratios, offering operators with superior instrumentation and devoted facility engineers to make sure absolutely the highest degree of reliability for occasions the place failure is just not an choice.
