Samuel Yeboah, Francesco Di Chiara and Mingliang Liu
At the moment, Netflix runs two Flink autoscalers. That’s precisely yet another than we wish. We constructed the primary one in-house years in the past, when there was no mature possibility suited to our platform. The second got here from the Apache Flink group, and it might scale workloads our homegrown system was by no means designed for. We now run each in manufacturing and are steadily converging on the open-source one. Alongside the way in which we discovered some exhausting classes about metrics, value, and the true value of sustaining infrastructure you can as a substitute undertake, and we hope they’re helpful whether or not you run a handful of Flink jobs or tens of 1000’s.
Why autoscaling isn’t non-obligatory at our scale
Netflix has run stream processing on Apache Flink since 2017. As of 2026 we function greater than 30,000 Flink jobs throughout a number of AWS areas. Most usually are not deployed by hand; they’re generated by our managed platform Knowledge Mesh, so the vast majority of customers by no means contact a Flink job straight. A smaller however rising set are customized jobs, constructed and operated by groups throughout the corporate to be used instances like personalization, Advertisements, and Stay occasions. They vary from single-operator jobs that shuttle data between Kafka matters to stateful pipelines with branches, joins, and terabytes of state, and their load swings with day by day cycles, launches, and regional failovers.
Provisioning each a type of jobs for its peak is wasteful; provisioning for the typical causes lag throughout surges. And in our platform a scaling motion isn’t free: by default it means taking a savepoint, stopping the job gracefully, and restarting it on the new dimension, which for a big stateful job can take minutes. That leaves a genuinely exhausting query: how do you give every job the sources it wants, when it wants them, and not using a human within the loop and with out breaking something?
The primary autoscaler: watching from outdoors
Our first reply, constructed round 2019, was an autoscaler formed like a stream-processing job. It ran on Mantis, consuming a dwell feed of cluster-level metrics from Atlas, our telemetry platform, together with CPU, community, Kafka lag, input-rate, and consume-rate alerts for each job. The scaler mixed lag-derived catch-up time, CPU/community utilization thresholds, noticed efficiency historical past, and regression over current enter price to resolve when to scale up or whether or not a smaller cluster might deal with the lookahead window. As a result of the autoscaler operates independently of the Flink platform, it stays unaffected by points inside Flink itself. Constructing it as a streaming job additionally made it straightforward to scale. Every autoscaler node dealt with the metrics for a subset of Flink jobs, and we by no means needed to write customized sharding or coordination logic to maintain up with a rising Flink fleet. It reliably minimize useful resource utilization by 25–45% throughout 1000’s of managed pipelines. Verify our earlier speak at Flink Ahead 2020.
However watching from outdoors has a ceiling. The system reasoned about an entire cluster via coarse container metrics, and it scaled a single knob, the overall TaskManager rely, so each operator in a job moved collectively. That match the straightforward, single-operator pipelines it was constructed for, however not the multi-operator, stateful DAGs that groups had been more and more bringing to us for Advertisements, suggestions, and video games. These had been precisely the roles it couldn’t cause about, and supporting every new case meant extra customized logic somewhat than any common functionality.
The autoscaler is simply nearly as good because the metrics served by exterior programs beneath it. These metrics might miss actual hassle: a job might be fully busy with none of it displaying up as CPU utilization, leaving the job caught in a degraded state the scaler had no solution to see. Not too long ago a networking migration quietly modified how some site visitors was reported, and a subset of the Atlas metrics the scaler relied on stopped capturing every part precisely. The hole stayed invisible till it surfaced in manufacturing a lot later.
It was time to rethink construct versus purchase.
The second autoscaler: reasoning from inside
After we began, the Flink group had no mature autoscaler to supply. By the point we re-evaluated, it did: the Apache Flink Autoscaler. As a substitute of watching containers from outdoors, it causes from contained in the job.
Its key thought is to estimate every operator’s true processing price (TPR): the throughput it might maintain if it had been absolutely busy. Flink studies, per subtask, the fraction of every second spent doing precise work, separate from time spent backpressured or idle. Dividing noticed throughput by that busy fraction extrapolates capability to full utilization: an operator dealing with 700 data/sec whereas busy 70% of the time has a TPR of 700 / 0.7 = 1,000 data/sec. Ranging from the sources, the autoscaler walks the job graph and makes use of every operator’s TPR, its enter/output ratios, and a goal utilization to compute the parallelism each vertex wants in order that no operator turns into the bottleneck, somewhat than resizing the entire cluster as a unit.
The 2 approaches make a special contract, summarized beneath.
The decisive distinction for us is the final two rows: the OSS autoscaler can scale precisely the stateful, multi-operator jobs our homegrown system couldn’t, and it lets every job carry its personal configuration — stabilization durations, thresholds, and different scaling habits tuned to the workload. That made it the pure match for the customized jobs groups had been scaling by hand.
Making it work at Netflix scale
Adopting the algorithm was easy; the group had executed the exhausting half. The work for us was operating it reliably throughout our personal jobs, and that is the place our system differs most from the inventory open-source deployment.
Firstly, the OSS autoscaler was initially architected to reside throughout the Kubernetes Operator for Flink, however our Flink platform runs by itself management aircraft, not that operator (see our earlier speak at Present Convention 2024). The group later made a unbelievable resolution to maintain the core logic as a standalone library. They refactored 4 generic interfaces that made it straightforward to plug straight into our inner ecosystem: a context carrying job metadata and REST API information, a state retailer, an occasion handler, and a realizer that applies scaling selections.
That service is a Spring Boot utility whose orchestration runs on Temporal, the sturdy workflow engine. An orchestrator workflow polls our Flink management aircraft about as soon as a minute for the roles with autoscaling enabled, and begins one long-running workflow per job. Every per-job workflow pulls that job’s per-vertex metrics from its Flink JobManager, runs the OSS analysis algorithm, and, when a scaling resolution outcomes, fingers it to a realizer that actuates the change via our Flink management aircraft.
The workflow-per-job design was a direct response to ache. We first ran evaluations in a single batch loop over the entire set of jobs, and it was fragile: one gradual or misbehaving job might stall metric assortment and scaling for each job behind it. Giving every job its personal sturdy workflow remoted that blast radius, so a single problematic job now fails and retries by itself, and the runtime scales out as we onboard extra jobs.
Secondly, three engineering gaps stood between “works in group” and “works at Netflix scale”:
- Metric assortment at excessive parallelism. On massive jobs, pulling metrics from the JobManager turned a bottleneck, and a part of the trigger was in Flink’s runtime. To handle that, we modified the JobManager to cache transient metric names and clear them up as soon as as a substitute of rescanning on each fetch, and we added server-side filtering so the autoscaler asks just for the metrics it wants. This let the autoscaler work on jobs as much as 3,000 Flink subtasks, the place it had beforehand struggled above roughly 1,000. These are in our inner fork of Flink launch, whereas some are contributed upstream resembling FLINK-36172.
- Preserving ahead chaining. Two separate vertices joined by a ahead connection should run on the identical parallelism, as a result of data are handed over in reminiscence on a set native channel. Scale considered one of them alone and Flink doesn’t fail; it silently converts that edge right into a community shuffle. Our fork detects forward-connected subgraphs and scales every as a unit.
- Respecting sink limits. Some sinks have finite write capability, so we added detection for async-sink backpressure (additionally a fork change) to maintain the autoscaler from scaling a job up right into a sink that can’t take up extra.
Earlier than it actuates something, the realizer runs a set of security checks. For instance, it refuses to scale a job down in a area being evacuated throughout a company-wide area failover. It additionally verifies there may be sufficient disk for the brand new cluster to carry the job’s checkpoint state, and it provides a small standby buffer for bigger clusters.
The street to 1 autoscaler
Final yr, the OSS-based autoscaler achieved common availability for customized jobs at Netflix, yielding promising preliminary outcomes. For example, our consumer telemetry and logging group achieved a 58% discount in its annualized Flink compute expenditures, saving roughly $1.1 million yearly. This effectivity is pushed by three key components. First, whereas static provisioning should all the time account for peak masses, autoscaling dynamically adapts to day by day cycles, capturing the drop in site visitors throughout nights and weekends in comparison with weekday peaks. Second, somewhat than counting on groups to manually optimize sources following efficiency enhancements or post-holiday slowdowns, the autoscaler frequently adjusts capability. Lastly, adopting uniform container dimensions allows superior bin-packing and extra granular scaling increments.
Moreover, cutting down too eagerly is its personal entice. Minimize too deep and CPU saturates, lag spikes, and the system can not react immediately as a result of its metric window and stabilization interval need to rebuild after every restart. We now run a goal utilization of 0.45, beneath the group default of 0.7, intentionally buying and selling slightly effectivity for stability. Fewer and calmer rescales are definitely worth the marginal value for big stateful jobs.
Whereas our scaler supplies fine-grained alerts and vertex-level resolution models for stateful DAGs, quick rescaling nonetheless closely is determined by Flink Core’s state restoration efficiency. At the moment, the most important remaining value in scaling a stateful job isn’t the scaler’s logic — it’s the restart and state restoration course of itself. Flink 2 addresses this via its disaggregated state structure, preserving state in exterior storage somewhat than on native disk, which might sharply cut back how a lot a rescale or restoration is determined by whole state dimension. Having began supporting Flink 2.2 at Netflix, we plan on experimenting with this new state backend to see if it might assist get rid of state restoration bottlenecks when scaling massive stateful jobs.
Wanting forward, we goal emigrate all inner scaler use instances onto the brand new one based mostly on OSS autoscaler to simplify our operational floor space.
Key Takeaways
Alongside the way in which, three classes that generalize past Flink:
- Metric selection issues greater than algorithm sophistication. Our most helpful debugging was not often concerning the scaling math; it was about which sign to belief most. Perceive your metrics earlier than you tune your algorithm.
- Set wise defaults, however depart room to tune. Our managed jobs are related sufficient that one good default covers most of them untouched, which is the purpose of a platform. However forcing a single configuration on each job punishes those that don’t match, so we pair defaults with per-job overrides and intentionally disguise the knobs that want deep experience. Most groups ought to by no means have to consider the autoscaler.
- Undertake, then prolong. We constructed in-house as a result of in 2019 nothing mature match our platform. When a robust group mission appeared, the correct transfer was neither to defend our funding ceaselessly nor to tear it out in a single day, however to undertake it for brand spanking new workloads, contribute fixes again, and plan a deliberate migration.
Due to the Flink and Knowledge Mesh groups for the control-plane adjustments this work relied on, to the Temporal group and our early pilot groups, and to the Apache Flink autoscaler maintainers whose basis we constructed on. Particular because of Andy Zhang, Calvin Cheung, Daniel Trager, Guil Pires, Mark Cho, Matthew Kornitsky, Nikhil Sulegaon, Sujay Jain, and Tom Lee.
A Story of Two Flink Autoscalers was initially revealed in Netflix TechBlog on Medium, the place persons are persevering with the dialog by highlighting and responding to this story.
