In-House LLM Serving at Netflix

By AI Platform’s Mannequin Runtime workforce and Inference workforce

Introduction

Most organizations eat LLMs by way of hosted APIs. Netflix went additional — we run the complete stack ourselves, from mannequin deployment by way of inference, inside our present manufacturing surroundings reasonably than a separate ML silo. A few of these choices weren’t apparent, and some revealed their trade-offs solely below manufacturing load.

This submit focuses on the alternatives the place options have been significantly thought of: engine choice, mannequin packaging, API floor design, deployment technique, and output constraints enforcement. The objective is to share not simply what was constructed, however why — and what manufacturing revealed that the design part didn’t anticipate.

Structure Overview

Member-scale ML at Netflix is fronted by a unified JVM-based serving system that handles the end-to-end circulation for downstream shoppers: routing and A/B take a look at logic, candidate era, characteristic fetching, inference, post-processing, and logging at every stage. Each real-time and cached batch paths are supported. Determine 1 reveals the 2 methods callers attain inference at present: the gRPC path by way of this serving system and a direct HTTP path utilized by newer LLM-driven purposes.

The place inference runs is determined by the mannequin. Small CPU fashions run in-process, avoiding remote-call overhead. Bigger fashions want GPUs — the serving system handles pre- and post-processing regionally however delegates inference to a distant service, Mannequin Scoring Service (MSS). MSS is the shared inference backend, supporting XGBoost, TensorFlow, PyTorch, and LLMs behind a unified interface, with NVIDIA Triton Inference Server beneath managing mannequin loading, batching, and GPU scheduling.

On prime of Triton sits a Java management airplane that handles deployment, versioning, well being checking, autoscaling, and multi-region rollout. Mannequin authors bundle their artifacts and configure the deployment; the management airplane provisions GPU cases, configures Triton, and orchestrates zero-downtime upgrades.

Press enter or click on to view picture in full dimension

Design Choices and Implementation

4 choices form this platform — engine, packaging, API floor, and rollout — introduced in dependency order, since every one constrains the following.

vLLM because the Paved-Path Engine

The platform was initially constructed on TensorRT-LLM, a performant inference engine on the time and already built-in with Triton — the compute backend in use inside MSS.

By summer season 2025, two issues had shifted: open-source engines had largely closed the efficiency hole with specialised stacks, and our workload combine had broadened to incorporate embedding era, prefill-only inference for rating and retrieval, autoregressive decoding, and customized fashions with non-trivial per-step constraint logic. We re-benchmarked in opposition to this combine and chosen vLLM as our paved-path engine on operational match:

• Hundreds customized mannequin architectures with out a multi-step compilation pipeline — sooner iteration on non-standard fashions.
• Extensibility hooks for customized decoding logic — crucial for the constrained-decoding work described later.
• Debuggability — simpler to examine failures and intermediate state than with a compiled engine in earlier TensorRT-LLM.
• Familiarity — many ML practitioners have been already utilizing vLLM in analysis, which reduce the research-to-production handoff value.

Integrating vLLM into Triton

With vLLM picked, the following determination was tips on how to bundle fashions for it. Triton helps two methods, and the selection has vital implications for maintainability — particularly, how tightly mannequin artifacts are coupled to frontend upgrades.

• Python backend. The writer defines specific enter/output tensor specs at packaging time. These specs are frozen within the artifact and should match what the third-party vendor’s frontend’s request builder expects, so each frontend improve that touches I/O specs requires a coordinated change to packaging code; in any other case, requests fail at runtime.
• vLLM backend. The artifact is only a JSON config pointing to the mannequin weights and tokenizer. Triton’s vLLM backend reads this config and generates I/O tensor specs dynamically at deployment time — the writer by no means defines them. Fashions and frontend evolve independently.

The vLLM backend is the architecturally right default. Two issues bit us in manufacturing:

• Triton/vLLM model mismatch. Triton’s vLLM backend is compiled in opposition to a particular vLLM API floor. When the 2 drift — for instance, Triton 25.09 importing vllm.engine.metrics, a module eliminated in vLLM 0.11.2 — the backend fails to load solely. The platform has to pin suitable variations when baking the service picture, and forestall mannequin authors from overriding the vLLM model at packaging time.
• Customized mannequin logic. The vLLM backend expects a regular HuggingFace-compatible mannequin and handles the complete inference lifecycle. Fashions needing customized preprocessing, postprocessing, or non-standard execution — ensemble pipelines, customized tokenization — should use the Python backend, which supplies full management over execute(). This escape hatch will seemingly stay crucial for a subset of fashions.

Ecosystem-Appropriate HTTP Frontend

With engine and packaging settled, the following query is how callers attain the system. A key design objective of our system was that LLM fashions ought to NOT be particular snowflakes. Each mannequin — XGBoost ensemble or large-scale LLMs — is scored by way of the identical gRPC name, so we reuse the identical consumer libraries, well being checking, and deployment pipelines. Provided that the OpenAI-compatible API interface has turn out to be the de facto interface for the LLM ecosystem — inference engines, orchestration frameworks, analysis instruments, and consumer libraries all converse it — so we expose the OpenAI-compatible API as a further frontend alongside gRPC.

The payoff reveals up within the experimentation-to-production path: graduating from a hosted mannequin to a fine-tuned self-hosted one — for high quality, latency, value, or knowledge privateness — is almost seamless. Similar API, minimal code modifications.

Behind the API, the implementation reuses NVIDIA’s Triton OpenAI-compatible frontend. It begins an embedded Triton server, wraps it in a TritonLLMEngine that converts request schemas into Triton inference requests, and serves responses by way of FastAPI. KServe HTTP/gRPC frontends are enabled alongside, so the identical Triton occasion stays accessible to the Java management airplane over gRPC. Adopting Triton’s frontend immediately uncovered one hole: response_format — accepted by the schema — was silently dropped earlier than reaching vLLM, so {that a} caller requesting JSON output proceeded with out guided decoding constraints and will obtain malformed JSON with no error surfaced by the platform. We git-subtreed and patched the frontend to translate response_format into vLLM’s guided decoding parameters at request time.

Deployment Methods

With API floor and engine in place, the query that is still is how new variations roll out with out dropping requests. GPU deployments take longer to convey up than CPU companies, and the I/O schema could change between mannequin variations — including a coordination downside on prime. The platform presents two methods:

• Pink-Black deploys a brand new model alongside the present one. As soon as the brand new occasion passes well being checks, visitors shifts in phases — the brand new model scales up whereas the previous scales down on the identical price. If any step fails, the system triggers an atomic rollback. Pink-Black is the suitable selection when the mannequin interface is steady. Manufacturing revealed a coordination hole when a brand new model requires an I/O schema change (e.g., new tensor dimensions): the upstream shopper can’t replace its config till the brand new mannequin is totally dwell, so it inevitably sends “previous” requests to a “new” deployment throughout the migration window, and people fail.
• Versioned solves that hole by sustaining an impartial deployment for each (modelId, modelVersion) pair. A number of variations serve concurrently, decoupling mannequin deployment from shopper updates: the patron waits for the brand new model to be totally prepared earlier than switching its config, whereas the previous model retains serving legacy visitors. The platform cleans up older deployments after inactivity however at all times preserves the newest. The trade-off is a short-term enhance in GPU value throughout the transition overlap.

We suggest embedding variable configurations (e.g., tensor shapes) immediately into the inference mannequin to make it version-agnostic, so it could use the cheaper Pink-Black path. Versioned is reserved for the uncommon circumstances the place a breaking interface change is unavoidable.

Operational Notes

Past these 4 choices, two operational particulars are value flagging — each hit manufacturing gaps the design part didn’t anticipate.

Boot sequence

Bringing a vLLM-on-Triton occasion up entails a number of coordinated steps earlier than the gRPC port opens. Two are non-routine.

• Mannequin caching. Downloading massive LLMs immediately from S3 or Hugging Face at startup is gradual sufficient to inflate cold-start latency previous what schedulers tolerate. We materialize fashions on Amazon FSx on the time of mannequin announcement, so heat begins hit a high-performance file system as a substitute of object storage.
• Embedded vs standalone Triton. When shoppers want the OpenAI-compatible API, Triton runs as an embedded server contained in the OpenAI-compatible frontend course of; in any other case, it runs standalone. That is configured per-deployment at packaging time.

The remainder of the boot sequence is mechanical: extracting the mannequin bundle, putting in customized vLLM plugins by way of Python entry_points, cleansing the Prometheus multiprocess listing, and gating the gRPC port till the engine is prepared.

Unified metrics endpoint

The Prometheus cleanup above hints at a wider observability hole. vLLM writes metrics to PROMETHEUS_MULTIPROC_DIR as .db recordsdata; Triton reviews server-level metrics by way of its personal Prometheus endpoint. Neither is conscious of the opposite, and Triton’s built-in bridge surfaces solely 9 of 40+ vLLM metrics — lacking vital ones like token throughput, KV cache utilization, and prefix cache hit charges.

We added a light-weight HTTP proxy that merges each right into a single /metrics endpoint: it fetches Triton metrics by way of HTTP, reads vLLM metrics from disk utilizing Prometheus’s MultiProcessCollector, and returns the mixed output. Present dashboards and alerts work with out modification.

Deep-Dive: Constrained Decoding at Scale

Some Netflix manufacturing workloads rely closely on fine-grained management over token era. Quite than making use of enterprise logic after inference — paying for invalid generations, then retrying or repairing — we push constraints contained in the decode loop, so the mannequin generates outputs which are compliant by building. We implement this by way of vLLM’s customized logits processor interface, modeling every constraint as a state machine that evolves with the generated token historical past and emits token-eligibility masks at every step. Every request will get its personal configured processor, since completely different requests apply completely different guidelines.

Getting this to scale ran throughout two engine variations: we initially deployed on vLLM V0 (V1 had characteristic gaps), then migrated to V1 in This fall 2025 as soon as it matured. The 2 subsections that comply with are the before-and-after.

Why the primary implementation didn’t scale

Our preliminary pure-Python implementation labored functionally however hit a scaling bottleneck. In vLLM V0, customized logits processors run per-request: the GPU produces logits for the entire batch, the CPU copies them throughout and waits for the switch, after which constraint logic runs sequentially for every request — sequentially as a result of the GIL prevents Python from parallelizing the per-request work. CPU time in logit processing due to this fact grows linearly with batch dimension, hitting tail latencies. Finish-to-end latency turns into CPU-bound despite the fact that the mannequin’s ahead move is batched effectively on GPU. It’s a bottleneck invisible in single-request benchmarks that solely surfaces below reasonable concurrency. Determine 2 makes the serial sample seen.

Press enter or click on to view picture in full dimension

vLLM V1 enabled a batch-level design

The structural repair arrived in vLLM V1, which moved logits processing to batch degree. We rewrote our customized processor to function on batch-level knowledge constructions, computing masks throughout many requests collectively, and reimplemented the new path in C++ with multi-threading to step across the GIL. The V1 API requires specific monitoring of batch membership modifications by way of update_state(batch_update) — extra advanced than V0’s per-request interface, however crucial to take care of right state in a dynamically evolving batch. Determine 3 reveals logits processing time staying flat as batch dimension grows.

Press enter or click on to view picture in full dimension

Operational hardening

Now, efficiency was now not the bottleneck. However stateful constraint logic within the decode loop launched two points the design part didn’t anticipate:

• Partial prefills. V1 performs chunked prefilling, so a request may be prefilled over a number of engine steps. BatchUpdate lacks the granularity to inform whether or not a request was totally or solely partially prefilled, so we added inside monitoring.
• Preemption. Beneath reminiscence stress, vLLM could evict {a partially} accomplished request’s KV cache and reschedule it later with a unique immediate and output token listing. This breaks the state machine’s assumption that the output token listing grows monotonically. We detect when the token historical past shrinks between decode steps, reset the state machine, and reinitialize from the brand new immediate.

Wrap up

We got down to construct an LLM serving platform for broad manufacturing ML necessities — low latency, deep customization, and integration with present infrastructure. The result’s a system on vLLM and Triton, unified behind a constant API, designed to offer ML practitioners a quick path from experimentation to manufacturing.

The teachings have been typically within the particulars — model pinning, silent API gaps, packaging trade-offs — however addressing them has made the platform meaningfully extra sturdy and the developer expertise smoother. Subsequent investments mirror the place we count on friction:

• System immediate compression to cut back immediate size with out sacrificing high quality.
• Asynchronous scheduling of vLLM V1.
• Vectorized logits processors that run as fused GPU kernels as a substitute of CPU code.
• Decrease-precision mannequin variants to lower reminiscence footprint and enhance throughput.

We’ll proceed working intently with the open-source neighborhood as this house evolves.

Contributions

This method is the results of shut collaboration and contributions from many groups throughout the AI Platform org at Netflix. Specifically, Liping Peng designed and developed the mannequin packaging workflow and drove the combination of Triton and vLLM with MSS to allow a unified pathway for serving LLMs. Hakan Baba, Nicolas Hortiguera, and ZQ Zhang led GPU capability planning, system efficiency tuning, software integration and observability, in addition to A/B take a look at readiness and operational excellence efforts for all manufacturing fashions. Santino Ramos enabled vLLM for manufacturing fashions and optimized constrained decoding efficiency. Binh Tang developed the preliminary model of customized mannequin serving and benchmarked completely different LLM serving frameworks. Lanxi Huang and Daneo Zhang constructed the serving improvement instruments to allow consumer self-service. Lingyi Liu drove the general system structure and core technical choices. Abhishek Agrawal and Shaojing Li present administration management to make sure alignment, prioritization and execution.

Acknowledgements

This work closely leverages open-source ML libraries, reminiscent of Triton, vLLM and PyTorch, and many others. We’re particularly grateful to the groups and contributors from the neighborhood. We additionally thank our companion groups in Netflix AI for Member Programs for his or her shut collaborations and innovation on the modeling aspect.