The challenges and choices behind AI Uncover’s intelligence layer
By Varun Khaitan, Peng Ren, Gary Tang, Itay Teller, Atif Faheem, Nipun Kumar, Donnie DeBoer, Eduardo Lundgren, Matteo Rinaldi, Raghav Bakshi
We lately launched AI Uncover, a brand new tab in our TV app the place a small group of members can strive new methods to search out one thing to look at. For years, our suggestions have labored within the background, inferring what a member would possibly get pleasure from from their viewing historical past. AI Uncover strikes from that silent rating towards steerable discovery. A member picks a theme, say “Fast Watches,” and will get a row of matching titles, a brief be aware on why they match and some narrower themes to strive subsequent. From there, they will hold narrowing, head someplace new or tune what they see.
Constructing AI Uncover raised a query many groups will acknowledge: how do you ship a greater member expertise with generative AI on TV with out sacrificing velocity, reliability or security? Our reply is a brand new intelligence layer, the agent harness behind AI Uncover. For every request, it decides the best way to mix LLMs with our advice programs. It builds on years of labor understanding what members get pleasure from, and on our current work making use of LLMs to suggestions and generative experiences. The layer itself plans every request, runs it throughout these programs, serves the consequence and evaluates it offline.
This publish covers the challenges we designed for and the important thing choices we made alongside the means.

Why an intelligence layer?
Each Netflix web page combines suggestions with guidelines. We choose titles a member could get pleasure from, and eligibility, maturity and security guidelines determine what can seem. An expertise constructed on LLMs wants the identical guidelines, plus code that decides when to name an LLM and the best way to deal with its output. That code is named an agent harness, and it’s the core of our intelligence layer.
The best implementation fingers your complete request to an LLM and lets it choose the titles, write the labels and determine what occurs subsequent. Nevertheless, that leaves us counting on probabilistic output the place we’d like ruled outcomes, as a result of each flip in AI Uncover ends on a display screen in somebody’s lounge. At our scale, even uncommon failures occur many instances a day. A member sees each gradual name, unavailable title or off-brand line. Seven challenges formed our design:
- Becoming LLMs right into a product and structure designed with out them. Conventional recommenders rank titles primarily based on previous habits. LLMs can do extra: perceive what a member is asking for and plan the best way to reply. Our present expertise would go away most of that unused.
- Holding suggestions grounded. Each choose have to be an actual, obtainable title from our altering catalog. Retrieval and rankers work in tandem with LLMs to decide on titles that match the member’s style and intent.
- Working with gradual, unpredictable LLMs. LLM calls are the slowest and costliest a part of the system. Tail latencies can spike, and the output might not be what you requested for. So we use LLMs selectively, deal with what they return as untrusted enter and take a look at efficiency and high quality earlier than something ships.
- Adapting to fast-changing fashions, instruments and prompts. Modeling and engineering options hold evolving, so the product can’t rely upon any single mannequin, instrument or immediate.
- Constructing for TV. Typing on a TV is difficult, so members depend on the distant, and the app ought to reply to each press instantly on extensively various {hardware}. TV apps additionally replace slowly, so modifications should ship with out an app launch. A typical TV request returns in moments, whereas multi-step technology can take a number of seconds.
- Holding a security and reliability bar at scale. Members at all times get a web page, even when an LLM name fails or yields poor outcomes. Their decisions and our content material requirements maintain on each request, and when one thing breaks, structured data present why.
- Studying from what members say alongside what we already know. Suggestions have at all times discovered from viewing habits. AI Uncover additionally lets members inform us what they need, and that has to form what they see subsequent.
System Structure
When a member picks a theme, the TV app sends the request to the Web page Server, which builds pages for all Netflix apps and fingers the request asynchronously to the intelligence layer. At a excessive degree, this new layer has two parts: a controller and an executor.
The controller decides what to run. It gathers the context wanted for every request, together with real-time indicators such because the time of day. It then settles on a plan, both a longtime one or a brand new one drafted by its planner, validates it and retrieves solely the member particulars that this plan requires from Reminiscence Hub, our per-member reminiscence retailer. These particulars could embody member style summaries generated by MemberLLM, our in-house language mannequin for personalised suggestions, or suggestions shared by the member. We speak extra about Reminiscence Hub and member style summaries later.
A plan is a graph of permitted steps, every working a functionality. A functionality is a named unit of labor that may have a number of variations and belongs to a household comparable to retrieval, rating, technology or packaging. It declares its inputs and outputs, the context it could see and what occurs if it fails.
Functionality declarations make a plan verifiable. Earlier than execution, the controller validates the graph: each step is permitted, each enter has a producer, nothing loops and the plan ends in precisely one web page. Afterward, the controller checks the consequence earlier than it’s served.
The executor runs the plan. Steps move typed outputs to one another by means of the declared contracts, so every receives precisely what it requested for. Impartial steps run in parallel, every with its personal deadline and applicable fallback behaviors. Some steps are optimized LLM calls with one process every, comparable to writing row titles or suggesting subsequent themes, and so they act solely inside the validated plan. Checks earlier than the run and contracts throughout it give us ruled execution, the place each motion stays inside limits set in code.
Each elements write to the request data, the traces we use offline to guage and enhance the system.

Key Choices
Every problem led to at least one determination. Right here’s what we selected and what it purchased us.
1. Design the product and structure round LLMs
We reimagined the expertise round LLMs. The simpler path would have been to drop them in as rankers, hold right this moment’s latency budgets and precompute the place essential. AI Uncover is constructed on a back-and-forth round themes a member can steer, which supplies LLMs work solely they will do: producing themes, explaining rows, summarizing style and serving to select the titles that match the member and the second. Members choose themes with the distant, in order that they by no means should kind. On the TV, these themes and explanations get their very own native UI components. Behind the display screen, LLMs can draft plans, and asynchronous serving offers multi-step agentic orchestration the time it wants.
This opens up experiences our conventional stack couldn’t help on its personal.
2. Make present programs composable
Holding suggestions grounded meant constructing on the programs that already know our catalog and members. Most capabilities wrap advanced, confirmed Netflix programs comparable to search, large-scale rating and catalog metadata entry. A few of them could run their very own brokers within the future.
Reshaping them took cautious work, as a result of agentic orchestration challenges many long-standing assumptions of distributed programs. A functionality now has to work in combos nobody designed, for a caller that chooses to name it at runtime. Every one wants a transparent objective and a semantic interface, and shaping it means deciding how a lot to mix right into a single step and the way a lot to go away to the planner. Too positive, and the planner has extra methods to go fallacious; too coarse, and it loses room to compose richer plans.
In return, the planner will get constructing blocks it may possibly combine in new methods, and connecting companies takes far much less handbook wiring. As a result of plans can solely use these blocks, even one drafted on the fly stays grounded in our catalog and member style.
3. Plan first, then execute
To comprise LLM latency and errors, we plan each request earlier than something runs. The acquainted strategy to let an agent use instruments is a loop: cause, name a instrument, learn the consequence, cause once more. It’s versatile, however you may’t inform up entrance what number of steps it’s going to take or whether or not it has gone down a fallacious path. In agent phrases, ours is a plan-and-execute agent: an LLM can suggest the trail, however code decides whether or not it runs.
Plans come from two locations: established plans we’ve already examined, and plans an LLM drafts on the fly. A router we management decides which one is served. Both means, the plan passes the identical checks earlier than it runs. If the planner is gradual or its plan fails a verify, we fall again to a examined plan.
This permits for adaptive planning. The planner can discover new methods to serve a request, and we will stability price, latency and high quality for every process, from a lean single-call plan to a richer multi-step one.

4. Keep mannequin and functionality agnostic
The most effective mannequin or functionality for a process right this moment could also be second-best, or overkill, in a number of months, so we constructed the system to change between them. We assign every process to the mannequin or functionality model that matches it. At present that’s a per-task setting we will override dwell. Just some steps in a request use an LLM.
That solely works if switching is straightforward. All of the LLMs we use sit behind one widespread interface. Earlier than we change, every candidate is scored on the identical duties by automated checks and LLM judges, so each swap is a measured change.
A lot of the system’s habits lives in server-side prompts and configuration, so we will swap fashions, instruments and capabilities with out new code or a TV app launch.
Upgrades get low-cost and experiments get simple. Transferring to a greater mannequin, ranker or retrieval system is a plan or setting change, not a migration, so every part can hold bettering with out a big funding.
5. Make web page technology asynchronous
Inference retains getting quicker, however member expectations rise simply as shortly: richer responses, extra context and extra of the web page formed by LLMs. The road between what feels instantaneous and what’s price ready for will hold shifting, so we constructed the asynchronous path as a first-class means of serving.
Most of a TV app is synchronous: press a button, get a web page. Generative work breaks that sample. Holding one request open for a lot of seconds throughout companies would tie up too many threads. So requests that contain technology are asynchronous. The press returns instantly with an acknowledgement, and the TV switches to a pondering state that explains what’s being ready. In the meantime, the executor runs the plan within the background. When the result’s prepared, the server notifies the TV, which then fetches it.
Gradual technology not slows anything. The serving path stays wholesome nonetheless lengthy a flip takes, and each press nonetheless will get fast suggestions on display screen.

6. Implement security in code
Some guidelines should maintain it doesn’t matter what an LLM writes: content material and maturity requirements, what every step can entry, and what a member has requested us to not present them. Placing these guidelines in a immediate makes them probabilistic, so we implement them in code. Generated textual content is cleaned up earlier than it reaches the display screen. The planner sees a stripped-down request, and member particulars are restricted to what the validated plan wants.
For instance, the planner decides which inputs every step wants, such because the profile’s maturity ranking for retrieval. As a substitute of writing parameter values instantly, it makes use of placeholders. After the plan passes validation, the controller fills them with values from the request, matched by declared kind. This additionally protects in opposition to immediate injection.
Every request additionally leaves a structured report: the plan, every step and its final result, and which guidelines utilized. When a web page misses, we will inform whether or not the plan, the information, a rule or the mannequin was at fault. The identical data feed our offline analysis and replay.
Collectively, these guidelines and data let an experiment ship on the identical security and privateness bar as the remainder of the product.
7. Bear in mind what members inform us
A style abstract is a brief profile of what a member enjoys, comparable to favourite genres, tone and recurring themes. MemberLLM generates it forward of time from the member’s viewing historical past, which retains that LLM work off the request path. It helps us additional personalize their web page, and steps that write themes and explanations obtain it as context, so their wording displays the member. We’ll cowl MemberLLM in additional element in an upcoming publish.
In AI Uncover, members may tune their suggestions and inform us instantly what they need. These decisions apply instantly and carry throughout visits, and so they give us a brand new sign alongside what we already find out about their style. This can be a first step towards remembering extra about every member, from their preferences to the habits that present up over time and the corners of the catalog they prefer to discover. We hold all of it in Reminiscence Hub, our persistent, low-latency reminiscence retailer, subsequent to their style summaries, and we will add new sorts of reminiscence as we’d like them. We apply these recollections to later requests the place they’re related, along with real-time context.
The member holds the distant
On this model, members work together by deciding on themes and utilizing tuning choices comparable to “don’t suggest.” There’s no free-text enter or open-ended chat. That retains the LLM’s job slender by design.
- The standard protections nonetheless apply. Profile settings and our security and privateness insurance policies maintain right here as all over the place else.
- Generated textual content was examined earlier than launch. We ran security evaluations on theme labels, row titles and explanations to ensure they meet our requirements.
What’s subsequent
It will have been quicker to let an LLM do every part and cease there. The intelligence layer gave us an AI-native product, grounded constructing blocks, adaptive planning, faster experimentation, asynchronous technology, ruled execution and agent reminiscence for tunable personalization. Whereas AI Uncover is its first expertise, we designed it with extra in thoughts.
We’ve organized our efforts to reply two questions:
- How a lot do members need to form their suggestions? Whether or not they’d slightly steer or lean again, and the way a lot, which sorts of themes assist, and the way we should always mix what members inform us with our present personalization programs.
- What tradeoffs does this know-how convey? How price and latency maintain up in manufacturing as we ramp site visitors, how lengthy members will look forward to generated outcomes, and whether or not errors got here from the mannequin, the harness or our present programs. Multi-turn classes additionally want totally different analysis and observability than single requests.
Long term, our purpose is a system that improves itself, ranging from the report every request leaves behind. At present we replay previous requests in opposition to a brand new immediate, mannequin or rule to see what modifications, and we choose every change with automated checks, LLM judges and human evaluations. Finally, we wish the system to suggest enhancements, take a look at them safely and hold solely what proves higher, inside our privateness and data-use insurance policies. As LLMs enhance, extra of the work could transfer into them, and the intelligence layer lets us transfer that line intentionally.
That is a part of an even bigger shift. Advice programs have lengthy discovered quietly from what members watch and answered with ranked rows. LLMs allow them to pay attention and reply, and members could more and more anticipate that. We’ll hold exploring what meaning for discovery on Netflix.
Acknowledgements
We want to thank Adi Suresh, Akash Katipally, Allan Johns, Becks Wooden, Bhargav Mangipudi, Carrie Santiago, Chris Martin, Claire Campbell, Corey Waldin, Dana Sittler, Daneo Zhang, David Leen, Devon Bryant, Dharamendra Kumar, Dharini Chandrasekaran, Ebony Turner, Federico Roa, Gianfranco Cuoco, Guru Tahasildar, Haley Clanton, Haresh Chudgar, Ivan Provalov, James Schultz, JingXu Yang, Josie Shaw, Kamelia Aryafar, Kate Cummings, Kathleen Garrity, Kelley Robinson, Kevin Du, Kevin O’Connor, Kyla Moore, Kyle Lind, Laura Matsuura, Leah Cusella, Lingyi Liu, Lingyong Wang, Mallory Walkup, Matthew Pereira, Michael Galassi, Minsu Jang, Moumita Bhattacharya, Paran Jain, Paul Lee, Qusai Al-Rabadi, Scarlet Chen, Sergi Perez, Sriram Rao, Stephen Shaw, Sura Elamurugu, Varad Pathak, Yesu Feng, Zainab Mahar Mir, and Zhongli Cheng for his or her contributions to this work.
Engineering AI Uncover was initially revealed in Netflix TechBlog on Medium, the place persons are persevering with the dialog by highlighting and responding to this story.