By Praneeth Yenugutala, Victor Dusa, Anastasia Samoilova, Megan Ren, Justin Wei, Marcelo Mayworm
At Netflix, knowledge and machine studying practitioners run hundreds of thousands of workloads every day throughout orchestration techniques and compute engines corresponding to Titus, Spark, Trino, and Druid. On peak days, roughly 100,000 workloads fail. When workloads fail, the failure itself is simply the start. Somebody nonetheless wants to seek out the helpful sign in a number of logs, determine whether or not the trigger sits of their code or the platform, and decide whether or not to retry, reconfigure, or escalate. Even a small share of inauspicious failures can devour engineering time, waste compute, and interrupt crucial knowledge pipelines.
Our first reply to that downside was Pensive, an auto-diagnosis and remediation system in Netflix’s knowledge platform. Pensive collected logs from failed workloads, used common expressions and hand-authored Java guidelines to extract and classify errors, then handed retry or remediation steering to merchandise corresponding to Maestro. It made ~ 50% of failures self-service and, in some instances, computerized.
Guidelines work effectively for identified patterns, however their protection grows solely when somebody acknowledges a brand new failure mode, understands the logs, writes a rule, and retains it present as engines and log codecs change. Over one three-month interval, Pensive processed roughly seven million job-failure classifications, but about half fell into the unknown bucket. We wanted a option to increase protection with out giving up the deterministic components of Pensive that already labored.
To unravel this ambiguous and complicated downside at Netflix scale, we invested in Pensive AI so as to add AI-assisted error extractions and centered prognosis and remediation steering to our knowledge and ML practitioners. The objective was not merely to position a mannequin in entrance of a giant assortment of logs. We wanted bounded extraction, predictable analysis inputs, responsive product habits, and high quality checks that might observe the system from native improvement via manufacturing. These necessities formed each layer of the design, which we’ll talk about on this weblog. Pensive AI wraps AI round deterministic processes, which provides broader protection, low latency, managed AI prices, and simpler upkeep throughout the software program improvement lifecycle. Constructing on this basis, we plan to discover extra agentic debugging experiences, as Half 6 describes.
Early outcomes are encouraging. AI error extraction now covers 100% of failures, together with the failures that legacy Pensive labeled as unknown. Failed step makes an attempt the place nobody used Pensive AI averaged 321% extra handbook debug-log exploration actions than makes an attempt the place somebody did (an observational comparability, not a causal declare).
The 2 paths complement one another. Error extraction improves an present functionality and is all the time on. It lets the consumer of a failed Maestro workload shortly examine the components of the logs which will level to an instantaneous root trigger. It has low latency and low value, and it could assist in a big share of instances. On-demand debugging allows a brand new functionality. It seems to be at signs throughout the logs and provides the LLM’s personal information and reasoning for a deeper evaluation, at barely larger latency and price. Extraction offers a quick first have a look at each failure, and on-demand debugging triggered by the consumer goes deeper when that first look just isn’t sufficient.
Half 1: Discovering the helpful errors
Legacy Pensive guidelines are exact, however each captures a specific log form. A simplified rule seems to be like this:
- identify: SparkContainerOOM
logSource: GENIE_STDERR
restartable: false
regex: "Container.*is working past bodily reminiscence limits.*Killing container"
abstract: Spark AM container runs out of reminiscence
With Pensive AI, we needed to seek out helpful errors that had no corresponding rule and adapt extra simply when acquainted errors modified form after a platform improve. Information Portal (our UI for knowledge pipelines) surfaces these extracted errors when obtainable and hyperlinks again to the supply logs, giving practitioners a centered start line for triage. That raises a query: how can we ask a mannequin to determine an error with out asking it to repeat an extended stack hint token by token?
Producing output tokens is far slower than studying enter tokens. A direct transcription strategy would make lengthy errors costly and gradual. As a substitute, Pensive numbers the strains of every log and provides the mannequin a sequence of {lineNumber, textual content} data. The mannequin identifies the primary and final strains of the error span, and Pensive reconstructs the chosen textual content from its personal log copy, resulting in decrease value and higher latency. This manner, the mannequin all the time returns two integers (begin and finish line quantity), whether or not the chosen logs span just a few strains or lots of of strains. The displayed content material comes instantly from the unique log, decreasing hallucinations. The mannequin can nonetheless select the mistaken vary, which is why extraction high quality stays an integral measurement for our AI evals, as we’ll talk about later.
For orchestrated workflows, “Pensive AI Error Extraction” runs robotically every time a step fails. A step try is one execution of a step inside a Maestro workflow. Maestro already calls Pensive for a regex-derived error and restartability classification, so AI extraction runs in the identical request and returns its outcomes alongside the established fields. The general Pensive response should return inside Maestro’s laborious 90-second post-processing timeout for a failed step try.
That deadline formed the extraction path. Pensive extracts errors from the logs in parallel to cut back latency, and the per-log calls share one deadline set effectively inside Maestro’s timeout restrict. A per-log extraction name that misses the deadline is excluded from the outcome, permitting the general request to return on time.
The info contract between Pensive AI and the assorted shoppers wanted the identical care. The legacy knowledge mannequin contained one flattened error string; AI extraction as a substitute returns an inventory of errors, one for every log supply, with deep hyperlinks again to the originating logs. Altering the previous area in place would have damaged present purchasers. We launched the brand new knowledge mannequin beside the legacy non-AI error extraction knowledge mannequin and continued producing each. Information Portal, our consumer interface for monitoring and observing knowledge pipelines, most popular AI errors when obtainable and fell again to the legacy worth after they weren’t throughout the rollout. That allowed a gradual rollout and ensured that historic failures nonetheless show an error, since we selected to not backfill the previous knowledge as a result of log retention points.
The migration remains to be in progress right now. The Information Portal UI floor used for failed Maestro steps reads the brand new AI extracted error area right now, whereas the question editor (for modifying and working SQL), Workbench notebooks (managed compute platform and UI to edit and run jupyter based mostly notebooks) and a few slack purchasers calling Pensive nonetheless use the legacy error extraction path. Every consumer wants a suitable schema change earlier than it could devour the brand new knowledge. Conserving each fields populated lets groups undertake the brand new habits at their very own tempo with out forcing a coordinated cutover as we proceed evolving Pensive AI.
Half 2: Full failure prognosis and the product expertise
Past error extraction, we constructed Pensive AI Debugging, a instrument that analyzes failed step makes an attempt and gives root-cause evaluation and advised remediations. It’s obtainable via programmatic APIs, MCP, and the product UI (Information Portal), all backed by a typical entry level throughout the Netflix Information Platform. This shared mannequin helps constant habits and customized product experiences.
This prognosis nonetheless has to reach shortly and stay constant throughout investigations. A debugging response can take tens of seconds or a minute to generate, in contrast with just a few seconds for error extraction. We run full AI-based debugging on demand as a substitute of robotically, which retains prices underneath management as a result of most failed steps are by no means opened. We deal with the failed step try, not the one who triggered it, because the unit of labor, so everybody investigating the identical failed try sees the identical prognosis.
That alternative offers us a secure unit of labor for caching. We cache every debugging response by step try. A second request turns into a key-value learn as a substitute of one other AI inference name. The cache additionally prevents duplicate work, decreasing latency and reducing prices additional. This additionally implies that a debugging generated within the Information Portal is cached and instantly retrieved for the MCP path as a substitute of incurring one other AI inference name. A conditional write creates an in-progress report, one caller begins era, and different callers anticipate the shared outcome to keep away from conflicts when concurrent requests are available for a similar step try failure. If era stays in progress previous an abandonment timeout, Pensive treats the report as deserted and reclaims it. Accomplished responses expire after 90 days to assist management prices and optimize storage as a result of debugging requests for a lot older workloads are unusual.
Pensive AI helps a number of methods to provoke debugging responses. A caller can (1) learn a cached response, (2) generate one if no cache entry exists, or (3) regenerate and overwrite an present response. These semantics assist the product UI, programmatic entry, and the analysis harness whereas retaining the response related to a single immutable workload failure.
For customers, a very powerful habits is {that a} long-running reply feels alive. Pensive streams the response as it’s produced. The service emits server-sent occasions, and the GraphQL layer republishes them as a WebSocket subscription for the Information Portal consumer.
Every occasion carries the entire response generated thus far. That duplicates a small quantity of textual content, but it surely retains the consumer easy: it doesn’t want a replay buffer or token reassembly logic, and the subsequent occasion corrects the show after a dropped or late occasion. Technology is unbiased of the connection that requested it. A disconnect doesn’t cancel the work, and the ultimate reply nonetheless reaches the cache. A tough reload can present a loading state and ballot for the finished outcome. Inside the single-page utility backing the Information Portal UI, energetic streams are maintained independently of view and part lifecycle, so navigation doesn’t interrupt a response in progress.
Half 3: Measuring and enhancing the habits we need
LLM-assisted techniques want greater than a demo or just a few working examples. Earlier than transport in manufacturing, we’d like confidence in each error extraction and debugging steering, and we have to know that we have now not regressed on failures that legacy Pensive already dealt with. We constructed separate analysis frameworks centered on the 2 distinct jobs.
Constructing labeled knowledge is commonly the costly a part of analysis. For error extraction, legacy Pensive’s hand-written guidelines present reference labels for failures they acknowledge. Additionally they function wonderful floor reality typically. We run these guidelines over manufacturing logs and evaluate their extracted spans with Pensive AI’s output. This offers us a dependable regression set with out relabeling each instance by hand, although it isn’t an absolute floor reality (since our human-built guidelines might even have been stale). For now, we maintain these guidelines frozen as a label supply. This lets us proceed to construct comparability evals and run them on prime of the present system.
Steady inputs are important for repeatable analysis experiments the place we A/B take a look at new fashions/configurations or context injections to enhance response high quality. The logs Pensive AI makes use of can expire after as few as seven days or as many as 30 days. Manufacturing Pensive AI requests archive fetched logs to S3 object storage asynchronously throughout workload execution time so the add just isn’t on the user-facing crucial path. Uploads don’t overwrite an present object, preserving the precise snapshot used to create an analysis case. Evals use these archived logs, so rating adjustments stay attributable to the system underneath take a look at. The archive additionally prevents parallel experiments from overloading the supply techniques and lets us run them with out affecting manufacturing. Cached S3 log inputs additional cut back prices when working evaluations.
Concretely, we measure 4 complementary alerts for error extraction:
- Inclusion: An LLM decide compares the legacy Pensive error for a failure with the errors Pensive AI extracted and determines whether or not the identified error remains to be represented, even when wording differs.
- Precision: A second decide sees solely the extracted AI errors and evaluates whether or not every is significant and actionable on its personal.
- Repeated noise: A script seems to be throughout the entire dataset for strains that recur too typically. A generic standing line might look innocent in a single reply however develop into distracting if it seems in a big share of failures. Traces showing in additional than about 30% of workflows are candidates for specific exclusion. In follow, the exclusion listing has solely three strains, every of which in any other case appeared in practically each extracted error.
- Latency: We report the latency of each extraction name. High quality scores alone can not reveal when an obvious enchancment requires an excessive amount of extra work, which regularly correlates with elevated AI prices.
This mixture offers us an actionable suggestions loop. Immediate adjustments can cut back basic noise, whereas the dataset-level view exposes the cussed boilerplate that immediate adjustments don’t get rid of. Current guidelines act as a regression flooring, and archived logs give us a repeatable option to evaluate adjustments. At Pensive AI’s scale, mannequin and system decisions should fulfill high quality, latency, maintainability and price constraints collectively.
To seed the debugging ground-truth dataset for analysis, we mine inner engineering-support Slack threads and use an LLM to determine threads that hyperlink to an orchestrated workflow, describe a resolved downside, and match the category of situation Pensive AI handles. A second mannequin synthesizes a reference debugging response from the related dialogue. Paired with logs from that workflow failure, it turns into a concrete analysis row in our evals dataset. This strategy works solely retrospectively, as a result of Pensive AI in manufacturing goals to supply debugging steering earlier than such a Slack dialogue occurs.
We additionally made the debugging era course of seen. Tracing follows the request via Pensive, with little one spans for every log fetch and extraction instrument name. Our tracing backend based mostly on Braintrust brings the immediate, completion, instrument calls, and latency right into a single view. Throughout early product improvement, that visibility helped us examine low-performing debugging responses in context as a substitute of guessing which enter or operation prompted them. Tracing helped us enhance the responses and fill context gaps even earlier than we had complete evals. In manufacturing, the API and UI return the hint identifier with the response, so suggestions can lead on to the hint for in-depth evaluation.
Half 4: Conserving high quality within the launch path
Analysis just isn’t just for experiments with LLM orchestration, fashions, prompts, or context adjustments. Adjustments to log choice, trimming logic, or a supply system can change what the mannequin sees and due to this fact change extraction high quality. We use the identical analysis harness and evaluators all through native improvement and Pensive AI deployment so these results are seen earlier than customers encounter them.
For every pull request, the code department is deployed to an remoted preview atmosphere and a CI job runs an experiment with our evals in opposition to that remoted atmosphere. Precision and inclusion thresholds gate the merge. The job additionally runs after a construct is promoted, catching interactions that weren’t current when particular person pull requests had been evaluated in isolation. If the promoted construct misses its thresholds, it alerts the on-call engineer to research or roll again.
The evaluator runs on a schedule in opposition to manufacturing to floor gradual drift or exterior adjustments. Builders can even set off analysis experiments remotely whereas iterating, utilizing the identical deployed-environment path as CI. Constant infrastructure makes the outcomes comparable throughout native iteration, preview builds, promoted builds, and manufacturing.
Half 5: On the lookout for suggestions sign from customers
The system is simply precious if it reduces the quantity of handbook debugging and log chasing completed by our customers. We needed a measurement that was obtainable from product telemetry and near the habits Pensive AI was designed to vary. Guide debug-log exploration grew to become one of many major alerts. The Pensive interface already gives hyperlinks to logs for a failed step try. To estimate whether or not the characteristic lowered raw-log searching, we in contrast failed step makes an attempt the place somebody interacted with Pensive AI with these the place nobody did. The second group averaged 321% extra handbook debug-log exploration actions. This observational comparability doesn’t set up causality or a particular discount in imply time to decision, but it surely means that centered prognosis typically permits practitioners to examine fewer uncooked logs. Programmatic MCP entry additionally lets groups construct operational flows with their terminals or customized agentic experiences across the similar prognosis service.
One other extra specific sign we acquire is thumbs up and thumbs down on the debugging response. These suggestions alerts land instantly alongside the inner traces so we will shortly affiliate suggestions to the entire log of inputs/outputs and the trail the AI took to reach on the reply. Extra importantly, we created an inner business-hours alerting system the place every thumbs down response sends us a Slack alert. This was precious to resolve lots of tough edges and context gaps initially within the product and helps us enhance the product repeatedly even right now.
Half 6: Future Work constructing on these foundations
Pensive AI is the inspiration for extra work that goes past prognosis and really useful remediations. We’re exploring “Pensive AI Deepthink,” an opt-in managed agentic expertise with larger value and latency. It builds on the exact, low-latency evaluation produced by Pensive AI Debugging and gathers extra context from inner documentation, manuals, previous Slack discussions, supply code, read-only warehouse queries, and Spark execution particulars. A routing step directs the investigation to the suitable talent, corresponding to Spark failures, data-audit failures, or late workflows. This retains the shared start line secure and deterministic whereas permitting the investigation to develop into extra specialised, additional enhancing recall. We’re additionally in talks with the Spark crew to leverage Pensive AI responses for failure debugging evaluation and A/B comparability for upcoming platform migrations.
We additionally plan to spend money on “Pensive AI Error Classification”. Arduous-to-maintain legacy guidelines additionally classify failures by restartability and by whether or not they’re platform or consumer prompted. Maestro acts on these labels and adjustments its retry habits accordingly like failing quick or tuning the default variety of retries (platform errors are tried greater than consumer errors by default). Error classifications should meet a better bar than error extractions and debugging alone. Shifting nearly all of the error classifications to AI would allow us to retire a lot of the remaining rule-maintenance burden and keep away from retrying unknown failures that had been unlikely to succeed. For restartability, the most costly error is classifying a recoverable failure as non-restartable, as a result of Maestro will flip a step that might have been recovered into a tough failure, main to finish consumer alerts and handbook remediations. The analysis framework will should be tuned round that consequence earlier than AI-enabled classifications can tackle these choices. Equally necessary, we should keep away from retrying non-transient failures that won’t succeed, as a result of retries waste compute and delay knowledge pipelines.
Our work on Pensive AI in Information Platform addresses concrete consumer ache factors in debugging, improves maintainability for platform house owners, and improves precision and recall for error extraction. Pensive AI extends protection to failures that legacy Pensive beforehand labeled as unknown. Reliability comes from the system across the AI mannequin calls: bounded error extractions, suitable consumer knowledge contracts, responsive supply, reproducible evaluations, caching and tracing, and launch controls that maintain high quality and price seen because the platform adjustments.
Half 7: Acknowledgements
This work wouldn’t have been potential with out our beautiful colleagues throughout Information Platform and our Information Engineering and Analytics companions who supplied early suggestions. Whereas we can not identify everybody individually, we’re deeply grateful for his or her contributions, which formed each response high quality and the product expertise. Particular due to Adith Swaminathan for advising on evaluations and inference enhancements and performing as a thought associate. Thanks to the Orchestration crew members, particularly Deepak Ramalingam and Malcolm Ramirez for his or her extra engineering contributions.
For dogfooding and integrating Pensive AI as a part of consumer assist, we additionally thank the engineering assist group (ESO Staff). We additionally thank Abhishek Kapatkar (Information Developer Experiences Staff) and Emmie Dong (Information Motion Staff) for his or her suggestions to enhance the Pensive AI integrations with the remainder of the ecosystem. Particular due to the AI Platform crew (Kyle Ahn, Rob Mann) and Braintrust Staff for serving to with distributed tracing, evals infrastructure and mannequin gateway runtime assist.
We additionally thank Kamran Masood, Ely Spears, Ronald Leung, Anjali Norwood, Prashanth Ramdas, Charles Smith, Harikrishna Menon and different leaders for his or her constructive suggestions and assist on the Pensive AI mission.
Making Failure Prognosis Dependable with Pensive AI was initially revealed in Netflix TechBlog on Medium, the place persons are persevering with the dialog by highlighting and responding to this story.
