Prasanna Vijayanathan, Renzo Sanchez-Silva
When Jake Paul and Mike Tyson stepped into the ring in November 2024, we delivered the struggle to roughly 65 million concurrent streams. On an peculiar day our infrastructure takes in additional than 2 billion requests and handles upwards of 38 million real-time logging occasions each second. At that scale, the entire image doesn’t match right into a single engineering crew’s head or onto a single dashboard. With extra Stay content material, Podcasts and real-time streams, our want to know and replace our techniques as shortly as potential turns into crucial to supply a great consumer expertise. Understanding the system at this scale and inside the time constraints we now have, typically turns into a frightening train find a needle in a haystack, in a darkish jungle, blindfolded.
This submit is about that seek for the needle.
The 4 hours earlier than a repair
Contemplate a regression of the type we see in a given week. The house display the place you see the checklist of lists of films, begins taking about 1.5 seconds longer to render. An alert fires. After which we go searching for what triggered this difficulty and the way will we repair it?
Because it so occurs, a regression on the applying might be associated to wherever within the system — a shopper app regression from a brand new model deployment or A/B check rollout, a load shedding within the infrastructure, a GraphQL question responding with dangerous knowledge due to a service rollout 6 hops downstream, are all suspects at this level. So, answering “what triggered the difficulty?” wants insights from throughout the system, and this includes combing by layers of information in a meticulous method.
The MELT Layer
“What triggered the difficulty?”
Metrics, occasions, logs and traces, or MELT, within the business’s shorthand, are the appropriate uncooked supplies for such an investigation, and every of them solutions roughly 1 / 4 of the query. We gather observability knowledge throughout these 4 pillars, unified right into a single observability knowledge layer. This lets us observe our techniques in actual time, throughout what our customers are experiencing, how our functions are doing and what our backend infrastructure and companies are processing and responding to.
The investigation that follows could appear like this in every layer:
The timeline of an investigation
So how lengthy does it take to analyze a difficulty, determine the basis trigger and repair it, so our customers cease working into this difficulty once more?
In our chosen instance, 4 hours. Little or no of it went into computing something tough. Most of it went into reconstructing relationships that our observability techniques already knew however we had by no means written down wherever a human or machine might merely learn and uncover.
The disagreement beneath
The intuition right here might have been, round 2023, to succeed in for extra knowledge or higher dashboards, and whereas each assist on the margins, neither addresses what makes these investigations sluggish.
It’s 2026 now, so we’d say, “hey, brokers”, however with out the appropriate context delivered to the brokers, AI techniques hallucinate, and particularly when groups are on-call and want a repair “now”, we can not let inaccurate agentic evaluation lead us down the improper path. We want the appropriate needle, not any needle.
Every of our techniques is internally coherent and externally remoted. Atlas, our telemetry platform, is aware of functions and their time collection. Radar, our alerting system, is aware of thresholds and firing historical past. incident.io is aware of severities and responders. PagerDuty is aware of rotations. Spinnaker is aware of what shipped and when. And our Slack historical past is aware of, in a kind no question engine can attain, which engineer solved one thing very related final quarter and what they discovered.
The issue is that they don’t agree on what they’re speaking about. A factor that’s an utility in a single system is a service in a second and a cluster in a 3rd, with names differing by a suffix, a casing conference, or a rename solely a few folks keep in mind (SMEs, ring a bell?). Becoming a member of throughout them requires a human carrying the mapping of their head, and that particular person turns into the bottleneck in such cross-domain investigations. The laborious a part of end-to-end observability might be higher understood as a knowledge modelling downside.
What we lacked was an specific, machine-readable settlement concerning the entities in our operational world.
The Ontology
This settlement has a reputation: an ontology. Ontology is a proper specification of varieties, the properties they carry, and the relationships they might have. Its unit is the triple, one truth expressed as topic, predicate, object. Contemplate the next instance triples.
api-gateway rdf:sort ops:Utility .
api-gateway ops:ownedBy "Crew Bedrock" .
api-gateway ops:deployedIn "us-east-1" .
INC-5377 rdf:sort ops:Incident .
INC-5377 ops:impacts api-gateway .
INC-5377 ops:severity "SEV 2" .
Within the instance right here, we now have already encoded six information, and may reply in a single question, a query that used to span three instruments.
“Which incidents affected functions owned by Crew Bedrock in us-east-1?” is a single question instead of trying up incidents, Spinnaker and Radar.
We name our ontology layer, The Observability Ontology. It acts because the schema, rule set and shared settlement that standardizes how completely different techniques perceive entities like functions, companies, incidents, deployments, and so forth.
We constructed it on the W3C stack for this: RDF for the information mannequin, OWL for the ontology, SHACL for validation, and SPARQL for querying. These requirements are nicely specified and broadly carried out, which retains the mannequin transportable throughout graph shops.
The info from the ontology itself, is saved as a information graph, continued to QuipuDB, our homegrown RDF graph database, a distributed quad retailer primarily based on redb, a Rust-based embedded key worth db. Information right here is generated by parsing uncooked telemetry, logs, and incident discussions into triples (subject-predicate-object information).
Each harvest additionally produces a full Turtle serialization for archival, making it human-readable and diff-friendly, which has confirmed extra helpful for debugging than we anticipated.
Our ontology is organized into twelve namespaces, every proudly owning one operational area:
There are two selections about this ontology that mattered whereas we constructed it, in order that they’re value sharing.
The primary is that we refused a generic Service sort. “Service” means 4 various things throughout our tooling, so functions resolve to precisely one sort, ops:Utility, and each one in all them is validated in opposition to our software program catalog of roughly 49,000 entries earlier than the entity is created. Unvalidated strings like usernames, crew names, dependency names scraped from community topology, by no means develop into functions. We additionally cap dependency edges in each instructions, as a result of one fashionable service can carry 1000’s of reverse dependencies and would in any other case dominate the graph.
The second resolution considerations what we selected to mannequin. Incident chat carries the reasoning behind an incident. The hypotheses tried and discarded, the mitigation that labored, who knew why, is all context within the incident’s Slack channel and that information evaporates when the channel goes quiet. So a 200-message, 50-person, 15-service incident channel turns into a construction, changing the operational chaos of an incident into ordered triples:
Creating determinism from non-determinism
Constructing this graph means utilizing instruments which can be inherently stochastic. LLMs summarize discussions. Fuzzy matching resolves service names. A probabilistic engine proposes causal chains. None of those produce the identical output twice with certainty, however what comes out on the finish of our system is a deterministic graph that solutions a SPARQL question the identical method each time.
The ontology is the mechanism that creates this determinism by its pipelined strategy. We consider the pipeline as a ladder that drives entropy down, the place entropy means the proportion of data nonetheless trapped in unstructured kind. For instance, in the beginning of this pipeline, an LLM might say something, but it surely can not emit a triple that violates the kind system. We constrain the output house tightly sufficient and a stochastic course of yields a deterministic contract:
# The mannequin could produce any prose. Its output MUST match these predicates:
class IN ["incident", "question", "discussion", "decision"]
sentiment IN ["positive", "neutral", "negative", "urgent"]
# End result — a deterministic triple:
thread:123 slack:class "incident" .
thread:123 slack:isIncidentRelated "true"^^xsd:boolean .
There are 5 layers that get us down this ladder. Any single step of this ladder is non-deterministic. However in mixture, the chain turns into deterministic. That inversion is the entire trick. We describe them under:
Enrichers: Brokers as entropy reducers
We use what we name, enrichers, to scale back entropy throughout the information graph. Dozens of enrichers run over the graph. Every one in all them takes noisy enter and emits typed triples, and every reads what earlier enrichers wrote. The contract for an Enricher is intentionally slim:
class Enricher:
async def _enrich_async(self, ...):
uncooked = await self.fetch(...) # 1. learn noisy enter
outcome = self.analyze(uncooked) # 2. course of (could also be stochastic)
graph.add((uri, RDF.sort, OPS.X)) # 3. emit TYPED triples
graph.add((uri, OPS.pred, Literal(outcome.worth)))
return EnrichmentResult(success=True, triples_added=depend)
Three invariants which can be commonplace throughout the enrichers:
- each entity carries an rdf:sort,
- each URI follows a single naming scheme, and
- the catalog gates each ops:Utility.
Two operational selections are value naming right here, as nicely. The enrichment pipeline runs sequentially — load the graph, add triples, and save.
This sequencing removes a whole class of race situations at a price in wall-clock time we’re pleased to pay. And calls out to exterior techniques like metrics, logs, occasions and the remainder might be best-effort, so a sluggish downstream API doesn’t block the graph harvest, albeit some degradation. To that finish, we’re additionally working to translate these separate techniques into the ontology, such because the Metrics Ontology and the Alerts Ontology, so the whole lot might be queried from the Ontology itself.
The Correlation Engine: discovering the correlation chains
The correlation engine assimilates the huge variety of alerts from various enter sources and condenses them into causal or correlatory chains of occasions. Right this moment, the engine reads 13 sign varieties, together with log errors, metric anomalies, sluggish and failed traces, alerts, incidents, deployments, config modifications, experiments, autoscaling, commits, end-to-end bottlenecks, and error patterns. It then searches for paths amongst them. Some ordered pairs are recognized to be causal and carry a excessive prior weight. For instance, deployment earlier than log error, config change earlier than metric anomaly, failed hint earlier than incident, and so forth. Others, which present excessive correlation are thought-about correlation chains. Such chains are scored on temporal proximity and severity, inside a ten-minute window, with something below 0.6 confidence getting discarded.
The position of the engine right here is to suggest with proof, to not make conclusions. Therefore, the engine shops its chains together with the graph information that have been used to create them.
A 0.87 chain within the above instance, is a powerful speculation for a human or an agent to examine, and the graph information the arrogance alongside the declare so that customers can perceive why a sequence is what it’s and weigh their choices to make selections.
The information flywheel
Every harvest cycle turns the identical 4 steps, and every rotation leaves the graph richer for the subsequent. That is the Information Flywheel, resulting in steady enrichment and enchancment of the context about our system, saved within the information graph.
Observe collects the alerts. Enrich provides typed triples. Infer finds patterns invisible within the uncooked knowledge. Adapt feeds information again, in order that established patterns develop into detection guidelines. The compounding results of the 4 phases results in a self-improving system. A causal chain found in a single incident turns into a recognized sample for the subsequent, and the ontology regularly stops being a schema and begins being a reminiscence.
Contemplate the instance above. After we first see an incident in api-gateway triggered attributable to a timeout and resolve it, we convert this information into entries to the ontology. This turns into encoded information. As we undergo a number of cycles of this encoding, the graph begins studying patterns. After N cycles, patterns develop into knowledge — “api-gateway deploys trigger 40% of SEV-2s” and “30% of deployment points in api-gateway come from Connection Timeouts”. Floor this as an perception and we proactively begin lowering incidents, not simply react shortly to new ones.
Placing the graph to work
On high of the graph sits AutoRCA, our inner agentic AI answer, constructed with a set of specialist brokers coordinated round one query. Strolling the house web page regression from the highest of this submit, the agentic system’s reasoning chain may look one thing like this:
Every specialization to the inputs is constructed by individuals who perceive their respective area deeply. That division of labour solely works if the specialists can cross findings to at least one one other with out having to agree on the phrases of communication and even have information of one another. The ontology acts because the interface. Every specialist learns one vocabulary — the graph’s. A crew writing specialised use circumstances for a way they use traces file their findings as triples about an ops:Utility, the logs specialization reads triples about that very same ops:Utility, and neither must know the opposite exists. A brand new specialist learns the identical vocabulary as soon as and may instantly learn the whole lot the opposite six have written.
Set in opposition to the investigation we opened with, the change is narrower than it would sound. The alert nonetheless fires at 3:30pm, an engineer nonetheless has to weigh the proof, and the repair nonetheless must be written by somebody who understands the code. The hour or two that used to enter assembling context — discovering the proudly owning crew, reconstructing what shipped, correlating data from completely different components of the system, recalling that one thing related occurred final quarter — all that work is the half the ontology and the graph absorbs. We’re early sufficient that the ranked hypotheses are generally improper, which is why every one carries the proof that produced it. However when the engineer now begins the investigation, they’re already holding the image that used to take half a day to assemble.
If you wish to construct one in all these
A good quantity of what we now have constructed, we realized by getting it improper first. Primarily based on this expertise, these are the ideas we might carry right into a second try.
- Mannequin identification earlier than anything. Select one authoritative catalog per entity sort and validate in opposition to it earlier than creating an entity.
- Deal with the ontology as code. Model it, evaluate it, check it, and let modifications arrive as pull requests. Treating the ontology as code lends itself to agentic growth and lets groups scale and contribute to it.
- Separate ideas from cases. Ideas change on a human timescale and cases change constantly, conflating them makes each more durable to evolve.
- Construct on open requirements. RDF, OWL, SHACL and SPARQL are specified, carried out many occasions over, and transportable. A bespoke graph mannequin ties a future client to at least one implementation.
- Constrain stochastic instruments with a kind system. That is the precept we might defend hardest. Fashions are helpful exactly the place determinism is unattainable, and a sufficiently slim output contract converts their output into one thing a question engine can belief.
- Make enrichment best-effort and ordered. A sluggish dependency ought to utmost degrade the graph, by no means block it. And ordering enrichers so every builds on the final buys extra accuracy and that’s well worth the tradeoff with parallelism that buys pace.
- Mannequin human reasoning alongside machine telemetry. Probably the most priceless edges in our graph got here from dialogue. Telemetry information what the system did, dialogue information what folks understood and interpreted. The reasoning is extra priceless to retailer and function knowledge after we want it.
- Preserve people because the approvers. Let brokers suggest fixes, enrichers, and ontology modifications, and route each proposal by a evaluate course of with people within the loop.
What comes subsequent
The graph makes three issues potential. First, computerized root trigger evaluation, the place a regression arrives with a ranked set of graph-backed hypotheses hooked up. Subsequent, automated remediation, the place a sufficiently assured analysis carries a proposed motion like a focused rollback, a visitors shift, a code change. And ultimately infrastructure that heals itself, appearing on a predicted failure earlier than members really feel it.
Alongside this, we plan to combine the Observability Ontology with Higher, the domain-modeling language on the coronary heart of Netflix’s Unified Information Structure (UDA). Higher lets groups describe enterprise and system domains as interconnected, machine-readable fashions and protect their that means and constraints as these fashions are projected into concrete schemas and techniques. This integration would join operational ideas similar to functions, alerts, deployments, and incidents to the broader set of domains modeled throughout Netflix, giving engineers and brokers a shared semantic layer for querying, validation, and reasoning. We are going to start with a proof of idea that fashions incidents in Higher and use what we be taught to find out the place the 2 techniques ought to converge.
Acknowledgements
None of this could exist with out the groups who share their knowledge, experience, and endurance with an early system. We need to specifically acknowledge the efforts of our personal crew members who made this potential from very early on on this lifecycle, Kaela Marsheck and Aditya Gaur. We’re additionally particularly grateful to our collaborators in Netflix’s ontology working group, Peter Royal, Alex Hutter, and to Alexandre Bertails and members of the Unified Information Structure (UDA)/Information Foundations crew for sharing Higher and their domain-modeling experience, and for serving to us discover how the Observability Ontology can develop into a part of a broader semantic basis at Netflix.
Ontology-Pushed Observability: Constructing an Finish-to-Finish Information Graph at Netflix Scale was initially revealed in Netflix TechBlog on Medium, the place persons are persevering with the dialog by highlighting and responding to this story.
