By Shreya S Ramanujam, Yongce Li, Afshine Amidi, Sina Ghiassian, Benjamin Klein.
Introduction
Image this — you’re residence after a protracted day at work and might’t wait to atone for Bridgerton. You go online to Netflix, seek for “Bridgerton” and land on this display screen:
If this sounds acquainted, you already know what a synopsis is — it’s a brief glimpse into what a viewer can count on to see in a present. We see two kinds of synopses right here: a series-level synopsis describing the overarching plot, and a number of episode-level synopses describing every episode.
A synopsis performs an important position for us — it helps viewers determine whether or not a title is value watching or not. Each synopsis must strike a steadiness: sharing simply sufficient to provide viewers a way of what to anticipate, whereas leaving sufficient intrigue to attract them in. However after all, we will’t simply write something.
“Within the fog-drenched streets of Victorian London, Detective Daphne Bridgerton races to unmask a serial killer recognized solely because the elusive Woman Whistledown.”
Intriguing? Undoubtedly. However it’s most definitely not the premise of Bridgerton. Which leads us to the primary (and most vital) rule of writing a very good synopsis: it have to be factual. At Netflix’s scale, with 1000’s of titles, a number of episodes per season and a number of languages, fact-checking all of those by hand merely doesn’t scale. That is the place our factuality service is available in: an automatic approach to choose whether or not a synopsis is factual or not.
Our present factuality checker feeds a reasoning mannequin every thing we find out about a title (like metadata and textual content belongings) and asks it to guage synopsis factuality in accordance with a rigorous rubric. There are two primary limitations of this strategy that we need to optimize:
- Excessive value of long-context reasoning calls: Every resolution requires costly, long-context mannequin calls to reasoning fashions, which doesn’t scale nicely once we need to examine a number of titles, episodes, languages, and magnificence variants.
- Misaligned rationales: The mannequin’s explanations typically diverge from our annotators’ reasoning, which makes its factuality verdicts much less dependable. We wish these rationales to be higher aligned in order that (1) the mannequin can persistently flag nuanced errors, and (2) its explanations can be utilized downstream to routinely appropriate points within the synopsis.
Our new strategy tackles each issues without delay. On the associated fee aspect, it delivers a 52.4% discount per synopsis. On the standard aspect, it considerably improves each verdict accuracy and reasoning alignment: Recall–Fail (“Of all actually non-factual synopses, what number of does the system accurately reject?”) rises from 0.77 to 0.86, and the brand new factuality service’s explanations now present ~82% overlap with human rationales (up from ~69%). In different phrases, it’s not simply cheaper and extra correct — it’s additionally a lot better aligned with how our professional annotators really choose synopses.
So, what precisely does factuality imply?
Let’s return to the Bridgerton instance. Say you see this synopsis about Bridgerton Season 1:
Synopsis A: “Bridgerton follows Daphne Bridgerton, the eldest daughter of the highly effective Bridgerton household as she makes her debut onto Regency London’s aggressive marriage market.”
It’s straightforward to guage that this synopsis is factually appropriate — no improper details, and it offers the reader a transparent, correct glimpse of what to anticipate from the present.
Let’s take one other instance:
Synopsis B: “Set in modern-day London, this gritty crime drama follows Detective Daphne Bridgerton as she hunts a serial killer whereas hiding her personal prison previous.”
One other straightforward one — the plot is so blatantly off the mark that it’s undoubtedly not factual.
However what about this?
Synopsis C: “Bridgerton follows the mysterious, sharp-tongued Woman Whistledown as she exposes the secrets and techniques of London’s excessive society via her scandalous gossip sheet.”
Technically, each assertion right here is true — Woman Whistledown does certainly publish a gossip sheet, and if there’s one factor she is understood for, it’s her scathing exposés. So then, is that this a factual synopsis?
Not fairly! Whereas technically true, it paints a deceptive image of the present. A brand new viewer may moderately count on it to revolve round Woman Whistledown, moderately than Daphne navigating the wedding market. This, too, is a kind of non-factuality we need to catch.
Lastly, right here’s the trickiest of all:
Synopsis D: “When Daphne Bridgerton makes her society debut, she catches the attention of the aloof Duke of Hastings, and the 2 start a passionate love affair that units the entire ton abuzz.”
Once more, each assertion is true, however the phrasing is deceptive. A brand new viewer may assume they fall in love instantly, lacking the truth that their relationship begins as a pretend, mutually useful association. That is one other kind of nuanced error our system ought to flag.
So now that we have now an thought of the sorts of errors we care about, the actual query is: how will we construct a system that is ready to catch each evident in addition to refined factuality violations?
Present Service and Its Limitations
Our present factuality pipeline is an easy, holistic LLM name. We feed a reasoning mannequin every thing we find out about a title (textual content belongings, metadata, and an in depth factuality rubric) and ask it to guage the factuality of a synopsis. We repeat the decision a number of instances to cut back output randomness, and if any run flags the synopsis as non-factual, it fails. This strategy has two primary limitations:
- Excessive value: Every judgment requires a number of (5, to be precise) calls to a big reasoning mannequin over extraordinarily lengthy contexts (100k+ tokens), which doesn’t scale nicely.
- Misaligned reasoning: The mannequin can fail synopses for nitpicky causes (e.g., “It is a present about great magic” will get failed as a result of the characters by no means actually name the magic “great”) whereas overlooking nuanced errors by convincing itself they “don’t actually make the synopsis false.” This misalignment makes verdicts much less dependable and rationales much less helpful for downstream correction.
Optimizing the Holistic Pipeline
To know precisely the place we should always optimize the present holistic pipeline, we research the present pipeline each qualitatively and quantitatively on a customized factuality benchmark that we constructed. This benchmark contains synopses hand-labeled by human annotators as factual or non-factual, together with their rationales. This evaluation reveals a number of key insights:
Perception 1: Do we actually want 5 holistic calls?
Let’s assume again to why we added a number of holistic calls within the first place: to cut back sampling noise. In apply, although, we see an fascinating dichotomy:
- Blatantly true / false synopses (Bridgerton A, B): All runs agree on the decision with near-identical reasoning.
- Nuanced errors (Bridgerton C, D): The mannequin is more likely to disagree with itself — some runs catch the error, some don’t. That is precisely the place extra sampling helps.
This implies a easy optimization: A and B don’t actually need all 5 calls. If we will by some means determine these “straightforward” synopses upfront, we will minimize down on the variety of requires them, whereas nonetheless giving tougher instances like C and D the additional calls they want.
The simplest method to do that is to ask the mannequin to output its confidence alongside its verdict and rationale. We will then separate “straightforward” (excessive confidence) synopses from “onerous” (low confidence) ones and solely ship the onerous instances via extra checks. Consider it like airport safety: most luggage go straight via, however those who set off the alarm (low confidence) get further screening.
Implementing this straightforward repair makes our pipeline go from the present holistic mannequin to a brand new “confidence gated” holistic mannequin:
Perception 2: 5 similar judges calls give us no new info
With confidence gating, our pipeline solely escalates tougher synopses to the complete 5 holistic calls. However further calls solely assist with sampling noise, not systematic reasoning errors: if each run follows the identical flawed rationale, they’ll all attain the identical improper verdict, rendering the additional calls ineffective. How can we be sure that every name counts?
Concept: Diversify the judges and make them specialised
As a substitute of operating 5 holistic judges with the identical generic factuality immediate, we introduce two specialised judges that individually goal the 2 elements of factuality we care about:
- Literal Choose: Is the synopsis actually true?
- Implicature Choose: Does this synopsis give a false image to a brand new reader who has not watched this present?
Explicitly looking for particular kinds of errors helps us higher flag each literal and subtler implicature errors. In apply, we observe that this works higher than repeatedly querying a single holistic choose, hoping that sampling stochasticity will floor nuanced errors.
Perception 3: The rationale reveals errors the decision hid
At this level, we have now a reasonably good pipeline, however we’re nonetheless underusing one of many richest alerts out there to us: the mannequin’s rationale behind its verdict. For nuanced errors, we regularly see the mannequin determine some extent of rivalry, then brush it apart primarily based on a defective assumption or as a result of it deems the difficulty too minor to have an effect on the decision. Can we use these shaky spots within the rationale to catch (and proper) errors the decision misses?
Concept: Secondary atomic-claim checking through a router:
Since our literal and implicature judges are already robust reasoning fashions, we ask them to moreover critically analyse their very own reasoning and floor shaky spots — any doubts, concessions, or assumptions which may want rechecking. These embody instances the place the mannequin:
- Makes an assumption.
- Raises a doubt that the out there proof can’t resolve (e.g., an exterior critic quote).
- Notices a contradiction however dismisses it as inadequate to vary the verdict.
We move all concessions from each judges to a router mannequin, which:
- Reframes them as self-contained, verifiable atomic claims, every with a materiality discipline specifying the proof wanted to help or refute it.
- Routes every declare to the suitable secondary Stage-2 choose: a internet choose for exterior details (e.g., citations, media quotes, or prior work), or a source-material recheck targeted on the reframed atomic declare. By pinpointing the precise level of rivalry, the recheck is extra focused, making it simpler to mark the declare as true or false.
Placing It All Collectively: Proposed Cascade Factuality Service
Our findings result in three key enhancements: specialised judges, confidence-gated escalation, and focused rechecking of shaky reasoning utilizing a secondary stage when escalated.
We implement these enhancements as a tiered system proven within the determine under: each synopsis first passes via each the literal and the implicature judges. Excessive-confidence verdicts are retained, whereas low-confidence instances escalate to the router and Stage-2 judges for additional checking. Stage 2 can overturn the preliminary verdict in two instances:
- FAIL → PASS: All concessions, together with the unique failure causes, are supported by the secondary examine.
- PASS → FAIL: At the least one concession is contradicted.
Outcomes
Based mostly on our discussions from above, we consider three completely different pipelines:
Analysis: Datasets and Metrics
We consider these programs on two datasets: show-level and episode-level synopses. Every datapoint features a synopsis, a human gold label (factual / not factual), and the annotator’s rationale. For episode-level synopses, we moreover require that the synopsis be particular to that specific episode, not only a generic synopsis of the general present.
We deal with two primary axes: High quality and Price. The standard metrics are listed under:
Recall-FAIL is our major metric: larger recall means fewer Gold FAILs are missed. We additionally monitor Precision-PASS, which measures how reliably PASS verdicts determine Gold PASSes, and alignment with human rationales.
On the associated fee aspect, we need to scale back each the common variety of LLM calls per synopsis and the end-to-end value of manufacturing a remaining verdict. Latency just isn’t our major optimization goal, however we do need to make sure the median (P50) latency doesn’t blow as much as impractical judgement instances.
Analysis Outcomes
The most effective numbers per metric per dataset are proven in daring.
Word: We in contrast the cascade in opposition to two holistic ablations — elevated reasoning effort and a smaller, cheaper mannequin — to check if easy ablations might obtain higher efficiency. For episode-level synopses, we additionally added a full-show abstract to the immediate to offer context from earlier episodes; we observe summaries considerably enhance efficiency when in comparison with non-summary context variants, so we report solely summary-based variants.
Our takeaways from this experiment are:
- Cascade improves high quality. Our major metric, Recall-FAIL improves on each present and episode stage synopses, with a restricted affect on Recall-PASS. In different phrases, we now catch extra Gold FAIL synopses with out sacrificing the reliability of our PASS verdicts.
- Cascade is cheaper. Cascade cuts complete value by 52.4% on the show-level dataset. Cascade (with abstract) can be 15% cheaper than the present (no abstract) holistic pipeline on episodic dataset.
- Cascade’s P50 latency will increase with respect to the holistic baseline. Even so, this stays inside what we think about an affordable vary for factuality judgment.
- Easy tweaks aren’t sufficient. Rising reasoning effort makes the factuality checker too lenient, whereas utilizing a smaller mannequin makes it too strict — neither addresses the underlying issues of the holistic baseline.
- Abstract context issues for episodic duties. With out a full-show abstract, the mannequin tends to over-fail as a result of it lacks the context to find out whether or not a element from the episode abstract suits the broader storyline.
Alignment with Human Rationale
Up to now, we’ve established that the cascade produces higher verdicts: it catches extra non-factual synopses and produces extra dependable PASS selections. However are these verdicts appropriate for the proper causes?
To measure this, we use an LLM-as-a-Choose to check every pipeline’s reasoning in opposition to the human rationale on Gold-FAIL (GF) synopses. The choose assigns a Reasoning Similarity Rating (RSS) of 0 (dissimilar), 0.5 (partially comparable), or 1 (very comparable) primarily based on rationale overlap.
We observe that cascade is considerably higher aligned with human rationale in all instances. It is a massive achieve — not solely is the cascade giving us extra correct factuality verdicts, nevertheless it additionally identifies the identical factors of rivalry within the synopsis as a specialised human annotator.
Conclusion + Future Work
Total, our proposed cascade pipeline improves each value and high quality of the prevailing factuality service:
- On the value aspect, confidence gating lets us keep away from pointless calls, decreasing the common value of factuality analysis.
- On the high quality aspect, two numerous judges enable us to gather complementary informative verdicts, whereas the Stage-2 judges recheck and proper the assumptions behind unsure selections. This leads to larger verdict accuracy and considerably higher alignment between mannequin and professional annotator reasoning.
The cascade offers us a path towards a factuality system that’s not simply extra correct, but additionally extra interpretable and self-correcting. Some future instructions to contemplate would be:
- Including multimodal inputs to assist confirm claims that textual content belongings alone can’t affirm.
- Use the rationale from factuality checking to appropriate the defective synopsis.
Reality-Checking at Scale: Constructing a Extra Correct and Environment friendly Synopsis Factuality System was initially printed in Netflix TechBlog on Medium, the place individuals are persevering with the dialog by highlighting and responding to this story.
