by Akshay Garg, Roger Quero
Maximizing immersion for our members is a vital purpose for the Netflix product and engineering groups to maintain our members entertained and totally engaged in our content material. Leveraging an excellent mixture of mature and cutting-edge consumer system applied sciences to ship a easy playback expertise with glitch-free in-app transitions is a vital step in direction of reaching this purpose. On this article we clarify our journey in direction of productizing a greater viewing expertise for our members by using options and capabilities in client streaming gadgets.
If in case you have a streaming system related to your TV, similar to a Roku Set Prime Field (STB) or an Amazon FireTV Stick, you will have come throughout an possibility within the system show setting pertaining to content material body charge. Machine producers usually name this characteristic “Match Content material Body Charge”, “Auto modify show refresh charge” or one thing related. In the event you’ve ever questioned what these options are and the way they’ll enhance your viewing expertise, preserve studying — the next sections cowl the fundamentals of this characteristic and clarify the main points of how the Netflix software makes use of it.
Netflix’s content material catalog consists of video captured and encoded in one among numerous body charges starting from 23.97 to 60 frames per second (fps). When a member chooses to observe a film or a TV present on a supply system (ex. Set-top field, Streaming stick, Sport Console, and so forth…) the content material is delivered after which decoded at its native body charge, which is the body charge it was captured and encoded in. After the decode step, the supply system converts it to the HDMI output body charge which was configured based mostly on the capabilities of the HDMI enter port of the related sink system (TV, AVR, Monitor and so forth). Generally, the output body charge over HDMI is routinely set to 50fps for PAL areas and 60fps for NTSC areas.
Netflix presents restricted excessive body charge content material (50fps or 60fps), however the majority of our catalog and viewing hours may be attributed to members watching 23.97 to 30fps content material. This primarily implies that more often than not, our content material goes by a course of referred to as body charge conversion (aka FRC) on the supply system which converts the content material from its native body charge to match the HDMI output body charge by replicating frames. Determine 1 illustrates a easy FRC algorithm that converts 24fps content material to 60fps.
Changing the content material and transmitting it over HDMI on the output body charge sounds logical and simple. In actual fact, FRC works nicely when the output body charge is an integer a number of of the native body charge ( ex. 24→48, 25→50, 30→60, 24→120, and so forth…). However, FRC introduces a visible artifact referred to as Judder when non-integer a number of conversion is required (ex. 24→60, 25→60, and so forth…), which manifests as uneven video playback as illustrated under:
You will need to notice that the severity of the judder depends upon the replication sample. Because of this, judder is extra distinguished in PAL areas due to the method of changing 24fps content material to 50fps over HDMI (see Determine 2):
- Whole of fifty frames have to be transmitted over HDMI per second
- Supply system should replicate the unique 24 frames to fill within the lacking 26 frames
- 50 output frames from 24 unique frames are derived as follows:
- 22 frames are duplicated ( complete of 44 frames )
- 2 frames are repeated 3 times ( complete of 6 frames )
As a evaluation, judder is extra pronounced when the frequency of the variety of repeated frames is inconsistent and unfold out e.g. within the situation talked about above, the body replication issue varies between 2 and three leading to a extra distinguished judder.
Now that we’ve got a greater understanding of the difficulty, let’s evaluation the options that Netflix has invested in. Because of the fragmented nature of system capabilities within the ecosystem, we explored a number of options to handle this difficulty for as many gadgets as potential. Every distinctive resolution leverages present or new supply system capabilities and comes with numerous tradeoffs.
The primary resolution we explored and just lately enabled leverages the aptitude of present supply & sink gadgets to vary the outgoing body charge on the HDMI hyperlink. As soon as this characteristic is enabled within the system settings, gadgets will match the HDMI output body charge with the content material body charge, both precisely or an integer a number of, with out consumer intervention.
Whereas this seems like the proper resolution, gadgets that assist older HDMI applied sciences e.g. HDMI v<2.1, can’t change the body charge with out additionally altering the HDMI information charge. This ends in what is usually referred as an “HDMI bonk” which causes the TV to show a clean display momentarily. Not solely is that this a disruptive expertise for members, however the period of the clean display varies relying on how briskly the supply and sink gadgets can resynchronize. Determine 3 under is an instance of how this transition seems:
Enhancements within the current HDMI requirements (HDMI 2.1+) now enable a supply system to ship the video content material at its native body charge with no need an HDMI resynchronization. That is potential by an revolutionary know-how referred to as Fast Media Switching (QMS) which is an extension of Variable Refresh Charge (VRR) focused for content material playback eventualities. QMS permits a supply system to keep up a continuing information charge on the HDMI hyperlink even throughout transmission of content material with completely different body charges. It does so by adjusting the quantity of non-visible padding information whereas maintaining the quantity of seen video information fixed. Because of the fixed HDMI information charge, the HDMI transmitter and receiver don’t must resynchronize, resulting in a seamless/glitch-free transition as illustrated in Determine 4.
HDMI QMS is positioned to be the perfect resolution to handle the issue we’re presenting. Sadly, at current, this know-how is comparatively new and adoption into supply and sink gadgets will take time.
Aside from the above HDMI specification dependent options, it’s potential for an software like Netflix to control the presentation time stamp worth of every video body to attenuate the impact of judder i.e. the appliance can current video frames to the underlying supply system platform at a cadence that may assist the supply system to attenuate the judder related to FRC on the HDMI output hyperlink.
Allow us to perceive this concept with the assistance of an instance. Let’s return to the identical 24 to 50 fps FRC situation that was lined earlier. However, as an alternative of serious about the FRC charge per second (24 ⇒ 50 fps), let’s develop the FRC calculation time interval to three seconds (24*3 = 72 ⇒50*3 = 150 fps). For content material with a local body charge of 24 fps, the supply system must get 72 frames from the streaming software in a interval of three seconds. Now as an alternative of sending 24 frames per second at an everyday per second cadence, for every 3 second interval the Netflix software can resolve to ship 25 frames within the first 2 seconds (25 x 2 = 50) and 22 frames within the third second thereby nonetheless sending a complete of 72 (50+22) frames in 3 seconds. This method creates an excellent FRC within the first 2 seconds (25 frames replicated twice evenly) and within the third second the supply system can do a 22 to 50 fps FRC which can create much less visible judder in comparison with the 24->50 fps FRC given a extra even body replication sample. This idea is illustrated in Determine 5 under.
NOTE: This resolution was developed by David Zheng within the Accomplice Expertise Expertise workforce at Netflix. Be careful for an upcoming article going into additional particulars of this resolution.
Given the potential options obtainable to make use of and the related advantages and limitations, the Netflix software working on a supply system adapts to make use of one among these approaches based mostly on elements similar to supply and sink system capabilities, consumer preferences and the precise use case inside the Netflix software. Let’s stroll by every of those elements briefly.
Each supply system that integrates the Netflix software is required to let the appliance know if it and the related sink system have the power to ship and obtain video content material at its native body charge. As well as, a supply system is required to tell whether or not it might probably assist QMS and carry out a seamless playback begin of any content material at its native body charge on the related HDMI hyperlink.
As mentioned within the introduction part, the presence of a system setting like “Match Content material Body Charge” sometimes signifies {that a} supply system is able to this characteristic.
Even when a supply system and the related sink can assist Native content material body charge streaming (seamless or non-seamless), a consumer might need chosen not to do that through the supply system system settings e.g. “Match Content material Body Charge” set to “By no means”. Or they may have indicated a desire of doing this solely when the native content material body charge play begin can occur in a seamless method e.g. “Match Content material Body Charge” set to “Seamless”.
The Netflix software must know this consumer choice in an effort to honor their desire. Therefore, supply gadgets are anticipated to relay this consumer desire to the Netflix software to assist with this run-time resolution making.
Despite supply system functionality and the consumer preferences collectively indicating that the Native Content material Body Charge streaming needs to be enabled, the Netflix software can resolve to disable this characteristic for particular member experiences. For instance, when the consumer is looking Netflix content material within the residence UI, we can’t play Netflix trailers of their Native body charge as a result of following causes:
- If utilizing Answer # 1, when the Netflix trailers are encoded in various content material body charges, switching between trailers will end in display blanking, thereby making the UI looking unusable.
- If utilizing Answer # 2, sending Netflix trailers of their Native body charge would imply that the related UI parts (motion of cursor, asset choice and so forth) would even be displayed on the lowered body charge and this can end in a sluggish UI looking expertise. It is because on HDMI output from the supply system, each graphics (Netflix software UI) and video parts will exit on the similar body charge (native content material body charge of the trailer) after being blended collectively on the supply system.
To deal with these points we observe an method as proven in Determine 6 under the place we allow the Native Body Charge playback expertise solely when the consumer selects a title and watches it in full display with minimal graphical UI parts.
This text offered options that purpose to enhance the content material playback expertise on HDMI supply gadgets. The breadth of accessible technical options, consumer selectable preferences, system capabilities and the appliance of every of those permutations within the context of assorted in-app member journeys symbolize a typical engineering and product resolution framework at Netflix. Right here at Netflix, our purpose is to maximise immersion for our members by introduction of latest options that can enhance their viewing expertise and preserve them totally engaged in our content material.
We wish to acknowledge the arduous work of quite a lot of groups that got here collectively to ship the options being mentioned on this doc. These embrace Core UI and JS Participant growth, Netflix Utility Software program growth, AV Check and Tooling (earlier article from this workforce), Accomplice Engineering and Product groups within the Shopper Engineering group and our information science associates within the Knowledge Science and Engineering group at Netflix. Diagrams on this article are courtesy of our Accomplice Enterprise Platform XD workforce.
