July 17, 2024
In your eyes solely: enhancing Netflix video high quality with neural networks | by Netflix Expertise Weblog | Nov, 2022
  1. Video preprocessing, which encompasses any transformation utilized to the high-quality supply video previous to encoding. Video downscaling is essentially the most pertinent instance herein, which tailors our encoding to display screen resolutions of various units and optimizes image high quality underneath various community situations. With video downscaling, a number of resolutions of a supply video are produced. For instance, a 4K supply video might be downscaled to 1080p, 720p, 540p and so forth. That is usually finished by a standard resampling filter, like Lanczos.
  2. Video encoding utilizing a standard video codec, like AV1. Encoding drastically reduces the quantity of video knowledge that must be streamed to your system, by leveraging spatial and temporal redundancies that exist in a video.
  • A realized strategy for downscaling can enhance video high quality and be tailor-made to Netflix content material.
  • It may be built-in as a drop-in answer, i.e., we don’t want some other modifications on the Netflix encoding aspect or the shopper system aspect. Tens of millions of units that help Netflix streaming mechanically profit from this answer.
  • A definite, NN-based, video processing block can evolve independently, be used past video downscaling and be mixed with completely different codecs.
Structure of the deep downscaler mannequin, consisting of a preprocessing block adopted by a resizing block.
Left: Lanczos downscaling; proper: deep downscaler. Each movies are encoded with VP9 on the identical bitrate and have been upscaled to FHD decision (1920×1080). You could have to zoom in to see the visible distinction.
  • The neural community structure was designed to be computationally environment friendly and likewise keep away from any damaging visible high quality impression. For instance, we discovered that only a few neural community layers have been enough for our wants. To cut back the enter channels even additional, we solely apply NN-based scaling on luma and scale chroma with a typical Lanczos filter.
  • We carried out the deep downscaler as an FFmpeg-based filter that runs along with different video transformations, like pixel format conversions. Our filter can run on each CPU and GPU. On a CPU, we leveraged oneDnn to additional cut back latency.
A top-down view of integrating the deep downscaler into Cosmos.