MPEG-I part 8, Network-Based Media Processing (NBMP) vision paper
Media processing continues to evolve to address ever more complex tasks and services. Applications are very varied and only limited by the imagination of the developers and the processing power of the target devices. Applications ranging from smart upscaling to complex AR and 6DoF scene composition and pre-rendering are however very processing intensive and rely on advanced deep learning algorithms that require dedicated hardware to run in real-time. Obviously, processing capabilities in handheld or mobile devices are evolving at a fast pace. However, it is not expected that handheld or mobile devices will offer processing capabilities comparable to system equipment due to several factors, such as power consumption, cost, mobility, and dimensions.
In order to address the needs of advanced multimedia services and to deliver the latest and most immersive media experience to end users, leveraging processing capabilities and resources in the network comes to rescue. Multimedia service providers and network/cloud service providers work together to provide customized immersive services to their customers. Unfortunately, this approach is being hampered by fragmentation. Multimedia service providers are faced with the challenge of adapting their services to multiple cloud and network service providers to reach their customers. These cloud and network service providers oftentimes define their own APIs to assign compute resources to their customers.
The Network-based Media Processing (NBMP) standard has been developed to address fragmentation and offer a unified way to perform media processing on top of any cloud platform and on any IP network. NBMP defines interfaces, media and metadata formats to facilitate instantiating any type of media processing in the network/cloud. NBMP relies on a Workflow Manager, an entity that will typically be virtualized, to start and control media processing. The Workflow Manager receives a Workflow Description from the NBMP Source, which instructs the Workflow Manager about the desired processing and the input and output formats to be taken and produced, respectively.
The Workflow Manager may use ready-made implementations of media processing functions and compose them together to create the media processing workflow. NBMP defines a Function Discovery API that it uses with a Function Repository to discover and load the desired Functions. A Function, once loaded, becomes a Task, which is then configured by the Workflow Manager through the Task API and can start processing incoming media.
MPEG expects that NBMP will reach the FDIS stage in January 2020.
