MPEG Auxiliary Video Data Representations
MPEG doc #: N8039
Date: April 2006
Authors: Arnaud Bourge (Philips), Christoph Fehn (Fraunhofer HHI)
1. Introduction
The purpose of ISO/IEC 23002-3 Auxiliary Video Data Representations is to support all those applications where additional data needs to be efficiently attached to the individual pixels of a regular video. In ISO/IEC 23002-3 it is described how this can be achieved in a generic way by making use of existing (and even future) video codecs available within MPEG.
2. Applications
A good example of an application that requires additional information associated with the individual pixels of a regular (2D) video stream is stereoscopic video presented on an auto-stereoscopic, single or multiple-user display. At the MPEG meeting in Nice (October 2005), the arrival of such displays on the market has been stressed, and several of them were even shown and demonstrated [1]. Because different display realizations vary largely in: (a) the number of views that are represented; (b) the maximum parallax that can be supported, an input format is required that is flexible enough to drive all possible variants. This can be achieved by supplying a depth or parallax values with each pixel of a regular video stream, and by generating the required stereoscopic views at the receiver side [2]. The standardization of a common depth, resp. parallax format within ISO/IEC 23002-3 Auxiliary Video Data Representations will thus enable interoperability between content providers, broadcasters and display manufacturers.
ISO/IEC 23002-3 is flexible enough to easily add other types of auxiliary video data in the future. One example could be the annotation of temperature maps coming from a infrared camera to regular video coming from a regular camera.
3. Format
The Auxiliary Video Data format defined in ISO/IEC 23002-3 consists of an array of N-bit values which are associated with the individual pixels of a regular video stream [3]. These data can be compressed like conventional luminance signals using already existing (and even future) MPEG video codecs. The format allows for optional subsampling of the auxiliary data in both the spatial and temporal domain. This can be beneficial depending on the particular application and its requirements [5] and allowing for very low bitrates for the auxiliary data.
The specification is very flexible in the sense that it defines a new 8-bit code word aux_video_type that specifies the type of the associated data, e.g., currently a value of 0x10 signals a depth map, a value of 0x11 signals a parallax map. New values for additional data representations can be easily added to fulfill future demands.
4. Transport Issues
The transport of auxiliary video data within an MPEG-2 Transport or Program Stream is defined in an Amendment [4] to the MPEG-2 Systems standard. It specifies new stream_id_extension and stream_type values that are used to signal an Auxiliary Video Data stream. An additional auxiliary_video_data_descriptor is utilized in order to convey in more detail how the data should to be interpreted by the application that uses them. Meta data associated with the auxiliary data is carried on system level allowing the use of unmodified video codecs (no need to modify silicon).
5. Conclusions
ISO/IEC 23002-3 Auxiliary Video Data Representations will provide an efficient way for attaching additional information, e.g., depth values, parallax values, etc., to the individual pixels of a regular video stream and to signal how these associated data should be interpreted by the application that uses them.
References
1. Philips 3D Solutions [Online] (2005). Available: http://www.business-sites.philips.com/ 3dsolutions.
2. C. Fehn, P. Kauff, M. Op de Beeck, F. Ernst, W. IJsselsteijn, M. Pollefeys, L. Van Gool, E. Ofek and I. Sexton. An Evolutionary and Optimised Approach on 3D-TV. In Proceedings of International Broadcast Conference, pages 357-365, Amsterdam, The Netherlands, September 2002.
3. ISO/IEC JTC 1/SC 29/WG 11. Working Draft 1.0 of ISO/IEC 23002-3 Auxiliary Video Data Representations. WG 11 Doc. N7898. Bangkok, Thailand, January 2006.
4. ISO/IEC JTC 1/SC 29/WG 11. Working Draft 1.0 of ISO/IEC 13818-1:200X/AMD 2. WG 11 Doc. N7818. Bangkok, Thailand, January 2006.
5. ISO/IEC JTC 1/SC 29/WG 11. Applications and Requirements for StereoScopic Video (SSV), WG 11 Doc. W7777, Bangkok, Thailand, January 2006.