MPEG-21 Evaluation Methods for Persistent Association Technologies (PAT)

MPEG doc#: N7455
Date: July 2005
Author: Niels Rump

1         Overview

MPEG-21 provides a framework within which many elements of multimedia are brought together. To handle such content a requirement for tools has been identified that can create and manage (e.g. detect or extract) an association between content and metadata pertaining to the content within MPEG-21. Tools based on the techniques known as “watermarking” and “fingerprinting” offer a means to form such associations, whereby the information can be directly embedded within or inferred from the content itself. Such tools are termed Persistent Association Technologies (PAT) and, within MPEG-21.

Recognising, however, that the standardisation of Persistent Association Technologies (PAT) is not currently thought to be viable in the context of MPEG-21 and the wider international standardisation community[1], ISO/IEC JTC 1/SC 29/WG 11 (MPEG) examined options which would allow it to assist the adoption of PAT by industry. This analysis also showed both a need and a possibility to establish a Technical report describing Evaluation Methods of Persistent Association Technologies. ISO/IEC TR 21000-11 provides best practices for such evaluations in a Technical Report.[2]

The purpose of ISO/IEC 21000-11 is to allow such evaluations to be conducted using a common evaluation framework with specific test methodologies for each of the discussed persistent association technology types or paradigms. This is intended to give confidence to those relying on the results that they are: 

• Appropriate tests of the technology that will predict its performance under real-world conditions and
• Comparable with results obtained from other tests conducted using the same methodology. 

2         Characteristics for PAT

The evaluations of Persistent Association Technologies as described in ISO/IEC 21000-11 rely on seven parameters that to aid users of such technologies when choosing (i) the class of PAT to use and (ii) a specific solution within the chosen class of technologies. These characteristics have been identified as being the most important. While this list of criteria is not exhaustive, a properly conducted test using them will enable the user to make an informed decision. They are: 

• Fingerprint Size;
• Watermark Payload;
• Granularity;
• Perceptibility;
• Robustness;
• Reliability; and
• Computational Performance.

3         Generic Framework and Methodology for Evaluation of PAT

The first step in any PAT evaluation concerns setting out the framework within which the application will be utilised, and then qualitatively examining different approaches to PAT within that framework. Initial questions will concern the applicability of the PAT paradigms. Questions may be asked such as: 

• Is the information in the PAT self-contained, or is a database required?
• Does the PAT need to scale across a large repertoire of distinct content files?
• Are false positives and false negatives of equal importance in the application?
• Will the PAT be subject to malicious attacks?
• Does the PAT need to carry cryptographic elements, or be embedded using cryptographic techniques?
• What computational resources are required for embedding or recognising the PAT?

The process described in ISO/IEC 21000-11 provides, for each of the seven characteristics listed above, and for each of the two PAT paradigms (i.e. watermarking and fingerprinting), a set of recommendations for (i) creating a test environment and (ii) for executing tests that will be able to show how well a particular PAT would fare in a particular application scenario.