MPEG Symbolic Music Representation
MPEG doc#: N7379
Date: July 2005
Authors: Pierfrancesco Bellini, Maurizio Campanai, Paolo Nesi, Giorgio Zoia
A symbolic representation of music is a logical structure based on symbolic elements representing audiovisual events, the relationships among those events, and how they can be rendered and synchronized with other media types. Many notations have been developed over the years and ages to represent visually or by other means the information needed by a performer to play the musical piece and produce music as composed and imagined by the author.
Symbolic music representation generalizes music notation concepts to model the visual aspects of a music score, and audio information or annotations related to the music piece. Symbolic Music Representation overcomes the limitations of MIDI, which is good enough to transport music event information (its main purpose), and it has limitations in producing satisfactory results on the audio and visual representation sides.
The evolution of information technology has more recently produced changes in the practical use of music representation and notation, transforming them from a simple visual coding model for sheet music into a tool for modeling music in computer programs and electronic devices in general. As a consequence, symbolic music representation is currently used for several purposes other than sheet music production and music teaching, such as for audio rendering, entertainment, music analysis, database query, performance coding, etc.
Symbolic Music Representation and MPEG
The MPEG-4 technology covers a huge media domain. In particular, the Audio part of this standard offers the possibility to include standard MIDI content synchronized with other forms of coding; it allows structured descriptions of audio content through a normative algorithmic description language associated with a score language more flexible than the MIDI protocol (MPEG-4 Structured Audio). These tools, though allowing to derive in someway a symbolic representation out of the information they carry, are to a large extent not enough to guarantee a correct coding of notation as they lack for instance any kind of information about visual and graphic aspects, many symbolic details, a thorough music notation modeling, and many necessary hooks for a correct human-machine interaction through the SMR decoder. MPEG-7 also provides some symbolic music related descriptors; but they are not meant to be a means for coding SMR as a form of content. On the other hand SMR content is a complete symbolic music representation and it may be rendered in synchronization with other audio-visual elements.
The MPEG SMR work item is trying to open the way for all the new applications summarized in Fig. 1 below. Many music-related software and hardware products are currently available in the market and they may greatly benefit from it, since it will foster new tool development by allowing highly increased functionality at reduced cost. Examples of applications that may rapidly be affected by this increased functionality include:
- Interactive music tutorials
- Multimedia music publication
- Software for entertainment (sound, text and symbolic information)
- Play training, performance training, ear training
- Compositional and theory training
- Software for music management in libraries
- Piano keyboards with symbolic music representation and audiovisual capabilities
- Mobile devices with music display capabilities
Fig.1 - General scenarios for MPEG SMR in entertainment and education. MPEG-4 with SMR support distribution by means of satellite data broadcast, internet, wireless or traditional communication and storage towards theatres, homes, archives, to devices such as i-TVs, tablet PCs, PCs, PDAs, smartphones.
Integration of SMR in the MPEG framework
Symbolic Music Representation (SMR) will be integrated into MPEG-4 by:
- defining an XML format for a text based symbolic music representation, to be used for interoperability with other symbolic music representation/notation formats and as a source for the production of an equivalent binary information that may be stored in files and/or streamed through a suitable transport layer;
- adding an SMR Object Type for the delivery of a binary stream containing SMR, synchronization information, and rendering rules; the associated decoder will allow to manage the received information to add the necessary “musical intelligence” for the interaction with humans;
- specifying the interface and the behavior for the symbolic music representation decoder and its relationship with the MPEG-4 synchronization and interaction layer (MPEG-4 BIFS nodes)
The SMR XML content can be produced using appropriate converters and/or a native SMR music editor. Then, an MPEG-4 SMR-enabled encoder tool can multiplex the SMR XML file into an MPEG-4 binary stream (standard XML binarization is also available in MPEG). The SMR binary stream contains information about music symbols, their synchronization with other media in time and space, and possible rendering rules for formatting music symbols. A decoder in the user’s terminal (player) converts this stream into a visual representation, which can be for instance rendered inside a BIFS scene.
Figure 2 reports a simple example of an MPEG-4 Player supporting MPEG-4 SMR. The SMR node(s) in the BIFS scene is used to render the symbolic music information in the scene (this could be performed by exploiting functionality of other BIFS nodes) as it is decoded by the SMR Decoder. The end user can interact with the symbolic music representation (change page, change view, transpose etc.) through the SMR interface node, using sensors in association to other nodes defining the audiovisual, interactive content. User commands are sent out from the SMR node fields to the SMR decoder (dashed lines), which generates a new view to be displayed in the scene.
Fig. 2 -- SMR inside an MPEG-4 Player
The structure of the SMR decoder is reported in Figure 3; steps of the decoding process include:
1. the Binary decoder decodes the binary stream; the decoder extracts the optional SMR rendering rules and the synchronization information from the SMR access units, loading the SMR Rendering Rules data structure to any SMR Rendering Rules engine, and sending the synchronization info to the SMR Manager
2. the SMR Model includes only SMR parameters, while the images, audio, video, etc. (other object types) are simply referred to other MPEG objects
3. the SMR renderer, controlled by the SMR Manager, uses the SMR Model with its parameter values and the SMR Rendering Rules to produce a view of the symbolic music information in the SMR Decoder Buffer.
4. the SMR Decoder Buffer may contain pixels and/or vector graphics information; this may be a solution dependent issue
5. the SMR Manager coordinates the behavior of the SMR decoder; it (i) receives and interprets the events coming from the SMR node interface. According to the command type, it can modify parameters in the SMR Model (e.g., transposition) and/or control the SMR Renderer (e.g., change view, change page, etc.), and (ii) it controls the synchronized rendering using the synch info
6. an SMR node and other BIFS nodes attach the content in the scene and specify the interface to the rest of the BIFS scene and to the user
A particular care is dedicated by SMR to the relationships with MIDI information; if a publisher wants to use only some MIDI files in MPEG-4 compliant devices (this is possible through the simplest Object Type defined in the Structured Audio subpart) and if these devices support SMR visualization, the specification will permit MIDI files to be automatically converted (through some specific algorithm) into SMR at the client and rendered. Similarly only the SMR may be available and delivered. In those cases, the MIDI information can be generated at the client from SMR to be used with MIDI compliant devices. This is particularly important to guarantee straightforward adaptation of current devices.
Information about the ongoing MPEG SMR activity can be obtained from the web pages of the MPEG ad hoc group on SMR (http://www.interactivemusicnetwork.org/mpeg-ahg). A large collection of documents containing requirements, scenarios, examples, and links can be accessed easily.
Fig.3 – The structure of a MPEG SMR decoder