feature

MPEG-7: Solutions for Rich Content Management

Neil Day

ONLINE, September 2001
Copyright © 2001 Information Today, Inc.

As the amount of digital content around the world grows each year, there is an increasing need for tools to help users control and manage this content. Content too is changing; more and more audio-visual content is being indexed, stored, and streamed over networks where bandwidth continues to expand. As digital libraries become more common, audio-visual content will likewise gain prominence. How do we search and browse this rich content efficiently and effectively? What technology can help us manage rich content that goes beyond the hitherto standard text-based query systems? MPEG-7, making its debut in September 2001, is ISO's newest standard. It specifically addresses these questions, and its impact will be felt in many multimedia application domains where content management and retrieval are important. Increasingly, this includes libraries and information departments.

INFORMATION PRODUCTION ESCALATING

According to University of California at Berkeley professors Peter Lyman and Hal R. Varian ("How Much Information"; http://www.sims.Berkeley.edu/how-much-info), "the world's total yearly production of print, film, optical, and magnetic content would require roughly 1.5 billion gigabytes (1.5 exabytes) of storage. This is the equivalent of 250 megabytes per person for each man, woman, and child on earth." Lyman and Varian claim that 93% of all information produced worldwide is stored in digital form–hard drives in standalone PCs account for 55% of total storage shipped each year.

They go on to say that "published" individual information is 600 times larger than traditional published information. There are 80 billion new photographic images per year and, from 1990 to 1995, UNESCO reports there were 4,250 films produced annually worldwide. Consider that it takes approximately two gigabytes to store an hour of motion-picture images in digital form using the MPEG-2 compression standard. If the images in 4,500 full-length movies produced each year were converted into bits, the world's annual original cinematic production would, therefore, consume about 16 terabytes. Much of the non-textual digital information is part of the invisible, or deep, Web.

It is clear that not only is information growing hugely each year, but also that digitization has become the preferred means of storage. With this growth of digital content–particularly multimedia content–a need for more sophisticated content management applications has developed that will facilitate effective and efficient management, searching, and browsing of the content. MPEG-7 provides the tools needed to create these applications in an open and non-proprietary environment, and is the gold standard for content management interoperability, not just for entertainment companies, but for every company, every industry, everywhere.

Defining MPEG-7

MPEG-7 is a standard for describing features of multimedia content that provides the world's most comprehensive set of audio-visual descriptions. These descriptions are based on catalog (title, creator, rights); semantic (the who, what, when, and where information about objects and events); and structural (the color histogram, measurement of the amount of color associated with an image, or the timbre of a recorded instrument) features of the AV content and leverages on AV data representation defined by MPEG-1, 2, and 4. MPEG-7 also uses XML Schema as the language of choice for content description, and will be interoperable with other leading standards such as Dublin Core (http://www.dublincore.org) and TV Anytime (http://www.tvanytime.org).

MPEG-7, formally named "Multimedia Content Description Interface," is the standard that describes multimedia content so users can search, browse, and retrieve that content more efficiently and effectively than they could using today's mainly text-based search engines. It's a standard for describing the features of multimedia content. However, MPEG-7 will not standardize the (automatic) extraction of AV descriptions/features. Nor will it specify the search engine (or any other program) that can make use of the description. It will be left to the creativity and innovation of search engine companies to manipulate and massage the MPEG-7-described content into search indices that can be used by their browser and retrieval tools.

MPEG-7 Technical Activities

It is important to note that MPEG-7 addresses many different applications in many different environments, which means that it needs to provide a flexible and extensible framework for describing audio-visual data. Therefore, MPEG-7 will define a multimedia library of methods and tools. It will standardize:

• A set of descriptors: A descriptor (D) is a representation of a feature that defines the syntax and semantics of the feature representation.

• A set of description schemes: A description scheme (DS) specifies the structure and semantics of the relationships between its components, which may be both descriptors and description schemes.

• A language that specifies description schemes, the Description Definition Language (DDL): It also allows for the extension and modification of existing description schemes. MPEG-7 adopted XML Schema Language as the MPEG-7 DDL. However, the DDL requires some specific extensions to XML Schema Language to satisfy all the requirements of MPEG-7. These extensions are currently being discussed through liaison activities between MPEG and W3C, the group standardizing XML.

• One or more ways (textual, binary) to encode descriptions: A coded description is a description that's been encoded to fulfill relevant requirements such as compression efficiency, error resilience, and random access (http://www.expway.tv/bim/bim.html).

MPEG-7 Application Domains

All domains making use of multimedia will benefit from MPEG-7 including:

• Digital libraries (image catalogue, musical dictionary, biomedical imaging catalogs)

• Multimedia editing (personalized electronic news service, media authoring)

• Cultural services (history museums, art galleries) p>
• Multimedia directory services (yellow pages, tourist information, geographical information systems)

• Broadcast media selection (radio channel, TV channels)

• Journalism (searching speeches of a certain politician using his name, his voice, or his face)

• Ecommerce (personalized advertising, online catalogues, directories of e-shops)

• Surveillance (traffic control, surface transportation, non-destructive testing in hostile environments)

• Investigation services (human characteristics recognition, forensics)

• Home entertainment (systems for the management of personal multimedia collections, including manipulation of content, such as home video editing, searching a game, karaoke)

• Social (dating services)

Typical applications enabled by MPEG-7 technology could include:

• Audio: I want to search for songs by humming or whistling a tune or, using an excerpt of Pavarotti's voice, get a list of Pavarotti's recordings and video clips in which Pavarotti sings or simply makes an appearance. Or, play a few notes on a keyboard and retrieve a list of musical pieces similar to the required tune, or images matching the notes in a certain way, e.g. in terms of emotions.

• Graphics: Sketch a few lines on a screen and get a set of images containing similar graphics, logos, and ideograms.

• Image: Define objects, including color patches or textures, and get examples from which you select items to compose your image. Or, check if your company logo was advertised on a TV channel as contracted.

• Visual: Allow mobile-phone access to video clips of goals scored in a soccer game, or automatically search and retrieve any unusual movements from surveillance videos.

• Multimedia: On a given set of multimedia objects, describe movements and relations between objects, and so search for animations fulfilling the described temporal and spatial relations. Or, describe actions and get a list of scenarios containing such actions.

SEARCH BY IMAGE

There are several possible ways to search for visual content using the inherent structural features of an image. In this example, there are four image features detailed. The color histogram feature of an image (1) allows me to search for images that have the same color. Note: the position of the colors is not important but rather the amount of similar color in the image. The next feature, spatial color distribution (2), allows me to search for images where the location of the same color is important. You can see that the added object in the right-bottom flag does not affect this type of search. You can additionally search for images that have a similar edge or contour profile as in the spatial edge distribution search technique (3). Note that color does not make a difference to this type of search. Finally, you can see an example of searching by object shape (4). Here, the color and edge profiles are not important.

This example of MPEG-7 technology is demonstrable today by visual search applications developed by ETRI of South Korea and Mitsubishi Electric (U.K.). More information about these and other MPEG-7 applications can be reviewed on the MPEG-7 Industry Focus Group Web site (http://www. mpeg-7.com).

BENEFITS OF MPEG-7

We live in the age of convergence, from the level of production through to distribution and consumption. The technical hardware and communication infrastructure is evolving and will soon reach the point where computing and communications will become embedded in everyday objects and environments. Media will also then become ubiquitous. Ubiquitous media will create a huge demand for new content, and meeting this demand must involve fundamental changes to all stages of media production, management, and delivery.

Media archives will become vast and interconnected pools of content, too large to be managed manually. Customization of content within programs–substitution of structural elements (characters, music, voices) according to viewer desires, content scaling for PDA, cell phones–will not only be possible, but also easy and pleasant. MPEG-7 will enable the creation of tools (through its structured combination of low-level features and high-level metadata) for coping with this "outbreak" of generic content. One benefit of MPEG-7 is interoperability, the rapid uptake of MPEG-7, because it is built on enabling technologies and standards. MPEG-7 is harmonizing with other standards that have demonstrated success and acceptance in both traditional media and new media businesses, e.g., W3C (XML, XML Schema), Dublin Core, and TV Anytime, etc. This will allow rapid integration into your company's products.

MPEG-7 will enable the content management system at one subsidiary to leverage the content of another subsidiary. For example, AOL, CNN, and Warner Bros. could categorize, exchange, process, and manage assets across boundaries or along an entire supply chain, possibly from production house to advertising agency to CNN.

MPEG-7 uniquely provides comprehensive standardized multimedia description tools for content. Descrip- tions for the catalogue level (e.g. title); the semantic level (who, what, when, where); and the structural level (spatio-temporal region, color histogram, timbre, texture) will provide tools for creative developers to generate new waves of multimedia applications. With MPEG-7's sister standard, MPEG-4 (streaming), an ideal combination is made for solutions that require efficient streaming of content, content manipulation, and indexing and retrieval of that content. In particular, mobile application developers (NTTDocomo) have already begun to adopt these two standards, and the trend is set to hugely increase as the demand for visual and audio information services continues to grow.

MPEG works closely with representatives of the creative industries to ensure that the best possible protection of the rights of stakeholders is maintained both in content and in metadata.

Stored audio-visual content, gathered over the years by broadcasters, libraries, and publishers, becomes more valuable because, with MPEG-7 indexing technology, more comprehensive methods are available for users to access and retrieve more detailed descriptions of that content.

Information more focused to industry, including a publicly accessible news mailing list, is also available at the MPEG-7 Industry Focus Group Web site (http://www.mpeg-industry.com).

• Interoperability and harmonization of standards
• Data exchange between subsidiaries
• Enabling a new generation of multimedia applications
• MPEG-4 and MPEG-7 tools for killer applications
• Intellectual property and management protection
• Making content more valuable

Neil Day ( neil@mpeg-industry.com) is chair of the MPEG-7 Applications and Promotions to Industry Ad Hoc Group and the MPEG-7 Industry Focus Group, while working for Digital Garage Inc. Strategic Research & Development Department (www.garage.co.jp).

Comments? Email letters to the Editor at marydee@infotoday.com.