In this final step the user can specify the related resources of the current resource item.

4. Available demonstrators

Different demonstrators have already been ingested and made available through the DELOS WP3 demo portal. Some of these demonstrators are introduced by their creators in the following sections.

4.1. MILOS - Multimedia Content Management System

MILOS (Multimedia dIgital Library for Online Search) is a general purpose software component tailored to support the design and effective implementation of digital library applications. MILOS supports the storage and content-based retrieval of any multimedia documents descriptions of which are provided by using arbitrary metadata models represented in XML.

Digital library applications are document-intensive applications where possibly heterogeneous documents and their metadata have to be managed effectively. We believe that the main functionalities required by DL applications can be embedded in a general purpose Multimedia Content Management System (MCMS), that is a software tool specialized to support applications where documents, embodied in different digital media, and their metadata are handled efficiently.

The minimum requirements of a Multimedia Content Management System are: flexibility in structuring both multimedia documents and their metadata; scalability; and efficiency.

Flexibility is required both at the level of management of basic multimedia documents and at the level of management of their metadata. The flexibility required in representing and accessing metadata can be obtained by adopting XML as standard for specifying any metadata (for example MPEG-7 can be used for multimedia objects, or SCORM (Shareable Content Object Reference Model Initiative) for e-learning objects). Proper regard for scalability and efficiency is essential to the deployment of real systems able to satisfy the operational requirements of a large community of users over a huge amount of multimedia information.

We believe that the basic functionalities of a MCMS are related to the issues of storage and preservation of digital documents, their efficient and effective retrieval, and their efficient and effective management. These functionalities should be guaranteed by appropriate management of documents and related metadata, according to the following prerequisites:

1. capability of managing different documents embodied in different media and stored with different strategies;
2. capability of describing documents by way of arbitrary, and possibly heterogeneous, metadata;
3. capability of providing DL applications with custom/personalised views on the metadata schema actually handled.

We have designed and built MILOS, a MCMS which satisfies the requirements and offers the functionalities discussed in previous section. The MILOS MCMS has been developed by using Web Service technology, which in many cases (e.g. .NET, EJB, CORBA, etc.) already provides very complex support for "standard" operations such as authentication, authorization management, encryption, replication, distribution, load balancing, etc. Therefore we need not elaborate further on these topics, but will concentrate mainly on the aspects discussed above.

MILOS is composed of three main components:

• the Metadata Storage and Retrieval (MSR) component
• the Multi Media Server (MMS) component
• the Repository Metadata Integrator (RMI) component

All these components are implemented as Web Services and interact by using SOAP (Simple Object Access Protocol). The MSR manages the metadata of the DL. It relies on our technology for native XML databases and offers the functionality described at point 2 above. The MMS manages the multimedia documents used by the DL applications. MMS offers the functionality of point 1 above. The RMI implements the service logic of the repository providing developers of DL applications with a uniform and integrated way of accessing MMS and MRS. In addition, it supports the mapping of different metadata schemas as described at point 3 above. All these components were built choosing solutions able to guarantee the requirements of flexibility, scalability, and efficiency.

Case study applications

Reuters case study

The Reuters dataset contains text news agencies and the corresponding metadata. There are two types of metadata: Reuters specific metadata including titles, authors, topic categories, and extended Dublin Core metadata.

The Reuters dataset contains 810,000 news agencies (2.6 Gb) where text and metadata are both encoded in XML. We linked the full text index and the automatic topic classifier to the elements containing the body, the title, and the headline of the news. Other value indexes were linked to elements corresponding to frequently searched metadata, such as locations, dates, countries.

ACM Sigmod Record and DBLP case study

Both the ACM Sigmod (Association for Computing Machinery Special Interest Group on Management of Data) Record dataset and the DBLP (Digital Bibliography & Library Project) dataset [3] consist of metadata corresponding to the description of scientific publications in the computer science domain. The ACM Sigmod record is relatively small. It is composed of 46 XML files (1Mb), while the DBLP dataset is composed of just one large (187Mb) XML file. Their structure is completely different even though they contain information describing similar objects.

We built one DL application which could access both datasets. We made use of MILOS' mapping functionality to ensure requests on the application were correctly translated for the two schemas. We linked a full text index to the elements containing the titles of the articles, and other value indexes to the more frequently searched elements, such as authors, dates, years, etc. 

ECHO case study

The ECHO dataset includes historical audio/visual documents and corresponding metadata. ECHO is a significant example of MILOS' ability to support the management of arbitrary metadata schemas. The metadata model adopted in ECHO, based on the IFLA/FRBR model, is rather complex and highly structured. It is used to represent the audio-visual content of the archive and includes, among others:

• the description of videos in English and in the original language
• specific metadata fields such as Title, Producer, year, etc.
• the boundaries of scenes detected (associated with a textual descriptions)
• the audio segmentation (distinguishing among noise, music, speech, etc.)
• the Speech Transcripts
• visual features for supporting similarity search on key-frames

The collection is composed of about 8,000 documents for 50 hours of video described by 43,000 XML files (36 MB). Each scene detected is associated with a JPEG-encoded key frame for a total of 21GB of MPEG-1 and JPEG files. Full text indexes were linked to textual descriptive fields, similarity search indexes were linked to elements containing MPEG-7 image (key frames) features, and other value indexes with frequently searched elements.

Milos Web site: http://milos.isti.cnr.it/

4.2. 3D Content-Based Retrieval

Objective

This demonstrator implements some approaches to retrieval of 3D objects based on their visual similarity. Its main goal is to test and compare the retrieval effectiveness of different solutions for 3D object modelling.

Research activity

Activity undertaken during the first year of the project concentrated on defining a test environment in which different 3D retrieval approaches could be compared.Within this work, retrieval by similarity was achieved through using a number of techniques for object description and similarity computation. Currently, the description techniques implemented include 3D moments, curvature histograms and shape functions. Similarity of content descriptors can be evaluated according to six different distance functions: Haussler Mu, Minkoswki L1, Kullback-Leibler, Kolmogorov-Smirnov, Jeffrey divergence and X² statistics.

In particular:

Curvature histograms are constructed by evaluating the curvature of vertices of the mesh representing the 3D object. Curvature values are discretized into 64 distinct classes.

To evaluate 3D moments of a 3D object defined by a polygonal mesh, a limited set of points P_i is considered, where the relevance of each point is weighted by the area of the portion of surface associated with the point. To make the representation independent from the actual position of the model, the first order moments m₁₀₀, m₀₁₀ and m₀₀₁ are first evaluated, and higher order moments are then evaluated with respect to the first order moments. In our experiments, moments up to the 6-th order have been computed for each model. This aims to attain a sufficient discrimination among different models.

Shape functions are evaluated by computing the histogram of Euclidean distances between all possible vertex pairs on the object mesh. Distances are normalized with regard to the maximum distance between two vertices, and discretized into 64 distinct class values.

Plans for next activities

The work on retrieval by content of 3D objects is currently proceeding under a project proposal approved for the next 18 months of the JPA (RERE 3D: Description, Matching and Retrieval by Content of 3D Objects). In particular, we are currently investigating a 3D representation capable of supporting the spatial localization of the properties of an object surface. This is expected to improve exisiting approaches which currently do not consider local properties of mesh vertices.

Demonstrator

The 3D CBR (Content-Based Retrieval) demonstrator allows users to test out retrieval by visual similarity over an archive of 3D object models. The system is fully developed in Java Technology and accessible through a Web interface available at: http://delos.dsi.unifi.it:8080/CV/.

The archive includes four classes of models:

• taken from the web,
• manually authored (with a 3D CAD software),
• high quality versions of models from the De Espona 3D Models Encyclopedia (http://www.deespona.com) and
• variations of the previous three classes (obtained through geometric deformation or application of noise, which caused surface points to be moved from their original locations).

Objects in the database cover a variety of classes, including statues, vases, household goods, transport, simple geometric shapes, and many others.

Each database model is represented in VRML (Virtual Reality Modelling Language) format through the IndexedFaceSet data structure.

The system supports retrieval according to the three content descriptors and the six similarity measures previously described. On the left part of the Web interface, three menus allow the user to:

• request a subsample of database models randomly selected;
• specify which content descriptor to use;
• specify which distance to use in order to compute the similarity between content descriptors.

The type of content descriptor and the similarity measure the user has currently selected are shown on the upper part of the interface. The user can query the system by activating the search button available below every model thumbnail. Once the search process is completed, the system presents retrieved items in decreasing order of similarity from top to bottom and from left to right (the most similar model being displayed on the upper left corner of the results panel).

In order to analyse the effect of using different content descriptors or similarity measures, once a search process is completed the user can change the type of content descriptor or similarity measure. In this case, the system automatically performs a new search evaluating the similarity between every database item and the upper left model, using the newly selected content descriptors and similarity measures.

4.3. VideoBrowse

Overview

This is a tool for fast video accessing and browsing. It provides functionalities for fast decoding and playback of MPEG-1 and MPEG-2 compressed streams, without the need of an external codec. It can also handle reverse playback, one frame forward and one frame backward operations.

Two algorithms for automatic shot detection are included in this tool, one directly operating on compressed data, and the other on uncompressed data. The result of shot detection processing is an index written in the MPEG-7 standard. The index is then parsed at the successive accesses to the same video file, and then used to generate a storyboard by selecting a single keyframe for each shot in the index.

Shot Detection

Two different algorithms are included in the tool:

1. Shot detection with MPEG features: a number of features extracted from the stream are considered, namely the DC coefficients of I-frames, the number of intra-coded macroblocks and the number of forward-backward-bidirectional predicted macroblocks in a GOP (Group of Pictures). Derivatives of these quantities are also considered. Then Linear Discriminant Analysis has been used to calculate the weights of each feature in a linear combination, whose final value is used (with a threshold) to discriminate between GOPs containing and not containing shot changes. This algorithm is extremely fast, but it cannot determine the exact location of the shot change within the GOP.
2. Shot detection with uncompressed features: this algorithm is slower than the previous one, but it gives more accurate results. For each couple of frames we calculate the mean R,G,B values, and the maximum difference between the three channels is used as the distance measure. This value is then compared with the median distance calculated on a window of 20 frames centred in the current frame. If the ratio is greater than a manually imposed threshold, the current frame is marked as a shot change. Whenever multiple adjacent frames satisfy this condition, the frame with the maximum difference measure is selected.

Both algorithms depend on the choice of the threshold, which the user can adjust manually. To help in this operation, at the end of the shot detection processing some statistics on the value to threshold are shown in a dialog box. Furthermore, the value in each frame is written to a CSV file.

Interface

The graphical interface includes the common playback controls such as play, forward one frame, etc., and a trackbar for fast movements in the video stream. When an index is available for the current video, a browsing window allows users to navigate through the representative keyframes, and to start the playback from a specific shot.

4.4. UvA Parallel Visual Analysis in TRECVID 2004

Introduction

The Parallel-Horus framework, developed at the University of Amsterdam, is a unique software architecture that allows non-expert parallel programmers to develop fully sequential multimedia applications for efficient execution on homogeneous Beowulf-type commodity clusters. Previously obtained results for realistic, but relatively small-sized applications have shown the feasibility of the Parallel-Horus approach, with parallel performance consistently being found to be optimal with respect to the abstraction level of message passing programs. Our demonstrator shows the most serious challenge Parallel-Horus has had to deal with so far: the processing of over 184 hours of video included in the 2004 NIST TRECVID evaluation.

The 2004 NIST TRECVID Evaluation

TREC is a conference series sponsored by the National Institute of Standards and Technology (NIST) with additional support from other U.S. government agencies. The goal is to encourage research in information retrieval by providing a large test collection, uniform scoring procedures, and a forum for organizations interested in comparing their results. An independent evaluation track called TRECVID was established in 2003 devoted to research in automatic segmentation, indexing, and content-based retrieval of digital video streams.

The 2004 NIST TRECVID evaluation defines four main tasks, at least one of which must be completed to participate in the evaluation. The University of Amsterdam participated in TRECVID 2004 by completing the feature extraction task.

This task was defined as follows: Given the 2004 NIST TRECVID video dataset, a common shot boundary reference for this dataset, and a list of feature definitions, participants must return for each feature a list of at most 2000 shots from the dataset, ranked according to the highest probability of detecting the presence of that feature.

The 2004 NIST TRECVID video dataset consisted of over 184 hours of digitized news episodes from ABC and CNN. In addition, ten feature definitions were given, including 'Bill Clinton', 'beach', 'airplane takeoff', and 'basket scored'.

Generic Semantic Concept Detection

Our approach to the feature extraction problem is based on the so-called Semantic Value Chain (SVC), a novel method for generic semantic concept detection in multimodal video repositories. The SVC extracts semantic concepts from video based on three consecutive analysis links, i.e. the Content Link, the Style Link, and the Semantic Context Link. The Content Link works on the video data itself, whereas the Style Link and the Semantic Context Link work on higher-level semantic representations.

In the Content Link we view video documents from the data perspective. In general, three modalities can be identified in video documents, i.e. the auditory, textual, and visual modality. In our approach, detectors are first applied to individual modalities. The results are then fused into an integrated Content Link detector. Based on validation experiments the best hypothesis for a single concept serves as the input for the next link.

Our demonstrator shows the processing of the visual modality only, as this is by far the most time-consuming part of the complete system.

Visual Analysis

The visual modality is analyzed at the image (or video frame) level. After obtaining video data from file, for each 15th video frame visual features are extracted by using Gaussian colour invariant measurements. RGB colour values are decorrelated by transformation to an opponent color system. Then, in succession, acquisition and compression noise are suppressed by Gaussian smoothing. A colour representation consistent with variations in target object size is then obtained by varying the size of the Gaussian filters. Global and local intensity variations are suppressed by normalizing each color value by its intensity, resulting in two chromaticity values per color pixel. Furthermore, rotationally invariant features are obtained by taking Gaussian derivative filters, and combining the responses into two chromatic gradient magnitude measures. These seven features, calculated over three scales, yield a combined 21-dimensional feature vector per pixel.

The obtained invariant feature vector serves as the input for a multi-class Support Vector Machine (SVM) that associates each pixel to one of the predefined regional visual concepts. The SVM labelling results in a weak semantic segmentation of a video frame in terms of regional visual concepts. This result is written out to file in condensed format (i.e.: a histogram) for subsequent processing.

Note that this segmentation of video frames into regional visual concepts at the granularity of a pixel is computationally intensive. This is especially the case if one aims to analyze as many frames as possible.

In our approach the visual analysis of a single video frame requires around 16 seconds on the fastest sequential machine at our disposal. Consequently, when processing two frames per second at a frame rate of 30, the required processing time for the entire TRECVID dataset would be around 250 days. Application of the Parallel-Horus framework, in combination with a distributed set of Beowulf-type commodity clusters significantly reduced this required processing time to less than 60 hours. These performance gains were obtained without any parallelization effort whatsoever, which was an important contributing factor in our top ranking in the TRECVID results.

4.5. Audio feature extraction with Rhythm Patterns

Content-based access to audio files, particularly music, requires the development of feature extraction techniques to capture the acoustic characteristics of the signal and so permit the computation of similarity between pieces of music, reflecting the similarities perceived by human listeners.

'Rhythm Patterns' are feature sets derived from content-based analysis of musical data and reflect the rhythmical structure in the musical pieces. Classification of sound into musical genres as well as automatic organization of music archives according to sound similarity are made possible through the psycho-acoustically motivated 'Ryhthm Patterns' features.

The feature extraction process for the Rhythm Patterns is composed of two stages. Firstly, the specific loudness sensation in different frequency bands is computed, by using a Short Time FFT (Fast Fourier Transform). The resulting frequency bands are then grouped into psycho-acoustically motivated critical-bands, applying spreading functions to account for masking effects and successive transformations into the decibel, Phon and Sone scales. This results in a power spectrum that reflects the human sensation of loudness. In the second step, the spectrum is transformed into a time-invariant representation based on the modulation frequency; this is achieved by applying another discrete Fourier transform, resulting in amplitude modulations of the loudness in individual critical bands. These amplitude modulations have different effects on human hearing sensation depending on their frequency, the most significant of which, referred to as the fluctuation strength, is most intense at 4Hz, decreasing towards 15Hz. From that data, reoccurring patterns in the individual critical bands, resembling rhythm, are extracted, which - after applying Gaussian smoothing to diminish small variations - result in a time-invariant, comparable representation of the rhythmic patterns in the individual critical bands. The proposed feature set then serves as a basis for an unsupervised organization task, as well as for machine learning or classification tasks.

This feature set was submitted to the Audio description contest of the International Conference on Music Information Retrieval (ISMIR 2004), winning the rhythm classification track.

4.6. TZI Demonstrators for DELOS WP3

Video Content Manager

The Video Content Manager is a tool for analysis and annotation of digital videos. It has been developed in cooperation with researchers from the cultural sciences and from the arts.

The faculty of cultural sciences at the University of Bremen possesses thousands of hours of digitized videos. These include videos of lectures, digitized telecasts and video works from students. For students and teachers in cultural studies, access to this video footage is often needed for practising, lecture reruns or the creation of new videos. Annotation of the video material is required to provide such access which can be achieved through use of the Video Content Manager.

At the University of the Arts Bremen, a group of art historians interested in the medium of video built a prototype of an international archive of video arts. Currently it is very difficult to gain access to as is additional information on artistic media works, including video arts. The information about these works is scattered across the world and is very difficult to obtain outside conventional channels, such as exhibitions, festivals, or conferences. These problems are tackled by the prototype archive. The Video Content Manager is used to facilitate annotation of the video works and their ingestion into the archive.

Annotating a video with the Video Content Manager is a three-stage process. Firstly, an automatic shot boundary detection algorithm is run on the video. Its results yield a temporal segmentation of the video. For each shot, a key frame is automatically extracted from the video. Such a key frame allows for a quick overview of the content of a shot and is suitable for browsing the video without having to view it as a whole.

In the second step, successive shots that cover the same topic or show the same location are merged together to form what we call a "scene". This has to be done manually. The result of this step is a hierarchical temporal segmentation of the video with three levels of different granularity: shot, scene, and video.

The final step is a textual annotation follows an annotation scheme tailored to the users' needs, but based consistently on Dublin Core [1]. The annotation is guided by the temporal segmentation from the second step and may use the keyframes from the first step for efficiency purposes. Shots are annotated on a more syntactical level (what can currently be seen in the video?). Scenes are annotated on a more semantical level (what is going on, what is the topic?).

The results of the annotation process may be exported as XML for ingestion into a database. The video data itself are not manipulated. The Video Content Manager is available on the demonstrator website of the DELOS cluster 3 (A/V-NTO).

Automatic shot boundary detection for TRECVID 2004

The TRECVID workshop [2] is an annual meeting of users and researchers in the field of content-based video analysis, retrieval, and digital video libraries. The workshop began life as a "track" of the Text Retrieval Conference (TREC), but became a separate workshop in 2003. Its goal is to provide a forum for evaluation of video retrieval algorithms together with a common collection of videos. In 2003 and 2004, the video material provided consisted mainly of news broadcasts, including sports material, weather forecasts and commercials.

Several tasks are made available to participants: shot boundary detection, high-level feature extraction, and search. TZI- Bremen University has taken part in the high-level feature extraction task (2002) and the shot boundary detection task (2002-2004). The shot boundary detection tool used to produce the results [3] submitted in 2004 is available as a demonstrator on the DELOS WP 3 demonstrator website.

Automatic realtime text extraction

Telecasts, especially news and magazine broadcasts, are often, if not always, enhanced with text inserts. The information contained in these inserts may cover topics, names of presenters, interviewers or interviewees, news tickers or casts. Automatic recognition of the text displayed can represent a considerable benefit to content-based video search and retrieval.

The automatic recognition of text in text-based documents (OCR) is a well-researched field. However the recognition of text inserts in video often proves more difficult. It includes segmentation of the text from the background, which is usually much more complex than in black and white text-only documents. To simplify the task, detection of those areas of the video containing text can be very useful.

A fast detector for text areas which extracts the locations of text inserts in video, and which tracks these text areas over multiple frames for scrolled text, is provided in the demonstrator section in the DELOS WP 3 portal. It is based on statistics on the visual properties of text inserts and can be run in realtime. The resulting location data may be used as hints for a subsequent video-OCR. Alternatively they can be used alone as an indicator in the discrimination between different video segments, for example between an anchor shot and a credits sequence.

4.7. The Video Segmentation and Annotation Tool Demonstrator

"Video Segmentation & Annotation tool" is a full system that supports the segmentation, indexing and annotation of audiovisual content and the creation of segmentation metadata compliant with the TV-Anytime (Standard for digital video content) Segmentation Metadata Model. The system comprises a graphical application where the metadata are created or edited and a relational database where these metadata are stored. More specifically, its functionality includes the creation of video segments and video segment groups according to the TV-Anytime Segmentation Metadata Model as well as the semantic annotation of these segments through the application of domain-specific ontologies and transcription files.

The architecture of the tool follows a multi-tier approach consisting of the following tiers:

1. Application Tier: the graphical application shown to the user.
2. XML-DB middleware tier: a set of software components responsible for the management of TV-Anytime XML documents (hidden tier).
3. Database Tier: a database management system used to store relational data that derived from the TV-Anytime metadata.

The system offers the following functionality:

• Creation of a new project: Users of the tool can start the segmentation process of a video programme and create a new project in one of 3 ways:
  - a) search on the database for recorded programmes (this requires a connection with a database server)
  - b) load XML file containing TV-Anytime Segmentation metadata and
  - c) open a new unsegmented video programme. (Note that the segmentation process cannot proceed without starting a new project as almost all of the buttons and fields of the graphical application are disabled.)
     
• Play video programme: One can use the media player frame of the application to play a video file.
    
• Create/Edit segments: Users can create or edit segments of the video programme playing according to the TV-Anytime Segmentation Model by defining or modifying their start and end time, their keyframes, their description (title, synopsis, keywords and related material), their version and their unique segmentID. (For more information about the above definitions see the TV-Anytime Segmentation metadata model.)
   
• Create/Edit segment groups: Users can create or edit segment groups composed of segments previously created, according to the TV-Anytime Segmentation Model by defining or modifying the segments which contain them, their group interval, their keyframes, their description (title, synopsis, keywords and related material), their group type, their version and their unique segmentGroupID.
   
• Save created metadata: Every metadata created for a video programme (segments and segment groups) are saved in a structure called the "Segment Information Table" found in the main memory. In order to store this information in a permanent way the tool provides two methods:
  - a) store metadata onto a database (where a connection with a database server already exists) and
  - b) export metadata in an XML file compliant with the TV-Anytime Segmentation metadata model.
   
• Search within segments and segment groups: this tool provides three ways in which to browse within the created segment and segment groups:
  - a) Groups & Segments Schema (a synoptic view of the segments and segment groups of the current Segment Information Table) ,
  - b) Segment Information Table Explorer (a more analytical view of the segments and segment groups of the current Segment Information Table) and
  - c) Text-based search for Segments & Groups (search within segments and segment groups of the current Segment Information Table based on the segment or segment group title or keywords).
   
• On-line help: The tool has an on-line help system that describes in an analytical way the use of the application and may be able to provide immediate answers to users' questions.
    
• Security and Recovery: The tool provides a mechanism so as to prevent loss of unsaved metadata in case of system crash. In that case when the application is restarted the user can reload these metadata.
    
• Importing and Using Ontologies: This tool provides the ability to import domain-specific ontologies (the current version supports ontologies that are based on keywords) in order to achieve a more accurate segment annotation by using words from these ontologies.
     
• External Java API video programmes of soccer matches: This tool has been integrated with an external application that supports an exiating ontology covering the domain of soccer matches. Users can insert keywords based on a MPEG-7-compliant ontology for soccer matches. The API creates set of phrases according to this ontology which are then used as keywords in the description of segments and segment groups.
    
• Advanced Searching in transcription files: Users can employ a transcription file produced during the segmentation process to find in which part of the video specific phrases are heard. They can browse within the entire transcription file or search for specific words or phrases. By clicking on one of these phrases they can identify the section of video corresponding to the speech identified. The reverse procedure is also supported: users can find which phrases are heard in each video segment while the video is played.
    

4.8. The UP-TV Demonstrator

The UP-TV system is based on the TV-Anytime architecture for digital TV systems and follows the corresponding metadata specifications of audiovisual content and user descriptions. It is directly related to audiovisual digital libraries as it relates to the development of single-user and server systems for the management of audiovisual content compliant with TV-Anytime specifications.

The UP-TV system follows a multi-tier architecture [1]. The lowest tier handles the metadata management. The middleware tier includes all the logic for interfacing the system with the outside world. The application tier enables the exchange of information between the server and heterogeneous clients through different communication links. The core of the system is the metadata management middleware that takes over the storage of the TVAM program and user metadata descriptions as well as providing advanced information access and efficient personalization services. The implementation was based on the following decisions:

The metadata management system should be able to receive and create all kinds of valid XML documents with respect to the TVAM XML Schema.

The database management system should follow the relational model and support the SQL standard as the language for data manipulation and retrieval, so that it can be easily integrated with additional information on the servers, allow concurrent access etc.

The solutions developed include functionality for storing the program metadata onto relational databases, functionality for storing TVAM consumer metadata on databases and functionality for retrieving data from the relational databases and assembling valid TVAM documents or document fragments. Mapping the TVAM XML structure onto relational databases provides efficient mechanisms for matching program and profile metadata as well as user profile adaptation and data mining for viewing histories through the use of the SQL language, thus facilitating the implementation of powerful services for both final users and service providers. The XML-DB middleware (figure 1) is a set of software components responsible for the manipulation of TVAM XML documents and the mapping of TVAM XML Schema to the underlying relational schema. It is supported by a relational database management system along with the relational database, used to store the data of TVA metadata descriptions.

TVAM-compliant clients use XML documents to communicate with the system. These documents contain data that could be used in conjunction with data from other TVAM XML documents. Document (or document fragment) retrieval is supported by a special purpose Application Programmating Interface (API). In this environment the data management software should not rely on XML document modelling solutions (like DOM) but rather on a data binding approach. Data binding offers a much simpler approach to working with XML and supports effective separation between document structure and data modelling.

There are numerous XML data binding products capable of transferring data between XML documents and objects. Design-time binders (which require configuration based on a DTD or an XML Schema before they can be used) are usually more flexible in the mappings that they can support. The overall system architecture assumes a design-time binder. Therefore a configuration process was necessary to create the appropriate classes. The XML data binder considered for the implementation of our system is data-centric. It is capable of fully representing XML documents as objects or objects as XML documents (the serialization of the object tree to XML document is encapsulated in class (un)marshal methods). The data binder uses a SAX-based parser and the corresponding validator can be used to ensure that incoming and outgoing XML documents conform to the TVAM XML schema.

The communication with the relational database management system relies on the use of standard interfaces like JDBC. Standard SQL statements are used to store-retrieve data from the underlying relational database. To do so, the classes created during the data binding configuration process are extended with DB-Insert/Retrieve methods. DBInsert methods use the object tree to create INSERT/UPDATE statements to give persistence to data on the object tree. These methods can also query the database to avoid duplicates of data. DBRetrieve methods retrieve data from the database in order to build object trees that could be used to create TVAM XML documents. The DB-Insert/ Retrieve methods rely on both the class hierarchy created by the data binding configuration process and the relational schema of the underlying database. The relational database is responsible for the storage and retrieval of information that is represented in TVAM XML documents.

In order to support ubiquitous access, special device-specific components of middleware were developed for the UP-TV environment. Java technology was chosen for the application development in hand-held devices since it is adequate for dynamic delivery of content, provides satisfactory user interactivity and ensures cross-platform compatibility. Two components were built, one suitable for cellular phones compatible with the MIDP profile and one for PDAs that support the Personal profile. In order to keep the communication scheme simple and uniform over different devices, we have chosen to use HTTP since it is suitable for the transfer of XML documents and is the network protocol supported by the MIDP libraries. The front-end of the server consists of Java servlets that accept HTTP requests from the clients and embody software adapters that adapt appropriately the information that will be exchanged and the functionality that can be provided, depending on the kind of the device requesting service.

4.9. The Campiello Demonstrator

The Campiello system is a system for intelligent tourism information and interaction between visitors (or potential visitors) of cities possessing significant cultural heritage (e.g. Chania and Venice) and their local citizens. This system has been developed through the use of innovative technologies including non-traditional objects like 3D reconstructions of archaeological sites and interactive city maps.

This section describes the architecture of the Campiello PC Interface. By the term 'architecture' we mean the functionality that the interface was designed to offer, the layout of the screens used in the interface and the way the interface is structured, i.e. the different sections used and the navigation between these sections.

The Campiello website as it is right now fully supports the first requirement. It currently supports 4 languages (English, Greek, Italian and French) but their number can be increased arbitrarily without modification to the implementation.

All the text that appears on the interface is read from a database where it is organized in terms of "Interface Contexts" and "Interface Topics". Contexts refer to, as the name implies, discrete contexts, i.e. sections or subsections of the interface. An Interface Context usually corresponds to one screen of the interface. Topics, on the other hand, refer to specific items in a Context, i.e. to elements within a screen.

Using this convention one can describe all elements on each screen with an intuitive, easy to remember name. For example, the title of the Places page is referred to as Context: Places, Topic: Name/Title. The caption for the Search button (which appears on every page) is referred to as Context: Any, Topic: Search.

In the database we have a description in all the available languages for each Context/Topic, so the system can fetch the appropriate text based on the language the user has selected.

Through the notion of a "Step" parameter to the interface labels stored in the database, we can have multiple texts that refer to the same Context/Topic. This was done to support cases where we needed multiple messages that would correspond to the same Context/Topic pair. An example of this are customized error messages that can vary subtly based on some parameter.

It is also possible to provide different text for each language regardless of whether the user enters the Campiello system through a Mac with a Netscape browser or through a PC with Internet Explorer.

We should point out here the difference between the language of the interface and the language of the content of Campiello. The texts for the interface should be provided in all the available languages, but the content of Campiello will not necessarily display in all languages, i.e. if someone has not posted an appropriate translation. As a result of that, changing the language when viewing Campiello content may lead to an error message if the same content is not available in the language selected, whereas this is not the case in respect of text used for the interface.

5. References

1. Dublin Core Metadata Initiative: http://dublincore.org
2. TRECVID Video Retrieval Evaluation: http://www-nlpir.nist.gov/projects/trecvid/
3. Results can be seen here: http://www-nlpir.nist.gov/projects/tvpubs/tv.pubs.org.html

5.1 Publications

[1] G. Amato, C. Gennaro, F. Rabitti, P. Savino "Milos: A Multimedia Content Management System". Extended abstract, SEBD 2004, S. Margherita di Pula (CA), Italy, June 21-23, 2004.
   

[2] G. Amato, F. Debole, F. Rabitti, P. Savino, and P. Zezula "A Signature-Based Approach for Efficient Relationship Search on XML Data Collections". XML Database Symposium (XSym 2004) in Conjunction with VLDB 2004, Toronto, Canada, 29-30 August 2004
  

[3] F. J. Seinstra, D. Koelma, and A. D. Bagdanov: "Finite State Machine-Based Optimization of Data Parallel Regular Domain Problems Applied in Low-Level Image Processing". IEEE Transactions on Parallel and Distributed Systems, 15(10):865-877, 2004.
   

[4] C. G. M. Snoek, M. Worring, and A. G. Hauptmann: "Detection of TV News Monologues by Style Analysis". In Proceedings of the IEEE International Conference on Multimedia & Expo (ICME) - Special session on Multi-Modality-Based Media Semantic Analysis, Taipei, Taiwan, 2004.
   

[5] N. Sebe, M.S. Lew, T.S. Huang: "Computer Vision in Human-Computer Interaction". HCI/ECCV 2004. Lecture Notes in Computer Science, Vol. 3058, Springer-Verlag, ISBN 3-540-22012-7, 2004
   

[6] M. Bertini, A. Del Bimbo, A. Prati, R. Cucchiara, "Semantic Annotation and Transcoding for Sport Videos". In Proceedings of International Workshop on Image Analysis for Multimedia Interactive Services (WIAMIS), 2004.
  

[7] M. Bertini, R. Cucchiara, A. Del Bimbo, A. Prati, "Content-based Video Adaptation with User's Preference". In Proceedings of International Conference on Multimedia & Expo (IEEE ICME 2004), 2004.
   

[8] M. Bertini, R. Cucchiara, A. Del Bimbo, A. Prati, "Object-based and Event-based Semantic Video Adaptation". In Proceedings of International Conference on Pattern Recognition (IAPR-IEEE ICPR 2004), vol. 4, pp. 987-990, 2004.
   

[9] M. Bertini, A. Del Bimbo, A. Prati, R. Cucchiara, "Objects and Events Recognition for Sport Videos Transcoding". In Proceedings of 2nd International Symposium on Image/Video Communications over fixed and mobile networks (ISIVC), 2004.
   

[10] C. Grana, G. Pellacani, S. Seidenari, R. Cucchiara, "Color Calibration for a Dermatological Video Camera System". In Proceedings of The 17th International Conference on Pattern Recognition (ICPR 2004), Cambridge, UK, pp. 798-801, August 23-26, 2004.
   

[11] R. Cucchiara, C. Grana, G. Tardini, R. Vezzani, "Probabilistic People Tracking for Occlusion Handling". In Proceedings of The 17th International Conference on Pattern Recognition (ICPR 2004), Cambridge, UK, pp. 132-135, August 23-26, 2004.
   

[12] N. Orio, P. Zanuttigh, G.M. Cortelazzo. "Content-Based Retrieval of 3D Models Based on Multiple Aspects". Accepted at IEEE International Workshop on Multimedia Signal Processing, Siena, IT, 29 September 1 October, 2004. In press.
   

[13] G. Neve, and N. Orio. "Indexing and Retrieval of Music Documents through Pattern Analysis and Data Fusion Techniques". Accepted at International Conference on Music Information Retrieval, Barcelona, ES, 10-14 October, 2004. In press.
    

[14] D. Schwarz, N. Orio, and N. Schnell. "Robust Polyphonic MIDI Score Following with Hidden Markov Models". Accepted at International Computer Music Conference, Miami, USA, 1-6 November, 2004. In press.
    

[15] E. Bertino, E. Ferrari, D. Santi, A. Perego: "Constraint-based Techniques for Personalized Multimedia Presentation Authoring". Submitted for publication.
   

[16] S. Valtolina, S. Franzoni, P. Mazzoleni, E. Bertino: "Dissemination of Cultural Heritage Content through Virtual Reality and Multimedia Techniques: a Case Study". Accepted for publication in IEEE MMM 2005: 11th International Multi-Media Modelling Conference. Melbourne, Australia, 12 - 14 January 2005.
    

[17] S. Valtolina, S. Franzoni, E. Bertino, E. Ferrari, P. Mazzoleni: "A virtual reality tour in an Italian Drama Theatre: A journey between architecture and history during 19th century". Proceedings of the EVA 2004: Electronic and Imaging & the Visual Arts. London, United Kingdom, 26 - 31 July 2004.
   

[18] M. Bertini, R. Cucchiara, A. Del Bimbo, A. Prati: "Semantic Annotation and Transcoding for Sport Videos". WIAMIS 2004, Lisbona, April 2004.
    

[19] C. Colombo, D. Comanducci, A. Del Bimbo and F. Pernici: "Accurate automatic localization of surfaces of revolution for self-calibration and metric reconstruction". In Proceedings IEEE Workshop on Perceptual Organization in Computer Vision (POCV 2004), Washington, DC, USA, June 2004.
   

[20] M. Bertini, A. Del Bimbo W. Nunziati: "Common Visual Cues for Sports Highlights Detection". Proc. IEEE International Conference on Multimedia & Expo (ICME'04), Taipei, Taiwan, June 27-30 2004.
   

[21] S. Berretti, G. D'Amico, A. Del Bimbo: "Shape Representation by Spatial Partitioning for Content Based Retrieval Applications". Proc. IEEE International Conference on Multimedia & Expo (ICME'04), Taipei, Taiwan, June 27-30 2004.
    

[22] J. Assfalg, G. D'Amico, A. Del Bimbo, P. Pala: "3D content-based retrieval with spin images". Proc. IEEE International Conference on Multimedia & Expo (ICME'04), Taipei, Taiwan, June 27-30 2004.
    

[23] M. Bertini, R. Cucchiara, A. Del Bimbo, A. Prati: "Objects and Events Recognition for Sport Videos Transcoding". ISIVC 2004, Brest, France, July 2004.
    

[24] S. Berretti, A. Del Bimbo, P. Pala: "A Graph Edit Distance Based on Node Merging". Proc. International Conference on Image and Video Retrieval (CIVR'04), pp.464-472, Dublino, Ireland, July 21-23 2004.
   

[25] S. Berretti, A. Del Bimbo: "Multiresolution Spatial Partitioning for Shape Representation". Proc. IEEE International Conference on Pattern Recognition (ICPR'04), vol.II, pp.775-778, Cambridge, United Kingdom, August August 23-26 2004.
    

[26] J. Assfalg, A. Del Bimbo, P. Pala: "Spin Images for Retrieval of 3D Objects by Local and Global Similarity". Proc. IEEE International Conference on Pattern Recognition (ICPR'04) vol.III, pp.906-909, Cambridge, UK, August 23-26, 2004.
   

[27] M. Bertini, R. Cucchiara, A. Del Bimbo, A. Prati: "Semantic Video Adaptation based on Automatic Detection of Objects and Events". Proc. IEEE International Conference on Pattern Recognition (ICPR'04) vol.IV, pp.987-990, Cambridge, UK, August 23-26, 2004.
   

[28] Jacobs and Th. Hermes and O. Herzog: "Hybrid Model-based Estimation of Multiple Non-dominant motions". In: Proceedings of the 26th DAGM Symposium on Pattern Recognition, Tübingen, Germany, 2004.
    

[29] M. Crucianu, M. Ferecatu and N. Boujemaa: "Reducing the redundancy in the selection of samples for SVM-based relevance feedback". Research report INRIA 5258, May 2004.
   

[30] H. Shao, T. Svoboda, L. Van Gool: "Distinguished Color/Texture Regions for Wide Baseline Stereo Matching". Submitted for publication.
   

[31] Cotsaces, M.A. Gavrielides, and I. Pitas: "Video Shot Boundary Detection and Condensed Representation: a review". IEEE Transactions on Circuits and Systems for Video Technology, submitted, September 2004.
    

[32] M. Frantzi, N. Moumoutzis, S. Christodoulakis: "A Methodology for the Integration of SCORM with TV-Anytime for Achieving Interoperable Digital TV and e-Learning Applications". In the Proceedings of the International Conference on Advanced Learning Technologies (ICALT 2004 ), August 2004, Finland
   

Author Details

George Ioannidis
Technologie-Zentrum Informatik (TZI)
University of Bremen
Germany
email:
website http://www.tzi.de