Newsletter Issue 3

Main | Feature Articles | Cluster Reports | DLA | IAP | A/V-NTO | UIV | KESI | EVAL | Promotion | Workshop | Latest News

Evaluation

Sarantos Kapidakis provides a summary of activity across the differing tasks for the Evaluation cluster.

  

Task 1: Evaluation Forum

  

The Evaluation forum website (http://dlib.ionio.gr/wp7/testbeds.html) that brings together DL developers and evaluators was harmonized with the DELOS website and guidelines and further content was added. The forum is represented by two distinct virtual spaces, where the communication and collaboration of WP7 members takes place.

  

The first virtual space consists of the Evaluation cluster website which hosts collections of existing evaluation approaches and testbeds. The collection of existing evaluation approaches (http://dlib.ionio.gr/wp7/literature.html), as expressed in the form of a list of publications, related to the evaluation of digital libraries, is available in two bibliographic formats (Harvard and BibTex), which allows for easy inclusion in publications. The collection of existing testbeds and toolkits (http://dlib.ionio.gr/wp7/testbeds.html) operates as a linking point to testbed collections for the evaluation of digital libraries or to the results of other research projects. In order to ensure the holistic inspection of the research area and awareness of previous work, these results are carefully selected and reflect diverse forms of research, methodologies, measurements and metrics.

  

Finally the WP7 website is an area for publishing information to other research communities or to the public. Visitors can read about the aims of WP7, the work already completed, the partners and the events that are organized by the cluster.

  

The second virtual space implements the discussion forum, which serves as an area of communication among the members of the WP. A list of threads reflecting the general interests of the WP and the specific Tasks enables WP members to communicate in a centralized way.

  

In addition, a satellite website supporting the WP7 workshop on the evaluation of digital libraries was created (http://dlib.ionio.gr/wp7/workshop2004.html) which now contains the presentations from the workshop and the electronic proceedings.

  

Task 2: Evaluation Models and Methods

   

In our Workshop on DL evaluation, which took place in Padova over 4-5 October, 5 keynote speakers covered major aspects of DL evaluation:

• Tefko Saracevic: Evaluation of Digital Libraries: An Overview
• Ann Blandford: Understanding Users' Experiences: Evaluation of Digital Libraries
• Christine L. Borgman: Evaluating the Uses of Digital Libraries
• Edward A. Fox: Towards a Quality Model for Digital Libraries
• Derek Law: Content Evaluation

In addition, 3 representatives from DL-related Integrated projects currently funded by the EU described their plans and expectations concerning DL evaluation. The workshop gave a very good analysis of the state of the art on digital library evaluation. During the final panel session major issues for further research on DL evaluation were discussed. The presentations from the workshop together with the electronic proceedings are online at http://dlib.ionio.gr/wp7/workshop2004_program.html.

  

Task 3: INEX

  

In 2004, INEX consisted of three tracks: ad-hoc retrieval, iTrack focusing with interactive retrieval, and the heterogeneous track dealing with heterogeneous collections. Retrieval effectiveness and efficiency are the evaluation criteria currently considered. Appropriate evaluation measures that consider structural relationships between different answers are still to be developed, as well as usage-oriented measures for the interactive track.

  

Ad-hoc Retrieval Track

  

For the ad-hoc retrieval track, the following steps were performed: The creation and selection of topics, the submissions of runs, the pooling of runs, and relevance assessments by the participating groups. Computation of results is currently underway.

  

Interactive Track

  

In the interactive track, a base system has been developed and tested, topics were selected based on the ad hoc topics, questionnaires were created and guidelines written. Each of the 10 participants ran the base system with at least 8 users and submitted their results and their interaction logs along with the completed questionnaires. In addition, some participants ran their own interactive system with additional users. Evaluation of results and analysis of the interaction logs is underway.

  

Heterogeneous Track

  

For the heterogeneous track, six subcollections from different DLs were gathered. Then topic selection guidelines were created before topics were selected from the ad-hoc track; furthermore new topics were created. After topics had been distributed and submission guidelines had been formulated, participating groups submitted their runs.

  

INEX Annual Workshop

  

The INEX annual workshop took place in Schloss Dagstuhl on December 6-8. Working notes for this workshop are available at http://inex.is.informatik.uni-duisburg.de:2004/pdf/INEX2004PreProceedings.pdf. The final proceedings will appear in the Springer series 'Lecture Notes in computer science'. At the workshop, participants presented their work, results were discussed and the evaluation campaign for 2005 was planned.

  

Task4: CLEF

  

For the 2004 evaluation campaign, 6 different evaluation tracks were defined to assess different aspects of the MLIR (Multilingual Information Retrieval) paradigm:

• Ad Hoc (monolingual/bilingual/multilingual free text retrieval)
• GIRT(domain/specific retrieval with controlled vocabulary)
• iCLEF (interactive retrieval)
• QA@CLEF (question answering)
• ImageCLEF (image retrieval) and
• CL-SDR (spoken document retrieval)

CLEF activity included: setting up and managing the CLEF 2004 website; preparing and distributing Calls for Participation; preparing and extending the data collection (a Portuguese newspaper collection for 1994/95 has been added to the existing multilingual comparable corpus in nine European languages); preparing and distributing topics for the Ad Hoc and GIRT tracks; receiving and analysing the results. In addition, DELOS supported the overall coordination of the other six tracks - managed on a voluntary basis by research groups with expertise in the areas covered - and the organisation of the CLEF2004 workshop held in Bath, UK, 15-17 September (immediately following ECDL2004).

  

Of the 64 registered, 55 groups submitted results:

• Europe 38
• North America 13
• Asia 4

This represented an increase on the 42 groups in CLEF2003. 96 people attended the workshop. 15 European and 2 North American research groups collaborated in the organization under the overall coordination of ISTI-CNR, supported by DELOS. The test collections (consisting of data, queries and relevance assessments for a number of tasks) were expanded. The main multilingual comparable document collection now contains nearly 2 million news documents in ten European languages; new collections were added for cross-language image retrieval.

  

There was a shift of focus from textual document retrieval to information extraction and multimedia retrieval over languages. Evaluation methodologies were tested for the two new tracks (cross-language question answering and cross-language retrieval in image collections via a combination of text- and content-based methods). The aim has been to stimulate research towards next-generation CLIR (Cross-Language Information Retrieval) systems. A CLEF Steering Committee meeting was held in Bath on 15 September 2004.

  

Author Details

  

Sarantos Kapidakis
Laboratory on Digital Libraries and Electronic Publishing
Archive and Library Sciences Department
Ionian University
Platia Eleftherias, Palea Anaktora,
Corfu 49100, Greece
email:
Tel: +30 26610 87413
Fax: +30 26610 87436

 

---

Publication date: June 2005
File last modified: Monday, 22-May-2006

The Delos Newsletter is published by the Delos Network of Excellence
and is edited by Richard Waller of UKOLN, University of Bath, UK.

   

PDF version of the whole issue