Evaluation

Norbert Fuhr gives a survey of current activity in the Evaluation cluster.

Introduction

While supporting ongoing evaluation initiatives like the INEX and CLEF campaigns, the evaluation cluster is also working towards the development of new evaluation models and methods.

CLEF  

CLEF (Cross-language Evaluation Forum, http://www.clef-campaign.org/) provides test-beds for the evaluation of cross-language information retrieval. For the 2004 campaign, CLEF will consist of 8 different evaluation tracks: Mono-, Bi- and Multi-lingual Information Retrieval, Mono- and Cross-Language Information Retrieval on Structured Scientific Data, Interactive Cross-Language Information Retrieval, Multiple Language Question Answering, Cross-Language Retrieval in Image Collections, and Cross-Language Spoken Document Retrieval. Supported languages will be Dutch, English, Finnish, French, German, Italian, Portuguese, Spanish, Swedish, Russian, Japanese and Chinese. At the moment, data and topics have been distributed to the participants, who will submit their runs by 15 May. Results will be presented at the CLEF workshop during ECDL over 16-17 September 2004 in Bath, UK.

INEX

INEX (Initiative for the evaluation of XML Retrieval, http://inex.is.informatik.uni-duisburg.de:2004/) deals with the evaluation of information retrieval methods for XML documents. The major tasks involved have been identified as retrieval for content-only queries and retrieval for queries referring both to content and structure of the target elements. During 2004 INEX will undertake 4 additional tracks dealing with relevance feedback, natural language queries, heterogeneous collections and interactive retrieval. 49 participating groups have registered for this year's INEX campaign and are currently involved in topic creation. The results for 2004 will be presented at a workshop in Schloss Dagstuhl, Germany, over 6-8 December.

Evaluation Models and Methods

This task focuses on the specification of standard evaluation methods for digital libraries, starting with a comparison and evaluation of existing evaluation methodology, and then developing new techniques, methods and measures. The first step will be a workshop on DL evaluation over 4-5 October 2004 in Padova, Italy. The workshop will concentrate upon a survey on the state of the art in digital library evaluation and will identify major issues for further research in the area of DL evaluation.

Conclusion

Most of the current activities in this cluster are targeted towards providing the necessary infrastructure for the INEX and CLEF campaigns. As a general infrastructure for DL evaluation, an evaluation forum is under development which will support communication between DL researchers and evaluation specialists. In the future, research on evaluation models and methods will be enforced in DELOS, along with the development of appropriate evaluation toolkits and test-beds.

Author Details