Newsletter Issue 2   

Main | Feature Article | Cluster Reports | DLA | IAP | A/V-NTO | UIV | KESI | EVAL | Dissemination | Latest News  

Evaluation 

Evaluation

  

Sarantos Kapidakis provides a summary of activity for the Evaluation cluster and describes the objectives and proposed content on the Work Package's forthcoming workshop.

  

Meetings

 

After our Kick-off meeting in Duisburg, Germany on 23 Jan 2004, the next Work Package meeting took place in Corfu, over 3-4 May 2004 in which we planned the WP activities mentioned as well as our workshop.

  

WEB

  

The Evaluation - WP7 website http://dlib.ionio.gr/wp7 is working and shows the aims and the objectives of the Work Package, the WP7 partners, related events, our publications, bibliography and useful links. It also includes a discussion forum to enable communication on evaluation issues. The design of the website is an iterative process and the collection of its content is a continuous WP activity.

 

INEX

  

INEX has seen intense activity during the last few months. We looked at the issue of accessing the INEX testbed. DELOS members can get access to the documents and queries, but not the relevance assessments. Approximately 50 groups are now registered to use it this year and there were 4 evaluation tracks. We have selected the CO topics for INEX 2004. 40 queries were selected this year. Many topics need modifications or corrections to be usable for the various tracks including the ad hoc retrieval task. Topic Assignment has been finalized and two topics are assigned to each participant keeping in view the topic authors. The deadline is 8 October 2004.

  

CLEF

  

After the detailed design of evaluation tracks, approximately 60 groups registered for one or more of the 8 evaluation tracks. 50 topics are prepared for mono- / bi- / multilingual tracks in 13 languages and 25 topics for Structured Scientific Data in 3 languages. Finally, CLEF is organizing a 3-day Workshop, co-located with the ECDL2004 conference, at Bath University, UK (15-17 September 2004), where the results will be presented.

  

Evaluation Workshop

  

The DELOS Workshop on the Evaluation of Digital Libraries http://dlib.ionio.gr/wp7/events.html is going to be held at the Department of Information Engineering (http://www.dei.unipd.it/), University of Padua, Via Gradenigo 6/a, 35131 Padova, Italy over 4-5 October 2004.

  

The objective of the workshop is to formulate a new model for DL evaluation and the results and recommendations emerging from the workshop will provide input for the production of a White paper (which will be ready by mid-December 2004). The workshop will cover a discussion on all components of a DL. We have invited representatives from each DELOS WP, and also a few speakers to cover different areas including some people from FP6 DL projects. Participation in the workshop will be by invitation only.

  

The Workshop is co-located with the DIALOGUES conferences (http://thepadovadialogues.dei.unipd.it) Padova Dialogues 2004 comprises three conferences:

  

SPIRE 2004 (String Processing and Information Retrieval) http://thepadovadialogues.dei.unipd.it/spire04

  

ALT 2004 (Algorithmic Learning Theory) http://www.tcs.uni-luebeck.de/pages/alt04.jhtml

  

DS 2004 (Discovery Science)

  

and satellite meetings for exchange and integration in the modelling, design and implementation of advanced tools for the representation, encoding, storage, search, retrieval and discovery of information and knowledge.

  

The workshop will focus on the definition of a workable evaluation framework. Specifically, this workshop will try to find answers to the following questions:

  

Why are DLs evaluated?

  

What are standard approaches for DL evaluation?

  

Which aspects/components are evaluated in these approaches?

  

collections, users, usage, systems

  

content, community, services, technology

  

societies, scenarios, spaces, structures, streams

  

Which evaluation criteria are used in these approaches?

   

What are the strengths and weaknesses of these approaches?

  

What should be done (in the future) in DL evaluation?

  

In addition, two general issues are to be addressed:

  

What is an appropriate conceptual DL model to which evaluation could refer?

  

Are there conceptual frameworks for evaluation?

  

Author Details

  

Sarantos Kapidakis
Laboratory on Digital Libraries and Electronic Publishing
Archive and Library Sciences Department
Ionian University
Platia Eleftherias, Palea Anaktora,
Corfu 49100, Greece

  

email:
Tel: +30 26610 87413
Fax: +30 26610 87436

  

---

Publication date: October 2004
File last modified: Friday, 19-May-2006

The Delos Newsletter is published by the Delos Network of Excellence
and is edited by Richard Waller of UKOLN, University of Bath, UK.

  

PDF version of the whole issue