DL news
2007-12-03: DELOS Association established
The DELOS Association for Digital Libraries has been established in order to keep the "DELOS spirit" alive by promoting research activities in the field of digital libraries.
More info...
2007-06-08: Second Workshop on Foundations of Digital Libraries

The 2nd International Workshop on Foundations of Digital Libraries will be held in Budapest (Hungary) on 20 Septemeber 2007, in conjunction with the 11th European Conference on Research and Advanced Technologies for Digital Libraries (ECDL 2007).
Event website

DL Events
January 24-25, 2008 - Padova, Italy

4th Italian Research Conference on Digital Library Systems
Event website

December 5-7, 2007 - Pisa, Italy

Second DELOS Conference on Digital Libraries
Event website

Delos News as an
Home arrow Research arrow Evaluation (WP7)
PDF Print E-mail

Digital library evaluation (WP7)

Cluster homepage | List of publications

Cluster objectives

Digital libraries need to be evaluated as systems and as services to determine how useful, usable, and economical they are and whether they achieve reasonable cost-benefit ratios. Results of evaluation studies can provide strategic guidance for the design and deployment of future systems, can assist in determining whether digital libraries address the appropriate social, cultural, and economic problems, and whether they are as maintainable as possible. Consistent evaluation methods also will enable comparison between systems and services.

The evaluation cluster will work both on evaluation methodologies in general as well as on providing the infrastructure for specific evaluations. Thus, the following objectives are addressed:

  • Development of a comprehensive theoretical framework for DL evaluation, which can serve as reference point for evaluation studies in the DL area.

  • Research on new methodologies will be supported in order to overcome the lack of appropriate evalution approaches and methods.

  • Development of corresponding toolkits and test-beds in order to enable new evaluations and to ease the application of standard evaluation methods.

Cluster activities

In order to reach these goals, the following activities will be carried out:

  • Workshops on DL evaluation, for collecting exixting evaluation approaches and methods.

  • Evaluation support to the DL community, by creating an evaluation forum for enabling communication between evaluation specialists and DL developers.

  • Development of new evaluation approaches and methods, in order to overcome the weaknesses of current approaches and the lack of methods for new types of applications.

  • Development of evaluation toolkits, e.g. for collecting and analysing experimental data.

  • Creation of test-beds for new content and usage types in DLs, by starting from the existing test-beds for XML and cross-lingual retrieval and extending these towards new media, applications and usage types.

  • Creation of test-beds for usage-oriented evaluation, by extending existing test-beds or by creation of test-beds of user interactions.

  • CLEF (Cross Language Evaluation Forum): it supports global digital library applications by (i) developing an infrastructure for the testing, tuning and evaluation of information retrieval systems operating on European languages in both monolingual and cross-language contexts, and (ii) creating test-suites of reusable data which can be employed by system developers for benchmarking purposes.

  • INEX (Inititaive for the Evalution of XML Retrieval): it provides an opportunity for participants to evaluate their XML retrieval methods using uniform scoring procedures and a forum for participating organisations to compare their results.

Expected results

  • Survey of existing evaluation methodology

  • A conceptual framework for DL evaluation

  • Collection of evaluation approaches and methods

  • Collection of evaluation toolkits and testbeds

  • New test-beds for new content and usage types and for usage-oriented evaluation

Cluster report in DELOS Newsletter, Issue 1, April 2004
Cluster report in DELOS Newsletter, Issue 2, October 2004
Cluster report in DELOS Newsletter, Issue 3, June 2005

Cluster coordinator

Norbert Fuhr
Universität Duisburg-Essen Germany

DELOS Community


Remember me
Forgot your password?
Create new user
DELOS search
 DELOS site
 DELOS sites