Workshop on the Evaluation of Digital Libraries Department of Information Engineering, University of Padua, Padova, Italy October 4-5, 2004 General information For more information on the event, please click here. Digital libraries are destined to become an essential part of the information infrastructure in the 21st century. Given the rapid development in this field, there is an increasing need for evaluations in order to determine the usefulness, usability and economics of digital libraries. However, there is a lack of appropriate evaluation methodologies. This workshop will target at a survey on the state of the art in DL evaluation, and will identify major issues for further research in the area of DL evaluation. Specifically, this workshop will try to find answers to the following questions: - Why are DLs evaluated? - What are standard approaches for DL evaluation? - Which aspects/components are evaluated in these approaches? -- collections, users, usage, systems -- content, community, services, technology -- societies, scenarios, spaces, structures, streams - Which evaluation criteria are used in these approaches? - What are the strengths and weaknesses of these approaches? - What should be done (in the future) in DL evaluation? In addition, two general issues are to be addressed: - What is an appropriate conceptual DL model to which evaluation could refer to ? - Are there conceptual frameworks for evaluation? As a result of the workshop, a white paper will be prepared, giving a brief survey over the state of the art in DL evaluation, as well as a description of open research issues in this area. Participation in the workshop will be by invitation only. List of workshop participants Programme MONDAY OCT. 4 | MORNING SESSION | 09.00-10.30 | Dedicated to an Introduction of the Problem, Presentations and Discussions Aimed at Identifying a Digital Library Evaluation Framework | Chairs: Maristella Agosti and Norbert Fuhr | | The DELOS Evaluation Cluster | Norbert Fuhr | | DELOS NoE: Previous Work, Experiences and Recommendations from the Evaluation Forum. What Is Next? | Ingeborg T. Sølvberg | 10.30-11.00 | Coffee Break | | 11.00-12.30 | Evaluation of Digital Libraries: An Overview | Tefko Saracevic | 12.30-14.00 | Lunch | | AFTERNOON SESSION | 14:00-16.00 | Invited speakers - Facets of DL evaluation | Chair: S. Kapidakis | | Understanding Users' Experiences: Evaluation of Digital Libraries | Ann Blandford | | Evaluating the Uses of Digital Libraries | Christine L. Borgman | 16.00-16.30 | Coffee Break | | 16.30-18.30 | Invited speakers - Facets of DL evaluation (cont.) | Chair: Carol Peters | | Towards a Quality Model for Digital Libraries | Edward A. Fox | | Towards a Quality Model for Digital Libraries | Derek Law | 18.30 | Closing | | TUESDAY OCT. 5 | MORNING SESSION | 09.00-10.45 | Evaluation Requirements from the Perspectives of the Different DELOS Clusters and Current EU DL Projects | Chair: Ingeborg T. Sølvberg | | User Interfaces and Visualization Evaluation - WP4 | Hanne Albrechtsen | | Evaluating Preservation Aspects - WP6 (html file) | Manfred Thaller | | BRICKS EU Project | Umberto Pernice | | DILIGENT EU Project | Donatella Castelli | 10.45-11.15 | Coffee Break | | 11.15-12.30 | Towards an Evaluation Framework: Position Papers of WP7 Members | Chair: Norbert Fuhr | | Evaluation of User Interaction in Digital Libraries | Giannis Tsakonas, Sarantos Kapidakis, Christos Papatheodorou | | Test Collection Construction: the CLEF Experience | Carol Peters | 12.30-14.00 | Lunch | | AFTERNOON SESSION | 14:00 - 15.45 | Towards an Evaluation Framework: Position Papers of WP7 Members (cont.) | Chair: Norbert Fuhr | | Accessing XML Documents: the INEX Initiative | Mounia Lalmas, Thomas Rölleke, Zoltan Szlavik, Tassos Tombros | | Evaluation of a Digital Library System | Maristella Agosti, Giorgio Maria Di Nunzio, Nicola Ferro | | Evaluating Digital Library Information Models | Trond Aalberg | | The Evaluation Computer: a Model for Structuring Evaluation Activities | Lazlo Kovacs, Andras Miscsik | | An Evaluation Framework for E-books | Monica Landoni | 15.45-16.15 | Coffee Break | | 16:15-18.00 | Panel and Discussion on Future Directions for the Evaluation of DL | Panel Chair: Tefko Saracevic | 18.00 | Closing | |
|