“What is a Good Digital Library?” – A QualityModel for Digital LibrariesMarcos Andr´e Gonc¸alvesaB´arbara L. MoreiraaEdward A. FoxbLayne T. WatsonbaDepartment of Computer Science, Federal University of Minas Gerais, 31270-901 BeloHorizonte MG BrazilbDepartment of Computer Science, Virginia Tech, Blacksburg, VA 24061, USAAbstractIn this article, we elaborate on the meaning of quality in digital libraries (DLs) by proposinga model that is deeply grounded in a formal framework for digital libraries: 5S (Streams,Structures, Spaces, Scenarios, and Societies). For each major DL concept in the frame-work we formally define a number of dimensions of quality and propose a set of numer-ical indicators for those quality dimensions. In particular, we consider key concepts ofa minimal DL: catalog, collection, digital object, metadata specification, repository, andservices. Regarding quality dimensions, we consider: accessibility, accuracy, complete-ness, composability, conformance, consistency, effectiveness, efficiency, extensibility, per-tinence, preservability, relevance, reliability, reusability, significance, similarity, and timeli-ness. Regarding measurement, we consider characteristics like: response time (with regardto efficiency), cost of migration (with respect to preservability), and number of service fail-ures (to assess reliability). For some key DL concepts, the (quality dimension, numericalindicator) pairs are illustrated through their application to a number of “real-world” digitallibraries. We also discuss connections between the proposed dimensions of DL quality andan expanded version of a workshop’s consensus view of the life cycle of information indigital libraries. Such connections can be used to determine when and where quality issuescan be measured, assessed, and improved — as well as how possible quality problems canbe prevented, detected, and eliminated.1 IntroductionWhat is a good digital library? As was pointed out in (Fuhr et al., 2001), the answerto this question depends on whom you ask. Many consider that what differentiatesa good DL from a not so good one is the quality of its services and content. Inprevious work, we have sought to formally elaborate the notion of digital librariesusing the 5S framework (Gonc¸alves et al., 2004). Since one of the main goals of thatPreprint submitted to Elsevier Science 22 November 2006work with 5S was to try to answer (at least partially) the question “What is a digitallibrary?” our hypothesis in this article is that further development of the theory willallow us to define critical dimensions and indicators of DL quality. In contrast toits physical counterpart, the “digital” nature of digital libraries allows automaticassessment and enforcement of those quality properties, thereby supporting pre-vention and elimination of quality problems. 5S gives a standard terminology todiscuss these issues in a common framework. Moreover, the formal nature of ourDL theory allows us to add precision as we define specific DL quality dimensionsand corresponding numeric indicators.In this article, we will follow the standard terminology used in the social sciences(Babbie, 1990). We will use the term composite quality indicator1(or in shortquality indicator) to refer to the proposed quantities instead of the stronger termquality measure. Only after one has a number of indicators, and they are validated2and tested for reliability3, can they be composed into reliable “measures”. Despitepartial tests of validity (for example, through focus groups4) the proposed qual-ity indicators do not qualify as measures yet. Also, it should be stressed that theproposed quantities are only approximations of or give quantified indication of aquality dimension. They should not be interpreted as a complete specification of aquality dimension, since more factors/variables could be relevant than are specifiedhere. We will, however, reserve the right to use the term “measure” when talkingabout standard measures that have long been used by the CS / LIS communities.The distinction should be clear in context.This article is organized as follows. Section 2 provides background and contextnecessary to understand the remainder of the article. Sections 3 through 6 present allthe dimensions of quality, the proposed indicators, and their applications to key DLconcepts. Section 7 deals with the connections between the proposed dimensionsand Borgman et al.’s Information Life Cycle (Borgman, 1996). Section 8 showsthe evaluation of the proposed quality model with a focus group. Section 9 coversrelated work and Section 10 concludes the article.1An indicator composed of two or more simpler indicators or variables.2According to (Babbie, 1990), validity refers to the extent to which a specific measure-ment provides data that relate to commonly accepted meanings of a particular concept.There are numerous yardsticks for determining validity: face validity, criterion-validity,content validity, and construct validity.3Also according to (Babbie, 1990), reliability refers to the likelihood that a given measure-ment procedure will yield the same description of a given phenomena if that measurementis repeated.4A type of face validity.22 Background and ContextIn this section, we summarize the 5S theory from (Gonc¸alves et al., 2004). Here wetake a minimalist approach, i.e., we define, according to our analysis, the minimumset of concepts required for a system to be considered a digital library. Accordingly,let:• Streams be a set of streams, which are sequences of arbitrary types (e.g., bits,characters, pixels, frames);• Structs be a set of structures, which are tuples, (G, L, F ), where G = (V, E) isa directed graph and F : (V ∪ E) → L is a labeling function;• Sps be a set of spaces each of which can be a measurable, measure, probability,topological, metric, or vector space.• Scs = {sc1, sc2, . . . , scd} is a set of scenarios where each sck= he1k({p1k}),e2k({p2k}), . . . , edkk({pdkk})i is a sequence of events tha talso can have a number of parameters pik. Events represent changes in compu-tational states; parameters represent specific variables defining a state and theirrespective values.• St2be a set of functions Ψ : V × Streams → (N × N) that associate nodes ofa structure with a pair of natural numbers (a, b) corresponding to a segment of astream.• Coll = {C1, C2, . . . , Cf} be a set of DL collections where each DL collectionCk= {