Verse Chorus Verse: Iterative Usability Studies for the IN Harmony: Sheet Music from Indiana ProjectTalk ObjectivesIN Harmony Project BackgroundOverview of the Usability StudiesWebsite/Server Logs Analysis: IntroductionLogs Analysis : Purpose of Study (General)Logs Analysis: Purpose of Study (Specific)Logs Analysis: Background InformationLogs Analysis: MethodologyLogs Analysis: Data AnalysisLogs Analysis: Data Analysis2Logs Analysis: Strengths and WeaknessesLogs Analysis: SummaryCard Sort : IntroductionCard Sort : Purpose of StudyCard Sort : Background InfoCard Sort : MethodologyCard Sort : Data AnalysisCard Sort : Data Analysis2Card Sort : Data Analysis3Card Sort : Strengths and WeaknessesCard Sort : SummaryContent Analysis : IntroductionContent Analysis : Purpose of StudyContent Analysis : Background InfoContent Analysis : MethodologyContent Analysis : Methodology2Content Analysis : Data AnalysisContent Analysis : Strengths & WeaknessesContent Analysis : SummaryWhat’s Next?ReferencesMore InformationVerse Chorus Verse:Iterative Usability Studies for the IN Harmony: Sheet Music from Indiana ProjectOr, How I spent too much time in Exc(H)ell!Michelle DalmauInterface & Usability Specialist Indiana University Digital Library [email protected]/14/2005 DLP Brown Bag, Fall 2005 2Talk Objectives•Focus on the usability process (methods for data gathering and analysis), not findings •Show that conducting usability studies can impact more than software development; they impact the design of the metadata model and further usability studies •Explore how methodologies fit in the development cycle of a project•Discuss the strengths and weakness of the methods to be summarized12/14/2005 DLP Brown Bag, Fall 2005 3IN Harmony Project Background•IMLS funded 3-year project to digitize ~10,000 pieces of Indiana-related sheet music•Collaboration between Indiana State Library, Indiana State Museum, Indiana Historical Society and Indiana University•Project deliverables include:•Creation of shared metadata model/guidelines and sheet music cataloging tool (year 1 & 2)•Creation of shared digitization standards and image processing system (year 1)•Collection website (year 2 & 3)12/14/2005 DLP Brown Bag, Fall 2005 4Overview of the Usability Studies •Website/Server Query Logs Analysis•Card Sort •Email Content Analysis12/14/2005 DLP Brown Bag, Fall 2005 5Website/Server Logs Analysis: Introduction•Server logs provide details about file requests to a server and the server response to those requests –Transaction Logs, often processed by server-side software such as Apache’s Webalyzer•Focus on page hits, referrers, hostname, browser type, querystring capture, etc–Query Logs, often custom logging of queries using technologies like Java’s “log4j” •Focus on discovery patterns (browse links, search terms entered in simple v. advanced search pages, etc.)•Used to monitor on-going website usage and inform design changes depending on patterns uncovered12/14/2005 DLP Brown Bag, Fall 2005 6Logs Analysis : Purpose of Study (General)•Need to design a metadata model (and in turn, a cataloging tool) that meets user needs. •Example scenarios under investigation:–Known item searching: how are titles and names searched? Represent all aspects of this in the metadata model. –Subject searching: Music subject description is complicated; topical, genre, style, form, etc. are often not mutually exclusive. Understand how users conduct subject-related searches in order to define appropriate fields and controlled vocabularies. –Uncover unanticipated search parameters that should be represented in the metadata model (e.g. key or catalog/sheet music plate ID numbers)12/14/2005 DLP Brown Bag, Fall 2005 7Logs Analysis: Purpose of Study (Specific)•Harvest real-life queries and discovery patterns in order to understand: –How often users conduct a browse, search or advanced search for sheet music–How often users conduct known-item versus unknown-item searching–What kinds of searches are being conducted (keyword, title, name, subject, etc.)–What kinds of subject-related queries are being conducted (e.g. topical, genre, style, etc.)12/14/2005 DLP Brown Bag, Fall 2005 8Logs Analysis: Background Information •Collected a 10% random sample of query logs from a 6 month period from 2 sheet music collections (2,542 total log entries)–IU Sheet Music (Homogeneous collection) –Sheet Music Consortium (Heterogeneous collection delivered via OAI-PMH)•Different interfaces affect usage patterns and therefore affect the data. –Comparative analysis must be conducted in light of the differences (reconcile data, discard data or provide context for the data)12/14/2005 DLP Brown Bag, Fall 2005 9Logs Analysis: Methodology •Establish parse rules for logs •Establish data analysis goals:–Determine relative frequency of browse, search and advanced searches conducted–Compare number of known-item to unknown-item queries –Sort queries into identifiable access points for further evaluation: creator, title, subject, etc.–Determine further categories for subject-related search strings (topical, form, etc.)12/14/2005 DLP Brown Bag, Fall 2005 10Logs Analysis: Data Analysis •Establish data analysis rules/guidelines:–Coding underwent two passes: by researcher and domain expert–Define known (name, title and publisher) v. unknown items (subject, year, keyword) –Define subject types for encoding: instrumentation, genre/form/style, topical, geographic, temporal, language … –Define how and when queries can be encoded with two or more distinct fields (e.g. “Statue of Liberty” could be subject or title).–And so on …12/14/2005 DLP Brown Bag, Fall 2005 11Logs Analysis: Data Analysis2 •Excel works for quantitative analysis (duh!)–Non-numeric data is easily sorted and counted using Excel’s advanced filter features–Generate graphs and charts for those who don’t want to “read” the final report12/14/2005 DLP Brown Bag, Fall 2005 12Logs Analysis: Strengths and Weaknesses •Strengths–Provides a good foundation – Overview of usage and discovery patterns–Objective – “real” data–Quick capture – Data collection is automatic–Straightforward – In general, quantitative data is easy to analyze using tools like Excel•Weaknesses–Analysis can be time consuming – Not all data is straightforward, interpretation requires rules and