Usability AssessmentLecture Slides Based on Usability Engineering by Laventhal and BarnesMacro Media’s Usability TipsSample Test for Web Site UsabilityUsability Performance MeasuresAdvantages of Usability MetricsHow many users do you need to measure?Measuring SuccessWhy Evaluate?Evaluation Is a ProcessWhat Are the Steps?Detailed Steps in the ProcessWhen to Evaluate?Formative EvaluationBenefits of Formative EvaluationSummative EvaluationSummative Evaluation ObjectivesWhat Kind of Evaluation?Analytic AssessmentChallenges of Analytic ModelsUnderstanding the EvaluationHigh-level Design EvaluationLow-level Design EvaluationTypes of MeasurementsSlide 25How Do Evaluations Differ?Benchmark TasksGetting Started with Benchmark TasksUsability Asessment01/14/19 1Usability AssessmentCIS 577Bruce R. MaximUM-Dearborn01/14/19 2Lecture Slides Based onUsability EngineeringbyLaventhal and Barnes01/14/19 3Macro Media’s Usability Tips•Remember user goals•Remember site goals•Avoid unnecessary introductions•Provide logical navigation and interactivity•Design for consistency•Don’t overuse animation•Use sound sparingly•Target low bandwidth users•Test for usability01/14/19 4Sample Test for Web Site Usability•Ask users to answer two factual questions based on the web site information•Ask users to answer two task oriented questions from the web site information•Ask users to share the subjective impressions about the ability of the web site content to meet their needs in an interview setting•Have users complete a post test survey regarding their overall experience01/14/19 5Usability Performance Measures•Time to learn• Speed of performance• User error rates• Retention over time• Subjective satisfaction01/14/19 6Advantages of Usability Metrics•You can track progress between releases•You can assess your competitive position•They help you make stop or go decisions prior to product launch•They can serve as the basis of bonus plans for design managers01/14/19 7How many users do you need to measure?•If you are doing qualitative testing, it is enough to obtain data from 3 to 5 users•If you are collecting quantitative data, you should test 20 or more users01/14/19 8Measuring Success•% improvement can be useful (eg. time to complete task in old and new design)•Several % improvement scores are may be combined by computing their geometric mean•Geometric mean in the Nth root of the product of N numbers01/14/19 9Why Evaluate?•Good design does not guarantee a good user interaction.•Quality assessment is a key element, in combination with good design, for building highly usable user interfaces.•Experience has shown that the best design efforts can not guarantee quality evaluation involving representative users.01/14/19 10Evaluation Is a Process•What steps are part of the evaluation process?•When should usability assessment be performed in the development cycle?•What kind of evaluation (or evaluations) can be used?01/14/19 11What Are the Steps?•Understand the problem (assessment)•Design a solution (the assessment)•Implement the solution (the assessment)•Test the solution01/14/19 12Detailed Steps in the Process•Understand the evaluation.•Design the evaluation at a high level. •Design the evaluation at a low level.•Design your procedure•Prototype your evaluation•Do it! Perform the evaluation.•Analyze the results.•Make recommendations based on the results of the evaluation.01/14/19 13When to Evaluate?•Formative Evaluation–Assessment conducted during the creation of a product prototype•Summative Evaluation–Assessment conducted during following the completion of a product prototype01/14/19 14Formative Evaluation •Formative evaluation occurs in order to help designers refine and revise their designs (preventive maintenance).•The focus in formative evaluation is to identify problems and potential solutions.•Do it as early as possible.•Do it several times during the development process.01/14/19 15Benefits of Formative Evaluation•It improves analysis and understanding of requirements for the eventual user interface.•It allows modification of the interaction and interface design.•It tests the interaction and interface design.01/14/19 16Summative Evaluation•Summative evaluation is concerned with summarizing the overall impact and effectiveness of a system.•These should be quantifiable goals or characteristics that can be used as a criterion for “good enough.”•This is the acceptance contract with the client.01/14/19 17Summative Evaluation Objectives•Determine what acceptable levels are for each of the usability goals and constraints (identified in the specification).•Determine how to measure each usability characteristic.•If the specification for the project included “usability requirements” these can often be used in summative evaluation.01/14/19 18What Kind of Evaluation?•Analytic (predictive) evaluation•Heuristic evaluation by experts•Cognitive walkthroughs•User testing01/14/19 19Analytic Assessment•The developer builds a design or competing designs and identifies prototypic tasks. •Based on human performance data, the developer can estimate the time to complete the task on the design or designs.•An example is GOMS analysis.01/14/19 20Challenges of Analytic Models•GOMS assumes idealized or expert performance on the part of the user. •The GOMS hierarchy is somewhat subjective. Two different designers may develop two different GOMS models for the same interface task and come up with different predictions.01/14/19 21Understanding the Evaluation•What is it that we are trying to evaluate?•Questions to answer–When to evaluate?•Formative evaluation•Summative evaluation–What kind of evaluation? •Analytic•Heuristic/walkthrough•User test01/14/19 22High-level Design Evaluation •What factors to evaluate? –Task match–Support of usability concerns•Questions to answer–User opinions and/or information about performance? –Will your measurements be based on observation of users in a naturalistic setting or will they be based on data from a controlled or laboratory setting?01/14/19 23Low-level Design Evaluation•Low-level design activities are focused on filling in the details to the high-level design.•What details? –Define your criteria!–Operational assessment criteria.–Identify standardized/benchmark