UTK IS 588 - Introduction to System Evaluation

Unformatted text preview:

Introduction to System EvaluationDesign PhilosophiesWhy EvaluateWhat Users Need? Class DiscussionWhat Users Need from a System?Investing in User TestingWhy NOT Evaluate? Class DiscussionWhy NOT Evaluate?When to Evaluate?What to Evaluate?HutchWorld Case StudySlide 12HutchWorld Case Study: First TestFirst Test: Usability Issues IdentifiedFirst Test: Redesign Based on Issues IdentifiedSecond Test: Usability StudySlide 17Second Test: Lessons Learned from Usability StudyAfter the Second TestThird Test: Focus Group SettingFinal Test: Natural SettingWhen to Stop System Testing?Introduction to System EvaluationIS 588Dania Bilal & Lorraine NormoreSpring 2007Design Philosophies•System-centered design philosophy–Software works well; why evaluate from users’ perspectives?•User-centered design philosophy–Software should be usable: based on users’ needs, tasks, and cognitive and affective behaviors.Why Evaluate•Following design guidelines does not guarantee usability•Need to assess the user’s experience with system, cognitively and affectively–Allow identification of level of usability•Saves money–Problems can be fixed before system is releasedWhat Users Need?Class DiscussionAS users of various information systems (Web engines, OPAC, databases, etc.) what do you need a system that you use to do for you, in general?What Users Need from a System?•A system that is–Pleasing; satisfying–Easy to learn; friendly–Effective; challenging–Efficient; saves user’s time and efforts–Supported by positive, corrective feedback mechanisms –Predictable; consistentInvesting in User Testing•Tognazzini:–Problem can be fixed before product is released–Designers can identify “real” problems with system–Provide objective assessment for redesigning system–Save money and marketing time–Robust design to sell •High confidence in productWhy NOT Evaluate?Class DiscussionWhy most designers are against involving users in the design and evaluation of products/systems?Why NOT Evaluate?•Save time•Save money•Philosophy: users don’t know much about system capabilities and some may not articulate well their needs.•Other???When to Evaluate?•During product development–After market research, mock-ups, low-fidelity prototypes, etc. are designed and users’ reactions are elicited–Formative evaluation–Summative evaluation•Before release of final product–After prototypes and actual designs have been tested and problems have been fixed–Summative evaluation•After release of final product–To measure the usability and success of the product with users in a variety of settings.What to Evaluate?•The user’s experience with the product–User needs and requirements –Success, ease of use, friendliness, and other basic usability features –Affective states•Likes, dislikes, satisfaction, frustration, attitude–Support of use patterns–Other interaction behaviors based on purpose of productHutchWorld Case Study•Virtual community chat system •Developed by Microsoft Virtual Worlds Research Group and Fred Hutchinson Cancer Research Center•Purpose–To enable cancer patients, caregivers, family, and friends to chat and gain emotional and information support from one another.HutchWorld Case Study•Early design ideas–Learn about the patients’ experiences…•How was this done? –Students to replyHutchWorld Case Study: First Test•Informal on-site test–Computers were set up in hospital and a scaled-back prototype of V-chat was installed.–A sample of patients and their families interacted with prototypeFirst Test: Usability Issues Identified•User preference for asynchronous communication•Addition of websites about cancer information•Addition of games and entertainment featuresFirst Test: Redesign Based on Issues Identified•Software was redesigned based on user needs•HutchWorld became a portal containing a variety of information, communication, and entertainment areas–Bulletin boards, email, text chat, web page creation wizard, and other featuresSecond Test: Usability Study•Seven participants (male & female)•Prior experience was assessed•Interaction with system •Performed at Microsoft usability labs–Structured tasks (see Text, p. 329)–Verbalization (think aloud protocol)–Participants activities were captured online–Observer/evaluator interacted with participant in lab–Evaluator took observational notesSecond Test: Usability Study•Development team member interacted with system as a participant•Short questionnaire was completed by each participant after completing tasks •Participants rated difficulties in completing each task (see Text, p. 331)Second Test: Lessons Learned from Usability Study•Problems with software and their severity were identified–Rating of problems: low, medium, high–Problems can be seen in Text, pp. 332-333.–Fixes were madeAfter the Second Test•Additional observations and testing were done •Six new participants–Simultaneous interaction in labs–More detailed and focused–Structured similar to previous test–Fewer problems were identified–Fixes were madeThird Test: Focus Group Setting•Focus group interacted with system in Cancer Research Center •Patients and caregivers observed interaction•Feedback on final version was obtainedFinal Test: Natural Setting•System was used in a residential building with an Internet access•Building housed patients and their families•Informal observation of patterns of use •Assess how HutchWorld integrated with patients’ lives (medical care routines; access to social support, etc.)When to Stop System Testing?Class Discussion1. When do you think testing a system or a product should stop and why?2. What are the various methods employed in HutchWorld Study? Are these multiple methods useful and


View Full Document

UTK IS 588 - Introduction to System Evaluation

Download Introduction to System Evaluation
Our administrator received your request to download this document. We will send you the file to your email shortly.
Loading Unlocking...
Login

Join to view Introduction to System Evaluation and access 3M+ class-specific study document.

or
We will never post anything without your permission.
Don't have an account?
Sign Up

Join to view Introduction to System Evaluation 2 2 and access 3M+ class-specific study document.

or

By creating an account you agree to our Privacy Policy and Terms Of Use

Already a member?