Evaluation MethodsIn 160 We’ve Covered…There are many more methods…How to chose a method?Pros & ConsMethodsSurveySlide 8InterviewSlide 10Controlled Lab ExperimentSlide 12In-lab ObservationSlide 14Controlled Field ExperimentControlled Field ExperimentField Observation StudySlide 18Heuristic EvaluationCognitive WalkthroughAutomate Observation StudySlide 22Experimental SimulationSlide 24Claims AnalysisGOMSSlide 27Computer SimulationComputer SimulationFormal TheorySlide 31Slide 32Early StageSlide 34Iterative & Summative StagesSlide 36RealismSlide 38PrecisionSlide 40GeneralizabilitySlide 42Time & CostSlide 44Researcher PerspectiveMetrics: examplesPeripheral DisplaysQuestions?Evaluation MethodsApril 20, 2005Tara MatthewsCS 160In 160 We’ve Covered…•Task Analysis & Contextual Inquiry•Cognitive Walkthrough•Heuristic Evaluation•WOZ usability study w/ paper prototypesThere are many more methods…•Survey•Interview•Controlled-lab experiment•In-lab observation•Controlled field experiment•Field observation study•Automated observation user study•Experimental simulation•Claims analysis•GOMS•Computer simulation•Formal theoryHow to chose a method?•Stage of study–formative, iterative, summative•Pros & cons•Metrics–depends on what you want to measure•Qualitative vs. quantitative•Research perspective–CS vs. psychology vs. sociologyPros & Cons•Realism•Precision•Generalizability•Time & cost•Researcher expertiseMethods•Survey•Interview•Controlled-lab experiment•In-lab observation•Controlled field experiment•Field observation study•Automated observation user study•Experimental simulation•Claims analysis•GOMS•Computer simulation•Formal theorySurvey•Online / paper questionnaires distributed to target audience•Can be used to–tabulate quantitative data–gather qualitative feedback (opinions, feelings, etc.)•Useful at any time in studySurvey•Pros–Easy to get a large number of responses.–Quick and easy to conduct.–Highly generalizable.•Cons–Self-selection.–Participants often only offer enough information to answer the question.–Can miss details.–Low in realism and precision.Interview•Evaluators formulate questions on the issues of interest.•Interview representative users, asking them these questions in order to gather information desired.•Interviewer reads questions to user, who replies verbally; interviewer records responses.Interview•Pros–Quick and easy to conduct.–Gives designer quick feedback on a range of ideas.–Can get a person’s initial reaction to an idea.–Can get detailed information from a person.•Cons–Often takes place away from natural setting.–Question wording or interviewer “body language” can bias answers.–High probability of false positives and missed problems (e.g., users may not have a clear idea of how an app will be used).–Can miss details if interviewer doesn’t know what issues to draw out.Controlled Lab Experiment•In lab, manipulate one feature of a system to assess the causal effects of the difference in that manipulated feature on other behaviors of the system.•Example:–in lab, show users 4 versions of a website:•blue, yellow, red, and black text–measure time to find specific words–compareControlled Lab Experiment•Pros–Provides precise, quantifiable data.–Easier to draw inferences from data.–Relatively quick.–Can get a medium-sized number of participants.•Cons–Short duration of a lab experiment may not be enough to allow users to become accustomed to an app.–Not a natural setting – interaction may not be normal.In-lab Observation•Participants come to lab to "use" an interface•Given sample tasks to complete with it•Evaluators observe and possibly audio- or videotape•Participants may "think out loud"•Can use lo-fi prototype (for a project in the design stage) to an almost-complete interface•Evaluators note participants’–emotions, exclamations, facial expressions, and other "qualitative" data–take note of quantitative data such as time to complete a task or number of errorsIn-lab Observation•Pros–Relatively quick.–Can get a medium-sized number of participants.•Cons–Observations are subjective and error prone.–Short duration of lab observation is not enough time for user to get accustomed to using the interface.–Not a natural setting – interaction may not be normal.Controlled Field Experiment •In natural setting, manipulate one feature of a system to assess the causal effects of the difference in that manipulated feature on other behaviors of the system.•Example:–Participants use 3 different input devices in their own office: mouse with 1, 2, or 3 buttons–Perform a set of tasks–Measure differencesControlled Field Experiment•Pros–Less intrusive than most other evaluation methods.–Provides more precise data than field observation.–Can observe natural behavior of user (though some part of the system will be controlled/unnatural).•Cons–More intrusive than field observation.–Less natural than field observation.Field Observation Study•Evaluator makes direct observations of “natural” systems•Takes care to not intrude on / disturb those systems•A.K.A. “ethnography”Field Observation Study•Pros–Only way to observe natural behavior of user & interaction between user & tools.•Cons–Difficult and time consuming.–Hard to get permission to observe people.–Observations are subjective and error prone.–Cannot make strong interpretations from observations.–Not very generalizable.Heuristic Evaluation•Pros–Quick and easy.•Cons–Nielson’s heuristics may not be as relevant to non-GUIs.–Results in false positives in missed problems, especially when experts are not part of target audience.Cognitive Walkthrough•Pros–Quick and easy.•Cons–Results in false positives and missed problems when evaluator is different from target audience.Automate Observation Study•Techniques include–video or audio recording of user–pop-up screens–screen shots–time logging–log users actions (collecting statistics about detailed system use)Automate Observation Study•Pros–Eases burden on observers for data collection & analysis.•Cons–Setup is often more time-consuming to complete.–Harder to get approved if it involves analysis of videotape or audiotape.–May miss nuanced/interpretive details.Experimental Simulation•In-lab experiment
View Full Document
Unlocking...