APPENDIXAARunning Usability EvaluationsRunning a usability test with real customers is essential to good design.You may know a lot about your customers, but it is hard to predict howpeople will react to and interact with a Web site. Usability tests are alsoeffective in ending those endless opinion wars in which members of thedesign team argue about what people like and don’t like. The best way toanswer this question is to recruit some participants) run a quick test, andsee what they say and do. This appendix lays out the steps for runningboth formal and informal usability tests, from setting up the test to run-ning the test to analyzing and presenting the results.Our assumption here is that you want to run a usability test in whichboth you and the participant are in the same place. You should also beaware that an alternative approach is remote usability testing-that is,recruiting and testing many participants online without your having tobe there. We discuss how to do this in Appendix E-Online Research.A.1 Setting Target GoalsWhat Do You Want to learn from the Test? l The first thing you have to dois decide what you want to get out of the test. Do you want to find out ifpeople are having problems with a specific part of the Web site? Do youwant to see how well a proposed design works? Or do you just want toget general feedback about the existing Web site?How Will You Get the Information You Want? l After deciding what youwant to learn, think about how you will get this information. If people1In psychology and other fields, the term subjects is used instead of participants, but wehave always felt that the term subjects has a slightly sinister tone.hARunning Usability Evaluationsare having problems with a portion of the Web site, the straightforwardthing to do is to test tasks that rely on that part and see what the prob-lems are. If you want to test a new design, it is useful to compare it to theold design or to a competitor’s Web site. This approach is also useful forgetting general feedback about an existing Web site.Process Data versus Bottom-line Datal There are two kinds of data thatyou can get from a usability test: process data and bottom-line data.Process data consists of informal, qualitative observations of what peo-ple are thinking and doing-an overall feeling of what works and whatdoes not on a Web site. The key things to look for here are critical inci-dents, places on your Web site where participants are confused, frus-trated, or even swear. Critical incidents also include cases in which peopleare pleasantly surprised or say something positive about the site.In contrast, bottom-line data consists of formal, quantitative meas-urements of what happened, such as the time it takes to complete a task,the number of errors that occur, or the time it takes to learn a task.In general, you should focus on getting process data first because itgives a good overview of where the problems in a Web site are and be-cause it is easier to get. Process data can also be obtained from low-fidelity paper prototypes, making it a handy technique for the early stagesof design.It takes more work to get and make use of bottom-line data. One rea-son is that you need to have lots of participants to get statistically reliableresults. Another is that bottom-line data does not always tell you whatproblems need to be fixed, it just tells you that people are going tooslowly or are making too many errors. Bottom-line data is better for laterphases of design, when you’re tuning the performance of an existing Website. It is also better for comparing two Web site designs, to show that oneis superior to another in a particular measurable aspect. Such a compari-son can be especially important when you’re trying to convince manage-ment to make either a major change or a change on an important page,such as the homepage of a high traffic site.Setting Up the TasksA.2The next step is to choose several representative tasks. By this we meanrealistic tasks that your target customers are likely to do on your Website. Choose some tasks that are simple, some that are of medium diffi-culty, and some that are hard. Ideally, these tasks will have already been+ 628 AppendixesRunning Usability EvaluationsA1worked out in the task analysis you carried out when learning to knowyour customers and can just be taken from there (see Chapter 3-Know-ing Your Customers: Principles and Techniques).Simple Tasks Are Short and Performed Often l Simple tasks include thingslike “Find the latest news article about parenting” or “Find the phonenumber and e-mail address of the help desk.” Success on simple tasks is abinary result: The person either succeeds or fails.Tasks of Medium Difficulty Are a little longer and Harder than SimpleTasks l Examples of medium-difficulty tasks include “Purchase the cheap-est printer you can find,N “Print out a list of all your previous purchases,”and “Add a message to the gourmet cooking community board.” Thesetasks span a few Web pages, but they are reasonable things that peoplewould do. Some medium-difficulty tasks will have binary success metrics;that is, they will either succeed or they will fail. The results of othermedium-difficulty tasks will be more open-ended and require furtherinterpretation of the results.Hard Tasks Span Many Web Pages and Are Fairly Involved l Examples ofhard tasks include “Make the Web site show you only the stocks you areinterested in,V “Buy a digital camera for a friend that he or she will like,”and “Buy a toy for your friend’s one-year-old child.” Most hard tasks arefree-form, so it will take some judgment to determine how successfulparticipants are.Tasks Should Be about What People Want to Do l Be careful not to tell peo-ple how to do the task. For example, instead of saying, “Go to ‘My profile’and find your previous purchases,” you should say something like, “Findall of your previous purchases.”Again, the task should be worded in theway people would ordinarily think about the problem-that is, what, nothow. Another example of careful wording is, “Make the Web site showyou only the stocks that interest you.” Not as realistic would be a tasklike “Customize your profile to show you the stocks that interest you,”because the words customize and profile are not likely to be part of people’sregular vocabulary. Another reason is that it might lead people on, espe-cially if there are links labeled “customize”