Popup Workshop:Preliminary Analysis of User TestingSusan L. HendrixDepartment of Computer ScienceUniversity of Colorado at Boulder28 Feb 20071IntroductionThis analysis of my user testing so far comes from a discussion with Clayton Lewis, in which I posed t he questionof whether I had done enough user testing. His suggestion, that I felt was a valuable one, was to take a look atuser testing so far, what I had set down as the aims of user testing, and see if the two matched. This documentis the result. It is an informal look at what I have done in this area and what I plan to do next. I come to theconclusion (quite spoiling the ending) that I have enough user testing at this point for the dissertation, andrequesting correction if anyone feels this is not the case.The evaluation plan is available at: http://www.cs.colorado.edu/ hendrixs/evalplan.pdf, and the reader mayrefer to it if necessary. I have also placed a companion document, that contains pictures of the popups for eachtest subject, at http://www.cs.colorado.edu/ hendrixs/committee/usersummary.pdf. This was too large to email.The next section describes the test subjects so far. In Section 3, changes made to the evaluation process aredescribed. Most of the process in the evaluation plan was followed and this section describes some additions,deletions and alterations. Section 4 looks at the observations made and whether they are adequate to support theareas discussed in the evaluation plan. The evaluation plan also listed some areas in which more experience wasneeded than that provided by earlier observations of users–the early work with 5th grade students, for example.These are discussed in Section 5. Finally, Section 6 details what work remains in terms of data analysis, and myconclusions about why I feel no more users are needed.2UsersThe original plan was to recruit 5-8 students of ages 10-13. At present, 5 students with a larger age range havebeen tested, as two of the children were 6, two were 11 (twins), and one was 12. The following table summarizesuser names, ages, number of popups and length of testing. The number of hours of testing is approximate, aseach tape is counted as one hour (some may be slightly less). Emily is still in testing; her totals represent thetesting so far, with the estimates of the final totals in parentheses.A short summary of the work of each child might be in order. The interested reader may find a .pdf documentwith pictures from each session of testing at http://www.cs.colorado.edu/ hendrixs/committee/usersummary.pdf.This will become Appendix D of the dissertation. A short summary of each session will be added in the finaldocument. I am currently going through the tapes, that have been converted to DVDs, recording times for later1Name Starting age No. Sessions Hours No. Pop-ups CompletedUrsula 6 (turned 7 during testing) 8 10 13Richard 6 5 5 7Daisy 12 7 11 7Peggy 11 (turned 12 during testing) 8 8 4Emily 11 (turned 12 during testing) 10 (12) 12 (15) 7(9)analysis, and preparing the summaries. Readers may wish to refer to this companion document, that was a bitlarge for email. (Or request a paper copy.)Ursula was 6 on beginning testing. I was somewhat hesitant about the use of the program with a 6 year-old,but I knew that Ursula did a lot of paper crafts on her own, and decided she would be an interesting subject.She was very shy and said little, even when prompted in early sessions. In later sessions, she answered questionsbut was still quiet. I was amazed at her ability t o make the pop-ups. Beginning with a simple frog (made byhand) and a simple abstract on the computer, she developed a facility with applied structures. She also had apredilection for bunnies and turtles. But it is interesting that after we returned to testing after a forced break ofseveral months, the bunnies and turtles disappeared and she began to make more abstract shapes.Richard was another 6 year-old. He is the brother of Daisy, an older subject. Richard also showed someability with applied elements, and from the start was adding to elements and changing them. He showed a lot ofimagination in his constructions, and liked telling elaborate stories about them. He could not write yet–this wasa problem for the younger subjects, but he asked me to write words under tabs on some of his pop-ups. He wasalso much more forthcoming than Ursula. He would have continued longer, but his family was moving overseas,so we had to stop at 5 sessions. (We did manage the final interview before he left.)Daisy was 12. She is Richard’s older sister, and also had to move, although since she had started earlier, wedid more. She showed a love of abstract shapes and faces–similar to those I saw with 5th graders earlier in thedevelopment of the program. At one point I suggested that she might illustrate a story she liked, as she was agreat reader. She brought in a fantasy book about owls, wanting to make an owl to put up in her room. This wasthe largest pop-up made in user testing. It is more a paper sculpture than a pop-up. Everything pops up andfolds except the talons, that were attached with double-sided tape in order to allow it to be folded f or traveling.Peggy and Emily are fraternal twins. They showed very different approaches, and I plan to devote a singlechapter in the dissertation to their work, as opposed to choosing one child as a case study. Peggy was notsomeone who likes arts and crafts. She made only 4 pop-ups, but they were quite complex, and she spent a longtime on them. She used applied elements, and added her own elements taken from books. Her pop-ups werethe most mechanically complex of any user’s, and she showed an interest in looking at the how-to books andunderstanding the more complex elements.Emily loves art, and takes oil painting classes. Her testing is not yet complete; I estimate 2 more sessions of2 hours each will be required at this point, that includes the final interview. She is making a book, with a seriesof characters all connected to each other–a man, his cat, the mouse eaten by the cat, and so on. She has addedmaterial to the computer generated elements, reshaped them, and added her own elements like flaps and tabs.She should complete the last animal (an elephant) in the next session, and plans one more pop-up with all theanimals together. She is unusual in that she planned a book from the start, and every pop-up has been madewith that in mind.To summarize the users, the lower number of users were