Information VisualizationEvaluation and User StudyCSE591 Visual AnalyticsLujin WangEvaluation for Information VisualizationEnrico Bertini http://www.dis.uniroma1.it/~beliv06/infovis-eval.html Component/system level evaluation  Low level components/perceptual studies  Longitudinal studies, case studies Metrics, benchmarks, model-based evaluation, frameworks, taxonomies  Novel evaluation methodologies, non-conventional methods/ parameters  ReviewsEvaluation for Information Visualization Component/system level evaluation Low level components/perceptual studies Longitudinal studies, case studies Martin Wattenberg. Baby names, visualization, and social data analysis.INFOVIS '05 Metrics, benchmarks, model-based evaluation, frameworks, taxonomies Robert Amar, James Eagan, and John Stasko. Low-level components of analytic activity in information visualization. INFOVIS '05 Novel evaluation methodologies, non-conventional methods/ parameters Reviews Catherine Plaisant. The challenge of information visualization evaluation.Advanced Visual Interfaces '04The Challenge of Information Visualization EvaluationCatherine PlaisantHuman-Computer Interaction LaboratoryUniversity of Maryland at College ParkAdvanced Visual Interfaces '04Introduction The tools and ideas in information visualization research publications are reaching users Commercial products– Spotfire, Inxight, and HumanIT Additions to statistical packages– SPAA/SigmaPlot, SAS/GRAPH, DataDesk Commercial development environments– ILOG JViews Others– SmartMoney financial maps, Peet’s Coffee Selector, health information maps, highway traffic informationInformation Visualization Evaluation The reports of usability studies and controlled experiments are helpful to understand the potential and limitations of tools However, we need  Consider other evaluation approaches that take into account – the long exploratory nature of users tasks– the value of potential discoveries – or the benefits of overall awareness Better metrics and benchmark repositories to compare tools Seek reports of successful adoption and demonstrated utility We need to understand how to improve evaluation methods in order to present actionable evidence of measurable benefits that will encourage more widespread adoptionCurrent Evaluation PracticesFour thematic areas of evaluation: (fifty user studies of InfoVis system) Controlled experiments comparing design elements Comparing specific widgets, mappings of information to graphical display Usability evaluation of a tool Provide feedback on problems of a tool, show how to refine the design Controlled experiments comparing two or more tools Compare a novel technique with the state of the art Case studies of tools in realistic settings Do real tasks, demonstrate feasibility and in-context usefulness Time consuming, results may not be replicable and generalizableChallenges Matching tools with users, tasks and real problems Using real datasets and demonstrating realistic tasks is important Improving user testing Looking at the same data from different perspectives , over a long time Answering questions you didn’t know you had Factoring in the chances of discovery and benefits of awareness Addressing universal usability General public, diverse usersPossible Next Steps Development of repositories of data and tasks Create benchmark datasets and tasks InfoVis 2003, 2004, 2005, 2006 contest, VAST 2006 contest Information Visualization Benchmark Repositoryhttp://www.cs.umd.edu/hcil/InfovisRepository/index.shtml Gathering of case studies and success stories Strengthening the role of toolkits and development tools Making a technique a part of a toolkit greatly improves its chance of successExamples of Technology Transfer A common evaluation measure for any technology is adoption by others, and the move into commercial products Commercial success depends on The quality of a product Financial alliances Marketing strategies Personal networking Luck Gain lessons from examples of transformations from prototypes to products and applicationsExample 1: From the Film Finder to Spotfire A 11-year-long voyage Dynamic scattergrams Other visualizations such as parallel coordinates, table views or standard business graphics were combinedExample 2: Treemap SmartMoney Map of Market The first widely known commercial application of treemaps Hive Group developed an application used by US marines for inventory management The availability of tools for data preparation and publishing, provides automatically updated views  A simplified interface allows end-users to view the data and perform limited filtering and grouping operationsExample 3: DataMap 1993 Dynamap For the National Center for Health Statistics DataMap Might be used by the Census Bureau to release data on CDsSummary Research prototypes are finding ways into commercial products We must carefully integrate the visualization tools into solutions for real problems Facilitating the importation of data Coping with large volumes of incomplete data Enabling users to integrate with other tools and collaborate with others Reporting on long term use in natural settings We should promote field studies, investigate new evaluation procedures, and celebrate successesLow-Level Components of Analytic Activity in Information VisualizationRobert Amar, James Eagan, and John StaskoCollege of Computing/GVU CenterGeorgia Institute of TechnologyINFOVIS '05Goal Present a set of primitive analysis task types Representative of the kinds of specific questions that a person may ask when working with a data set Serves as a form of common language or vocabulary when discussing the capabilities, advantages, and weakness of different information visualization systems Fosters an emphasis on the importance of analytic measures of information visualization systems Aids designers in creating novel presentations that amplify users’analytic abilities Serves as an informal checklist to assess and evaluate new systems and techniquesThe Nature of Analytic Activity User analyzes questions and tasks as part of analytic activity typically range from broader “high-level” goals to much more specific “low-level” inquiries For example, a person studying the history of