In M. Meybury (ed.), Intelligent Multimedia Information Retrieval. AAAI/MIT, 1997, 83-111.Sketching, Searching, and Customizing Visualizations: aContent-based Approach to Design RetrievalMei C. Chuah, Steven F. Roth, Stephan KerpedjievSchool of Computer ScienceCarnegie Mellon UniversityPittsburgh, PA, 15213, USATel: +1-412-268-2145E-mail: {mei+, steven.roth, kerpedjiev}@cs.cmu.eduhttp://www.cs.cmu.edu/~sageABSTRACTWe present new techniques for retrieval of data-graphics and a system, SageBook, thatemploys these techniques to facilitate the process of visualization design. Design is animportant activity in many different disciplines, including engineering, science, and business,but current systems provide little support for non-expert users to design new graphics for usein data analysis. SageBook’s approach is to provide expertise through the retrieval and reuseof previously successful designs. The design task places new demands on retrieval technologybecause it requires not only a good search engine but also effective tools to pose queries,browse results, and adapt previous designs for reuse. Despite our focus on data-graphicdesign, the concepts presented can be transferred to other design activities.1. INTRODUCTIONThis chapter will discuss retrieval as it relates to the problem of graphic design, an importantactivity in many disciplines and tasks. Graphics are used by analysts in many domains toanalyze trends, detect patterns and anomalies, and answer focused questions. Data analysesare also performed by statisticians to identify relationships or detect problem areas. Theseactivities are classified as exploratory data analysis (Tukey 1977). In addition to analysis,Sketching, Searching, and Customizing Visualizations: a Content-based Approach to Design Retrieval • 2graphics are useful for succinctly and clearly communicating information to others inpresentations. Whether for analysis or presentation the success of these tasks depends on theability of people to design effective graphics of their data quickly.In the area of visualization design, the last decade saw significant progress in developingintelligent tools that help users construct data-graphics (e.g. Mackinley 1986; Roth andMattis 1990; Casner 1991). A common feature of these systems is their ability to generategraphics that integrate multidimensional data. Most commercial applications, includingpopular spreadsheet systems, produce only simple graphics that tend to isolate data attributesin separate charts. In contrast, these intelligent systems integrate multiple attributes into onegraphic using a variety of composition techniques: multiparameter graphical objects, spacealignment, and grapheme clustering.Another major problem that has been addressed only recently in the SAGE research isproviding users with an interface that helps them design custom visualizations (Roth et al.1994). This research stresses the need for a combination of user-controlled interactive designtools and automatic design mechanisms. Their assumption is that design is inherently a dualprocess of constructing or assembling graphic elements into composites in a bottom-upfashion, as well as a process of considering previous examples that might be relevant tocurrent needs. Thus, these processes suggest complementary tools for specifying graphicsconstructively and browsing previously created graphics to reapply them.Sketching, Searching, and Customizing Visualizations: a Content-based Approach to Design Retrieval • 3Raw dataInternalStructureInternalStructureQueryNewGraphicConversionResultObjectsUserQuery refinementAdaptationStorage RetrievalMatchingSelectedgraphicBrowsingLibraryFigure 1: The retrieval and reuse process for designsTo support these processes, two tools were created. SageBrush is a tool with which userssketch their design ideas; the intelligent design engine of SAGE then converts this sketch intoa data-graphic. Moreover, the sketch may be incomplete, in which case SAGE attempts tocomplete the graphics by selecting and composing additional graphical elements andproperties. Although the interactive graphic design techniques supported by SageBrush areuseful when people know the graphic they wish to create, it is still necessary to provide themwith alternative graphics when they are unsure how to visualize data and need to browsethrough design alternatives. To provide this type of design support, we developed anothertool called SageBook. It is closely integrated within the SAGE system and provides retrievalcapabilities that help users extract relevant visualization designs and adapt them to theirneeds. The major processes occurring in SageBook are shown in Figure 1.Central in the retrieval and reuse process shown in Figure 1 is a library (store) ofvisualization designs. This storage infrastructure is expressive in such a way that itaccommodates important design elements and supports easy conversion of existing designsinto its internal storage language. In Figure 1, the rectangles labeled "internal structure"represent designs or queries in the internal language. Users can query the stored designsSketching, Searching, and Customizing Visualizations: a Content-based Approach to Design Retrieval • 4based on their current tasks and data. Query interfaces help users communicate to the systemthe types of designs that are desired. Therefore, the interface should closely match the users’mental model of design elements in the intended domain. Before matching a query withlibrary entries, the system converts that query to the system’s internal storage language in away similar to the conversion of the original designs.The internal description of the query is then used to retrieve library entries. There are twotypes of retrieval algorithms based on exact and similarity matching. Exact matching returnsdesigns that fulfill all the criteria specified in the query, while similarity matches are lessstringent and may return designs that contain enough, but not all, of the query elements. Indesign, one important use of graphic libraries is for getting new ideas from past successes.Since exact matching would severely limit the number and types of designs retrieved, it iscritical to be able to retrieve designs based on some similarity measure. Similarity matching,however, can result in a large number of hits, especially when the library is well-populated.In order to effectively process the search results, users need tools that can help them organizethe