EUROGRAPHICS 2004 / M.-P. Cani and M. Slater(Guest Editors)Volume 23 (2004), Number 3SMARTPAPER: An Interactive and User FriendlySketching SystemAmit Shesh and Baoquan ChenUniversity of Minnesota at Twin Cities†AbstractThis paper describes an interactive sketching system for 3D design/modeling that diverts from the conventionalmenu-and-button interfaces of CAD tools. The system, dubbed SMARTPAPER, offers a unified sketching environ-ment that supports direct sketching as well as gestured sketching with more emphasis on the former to encouragenatural sketching styles. SMARTPAPER also provides a unified 2D and 3D drawing domain by allowing the userto sketch directly on a 3D model in addition to the usual 2D sketching from scratch. A natural sketching ex-perience is offered by supporting casual sketching consisting of wiggly, discontinuous, overlapping strokes. Thesystem is empowered by an array of seamlessly integrated 2D and 3D features such as 2D sketch cleaning, 3Dreconstruction from 2D sketch, 3D transformations, sketching on 3D, and conventional 3D CSG operations likecutting and joining. The key to the success of SMARTPAPER is efficient and robust 3D reconstruction from a singlefreehand 2D sketch with minimal hints. We have employed and improved Lipson’s optimization method, originallydesigned for offline reconstruction of engineering drawings, in our interactive system by leveraging additionalclues obtained by interaction during sketching.Categories and Subject Descriptors (according to ACM CCS): I.3.3 [Computer Graphics]: Interaction Techniques,Pen-based Interaction1. IntroductionDesigners in almost all professions use a paper and pencil tomake sketches during the early stages of design. However,designing by sketches on computer has been quite difficultbecause of hardware usability issues. Also, the lack of 3Dgeometric information in sketches and the imprecision asso-ciated with them makes them very difficult to interpret al-gorithmically. In particular, 3D model reconstruction from asingle projective sketch is a mathematically insoluble prob-lem.Considerable research effort has been devoted to pro-mote design by sketches. Work has been done tosupport sketching schematics [GD96] and actual 3Dmodels in architecture design. Lipson and Shpitalni[LS00, SL97, LS02, SL96, LS96] generate 3D models from2D sketches, but their systems are non-interactive in nature.†Email: {ashesh,baoquan}@cs.umn.eduSKETCH [ZHH96] represents a fully gesture-based sketch-ing system, which however, can be unintuitive to use if thegestures are not carefully designed and are large in number.Teddy [IMT99] employs fewer and more intuitive gestures,but it focuses on design of free-form objects. We attemptto minimize these shortcomings by providing an interactivesketching system, SMARTPAPER, that is capable of recon-structing and operating on arbitrary rigid solid geometricalshapes and is intuitive to use. SMARTPAPER draws inspi-ration from all the seminal work in sketch recognition men-tioned above to achieve these goals. The system is imple-mented on a Tablet PC for a more natural sketching experi-ence (Figure 1(a)).We make four significant contributions through SMART-PAPER. First, SMARTPAPER presents a unified sketchingenvironment that supports both direct and gestured sketch-ing, with emphasis on the former. Secondly, SMARTPAPERgives more freedom to the user by supporting casual sketch-ing styles, where several overlapping discontinuous strokesc° The Eurographics Association and Blackwell Publishing 2004. Published by BlackwellPublishing, 9600 Garsington Road, Oxford OX4 2DQ, UK and 350 Main Street, Malden,MA 02148, USA.Amit Shesh & Baoquan Chen / SMARTPAPERFigure 1: SMARTPAPER at a glance: (a) SMARTPAPER on a Tablet PC, (b) a scene created using SMARTPAPER.could be sketched. Thirdly, it allows a user to sketch di-rectly on a 3D model in addition to making a normal 2Dsketch. Fourthly, SMARTPAPER provides a feedback sys-tem that allows a user to examine the interpretation made andprovide hints accordingly to improve its performance, lead-ing to greater user satisfaction. In addition to sketch recog-nition, SMARTPAPER offers a compendium of Computa-tional Solid Geometry (CSG) operations, synergistically re-sulting in a practical proof-of-concept system. Also, it em-ploys non-photorealistic rendering techniques to give the re-constructed objects a sketchy look. As will become evidentin section 3, from a user interface perspective, the systemcombines seamlessly various 2D and 3D operations such 2Dsketching, sketching on 3D, 3D transformations, cutting andjoining.2. Previous WorkMost sketching systems focus on a particular class of entitiesaccording to their target application areas. Landay and My-ers [LM95] study the application of sketch recognition in2D user interface design. Electronic Paper Napkin [GD96]focusses on schematic diagrams in conceptual design, alsoin two dimensions. Teddy [IMT99] aims at design of 3Dfree-form objects using sketches. Many sketching systems[ZHH96, GD96, LS00, SL97, LS02, SL96, LS96] focus on3D rigid objects. SMARTPAPER generates 3D models ofarbitrary rigid solid objects.Since sketch recognition and interpretation is a mathe-matically insoluble problem, most sketching systems offerusers some constrained drawing environment to ease recon-struction. In the system designed by Tolba et al [TDM01]the user iteratively sketches objects on paper, scans theminto the system and aligns them on a provided "perspec-tive grid". Igarashi et al [IH01] interactively generate sug-gestions as the user sketches. Although this may result in agreater recognition accuracy as it is based on selection in-stead of direct recognition after the sketch is complete, thefrequent suggestions popping up on the screen while the useris sketching can be distracting. Also, these are rule-basedsystems and may not scale well to more general objects. Ges-tured interfaces like SKETCH [ZHH96] identify gesturesfrom the input strokes and interpret them according to a fixedset of rules. The performance of these systems and their easeof use critically depend upon their ability to design intuitiveand fewer gestures and accurately and faithfully recognizethem respectively. Some systems [IMT99, LB90] allow di-rect sketching which does offer more flexibility to the userwithout having to learn many gestures. SMARTPAPER sup-ports both direct and gestured sketching and reconstructs 3Dobjects from sketchy inputs.Lipson et al