1Fall 2007 CAP 5937 – Topics in Pen-Based User Interfaces ©Joseph J. LaViola Jr.Pen-Based Gestural User InterfacesLecture #6: GesturesJoseph J. LaViola Jr.Fall 2007Fall 2007 CAP 5937 – Topics in Pen-based User Interfaces ©Joseph J. LaViola Jr.What is a Pen Gesture? Simple ink stroke or strokes to convey an idea fast to perform easy to remember Typically disappear after they are recognized  Supports in-band interaction2Fall 2007 CAP 5937 – Topics in Pen-based User Interfaces ©Joseph J. LaViola Jr.Gesture Types Single stroke Multi-stroke compound gestures punctuated gestures Trade-off in recognition between single and multiple stroke gestures Used in  modeling command languages invoking interface widgetsSingle stroke gestureMulti-stroke gestureFall 2007 CAP 5937 – Topics in Pen-based User Interfaces ©Joseph J. LaViola Jr.Gestures in Modeling Used in 2D/3D object modeling Distinction between sketch-based modeling and gestures in modeling Used to create geometry manipulate geometry guidance for computational algorithmswww-ui.is.s.u-tokyo.ac.jp/~takeo/research/teddy/teddy.htm3Fall 2007 CAP 5937 – Topics in Pen-based User Interfaces ©Joseph J. LaViola Jr.SKETCH Seminal work by Zeleznik et al. (1996) Conceptual modeling Uses simple lines and curves to build geometric primitives cubes, cylinders. pyramids, etc... No machine learning-based recognition used simple FSA Does make use of modifier keysFall 2007 CAP 5937 – Topics in Pen-based User Interfaces ©Joseph J. LaViola Jr.SKETCH Gesture Set (1)Dragging objectsScaling objectsCopying objects4Fall 2007 CAP 5937 – Topics in Pen-based User Interfaces ©Joseph J. LaViola Jr.SKETCH Gesture Set (2)Fall 2007 CAP 5937 – Topics in Pen-based User Interfaces ©Joseph J. LaViola Jr.Teddy Seminal work by Igarashi et al. (1999) did for organic modeling what Zeleznik et al. did for primitive-based modeling Supports  Object creation Cutting Extrusion Smoothing No machine learning used Simple FSA and geometric construction techniquesSmooth Teddy5Fall 2007 CAP 5937 – Topics in Pen-based User Interfaces ©Joseph J. LaViola Jr.Surface/Mesh Editing Fine line between sketching and gestures Uses simple gesture as input to a surface editing algorithm This type of approach has been used for image processing as well see work of SalesinNealen et al. (2005)Fall 2007 CAP 5937 – Topics in Pen-based User Interfaces ©Joseph J. LaViola Jr.Gestures as Command Languages Gestural commands replace traditional WIMP user interfaces also used to invoke interface widgets Notion of in-band gestures invoking commands and operations at the location of interaction contrasts with having to move to top/side of the screen to press a button or find a menu item Used in entering text text editing note taking mathematical apps etc…6Fall 2007 CAP 5937 – Topics in Pen-based User Interfaces ©Joseph J. LaViola Jr.Graffiti Language for entering text Maps to keyboard Used with Palm Pilot Single stroke language Has prefix for some symbols Takes a while to learnFall 2007 CAP 5937 – Topics in Pen-based User Interfaces ©Joseph J. LaViola Jr.Text Editing Example of a gesture set taken from real world and developed for pen computers Natural connection between pencil and paper and computerwww.jumpingminds.com7Fall 2007 CAP 5937 – Topics in Pen-based User Interfaces ©Joseph J. LaViola Jr.MathPad2 Simple gesture set for invoking operations manipulating ink Uses notion of punctuated gestures multi-stroke (gesture + punctuation) makes use of context Why? reduce number of gestures overload appropriate gestures reduce conflictsFall 2007 CAP 5937 – Topics in Pen-based User Interfaces ©Joseph J. LaViola Jr.MathPad2Gesture Set (1)8Fall 2007 CAP 5937 – Topics in Pen-based User Interfaces ©Joseph J. LaViola Jr.MathPad2Gesture Set (2)Fall 2007 CAP 5937 – Topics in Pen-based User Interfaces ©Joseph J. LaViola Jr.Fluid Inking (Zeleznik and Miller 2006) Approach to augment free-form inking with gestures (collection of techniques) Guidelines hardware impartiality (no buttons) performability (minimal targeting) extensibility discoverability Uses  terminal punctuation flicksGestures in top row, regular ink in bottom row.Flick, mnemonic flick, ink9Fall 2007 CAP 5937 – Topics in Pen-based User Interfaces ©Joseph J. LaViola Jr.Recognizing Gestures FSA’s and simple primitive operators conditionals and saving state from one event trigger to another Operators can be features same features used in machine learning! features must be excellent discriminators Machine learning techniques SVMs, K-nearest neighbor, AdaBoost more on this soon!Fall 2007 CAP 5937 – Topics in Pen-based User Interfaces ©Joseph J. LaViola Jr.Anatomy of a GestureDetecting and equal signNote that as the gesture set increases the more tests you typically have to employ to avoid conflicts.10Fall 2007 CAP 5937 – Topics in Pen-based User Interfaces ©Joseph J. LaViola Jr.Learning Gestures How many gestures is too many? Learning strategies Simple tutorials/manuals Gesture practice tools color coding (useful for multi-stroke gestures) Showing gestures through animations Techniques not proven – open research areaMusic NotePad gestural tutorial systemFall 2007 CAP 5937 – Topics in Pen-based User Interfaces ©Joseph J. LaViola Jr.Readings Zeleznik, R., K. Herndon, and J. Hughes. SKETCH: An Interface for Sketching 3D Scenes. Proceedings of SIGGRAPH'96, ACM Press, 163-170, 1996.  Igarashi, T., S. Matsuoka, and H. Tanaka. Teddy: A Sketching Interface for 3D Freeform Design. Proceedings of SIGGRAPH'99, ACM Press, 409-416, 1999.  Robert Zeleznik and Timothy Miller. Fluid Inking: Augmenting the Medium of Free-Form Inking with Gestures. In Graphics Interface. Canadian Human-Computer Communications Society, p 155-162, June