To appear, FirstInternational Symposium on Non-Photorealistic Animation and Rendering, June5-7, 2000Painterly Rendering for Video and InteractionAaron Hertzmann Ken Perlin Media Research LaboratoryDepartment of Computer ScienceNew York UniversityAbstractWe present new methods for painterly video processing.Based on our earlier still image processing technique, we“paint over” successive frames of animation, applying paintonly in regions where the source video is changing. Imageregions with minimal changes, such as due to video noise,are also left alone, using a simple difference masking tech-nique. Optionally, brush strokes may be warped betweenframes using computed or procedural optical flow.These methods produce video with a novel visual styledistinct from previously demonstrated algorithms. Withoutoptical flow, the video gives the effect of a painting that hasbeen repeatedly updated and photographed, similar to paint-on-glass animation. We feel that this gives a subjective im-pression of the work of a human hand. With optical flow, thepainting surface flows and deforms to follow the shape of theworld.We have constructed an interactive painting exhibit, inwhich a painting is continually updated. Viewers have foundthis to be a compelling experience, suggesting the promiseof non-photorealistic rendering for creating compelling in-teractive visual experiences.CR Categories: I.3.3 [Picture/Image Generation]: Com-puter Graphics—Display Algorithms; J.5 [Computer Appli-cations]: Arts and Humanities—Fine ArtsKeywords: Non-photorealistic rendering, painterly ren-dering, video processing, animation WWW: http://www.mrl.nyu.edu/fhertzmann,perlingFigure 1: A viewer interacting with a “living” painting.1 IntroductionGraphics researchers have demonstrated many tech-niques for producing dazzling and expressive still non-photorealistic images, allowing users to quickly createimages that would previously require the labors of a skilledartist. Of course, these systems do not replace artists for thesame reasons that cameras did not.This paper builds on the still image research to address theproblem of producing non-photorealistic animations and in-teractions. Here, instead of making a task faster and easier,we are focused on making a task possible. Some of the inno-vations required for non-photorealistic interaction are purelyalgorithmic (e.g. silhouette detection [13, 5]). In other cases,technological problems are tightly coupled to artistic prob-lems (e.g. [2]). In particular, we are concerned with thequestion: what can a painterly animation look like? This pa-per presents several new tools that can be used in exploringthis design space. Similar explorations have occurred in ex-perimental animation, although such work is limited by theextreme amount of effort required to hand paint each frame.Previous research can be categorized as either generatingrepresentationsof a painted world, or painted representationsTo appear, FirstInternational Symposium on Non-Photorealistic Animation and Rendering, June5-7, 2000of a world. The first of these approaches attaches stroke po-sitions and sizes over time to 3D geometry [14, 12, 4]. Thisusually gives the appearance of a “Painted World,” i.e. aworld inhabited by brush strokes. In contrast, Litwinowicz[11] and Curtis [2] produce painted representations of theworld, by allowing strokes to detach from geometry, in bothspace and time. We continue to explore this latter line ofresearch.Our goal in this research is to make tools to allow artists tocreate painterly animations and interfaces. For many appli-cations, non-photorealisticanimations and interfacesprovidedistinct advantages: visual appeal, the visual and emotionalexpression of physical media (e.g. brush strokes), informaland compelling interfaces, reduced requirementsfor geomet-ric modeling (when processing 3D), and cultural references(e.g. animating existing paintings and styles).1.1 Related WorkWe now give a brief overview of research directly relatedto this paper; see [3] for a more thorough survey of non-photorealistic rendering. Haeberli [6] described automaticand semi-automatic painterly rendering algorithms, exten-sions of which are now commonly used in commercial desk-top publishing software. Meier [14] attaches particles to 3Dobjects by an automatic procedure, and places brush strokesto coincide with the particles; the Deep Canvas system [4]allows a skilled artist to paint curved strokes onto a model,and then animate the painting. Litwinowicz [11] uses opti-cal flow to push short brush strokes along scene movements,and provides various tools [12] for editing/correcting flowand layering. Hertzmann [7] automatically paints still im-ages with various stroke sizes and shapes; in this paper, weextend this method to processing video.Previous work has applied interactive non-photorealisticrendering to 3D modeling [18, 8], technical illustration[13, 5, 15], education and entertainment [9]. The methodsin this paper are aimed at the last of these goals: buildingtools that will allow artists to create compelling, enjoyable,and expressive video and interfaces.1.2 OverviewThis paper is organized as follows. We begin by reviewingour still image processing system, and show how this systemis extended to video. We then describe difference maskingtechniques for reducing flickering in video, and the use ofoptical flow to push strokes so as to follow the scene motion.Several video experiments and an interactive exhibit are de-scribed, followed by discussion and future work.2 Still ImagesIn this section, we review a revised version of our still imageprocessing algorithm; a detailed description can be found in[7]. This version uses a summed-area table [1] to quicklycompute blurred images. Furthermore, the paint functiontakes an extra parameter, which allows us to choose whetherto force the first paint layer to cover the canvas.The algorithm paints a rough sketch of the image withlarge brush strokes, and then refines it with smaller brushstrokes, but only in regions where the painting does notclosely match the source image (e.g. where there is fine de-tail in the source).function paint(Is, // source imageIp, // canvas; initially blank for still imagesR1:::Rn, // brush sizesfirstFrame) // boolean; true for still imagesCreate a summed-area tableAfromIsrefresh firstFrameforeach brush sizeRi, from largest to smallest, doUseAto compute a blurred reference imageIRigrid fgRiClear depth bufferforeach