Character Animation from 2D Pictures and 3D Motion DataALEXANDER HORNUNG, ELLEN DEKKERS, and LEIF KOBBELTRWTH-Aachen UniversityThis article presents a new method to animate photos of 2D characters using 3D motion capture data. Given a single image of a person or essentially human-likesubject, our method transfers the motion of a 3D skeleton onto the subject’s 2D shape in image space, generating the impression of a realistic movement.We present robust solutions to reconstruct a projective camera model and a 3D model pose which matches best to the given 2D image. Depending on thereconstructed view, a 2D shape template is selected which enables the proper handling of occlusions. After fitting the template to the character in the inputimage, it is deformed as-rigid-as-possible by taking the projected 3D motion data into account. Unlike previous work, our method thereby correctly handlesprojective shape distortion. It works for images from arbitrary views and requires only a small amount of user interaction. We present animations of a diverseset of human (and nonhuman) characters with different types of motions, such as walking, jumping, or dancing.Categories and Subject Descriptors: I.3.3 [Computer Graphics]: Picture/Image Generation;I.4.9 [Image Processing and Computer Vision]: ApplicationsGeneral Terms: AlgorithmsAdditional Key Words and Phrases: 2D character animation, 3D motion data, camera and model pose determination, as-rigid-as-possible shape manipulationwith perspective correctionACM Reference Format:Hornung, A., Dekkers, E., and Kobbelt, L. 2007. Character animation from 2D pictures and 3D motion data. ACM Trans. Graph. 26, 1, Article 1 (January2007), 9 pages. DOI = 10.1145/1186644.1186645 http://doi.acm.org/10.1145/1186644.11866451. INTRODUCTIONIn recent years, research on 2D image manipulation has receiveda huge amount of interest. Very powerful solutions for problemssuch as matting [Sun et al. 2004], image completion [Drori et al.2003], texture synthesis [Efros and Leung 1999], and rigid imagemanipulation [Igarashi et al. 2005] have been presented. Based onthese and similar methods, it has now become possible to exploreinteresting new ideas to reanimate still pictures, for example, as doneby Chuang et al. [2005] in their article on animating pictures usingstochastic motion textures. They animate passive elements, suchas water and trees, that are subject to natural forces like wind. Inthis article, we want to take the idea of creating animations directlyin image space one step further by making photographed personsmove.One possible approach to address this problem would be the re-construction of a textured 3D model, and to animate this modelusing classical animation techniques. This, however, would requirecomplex, fully textured 3D models which have to be created andadapted per image. In particular, for highly detailed characters suchas the Scarecrow example shown in Figures 2 and 7, the requiredmanual model-adaption process would be impractical. Moreover, itwould be necessary to apply very sophisticated 3D rendering tech-niques to realistically embed the 3D model into the 2D image soThe CMU motion database used in this work was created with funding from NSF EIA-0196217.Authors’ address: A. Hornung, E. Dekkers, and L. Kobbelt, Computer Graphics Group, RWTH-Aachen University, Lehrstuhl f¨ur Informatik 8, RWTH-Aachen,Ahornstraße 55, 52074 Aachen, Germany; email: {hornung, dekkers, kobbelt}@cs.rwth-aachen.de.Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not madeordistributed for profit or direct commercial advantage and that copies show this notice on the first page or initial screen of a display along with the full citation.Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, to republish,topost on servers, to redistribute to lists, or to use any component of this work in other works requires prior specific permission and/or a fee. Permissions may berequested from Publications Dept., ACM, Inc., 2 Penn Plaza, Suite 701, New York, NY 10121-0701 USA, fax +1 (212) 869-0481, or [email protected] 2007 ACM 0730-0301/2007/01-ART1 $5.00 DOI 10.1145/1186644.1186645 http://doi.acm.org/ 10.1145/1186644.1186645as to preserve the photorealism or style of the original input image.By contrast, even simple 2D morphing and blending often leads tomore convincing results than using sophisticated 3D reconstructionand rendering. For instance, methods such as Poisson-matting or2D image completion allow for a smooth and realistic combinationof different image contents, which is much harder to achieve whentrying to augment 2D images with 3D models.Hence, we present a purely image-based approach to combinerealistic images with realistic 3D motion data in order to gener-ate visually convincing animations of 2D characters. This is moti-vated by the intention to preserve the realism or style of the originalimage data, without losing quality due to intermediate conversionsteps into a 3D representation. Several recent articles [Barrett andCheney 2002; Igarashi et al. 2005; Chuang et al. 2005] have shownthe promising potential of such solely image-based animation ap-proaches.The contribution of this article is a method to generate anima-tions of photographed or painted 2D characters based on 3D mo-tion data. For arbitrary input images, our method robustly recon-structs the camera and 3D model pose corresponding to the depictedsubject. Using generic shape templates, the character is decom-posed into animation layers. Occluded regions and the backgroundare reconstructed by texture synthesis. We show how the resultingcharacter-shape can be animated using an augmented version of theACM Transactions on Graphics, Vol. 26, No. 1, Article 1, Publication date: January 2007.2•A. Hornung et al.Fig. 1. Given only a single input image (left), our method generates 2Danimations from 3D motion data (right).as-rigid-as-possible shape manipulation technique [Igarashi et al.2005], which correctly handles projective distortion effects such asforeshortening. In combination, these techniques enable us to realis-tically change the pose of a character or create animations from sin-gle input images of arbitrary human and nonhuman subjects basedon 3D motion data.2. RELATED WORKOur work is inspired by a diverse set of interesting