1A Survey of Facial Modeling and Animation Techniques Jun-yong Noh Integrated Media Systems Center, University of Southern California [email protected] http://csuri.usc.edu/~noh Ulrich Neumann Integrated Media Systems Center, University of Southern California [email protected] http://www.usc.edu/dept/CGIT/un.html Realistic facial animation is achieved through geometric and image manipulations. Geometric deformations usually account for the shape and deformations unique to the physiology and expressions of a person. Image manipulations model the reflectance properties of the facial skin and hair to achieve small-scale detail that is difficult to model by geometric manipulation alone. Modeling and animation methods often exhibit elements of each realm. This paper summarizes the theoretical approaches used in published work and describes their strengths, weaknesses, and relative performance. Taxonomy groups the methods into classes that highlight their similarities and differences. Categories and Subject Descriptors: General Terms: Additional Key Words and Phrases: Introduction Since the pioneering work of Frederic I. Parke [91] in 1972, many research efforts have attempted to generate realistic facial modeling and animation. The most ambitious attempts perform the modeling and rendering in real time. Because of the complexity of human facial anatomy, and our natural sensitivity to facial appearance, there is no real time system that captures subtle expressions and emotions realistically on an avatar. Although some recent work [43, 103] produces realistic results with relatively fast performance, the process for generating facial animation entails extensive human intervention or tedious tuning. The ultimate goal for research in facial modeling and animation is a system that 1) creates realistic animation, 2) operates in real time, 3) is automated as much as possible, and 4) adapts easily to individual faces. Recent interest in facial modeling and animation is spurred by the increasing appearance of virtual characters in film and video, inexpensive desktop processing power, and the potential for a new 3D immersive communication metaphor for human-computer interaction. Much of the facial modeling and animation research is published in specific venues that are relatively unknown to the general graphics community. There are few surveys or detailed historical treatments of the subject [85]. This survey is intended as an accessible reference to the range of reported facial modeling and animation techniques. Facial modeling and animation research falls into two major categories, those based on geometric manipulations and those based on image manipulations (Fig. 1). Each realm comprises several sub-categories. Geometric manipulations include key-framing and geometric interpolations [33, 86, 91], parameterizations [21, 88, 89, 90], finite element methods [6, 44, 102], muscle based modeling [70, 96, 101, 106, 107, 110, 122, 131], visual simulation using pseudo muscles [50, 71], spline models [79, 80, 125, 126, 127] and free-form deformations [24, 50]. Image manipulations include image morphing between2photographic images [10], texture manipulations [82], image blending [103], and vascular expressions [49]. At the preprocessing stage, a person-specific individual model may be constructed using anthropometry [25], scattered data interpolation [123], or by projecting target and source meshes onto spherical or cylindrical coordinates. Such individual models are often animated by feature tracking or performance driven animation [12, 35, 84, 93, 133]. Fig. 1 - Classification of facial modeling and animation methods This taxonomy in Figure 1 illustrates the diversity of approaches to facial animation. Exact classifications are complicated by the lack of exact boundaries between methods and the fact that recent approaches often integrate several methods to produce better results. The survey proceeds as follows. Section 1 and 2 introduce the interpolation techniques and parameterizations followed by the animation methods using 2D and 3D morphing techniques in section 3. The Facial Action Coding System, a frequently used facial description tool, is summarized in section 4. Physics based modeling and simulated muscle modeling are discussed in sections 5 and 6, respectively. Techniques for increased realism, including wrinkle generation, vascular expression and texture manipulation, are surveyed in sections 7, 8, and 9. Individual modeling and model fitting are described in section 10, followed by animation from tracking data in section 11. Section 12 describes mouth animation research, followed by general conclusions and observations. InterpolationAnthropometryBilinearinterpolationScattereddatainterpolationProjection ontospherical orcylindricalcoords.PhysicsbasedmusclemodelPseudomusclemodelFiniteElementMethodsTexturemanipulation andimage blending(Layered)SpringmeshPure vectorbasedmodelSplinemodelFree formdeformationWrinklegeneration facial modeling / animationGeometrymanipulationsImagemanipulationsIndividualModelConstructionParameterization ImagemorphingVascularexpressionsHairAnimationModelacquisitionand fitting31. Interpolations Interpolation techniques offer an intuitive approach to facial animation. Typically, an interpolation function specifies smooth motion between two key-frames at extreme positions, over a normalized time interval (Fig. 2). Fig. 2 - Linear interpolation is performed on muscle contraction values Linear interpolation is commonly used [103] for simplicity, but a cosine interpolation function or other variations can provide acceleration and deceleration effects at the beginning and end of an animation [129]. When four key frames are involved, rather than two, bilinear interpolation generates a greater variety of facial expressions than linear interpolation [90]. Bilinear interpolation, when combined with simultaneous image morphing, creates a wide range of realistic facial expression changes [4]. Interpolated images are generated by varying the parameters of the interpolation functions. Geometric interpolation directly updates the 2D or 3D positions of the face mesh vertices, while parameter interpolation controls functions that indirectly move the vertices. For example, Sera et al. [115] perform a linear interpolation of the spring muscle force parameters, rather than the positions of the vertices, to achieve realistic mouth animation.