3D FACE PROCESSING Modeling, Analysis and Synthesis THE KLUWER INTERNATIONAL SERIES IN VIDEO COMPUTING Series Editor Mubarak Shah, Ph.D. University of Central Florida Orlando, USA Other books in the series: EXPLORATION OF VISUAL DATA Xiang Sean Zhou, Yong Rui, Thomas S. Huang; ISBN: 1-4020-7569-3 VIDEO MINING Edited by Azriel Rosenfeld, David Doermann, Daniel DeMenthon;ISBN: 1-4020-7549-9 VIDEO REGISTRATION Edited by Mubarah Shah, Rakesh Kumar; ISBN: 1-4020-7460-3 MEDIA COMPUTING: COMPUTATIONAL MEDIA AESTHETICS Chitra Dorai and Svetha Venkatesh; ISBN: 1-4020-7102-7 ANALYZING VIDEO SEQUENCES OF MULTIPLE HUMANS: Tracking, Posture Estimation and Behavior Recognition Jun Ohya, Akita Utsumi, and Junji Yanato; ISBN: 1-4020-7021-7 VISUAL EVENT DETECTION Niels Haering and Niels da Vitoria Lobo; ISBN: 0-7923-7436-3 FACE DETECTION AND GESTURE RECOGNITION FOR HUMAN-COMPUTER INTERACTION Ming-Hsuan Yang and Narendra Ahuja; ISBN: 0-7923-7409-6 3D FACE PROCESSING Modeling, Analysis and Synthesis Zhen Wen University of Illinois at Urbana-Champaign Urbana, IL, U.S.A. Thomas S. Huang University of Illinois at Urbana-Champaign Urbana, IL, U.S.A. KLUWER ACADEMIC PUBLISHERS NEW YORK, BOSTON, DORDRECHT, LONDON, MOSCOW eBook ISBN: 1-4020-8048-4 Print ISBN: 1-4020-8047-6 Print © 2004 Kluwer Academic Publishers All rights reserved No part of this eBook may be reproduced or transmitted in any form or by any means, electronic, mechanical, recording, or otherwise, without written consent from the Publisher Created in the United States of America Boston ©2004 Springer Science + Business Media, Inc. Visit Springer's eBookstore at: http://www.ebooks.kluweronline.com and the Springer Global Website Online at: http://www.springeronline.com Contents List of Figures List of Tables Preface Acknowledgments xi xv xvii xix 1. INTRODUCTION 1 2 Motivation Research Topics Overview 2.1 2.2 2.3 2.4 2.5 3D face processing framework overview 3D face geometry modeling Geometric-based facial motion modeling, analysis and synthesis Enhanced facial motion analysis and synthesis using flexible appearance model Applications of face processing framework 3 Book Organization 1 2 2 2 4 5 7 8 9 11 11 12 12 13 14 14 15 17 19 2. 3D FACE MODELING 1 State of the Art 1.1 1.2 1.3 Face modeling using 3D range scanner Face modeling using 2D images Summary 2 Face Modeling Tools in iFACE 2.1 2.2 Generic face model Personalized face model 3 Future Research Direction of 3D Face Modeling 3. LEARNING GEOMETRIC 3D FACIAL MOTION MODEL vi 3D FACE PROCESSING: MODELING, ANALYSIS AND SYNTHESIS 1 Previous Work 1.1 1.2 1.3 Facial deformation modeling Facial temporal deformation modeling Machine learning for facial deformation modeling 2 3 4 5 6 7 Motion Capture Database Learning Holistic Linear Subspace Learning Parts-based Linear Subspace Animate Arbitrary Mesh Using MU Temporal Facial Motion Model Summary 4. GEOMETRIC MODEL-BASED 3D FACE TRACKING 1 Previous Work 1.1 Parameterized geometric models 1.1.1 1.1.2 1.1.3 1.1.4 1.2 1.3 1.3.1 1.3.2 B-Spline curves Snake model Deformable template 3D parameterized model FACS-based models Statistical models Active Shape Model (ASM) and Active Appearance Model (AAM) 3D model learned from motion capture data 2 3 4 Geometric MU-based 3D Face Tracking Applications of Geometric 3D Face Tracking Summary 5. GEOMETRIC FACIAL MOTION SYNTHESIS 1 Previous Work 1.1 1.2 1.3 Performance-driven face animation Text-driven face animation Speech-driven face animation 2 3 4 5 Facial Motion Trajectory Synthesize Text-driven Face Animation Offline Speech-driven Face Animation Real-time Speech-driven Face Animation 5.1 Formant features for speech-driven face animation 5.1.1 Formant analysis 19 19 20 21 22 23 24 27 29 30 31 31 32 32 32 33 33 33 34 34 34 35 37 38 41 41 41 42 42 44 46 47 48 49 49 Contents vii 5.1.2 An efficient real-time speech-driven animation system based on formant analysis 5.2 ANN-based real-time speech-driven face animation 5.2.1 5.2.2 5.2.3 5.2.4 Training data and features extraction Audio-to-visual mapping Animation result Human emotion perception study 6 Summary 6. FLEXIBLE APPEARANCE MODEL 1 Previous Work 1.1 1.2 1.3 Appearance-based facial motion modeling, analysis and synthesis Hybrid facial motion modeling, analysis and synthesis Issues in flexible appearance model 1.3.1 1.3.2 1.3.3 Illumination effects of face appearance Person dependency Online appearance model 2 Flexible Appearance Model 2.1 Reduce illumination dependency based on illumination modeling 2.1.1 2.1.2 2.1.3 Radiance environment map (REM) Approximating a radiance environment map using spherical harmonics Approximating a radiance environment map from a single image 2.2 Reduce person dependency based on ratio-image 2.2.1 2.2.2 2.2.3 Ratio image Transfer motion details using ratio image Transfer illumination using ratio image 3 Summary 7. FACIAL MOTION ANALYSIS USING FLEXIBLE APPEARANCE MODEL 1 Model-based 3D Face Motion Analysis Using Both Geometry and Appearance 1.1 1.2 1.3 1.4 Feature extraction Influences of lighting Exemplar-based texture analysis Online EM-based adaptation 50 52 53 53 55 56 59 61 62 62 62 63 63 66 66 67 67 67 68 70 71 71 71 72 73 75 75 77 79 79 80 viii 3D FACE PROCESSING: MODELING, ANALYSIS AND SYNTHESIS 2 3 Experimental Results Summary 8. FACE APPEARANCE SYNTHESIS USING FLEXIBLE APPEARANCE MODEL 1 Neutral Face Relighting 1.1 Relighting with radiance environment maps 1.1.1 1.1.2 1.1.3 1.1.4 Relighting when rotating in the same lighting condition Comparison with inverse rendering approach Relighting in different lighting conditions Interactive face relighting 1.2 Face relighting from a single image 1.2.1 Dynamic range of images 1.3 1.4 Implementation Relighting results 2 3 Face Relighting For Face Recognition in Varying Lighting Synthesize Appearance Details of Facial Motion 3.1 3.2 Appearance of mouth interior Linear alpha-blending of texture 4 Summary 9. APPLICATION EXAMPLES OF THE FACE PROCESSING FRAMEWORK 1 Model-based Very Low Bit-rate Face Video Coding 1.1 1.2 1.3 1.4 Introduction Model-based face video coder Results Summary and future work 2 Integrated Proactive HCI environments 2.1 2.2 2.3 Overview Current status Future work 3 Summary 10. CONCLUSION AND FUTURE WORK 1 2 Conclusion Future Work 2.1 2.2 2.3 Improve geometric face processing Closer correlation between geometry and appearance Human perception evaluation of synthesis 83 87 91 91 92 92 93 93 94 94 95 96 97 100 103 103 104 105 107 107 107 108 109 110 110 111 112 113 113 115 115 116 116 116 117 Contents ix 2.3.1 2.3.2 Previous work Our ongoing and future work Appendices Projection of face images in 9-D spherical harmonic space References Index 117 120 123 125 137 List of Figures 1.1 1.2 2.1 2.2 2.3 2.4 2.5 3.1 3.2 3.3 3.4 3.5 3.6 Research issues and applications of face processing. A unified 3D face processing framework. The generic face model. (a): Shown as wire-frame model. (b): Shown as shaded model. An example of range scanner data. (a): Range map. (b): Texture map. Feature points defined on texture map. The model editor. An example of customized face models. An example of marker layout for MotionAnalysis sys- tem. The markers of the Microsoft data [Guenter et al., 1998]. (a): The markers are shown as small white dots. (b) and (c): The mesh is shown in two different viewpoints. The neutral face and deformed face corresponding to the first four MUs. The top row is frontal view and the bottom row is side view. (a): NMF learned parts overlayed on the generic face model. (b): The facial muscle distribution. (c): The aligned facial muscle distribution. (d): The parts over- layed on muscle distribution. (e): The final parts de- composition. Three lower lips shapes deformed by three of the lower lips parts-based MUs respectively. The top row is the frontal view and the bottom row is the side view. (a): The neutral face side view. (b): The face deformed by one right cheek parts-based MU. 3 4 14 15 15 16 16 22 23 24 25 26 26 [...]... aspects of 3D face processing, although all these areas are still subject of active research This book introduces the frontiers of 3D face processing techniques It reviews existing 3D face processing techniques, including techniques for 3D face geometry modeling, 3D face motion modeling, 3D face motion tracking and animation Then it discusses a unified framework for face modeling, analysis and synthesis. .. model-based 3D face tracking, and 3D face synthesis such as text- and speech-driven face animation 2.3 Geometric-based facial motion modeling, analysis and synthesis Accurate face motion analysis and realistic face animation demands good model of the temporal and spatial facial deformation One type of approaches use geometric-based models [Black and Yacoob, 1995, DeCarlo and Metaxas, 2000, Essa and Pentland,... we discuss the motivation for 3D face processing research and then give overviews of our 3D face processing research 2 1 3D FACE PROCESSING: MODELING, ANALYSIS AND SYNTHESIS Motivation Human face provides important visual cues for effective face- to -face humanhuman communication In human-computer interaction (HCI) and distant human-human interaction, computer can use face processing techniques to estimate... computer vision and computer graphics A 3D face model lays basis for modelbased face video analysis and facial animations In face video analysis, a 3D face model helps recognition of oblique views of faces [Blanz et al., 2002] Based on the 3D geometric model of faces, facial deformation models can be constructed for 3D non-rigid face tracking [DeCarlo, 1998, Tao, 1999] In computer graphics, 3D face models... (shape) and appearance (texture) in face analysis and synthesis The Active Appearance Model (AAM) [Cootes et al., 1998] and its variants, apply PCA to model both the shape variations of image patches and their texture variations They have been shown to be powerful tools for face alignment, recognition, and synthesis Blanz and Vetter [Blanz and Vetter, 1999] propose 3D morphable models for 3D faces modeling,. .. introduces the motivation and background of 3D face processing research and gives an overview of our research Several research topics will be discussed in more details in the following chapters First, we describe methods and systems for modeling the geometry of static 3D face surfaces Such static models lay basis for both 3D face analysis and synthesis To study the motion of human faces, we propose motion... overview of these 3D face modeling techniques Then we will describe the tools in our iFACE system for building personalized 3D face models The iFACE system is a 3D face modeling and animation system, developed based on the 3D face processing framework It takes the CyberwareTM 3D scanner data of a subject’s head as input and provides a set of tools to allow the user to interactively fit a generic face model... Chapter 6 to enhance the framework We use efficient and effective methods to reduce the the appearance model’ s dependency on illumination and person Then, in Chapter 7 and Chapter 8 we xviii 3D FACE PROCESSING: MODELING, ANALYSIS AND SYNTHESIS present experimental results to show the effectiveness of the flexible appearance model in face analysis and synthesis In Chapter 9, we describe applications in... surface deformation, and (2) how to apply these models for facial deformation analysis and synthesis In this section, we introduce previous research on facial deformation modeling 1.1 Facial deformation modeling In the past several decades, many models have been proposed to deform 3D facial surface spatially Representative models include free-form inter- 20 3D FACE PROCESSING: MODELING, ANALYSIS AND SYNTHESIS. .. Section 3 2 Face Modeling Tools in iFACE We have developed iFACE system which provides functionalities for face modeling and face animation It provides a research platform for the 3D face processing framework The iFACE system takes the CyberwareTMscanner data of a subject’s head as input and allows the user to interactively fit a generic face model to the CyberwareTM scanner data The iFACE system also . motivation for 3D face processing research and then give overviews of our 3D face processing research. 2 3D FACE PROCESSING: MODELING, ANALYSIS AND SYNTHESIS 1. Motivation Human. the frontiers of 3D face processing techniques. It reviews existing 3D face process- ing techniques, including techniques for 3D face geometry modeling, 3D face motion