International Journal of Computer Vision 56(3), 151–177, 2004c 2004 Kluwer Academic Publishers. Manufactured in The Netherlands.Object Detection Using the Statistics of PartsHENRY SCHNEIDERMAN AND TAKEO KANADE∗Robotics Institute, Carnegie Mellon University, Pittsburgh, PA 15213, [email protected]@cs.cmu.eduReceived April 20, 2001; Revised February 15, 2002; Accepted March 6, 2002Abstract. In this paper we describe a trainable object detector and its instantiations for detecting faces and carsat any size, location, and pose. To cope with variation in object orientation, the detector uses multiple classifiers,each spanning a different range of orientation. Each of these classifiers determines whether the object is present ata specified size within a fixed-size image window. To find the object at any location and size, these classifiers scanthe image exhaustively.Each classifier is based on the statistics of localized parts. Each part is a transform from a subset of waveletcoefficients to a discrete set of values. Such parts are designed to capture various combinations of locality in space,frequency, and orientation. In building each classifier, we gathered the class-conditional statistics of these part valuesfrom representative samples of object and non-object images. We trained each classifier to minimize classificationerror on the training set by using Adaboost with Confidence-Weighted Predictions (Shapire and Singer, 1999). Indetection, each classifier computes the part values within the image window and looks up their associated class-conditional probabilities. The classifier then makes a decision by applying a likelihood ratio test. For efficiency, theclassifier evaluates this likelihood ratio in stages. At each stage, the classifier compares the partial likelihood ratio to athreshold and makes a decision about whether to cease evaluation—labeling the input as non-object—or to continuefurther evaluation. The detector orders these stages of evaluation from a low-resolution to a high-resolution searchof the image. Our trainable object detector achieves reliable and efficient detection of human faces and passengercars with out-of-plane rotation.Keywords: object recognition, object detection, face detection, car detection, pattern recognition, machine learn-ing, statistics, computer vision, wavelets, classification1. IntroductionObject detection is a big part of people’s lives. We, ashuman beings, constantly “detect” various objects suchas people, buildings, and automobiles. Yet it remainsa mystery how we detect objects accurately and withlittle apparent effort. Comprehensive explanations havedefied psychologists and physiologists for more than acentury.∗This work was supported in part by the Advanced Research andDevelopment Activity (ARDA) under contract number MDA904-00-C-2109.Our goal in this research is not to understand howhumans perceive, but to create computer methods forautomatic object detection. Automated object detec-tion has many potential uses including image retrieval.Digital image collections have grown dramatically inrecent years. Corbis estimates it has more than 67 mil-lion images in its collection. The Associated Press col-lects and archives an estimated 1,000 photographs aday. Currently, the usability of these collections is lim-ited by a lack of effective retrieval methods. To find aspecific image in such a collection, people must searchusing text-based captions and primitive image featuressuch as color and texture. Automatic object detection152 Schneiderman and Kanadecould be used to extract more information from theseimages and help label and categorize them. Improvedsearch methods will make these databases accessibleto wider groups of users, such as law enforcementagencies, medical practitioners, graphic and multime-dia designers, and artists. Automatic object detectioncould also be useful in photography. As camera tech-nology changes from film to digital capture, cameraswill become part optics and part computer. Such a cam-era could automatically focus, color balance, and zoomon a specified object of interest, say, a human face. Also,detectors of a specific object have specialized uses: facedetectors for face identification and car detectors formonitoring traffic.1.1. Challenges in Object DetectionAutomatic object detection is a difficult undertaking. Inover30years of research in computer vision, progresshas been limited. The main challenge is the amountof variation in visual appearance. An object detectormust cope with both the variation within the objectcategory and with the diversity of visual imagery thatFigure 1. Multiple classifiers are built to deal with appearance changes due to pose. (a) For faces, classifiers are trained on 2 viewpoints and(b) for cars, classifiers are trained on 8 viewpoints.exists in the world at large. For example, cars vary insize, shape, coloring, and in small details such as theheadlights, grille, and tires. The lighting, surroundingscenery, and an object’s pose affect its appearance. Acar detection algorithm must also distinguish cars fromall other visual patterns that may occur in the world,such as similar looking rectangular objects.1.2. Object Detection Using ClassifiersOur method for object detection factors out variation inthe pose of the object. Our object detector uses a set ofclassifiers, each of which determines whether the objectis present at a specific pose in a fixed-size rectangularimage window. For faces, the detector uses classifiersfor three discrete poses: front, left profile, and rightprofile. Taking advantage of facial symmetry, we onlyneeded to train classifiers for the frontal and right pro-file viewpoints shown in Fig. 1(a), and we built a leftprofile detector by reflecting the right profile detector.For cars, we use 15 discrete viewpoints, and by exploit-ing symmetry again, we only trained classifiers for theeight viewpoints as shown in Fig. 1(b). These classifiersObject Detection Using the Statistics of Parts 153Figure 2. Detection by scanning classifier across image in both (a) position and (b) scale.tolerate a small range of variation in object orientation,size, and alignment within the image window.To perform detection, we scan each classifier over theoriginal image and a series of resized versions of theoriginal image, as illustrated in Fig. 2, where the rect-angular blocks indicate successive applications of theclassifier. This exhaustive scanning operation makes