55 Copyright 1995 Psychonomic Society, Inc.This research was conducted in partial fulfillment of the require-ments for the PhD in the Department of Brain and Cognitive Sciencesat the Massachusetts Institute of Technology. Financial support at MITwas provided by the James R. Killian Fellowship sponsored by theJames and Lynelle Holden Fund, a Fellowship from the WhitakerHealth Sciences Fund, and an NSF Graduate Fellowship. In addition,parts of this research were funded under NSF Grant BNS 8518774 toSteven Pinker, Professor of Brain and Cognitive Sciences at MIT, anda grant from the Sloan Foundation to the MIT Center for CognitiveScience. Final revisions were completed during a visit to the Max-Planck-Institut für biologische Kybernetik in Tübingen. Jigna Desaiand Carmita Signes ran many of the subjects and were stellar researchassistants. Many thanks to Kyle Cave, Jacob Feldman, Paul Bloom,Larry Maloney, William Hayward, Pierre Jolicoeur, James Pomerantz,two anonymous reviewers, and in particular to my thesis committeemembers, Irving Biederman, Ellen Hildreth, and David Irwin for theirtime, insights, and support. Michael Kubovy provided useful com-ments on an earlier version. Special thanks to Steve Pinker and toLaurie Heller. Requests for reprints may be sent to M. J. Tarr, Depart-ment of Psychology, Yale University, P.O. Box 208205, New Haven,CT 06520-8205 (e-mail: [email protected]).Rotating objects to recognize them: A case studyon the role of viewpoint dependency in therecognition of three-dimensional objectsMICHAEL J. TARRYale University, New Haven, ConnecticutSuccessful object recognition is essential for finding food, identifying kin, and avoiding danger, aswell as many other adaptive behaviors. To accomplish this feat, the visual system must reconstruct3-D interpretations from 2-D “snapshots” falling on the retina. Theories of recognition address thisprocess by focusing on the question of how object representations are encoded with respect to view-point. Although empirical evidence has been equivocal on this question, a growing body of surpris-ing results, including those obtained in the experiments presented in this case study, indicates thatrecognition is often viewpoint dependent. Such findings reveal a prominent role for viewpoint-dependent mechanisms and provide support for the multiple-views approach, in which objects areencoded as a set of view-specific representations that are matched to percepts using normalizationprocedures.Just as you could not fully reconstruct a house from pho-tos all taken from a single vantage point, “snapshots” atmany angles must be combined to reconstruct a Burgessorganism. Conway Morris told me that he managed to re-construct the curious Wiwaxia—an animal with no mod-ern relatives, and therefore no known prototype to use asa model—by passing countless hours “rotating the damnedthing in my mind” from the position of one drawing to thedifferent angle of another, until every specimen could bemoved without contradiction from one stance to the next.Then he finally knew that nothing major was missing orout of place.—Stephen J. Gould, Wonderful Life (1989)The human ability to recognize objects is remark-able—under all but the most degraded conditions, wesucceed in understanding the perceptual world aroundus. This performance is even more astounding when oneconsiders that we encounter 3-D objects from an infinitenumber of viewpoints producing potentially unique 2-Dprojections on the retina. The problem, as expressed byGould (1989) with reference to the paleontologist’s re-construction of 3-D organisms, is that one must find“guides to the three dimensional reanimation of squashedand distorted fossils”—no less than fossils, 2-D imagesmust be “reanimated.”To achieve shape constancy (object perception re-gardless of position, scale, or viewpoint), retinal imagesmust not only be organized into coherent percepts, butmust be compared with long-term representations of ob-jects previously seen. The format of these representa-tions may take many forms. For instance, objects mightbe encoded as spatial representations, descriptive lists offeatures, or Fourier decompositions of component sine-wave gratings. A growing body of research supports theconcept of spatial representations, although the precisemechanisms used to match these to percepts are not yetwell understood. Current theories of object recognitionhave varied widely, arguing for recognition by the loca-tion of small sets of unique features (Corballis, 1988),for recognition by the alignment of 2-D input shapeswith 3-D models (Ullman, 1989), for recognition by nor-malizing 2-D input shapes to viewpoint-specific 2-Dviews (Bülthoff & Edelman, 1992), or for recognition bycomparing recovered qualitative descriptions of 3-Dparts and their spatial relations (Biederman, 1987; Hum-mel & Biederman, 1992). Recognition may also rely ontexture, shading, color, or motion. It is almost certainthat many of these possibilities coexist as mechanisms invisual perception; however, because the latter attributesare often considered precursors to the recovery of shapePsychonomic Bulletin & Review1995, 2 (1), 55–8256 TARR(Horn & Brooks, 1989), theories of recognition have fo-cused on shape-based representations.Families of Recognition TheoriesTheories of object recognition may be characterizedin terms of four issues: (1) the coordinate system orframe of reference (viewpoint dependent or viewpointinvariant); (2) the nature of component features (e.g.,surfaces, segmented contours, or parts); (3) the encodingof relations between features (metrically specific orqualitative); and (4) the number of spatial dimensions(2-D, 2-D plus depth, or 3-D). These issues are theoret-ically independent, but in practice, they tend to clusterinto one of several families. Consequently, tests of a sin-gle issue—for example, the frame of reference used inthe representation—are often taken as generally dis-tinguishing between families. Indeed, approaches toshape constancy may be divided roughly into completeviewpoint-invariant theories, restricted viewpoint-invariant theories, and viewpoint-dependent theories.Complete viewpoint-invariant theories. This fam-ily includes both recognition by orientation-free uniquefeatures (Corballis, 1988; Jolicoeur, 1990a) and object-centered theories in which an object is described withinthe same coordinate system regardless of its orientation,size, or location (Marr &