Vol. 11, No. 11/November 1994/J. Opt. Soc. Am. A 3079Photometric stereo under a light source witharbitrary motionHideki Hayakawa*ATR Human Information Processing Research Laboratories, 2-2 Hikaridai, Seika-cho, Soraku-gun, Kyoto 619-02, JapanReceived September 27, 1993; revised manuscript received March 28, 1994; accepted April 6, 1994A new photometric-stereo method for estimating the surface normal and the surface reflectance of objectswithout a priori knowledge of the light-source direction or the light-source intensity is proposed. First, Iconstruct a p X f image data matrix I from p pixel image intensity data through f frames by moving a lightsource arbitrarily. Under the Lambertian assumption the image data matrix I can be written as the productof two matrices S and L, with S representing the surface normal and the surface reflectance and L representingthe light-source direction and the light-source intensity. Using this formulation, I show that the image datamatrix I is of rank 3. On the basis of this observation, I use a singular-value decomposition technique anduseful constraints to factorize the image data matrix. This method can also be used to treat cast shadows andself-shadows without assumptions. The effectiveness of this method is demonstrated through performanceanalysis, laboratory experiment, and out-of-laboratory experiment.1. INTRODUCTIONPhotometric stereo is a method for shape estimation thatuses several intensity images obtained under differentlighting conditions.' The method can be used for specu-lar surfaces if extended light sources with spatially vary-ing brightness are used.2'3Surface curvature can beobtained from photometric-stereo data if one uses bright-ness gradients in addition to brightness itself.4Surfacedepth can also be estimated from photometric-stereo datawhen a point light source is relatively near the surfaceand near the camera.5'6The photometric-stereo tech-nique has been practically applied to scanning electronmicroscopic images,7and data provided by photometricstereo have been used for some industrial inspection andpart-alignment tasks.8 9These applications show thatphotometric stereo is a good candidate for propelling thecommercialization of machine vision techniques. How-ever, with previous methods it is impossible to estimatethe surface normal and the surface reflectance withouta priori knowledge of both the light-source direction andthe light-source intensity.In this paper I propose a new photometric-stereomethod for estimating the surface normal and the sur-face reflectance of objects without a priori knowledge ofthe light-source direction or light-source intensity. Inthis method, assuming only that the object's surface isLambertian, the surface normal, the surface reflectance,the light-source direction, and the light-source intensitycan be determined simultaneously. For example, we canestimate these four parameters from intensity imagesthat are obtained under a light source arbitrarily movedby a human. This method does not rely on any smooth-ness assumptions for these parameters. Furthermore,as the number of intensity images increases, the surfacenormal, the surface reflectance, the light-source direction,and the light-source intensity errors become smaller evenif the intensity images are taken in a noisy environment.The foundation of this method is similar to that of a fac-torization method for shape and motion estimation fromimage streams.'0In this method, I construct a p X fimage data matrix I from p pixel image intensity datathrough f frames by moving a light source arbitrarily.Under the Lambertian assumption, the image data ma-trix I can be written as the product of two matrices S andL, with S representing the surface characteristics (sur-face normal and reflectance) and L representing the lightcharacteristics (light-source direction and intensity). Us-ing this formulation, I show that the image data matrixI is of rank 3. On the basis of this observation, I use asingular-value decomposition (SVD) technique and one ofthe two following constraints to factorize the image datamatrix. One constraint is the constraint of surface re-flectance. This constraint can be used when there areat least 6 pixels in which the relative value of the sur-face reflectance is constant or is known. The other isthe constraint of light-source intensity. This constraintcan be used when there are at least 6 frames in whichthe relative value of the light-source intensity is constantor is known. The idea of applying the SVD technique toobserved image intensities to separate them into funda-mental components was introduced previously. "2The present method also describes how to deal withshadow regions. In an intensity image, there are twotypes of shadows: self-shadows and cast shadows. Pre-vious methods could not deal with either shadow typewithout relying on certain assumptions. This methodcan easily treat both types of shadow without the useof assumptions. In the image data matrix, we select aninitial submatrix having no shadowed data. We can thenestimate the surface normal and reflectance in shadowregions by growing a partial solution obtained from theinitial submatrix.The effectiveness of this method is demonstratedthrough performance analysis and through a laboratoryexperiment on Lambertian reflectance objects. Further-0740-3232/94/113079-11$06.00 © 1994 Optical Society of AmericaHideki Hayakawa3080 J. Opt. Soc. Am. A/Vol. 11, No. 11/November 1994Hiei aykwlightsource-0-myV' viewerI/V reflectivit function 0Fig. 1. Geometric reflectance model for image generation inthe viewer-oriented coordinate system. Here ni, in, and v de-note the three-dimensional (3D) unit vectors of surface normal,light-source direction, and viewer direction, respectively.One simple idealized model of surface reflectance isgiven byi(x, y) = r(x, y)t[n(x, y) ml] (2)for n(x, y) m Ž ~ 0. This reflectance function correspondsto the phenomenological model of a Lambertian surface.Here r is the surface reflectance and (x,