CS152, Spr. 2007 Intro Computer VisionPhotometric StereoIntroduction to Computer VisionCSE152Lecture 16CS152, Spr. 2007 Intro Computer VisionShading reveals 3-D surface geometryCS152, Spr. 2007 Intro Computer Vision• Shape-from-shading: Use just one image to recover shape. Requires knowledge of light source direction and BRDF everywhere. Too restrictive to be useful.• Photometric stereo: Single viewpoint, multiple images under different lighting.1. Arbitrary known BRDF2. Lambertian BRDF, known lighting3. Lambertian BRDF, unknown lighting.CS152, Spr. 2007 Intro Computer VisionAn example of photometric stereo Input ImagesInput ImagesCS152, Spr. 2007 Intro Computer VisionBRDF• Bi-directional Reflectance Distribution Function ρ(θin, φin ; θout, φout)• Function of– Incoming light direction:θin, φin– Outgoing light direction: θout, φout• Ratio of incident irradiance to emitted radiance^n(θin,φin)(θout,φout)CS152, Spr. 2007 Intro Computer VisionCS152, Spr. 2007 Intro Computer VisionPhotometric Stereo:General BRDFand Reflectance MapCS152, Spr. 2007 Intro Computer VisionCoordinate systemxyf(x,y)Surface: s(x,y) =(x,y, f(x,y))Tangent vectors:⎟⎟⎠⎞⎜⎜⎝⎛∂∂=∂∂⎟⎠⎞⎜⎝⎛∂∂=∂∂yfyyxsxfxyxs,1,0),(,0,1),(Normal vector⎟⎟⎠⎞⎜⎜⎝⎛−∂∂∂∂=×=∂∂×∂∂=1,,yfxfssysxsyxnCS152, Spr. 2007 Intro Computer VisionGradient Space (p,q)xyf(x,y)Normal vectorTyfxfysxs⎟⎟⎠⎞⎜⎜⎝⎛−∂∂∂∂=∂∂×∂∂= 1,,nyfqxfp∂∂=∂∂= ,Gradient Space : (p,q)n()Tqpqp1,,11ˆ22−++=nCS152, Spr. 2007 Intro Computer VisionImage FormationFor a given point A on the surface, the image irradiance E(x,y) is a function of 1. The BRDF at A 2. The surface normal at A3. The direction of the light sourcens.aE(x,y)ACS152, Spr. 2007 Intro Computer VisionReflectance MapLet the BRDF be the same at all points on the surface, and let the light direction s be constant. 1. Then image irradiance is a function of only the direction of the surface normal.2. In gradient space, we have E(p,q).nsaE(x,y)CS152, Spr. 2007 Intro Computer VisionExample Reflectance Map: Lambertian surfaceFor lighting from frontE(p,q)CS152, Spr. 2007 Intro Computer VisionLight Source Direction, expressed in gradient space.CS152, Spr. 2007 Intro Computer VisionReflectance Map of Lambertian SurfaceWhat does the intensity (Irradiance) of one pixel in one image tell us?It constrains the surface normal projecting to that pont to a curveE.g., Normal lies on this curveCS152, Spr. 2007 Intro Computer VisionReflectance Map of Lambertian + Specular SurfaceCS152, Spr. 2007 Intro Computer VisionTwo Light SourcesTwo reflectance mapsEmeasured2Emeasured1A third image would disambiguate matchCS152, Spr. 2007 Intro Computer VisionPhotometric stereo: Step11. Offline, use known source direction & BRDF to construct reflectance map for each direction.2. Acquire three images with known light source direction.3. For each pixel location (x,y), find (p,q) as the intersection of the three curves.4. This is the surface normal at pixel (x,y). Over image, this is normal field.CS152, Spr. 2007 Intro Computer VisionNormal FieldCS152, Spr. 2007 Intro Computer VisionPlastic Baby Doll: Normal FieldCS152, Spr. 2007 Intro Computer VisionNext step:Go from normal field to surfaceCS152, Spr. 2007 Intro Computer VisionRecovering the surface f(x,y)Many methods: Simplest approach1. From normal field n =(nx,ny,nz), p=-nx/nz, q=-ny/nz2. Integrate p=df/dx along a row (x,0) to get f(x,0)3. Then integrate q=df/dy along each column starting with value of the first rowf(x,0)CS152, Spr. 2007 Intro Computer VisionWhat might go wrong?• Height z(x,y) is obtained by integration along a curve from (x0, y0).• If one integrates the derivative field along any closed curve, on expects to get back to the starting value.• Might not happen because of noisy estimates of (p,q)∫++=),(),(0000)(),(),(yxyxqdypdxyxzyxzCS152, Spr. 2007 Intro Computer VisionWhat might go wrong?yzxxzy ∂∂∂∂=∂∂∂∂xqyp∂∂=∂∂Integrability. If z(x,y) is the height function, we expect that In terms of estimated gradient space (p,q), this means:But since p and q were estimated indpendentlyat each point as intersection of curves on three reflectance maps, equality is not going to exactly holdCS152, Spr. 2007 Intro Computer VisionHorn’s Method[ “Robot Vision, B.K.P. Horn, 1986 ]• Formulate estimation of surface height z(x,y) from gradient field by minimizing cost functional:where (p,q) are estimated components of the gradient while zxand zyare partial derivatives of best fit surface• Solved using calculus of variations – iterative updating• z(x,y) can be discrete or represented in terms of basis functions.• Integrability is naturally satisfied.dxdyqzpzyx22Image)()( −+−∫∫CS152, Spr. 2007 Intro Computer VisionIterative update•Let Si,jbe the height function at point i,j and pi,jand qi,jbe the measurements. The error is:•Let mi,jbe a mask that is 1 for pixels where reconstruction is to occur and 0 otherwise. Total error is:• Iteratively update the surface, starting at first iteration withS0i,j= 0.()()[]2,1,1,2,,1,1,41jijijijijijijiqSSpSSe −−+−−=−+−+jiijjiemE,,∑∑=CS152, Spr. 2007 Intro Computer VisionII. Photometeric Stereo:Lambertian Surface, Known LightingCS152, Spr. 2007 Intro Computer VisionLambertian SurfaceAt image location (u,v), the intensity of a pixel x(u,v) is:e(u,v) = [a(u,v) n(u,v)] · [s0s ]= b(u,v) · swhere• a(u,v) is the albedo of the surface projecting to (u,v).• n(u,v) is the direction of the surface normal.•s0is the light source intensity.• s is the direction to the light source.^n^s^^ae(u,v)CS152, Spr. 2007 Intro Computer VisionLambertian Photometric stereo• If the light sources s1, s2, and s3are known, then we can recover b from as few as three images. (Photometric Stereo: Silver 80, Woodham81). [e1 e2 e3 ] = bT[s1s2 s3] • i.e., we measure e1, e2, and e3 and we know s1, s2, and s3. We can then solve for b by solving a linear system. • Normal is: n = b/|b|, albedo is: |b|[][]1321−=321Tsssb eeeCS152, Spr. 2007 Intro Computer VisionWhat if we have more than 3 Images?Linear Least Squares[e1 e2 e3…en ] = bT[s1s2 s3...sn] Rewrite as e = Sbwheree is n by 1b is 3 by 1S is n by 3Let the residual ber=e-SbSquaring this: r2 = rTr = (e-Sb)T(e-Sb)= eTe-2bTSTe + bTSTSb∇b(r2)=0 - zero derivative is a necessary condition for a minimum,