1Shadows and Flash15-463: Rendering and Image ProcessingAlexei EfrosSNU Graphics & Media LabSNU Graphics & Media LabProblem DescriptionProblem DescriptionShadow Matting and CompositingShadow Matting and CompositingExtracting and Compositing the shadow in image spaceExtracting and Compositing the shadow in image spaceEstimating Shadow DeformationsEstimating Shadow DeformationsNeeds to account for the context differences in 3DNeeds to account for the context differences in 3D2SNU Graphics & Media LabSNU Graphics & Media LabShadow Compositing EquationShadow Compositing EquationSLC )1(ββ−+=C: observed colorC: observed colorS: shadow colorS: shadow colorL: color of a pixel when not in shadowL: color of a pixel when not in shadowββ: shadow matte: shadow matteSNU Graphics & Media LabSNU Graphics & Media LabDefining the set Defining the set ΣΣΣΣΣΣΣΣLet Let ΣΣbe a set of pixels in the source image, be a set of pixels in the source image, that covers the shadow of our interestthat covers the shadow of our interest3SNU Graphics & Media LabSNU Graphics & Media LabShadow Matte ExtractionShadow Matte ExtractionObtain an image sequence so that, by selecting Obtain an image sequence so that, by selecting appropriate frames, each pixel in appropriate frames, each pixel in ΣΣhas both has both shadowed and shadowed and unshadowedunshadowedversions. Then, exclude versions. Then, exclude the foreground object (using the video matting the foreground object (using the video matting algorithm).algorithm).For each pixel in For each pixel in ΣΣ, find min and max over frames, find min and max over framesS=shadow color, L=S=shadow color, L=unshadowunshadowcolor of the pixelcolor of the pixelFind the shadow map Find the shadow map ββbybyββis the essential part we are looking for is the essential part we are looking for min and maxf fffS C L C= =2( ) ( )C S L SL Sβ− ⋅ −=−SNU Graphics & Media LabSNU Graphics & Media LabShadow Matte ExtractionShadow Matte Extraction4SNU Graphics & Media LabSNU Graphics & Media LabShadow CompositingShadow CompositingExtract SExtract S’’and Land L’’from the lit and shadowed from the lit and shadowed versions of the novel background scene.versions of the novel background scene.Use Use ββobtained from the shadow matte extraction obtained from the shadow matte extraction step, and use the following equation to composite.step, and use the following equation to composite.ffffCLCS max and min''=='')1( SLCββ−+=SNU Graphics & Media LabSNU Graphics & Media LabShadow ComputationShadow Computation5Estimating Shadow DeformationsEstimating Shadow DeformationsSNU Graphics & Media LabSNU Graphics & Media LabThe Problem Reduces to Finding The Problem Reduces to Finding the Displacement Map the Displacement Map W[pW[p]]ppW[pW[p]]Once we obtain the map W, then just useOnce we obtain the map W, then just useββ’’==ββ[W[p[W[p]]]]Source planar surface versionSource planar surface versionTarget version with 3D geometryTarget version with 3D geometry6SNU Graphics & Media LabSNU Graphics & Media LabAn AssumptionAn AssumptionSource and target images share a common Source and target images share a common reference plane reference plane ππSNU Graphics & Media LabSNU Graphics & Media LabHow to Find How to Find W[pW[p]?]?In 3D, W[P] = QIn 3D, W[P] = QIn 2D image space, In 2D image space, W[pW[p] = q] = qp = the pixel representing Pp = the pixel representing Pq = the pixel representing Qq = the pixel representing QAn ObservationAn ObservationIf projections of two 3D lines happened to If projections of two 3D lines happened to intersect at Pintersect at PQ is the intersection of two lines, formed byQ is the intersection of two lines, formed by……7SNU Graphics & Media LabSNU Graphics & Media LabPreparation for Finding Preparation for Finding W[pW[p]]““Directional ScanDirectional Scan””See the VideoSee the VideoSNU Graphics & Media LabSNU Graphics & Media LabPart 1: For each directional scan s doPart 1: For each directional scan s doA.A.Label each pixel location p with the continuous Label each pixel location p with the continuous time time ttss[p[p] at which it is first crossed by the ] at which it is first crossed by the shadow edge using temporal analysisshadow edge using temporal analysisB.B.For each frame f, fit a shadow line equation For each frame f, fit a shadow line equation x=x=mms,fs,fy+by+bs,fs,fto the points r in the reference to the points r in the reference plane region labeled with time plane region labeled with time ttss[r[r]=f.]=f.C.C.Perform outliner rejection and replacement on Perform outliner rejection and replacement on the line equation parameters (the line equation parameters (mms,fs,f, , bbs,fs,f) to ) to reduce noise.reduce noise.8SNU Graphics & Media LabSNU Graphics & Media LabIllustraionsIllustraionsSNU Graphics & Media LabSNU Graphics & Media LabPart 2: For each pixel location p doPart 2: For each pixel location p doA.A.For each scan s, interpolate the line equation For each scan s, interpolate the line equation parameters (parameters (mms,s,ts[pts[p]], , bbs,s,ts[pts[p]]) and ) and (m(ms, s, ts[pts[p]], , bbs,s,ts[pts[p]]) from the nearest integer ) from the nearest integer frames of the scan sequences to obtain the line frames of the scan sequences to obtain the line equation parameters (equation parameters (mms,ts[ps,ts[p]], , bbs,ts[ps,ts[p]]) for this ) for this pixelpixelB.B.Compute the intersection q of lines for all scansCompute the intersection q of lines for all scansC.C.Store q as the value of the displacement map Store q as the value of the displacement map for pixel p: i.e. for pixel p: i.e. W[p]W[p]←←qq..9SNU Graphics & Media LabSNU Graphics & Media LabVisualization of Displacement MapVisualization of Displacement MapSNU Graphics & Media LabSNU Graphics & Media LabResultsResults10Other Graphics-related Courses15-505: Animation Art and Technology, Hodgins/Duesing15-463: Rendering and Image Processing, Efros15-493: Computer Game Programming, Hodgins/???05-331: Building Virtual Worlds, Pausch15-863: Simulation for Animation, James15-864: Advanced Computer Graphics, James15-869: Physically Based Character Animation, Pollard15-???: Other specialized graduate courses in
View Full Document
Unlocking...