Passive 3D PhotographyVisual CuesSlide 3Slide 4Slide 5Slide 6Talk OutlineSingle View ModelingHow Do Humans Do This?Vanishing PointsMeasuring Height“Single View Metrology” [Criminisi 99]Slide 16“Morphable Models” [Blanz 99]Stereo ReconstructionStereoSlide 20Stereo CorrespondenceStereo Matching AlgorithmsStructure from MotionSlide 24Slide 25Slide 26Slide 27Slide 28ResourcesBibliographySlide 31Steve SeitzSteve SeitzCarnegie Mellon UniversityCarnegie Mellon UniversityUniversity of WashingtonUniversity of Washingtonhttp://www.cs.cmu.edu/~seitzhttp://www.cs.cmu.edu/~seitzPassive 3D PhotographyPassive 3D PhotographySIGGRAPH 2000 Course onSIGGRAPH 2000 Course on3D Photography3D PhotographyShadingShadingVisual CuesVisual CuesMerle Norman Cosmetics, Los AngelesVisual CuesVisual CuesShadingShadingTextureTextureThe Visual Cliff, by William Vandivert, 1960Visual CuesVisual CuesFrom The Art of Photography, CanonShadingShadingTextureTextureFocusFocusVisual CuesVisual CuesShadingShadingTextureTextureFocusFocusMotionMotionVisual CuesVisual CuesOthers:Others:•HighlightsHighlights•ShadowsShadows•SilhouettesSilhouettes•Inter-reflectionsInter-reflections•SymmetrySymmetry•Light PolarizationLight Polarization•......ShadingShadingTextureTextureFocusFocusMotionMotionShape From XShape From X•X = shading, texture, focus, motion, ...X = shading, texture, focus, motion, ...Talk OutlineTalk OutlineOverview Leading ApproachesOverview Leading Approaches1.1.Single view modelingSingle view modeling2.2.Stereo reconstructionStereo reconstruction3.3.Structure from motionStructure from motionSingle View ModelingSingle View ModelingHow Do Humans Do This?How Do Humans Do This?Good Guesswork Based on PriorsGood Guesswork Based on Priors•““these lines these lines looklook parallel” parallel”•““this this lookslooks like a cube” like a cube”•““this this lookslooks like a shadow” like a shadow”Computers Can Do This TooComputers Can Do This Too•Shape from shading Shape from shading [Horn 89][Horn 89]•User-aided modelingUser-aided modeling>““Tour into the Picture” [Horry 97]Tour into the Picture” [Horry 97]>““Facade” [Debevec 96] Facade” [Debevec 96] >““Single View Metrology” [Criminisi 99]Single View Metrology” [Criminisi 99]•Learning approachesLearning approaches>““Morphable Models” [Blanz 99]Morphable Models” [Blanz 99]Vanishing PointsVanishing PointsVanishingVanishingPointPointMeasuring HeightMeasuring Height123455.42.83.3Same Concepts EnableSame Concepts Enable•Reconstructing X, Y, and ZReconstructing X, Y, and Z•Computing camera projection matrixComputing camera projection matrix•Eliminating the rulerEliminating the ruler““Single View Metrology” [Criminisi 99]Single View Metrology” [Criminisi 99]““Single View Metrology” [Criminisi 99]Single View Metrology” [Criminisi 99]The Music Lesson, Jan Vermeer, 1662-65 Royal Collection of Her Majesty Queen Elizabeth II““Morphable Models” [Blanz 99]Morphable Models” [Blanz 99]VideoVideoStereo ReconstructionStereo ReconstructionThe Stereo ProblemThe Stereo Problem•Shape from two (or more) imagesShape from two (or more) images•Biological motivationBiological motivationknownknowncameracameraviewpointsviewpointsStereoStereoscene pointscene pointfocal pointfocal pointimage planeimage planeStereoStereoBasic Principle: TriangulationBasic Principle: Triangulation•Gives reconstruction as intersection of two raysGives reconstruction as intersection of two rays•Requires Requires point correspondencepoint correspondenceStereo CorrespondenceStereo CorrespondenceDetermine Pixel CorrespondenceDetermine Pixel Correspondence•Pairs of points that correspond to same scene pointPairs of points that correspond to same scene pointEpipolar ConstraintEpipolar Constraint•Reduces correspondence problem to 1D search along Reduces correspondence problem to 1D search along conjugateconjugate epipolar linesepipolar linesepipolar planeepipolar planeepipolar lineepipolar lineepipolar lineepipolar lineStereo Matching AlgorithmsStereo Matching AlgorithmsMatch Pixels in Conjugate Epipolar LinesMatch Pixels in Conjugate Epipolar Lines•Assume color of point does not changeAssume color of point does not change•PitfallsPitfalls>specularities specularities >low-contrast regionslow-contrast regions>occlusionsocclusions>image errorimage error>camera calibration errorcamera calibration error•Numerous approachesNumerous approaches>dynamic programming [Baker 81,Ohta 85]dynamic programming [Baker 81,Ohta 85]>smoothness functionalssmoothness functionals>more images (trinocular, N-ocular) [Okutomi 93]more images (trinocular, N-ocular) [Okutomi 93]>graph cuts [Boykov 00]graph cuts [Boykov 00]Structure from MotionStructure from MotionReconstruct Reconstruct •Scene Scene geometrygeometry •Camera Camera motionmotionUnknownUnknowncameracameraviewpointsviewpointsStructure from MotionStructure from MotionThe SFM ProblemThe SFM Problem•Reconstruct scene Reconstruct scene geometrygeometry and camera and camera motionmotion from from two or more imagestwo or more imagesTrack2D FeaturesEstimate3DOptimizeFit SurfacesStructure from MotionStructure from MotionStep 1: Track FeaturesStep 1: Track Features•Detect good featuresDetect good features>corners, line segmentscorners, line segments•Find correspondences between framesFind correspondences between frames>window-based correlationwindow-based correlationStructure from MotionStructure from MotionStep 2: Estimate Motion and StructureStep 2: Estimate Motion and Structure•Orthographic projection, e.g., Orthographic projection, e.g., [Tomasi 92][Tomasi 92]•2 or 3 views at a time 2 or 3 views at a time [Hartley 00][Hartley 00] n21f21f21XXXΠΠΠIIIImagesMotionStructureStructure from MotionStructure from MotionStep 3: Refine EstimatesStep 3: Refine Estimates•Nonlinear optimization over cameras and pointsNonlinear optimization over cameras and points>[Hartley 94][Hartley 94]•““Bundle adjustment” in photogrammetryBundle adjustment” in photogrammetryStructure from MotionStructure from MotionStep 4: Recover SurfacesStep 4: Recover Surfaces•Image-based triangulation Image-based triangulation [Morris 00, Baillard 99][Morris 00, Baillard 99]•Silhouettes Silhouettes [Fitzgibbon 98][Fitzgibbon 98]•Stereo Stereo