Image-based 3D Scene UnderstandingThis classWhy worry about 3d scenes?Reason 1: We may want to interact with the sceneReason 2: We need contextReason 2: We need context2D Object DetectionWhat the 2D Detector SeesComputers need context tooHow to represent scene space?How to represent scene space?How to represent scene space?Gibson’s Surface LayoutGibson’s Surface LayoutGibson’s Surface Layout3D Scene Understanding3D Spatial Layout3D Spatial Layout3D Spatial LayoutThe Main ChallengeOur World is StructuredLearn the Structure of the WorldInfer Most Likely Scene3D Spatial LayoutDescription of 3D SurfacesUse All Available CuesGet Good Spatial SupportImage SegmentationLabeling SegmentsImage LabelingConfidences from Boosted Decision TreesSurface Confidence MapsExperiments: Input ImageExperiments: Ground TruthExperiments: ResultSurface Estimates: OutdoorSample ResultsSurface Estimates: PaintingsSurface Estimates: IndoorFailures: Reflections and ShadowsImportance of Many CuesSpatial Support MattersAutomatic Photo Pop-upAutomatic Photo PopupSurfaces Not Enough3D Spatial LayoutGoalGoalGoalThe ChallengeOcclusion Reasoning as ClassificationGradual Inference of Scene StructureGradual Inference of Scene StructureGetting the Initial BoundariesOcclusion CuesOcclusion Cues: SurfacesOcclusion Cues: DepthOcclusion Cues: DepthLearn to Identify OcclusionsLearn to Identify OcclusionsLocal Estimates May Be InconsistentGlobal Consistency via CRFGlobal Boundary ReasoningGetting the Next BoundariesGradual Inference of Scene StructureFinal Estimate3D Cues Critical for Finding OcclusionsIterative Approach, 3D/CRF ImportantOcclusion ResultOcclusion ResultOcclusion Result3D Model with Occlusions3D Spatial LayoutThe Relation Between Size and ViewpointObject Size ↔ Camera Viewpoint Object Size ↔ Camera Viewpoint Object Size ↔ Camera Viewpoint Object Size ↔ Camera Viewpoint Object Size ↔ Camera Viewpoint Object Size ↔ Camera Viewpoint How surfaces and viewpoint help detectionHow surfaces and viewpoint help detectionInput to Our AlgorithmExact Inference over Tree with Belief Propagation Improved Viewpoint EstimateImproved Object EstimatePutting Objects in PerspectiveExperiments on LabelMe DatasetMore Info  Better DetectionBetter Detectors  Better ViewpointMore is BetterResultsResultsNeed more robust scene priorsRelations Defined So FarClosing the LoopCoherent Image InterpretationCoherent Image InterpretationAutomatic Photo Pop-up with OcclusionsAutomatic Photo Pop-up with OcclusionsAutomatic Photo Pop-up with OcclusionsSlide Number 117Things to rememberNext classImage-based 3D Scene UnderstandingComputer VisionCS 543 / ECE 549 University of IllinoisDerek Hoiem04/29/10This class• 3D scene understanding from one image (mostly my dissertation)• Puts into practice most of the single-image concepts learned in this classWork primarily Hoiem, Efros, Hebert in ICCV 2005, SIGGRAPH 2005, CVPR 2006, ICCV 2007, IJCV 2007, IJCV 2008, CVPR 2008Why worry about 3d scenes?Reason 1: We may want to interact with the sceneNavigation ManipulationReason 2: We need contextReason 2: We need context2D Object DetectionWhat the 2D Detector SeesComputers need context tooTrue DetectionTrue DetectionsMissedMissedFalse DetectionsLocal Detector: [Dalal-TriggsHow to represent scene space?How to represent scene space?Holistic Scene Space: “Gist”Oliva & Torralba 2001Torralba & Oliva 2002How to represent scene space?Depth MapSaxena, Chung & Ng 2005, 2007Gibson’s Surface Layoutslide from Aude Oliva • Gibson: “The elementary impressions of a visual world are those of surface and edge.” The Perception of the Visual World (1950)• Focus on texture gradientsSurface Layout (Gibson cont.)slide from Aude Oliva Gibson’s Surface LayoutSurface Layout (Gibson cont.)slide from Aude Oliva Gibson’s Surface Layout3D Scene Understanding3D Spatial LayoutSUPPORTVERTICALVERTICALSKY• Description of 3D Surfaces• Occlusion Relationships• Camera Viewpoint & Objects3D Spatial Layout• Description of 3D Surfaces• Occlusion Relationships• Camera Viewpoint & Objects 12343D Spatial Layout• Description of 3D Surfaces• Occlusion Relationships• Camera Viewpoint & ObjectsCar PersonCarThe Main Challenge• Recovering 3D geometry from single 2D projection • Infinite number of possible solutions! …Our World is StructuredAbstract World Our WorldImage Credit (left): F. Cunin and M.J. Sailor, UCSDLearn the Structure of the World…Infer Most Likely SceneUnlikely Likely3D Spatial LayoutSUPPORTVERTICALVERTICALSKY• Description of 3D Surfaces• Occlusion Relationships• Camera Viewpoint & ObjectsGoal: Label image into 7 Geometric Classes:• Support• Vertical– Planar: facing Left (), Center ( ), Right ()– Non-planar: Solid (X), Porous or wiry (O)• SkyDescription of 3D SurfacesUse All Available CuesVanishing points, linesColor, texture, image locationTexture gradientGet Good Spatial Support50x50 Patch50x50 PatchImage Segmentation• Single segmentation won’t work• Solution: multiple segmentations………For each segment:- Get P(good segment | data) P(label | good segment, data)Labeling SegmentsImage Labeling…Labeled SegmentationsLabeled Pixels∑∝segmentsdatasegmentgoodlabelPdatasegmentgoodPdatalabelP ),|()|()|(…Gray?High inImage?Many LongLines?YesNoNoNoNoYes YesYesVery High Vanishing Point?High in Image?Smooth? Green?Blue?YesNoNoNoNoYes YesYesConfidences from Boosted Decision TreesGround Vertical Sky[Collins et al. 2002]P(label | good segment, data)Surface Confidence MapsP(Support) P(Vertical) P(Sky)P(Planar Left) P(Planar Center) P(Planar Right)P(Non-Planar Porous)P(Non-Planar Solid)Test ImageExperiments: Input ImageExperiments: Ground TruthExperiments: ResultSurface Estimates: OutdoorInput Image Ground Truth ResultAvg. AccuracyMain Class: 88%Subclass: 62%Sample ResultsInput ImageGround Truth ResultSurface Estimates: PaintingsInput ImageOur ResultSurface Estimates: IndoorAvg. AccuracyMain Class: 93%Subclass: 76%Input Image Ground Truth ResultFailures: Reflections and ShadowsInput ImageResultImportance of Many CuesAll PositionOnlyColor OnlyTexture OnlyPerspective OnlyMain88%83% 72% 80% 68%Subclass61%43% 43% 55% 52%All All But PositionAll But ColorAll But TextureAll But PerspectiveMain88%84% 87% 87% 88%Subclass61%60% 60% 58% 57%Spatial Support MattersAutomatic Photo Pop-upAutomatic Photo PopupLabeled ImageFit Ground-Vertical Boundary with Line SegmentsForm Segments into PolylinesCut