Image-based 3D Scene UnderstandingThis classWhy worry about 3d scenes?Reason 1: We may want to interact with the sceneReason 2: We need contextSlide 62D Object DetectionWhat the 2D Detector SeesComputers need context tooHow to represent scene space?Slide 13Slide 14Gibson’s Surface LayoutSlide 16Slide 173D Scene Understanding3D Spatial LayoutSlide 22Slide 23The Main ChallengeOur World is StructuredLearn the Structure of the WorldInfer Most Likely SceneSlide 30Description of 3D SurfacesUse All Available CuesGet Good Spatial SupportImage SegmentationLabeling SegmentsImage LabelingConfidences from Boosted Decision TreesSurface Confidence MapsExperiments: Input ImageExperiments: Ground TruthExperiments: ResultSurface Estimates: OutdoorSample ResultsSurface Estimates: PaintingsSurface Estimates: IndoorFailures: Reflections and ShadowsImportance of Many CuesSpatial Support MattersAutomatic Photo Pop-upAutomatic Photo PopupSurfaces Not EnoughSlide 53GoalSlide 57Slide 58The ChallengeOcclusion Reasoning as ClassificationGradual Inference of Scene StructureSlide 62Getting the Initial BoundariesOcclusion CuesOcclusion Cues: SurfacesOcclusion Cues: DepthSlide 67Learn to Identify OcclusionsSlide 69Local Estimates May Be InconsistentGlobal Consistency via CRFGlobal Boundary ReasoningGetting the Next BoundariesSlide 75Final Estimate3D Cues Critical for Finding OcclusionsIterative Approach, 3D/CRF ImportantOcclusion ResultSlide 81Slide 823D Model with OcclusionsSlide 84The Relation Between Size and ViewpointObject Size ↔ Camera ViewpointSlide 88Slide 89Slide 90Slide 91Slide 92How surfaces and viewpoint help detectionSlide 94Input to Our AlgorithmExact Inference over Tree with Belief PropagationImproved Viewpoint EstimateImproved Object EstimatePutting Objects in PerspectiveExperiments on LabelMe DatasetSlide 102Better Detectors  Better ViewpointMore is BetterSlide 106Slide 107Need more robust scene priorsRelations Defined So FarClosing the LoopCoherent Image InterpretationSlide 113Automatic Photo Pop-up with OcclusionsSlide 115Slide 116Slide 117Things to rememberNext classImage-based 3D Scene UnderstandingImage-based 3D Scene UnderstandingComputer VisionCS 543 / ECE 549 University of IllinoisDerek Hoiem04/29/10This classThis class•3D scene understanding from one image (mostly my dissertation)•Puts into practice most of the single-image concepts learned in this classWork primarily Hoiem, Efros, Hebert in ICCV 2005, SIGGRAPH 2005, CVPR 2006, ICCV 2007, IJCV 2007, IJCV 2008, CVPR 2008Why worry about 3d scenes?Why worry about 3d scenes?Reason 1: We may want to interact with the sceneReason 1: We may want to interact with the sceneNavigation ManipulationReason 2: We need contextReason 2: We need contextReason 2: We need contextReason 2: We need context2D Object Detection2D Object DetectionWhat the 2D Detector SeesWhat the 2D Detector SeesComputers need context tooComputers need context tooTrue DetectionTrue DetectionsMissedMissedFalse DetectionsLocal Detector: [Dalal-Triggs 2005]How to represent scene space?How to represent scene space?How to represent scene space?How to represent scene space?Holistic Scene Space: “Gist”Oliva & Torralba 2001Torralba & Oliva 2002How to represent scene space?How to represent scene space?Depth MapSaxena, Chung & Ng 2005, 2007Gibson’s Surface LayoutGibson’s Surface Layoutslide from Aude Oliva • Gibson: “The elementary impressions of a visual world are those of surface and edge.” The Perception of the Visual World (1950)• Focus on texture gradientsSurface Layout (Gibson cont.)slide from Aude Oliva Gibson’s Surface LayoutGibson’s Surface LayoutSurface Layout (Gibson cont.)slide from Aude Oliva Gibson’s Surface LayoutGibson’s Surface Layout3D Scene Understanding3D Scene Understanding3D Spatial Layout3D Spatial LayoutSUPPORTVERTICALVERTICALSKY• Description of 3D Surfaces• Occlusion Relationships • Camera Viewpoint & Objects3D Spatial Layout3D Spatial Layout• Description of 3D Surfaces• Occlusion Relationships • Camera Viewpoint & Objects 12343D Spatial Layout3D Spatial Layout• Description of 3D Surfaces• Occlusion Relationships • Camera Viewpoint & ObjectsCar PersonCarThe Main ChallengeThe Main Challenge•Recovering 3D geometry from single 2D projection •Infinite number of possible solutions! …Our World is StructuredOur World is StructuredAbstract World Our WorldImage Credit (left): F. Cunin and M.J. Sailor, UCSDLearn the Structure of the WorldLearn the Structure of the World…Infer Most Likely SceneInfer Most Likely SceneUnlikely Likely3D Spatial Layout3D Spatial LayoutSUPPORTVERTICALVERTICALSKY• Description of 3D Surfaces• Occlusion Relationships • Camera Viewpoint & ObjectsGoal: Label image into 7 Geometric Classes:•Support•Vertical–Planar: facing Left (), Center ( ), Right ()–Non-planar: Solid (X), Porous or wiry (O)•SkyDescription of 3D SurfacesDescription of 3D SurfacesUse All Available CuesUse All Available CuesVanishing points, linesColor, texture, image locationTexture gradientGet Good Spatial SupportGet Good Spatial Support50x50 Patch50x50 PatchImage SegmentationImage Segmentation•Single segmentation won’t work•Solution: multiple segmentations………For each segment: - Get P(good segment | data) P(label | good segment, data)Labeling SegmentsLabeling SegmentsImage LabelingImage Labeling…Labeled SegmentationsLabeled PixelssegmentsdatasegmentgoodlabelPdatasegmentgoodPdatalabelP ),|()|()|(…Gray?High inImage?Many LongLines?YesNoNoNoNoYes YesYesVery High Vanishing Point?High in Image?Smooth? Green?Blue?YesNoNoNoNoYes YesYesConfidences from Boosted Decision TreesConfidences from Boosted Decision TreesGround Vertical Sky[Collins et al. 2002]P(label | good segment, data)Surface Confidence MapsSurface Confidence MapsP(Support) P(Vertical) P(Sky)P(Planar Left) P(Planar Center) P(Planar Right)P(Non-Planar Porous)P(Non-Planar Solid)Test ImageExperiments: Input ImageExperiments: Input ImageExperiments: Ground TruthExperiments: Ground TruthExperiments: ResultExperiments: ResultSurface Estimates: OutdoorSurface Estimates: OutdoorInput Image Ground Truth ResultAvg. AccuracyMain Class: 88%Subclass: 62%Sample ResultsSample ResultsInput ImageGround Truth ResultSurface Estimates: PaintingsSurface Estimates: PaintingsInput ImageOur ResultSurface Estimates: IndoorSurface Estimates: IndoorAvg. AccuracyMain Class: 93%Subclass: 76%Input Image Ground Truth ResultFailures: