Slide 1Slide 2Slide 3Slide 4Slide 5Slide 6Slide 7Slide 8Slide 9Slide 10Slide 11Slide 12Slide 13Slide 14Slide 15Slide 16Slide 17Slide 18Slide 19Slide 20Slide 21Slide 22Slide 23Slide 24Slide 25Slide 26Slide 27Slide 28Object Recognition by Integrating Multiple Image SegmentationsCaroline Pantofaru, Cordelia Schmid, Martial HebertECCV 2008E2Types of object recognitionCatBike FaceGoal: Accurate object recognition and object segmentation of deformable object classes3Too many possible shapes4Shape proposals•Silhouette masks of (semi-) rigid objects.–Edge-driven: •[Shape context,Belongie et al., PAMI’02]–Feature-driven: •[Marszalek and Schmid, CVPR’06]•Each pixel (patch) separately: –Noise•[Sivic et al., ICCV’05] [Ferrari et al. ECCV’04]–Constrained shape and appearance•[Borenstein and Malik, CVPR’06]•Fixed-shape parts: semi-rigid objects.–[Mori et al., CVPR’04]5Bottom-up grouping: Image Segmentation•[BP at CVPR’06] [BMVC’07]•[Hoiem et al., IJCV’07][Russell et al., CVPR’06][Todorovic and Ahuja, CVPR’06][Ren and Malik, ICCV’03]…6Classifying single regionsImageObject mapLearn single region feature classifierBottom-up image segmentationTraining data……Bottom-up image segmentationsSingle region classification by SVMRegion features:•centroid•histogram of quantized hue features•Region-based Context Feature•aggregate hue histogram over the image•aggregate RCF over the entire image“Good” segmentation:•Repeatable•Regions are “large enough” for feature computation but stay within object boundaries.7Human segmentations are inconsistent•Berkeley segmentation database–[Martin et al. ICCV’01]8[MS: Mean shift, Comaniciu and Meer, PAMI’02] [FH: Felzenszwalb and Huttenlocher, IJCV’04] [MS+FH: Pantofaru and Hebert, CMU’05]Automatic segmentations are very inconsistent9Types of regions and boundariesObject boundaryColor discontinuitySegmentation artifactOver- segmentationObject part Under- segmentation10Multiple segmentations•Mean Shift [Comaniciu and Meer, PAMI’02]•Normalized cuts with boundary estimates [Shi and Malik, PAMI’00; Fowlkes et al., CVPR’03]•Graph-based segmentation [Felzenszwalb and Huttenlocher, IJCV’04]Intersections of Regions11Multiple segmentations – Related work•Ways to generate multiple segmentations:–Hierarchical•[Hoiem et al., IJCV’07] [Borenstein and Malik, CVPR’06] [Todorovic and Ahuja, ICCV’07]–One algorithm, different numbers of regions •[Russell et al., CVPR’06] [Malisiewicz and Efros, BMVC’07]–Multiple algorithms, multiple parameters•[ECCV’08]•Ways to use multiple segmentations:–Pick best region•[Russell et al., CVPR’06]–Use segments as parts•[Todorovic and Ahuja, ICCV’07]–Soft segmentation, use shape•[Borenstein and Malik, CVPR’06]–Integrate information from all segmentations•[ECCV’08] [Hoiem et al., IJCV’07]12Ideas and Assumptions•Ideas:–Groups of pixels consistently clustered should be consistently classified.–The set of regions provides robust features for classification.•Assumptions:–Object edges are a subset of the region outlines.–Each pixel is contained in some region that is large enough for descriptive feature computation.1213Multiple SegmentationsIntersections of Regions……14ImageMultiple segmentations: ResultsConfidenceGround truth Multiple segmentationsGood single segmentationPoor single segmentation•MSRC 21-class data set–[Shotton et al., ECCV’06]15Multiple segmentations: ResultsImage ConfidenceGround truth Good single segmentationPoor single segmentationMultiple segmentations16Multiple segmentations: Results•Pixel accuracy on the MSRC 21-class data setShotton’06 Verbeek’07 Good single segPoor single segMultiple segsClass-averaged57.7% 64.0% 59.8% 49.6% 60.3%Overall 72.2% 73.5% 72.2% 63.3% 74.3%•[Shotton et al., ECCV’06]•[Verbeek and Triggs, CVPR’07 (Different data split)]17Multiple Segmentations: Resultsclass avgpixel avgbldg grass tree cow sheep sky plane water face carShotton 57.7 72.262 98 86 58 50 83 60 53 74 63*Verbeek 64.0 73.552 87 68 73 84 94 88 73 70 68Worst seg 49.6 63.348 80 69 51 61 87 73 71 57 47Best seg 59.8 72.261 89 79 57 66 92 81 80 67 63All segs 60.3 74.368 92 81 58 65 95 85 81 75 65bike flower sign bird book chair road cat dog body boatShotton75 63 35 19 92 15 86 54 19 62 7*Verbeek74 89 33 19 78 34 89 46 49 54 31Worst seg56 34 28 15 75 16 76 28 17 40 11Best seg66 52 31 26 88 27 80 52 32 45 30All segs68 53 35 23 85 16 83 48 29 48 1518Are all of the segmentations useful?•PASCAL VOC2007 segmentation challenge data set19Region reliabilityIntersections of Regions…[Homogeneity: Hoiem et al., IJCV’07]Classifier: Boosted decision trees 20Result of using multiple segmentationsGround Truth Multiple SegsIndividual segmentations•PASCAL VOC2007 segmentation challenge data set21Results on PASCAL VOC07 data setImageGround truth Good single segmentationPoor single segmentationMultiple segmentationsGood single segBad single seg Multiple segs Multiple segs w/ region weightsClass-averaged 18.2 12.7 19.1 19.622Spatial information – Related work•Silhouettes/Shape–[Levin and Weiss, ECCV’06] [Kumar et al., CVPR’05] [Borenstein and Malik, CVPR’06]–Rigid or semi-rigid objects•Relationships between parts–[Mori et al., CVPR’04] [Todorovic and Ahuja, ICCV’07]–Semi-rigid objects–Regions are whole parts•Pair-wise region information in a random field–[ECCV’08] [BMVC’07] [Hoiem et al., IJCV’07] [He et al., ECCV’06] –Deformable objects–Combine over-segmented regions23Pair-wise Region ModelReward confidence in the label:Penalize discontinuity:Solve using Graph Cuts [Kolmogorov and Zabih, ECCV’02]Energy minimization:IofRs…24Pair-wise Region Model ResultsSingle region Random field25PASCAL VOC2007 segmentation challengeWithout spatial infoWith spatial infoClass avg 19.6 19.3Person 80.7 87.3Bus 32.2 34.2Cat 13.7 15.7Background 59.2 47.1Dog 8.5 5.7Horse 24.3 23.3Pair-wise Region Model Results• Fully supervised training• Multiple segmentations• 21-class problem26Training with weak labels• Augment the training set. • 422 fully labeled images + 400 weakly labeled images.• Weak labels: bounding boxes or image labels.27Results of training with weak labelsImageSeg + Image TrainingSeg + BBox TrainingSeg TrainingGround Truth•PASCAL VOC2007 segmentation challenge data set28ConclusionsA study into the use of
View Full Document
Unlocking...