C280 Computer VisionC280, Computer VisionProf. Trevor [email protected] 23: Segmentation II & Computational Photography TeaserTwo presentations toda y:Two presentations today:Contours and Junctions in Natural ImagesJitendra MalikUniversity of California at BerkeleyUniversity of California at Berkeley(with Jianbo Shi, Thomas Leung, Serge Belongie, Charless Fowlkes, David Martin, Xiaofeng Ren, Michael Maire, Pablo Arbelaez)3From Pixels to PerceptionpGrassWateroutdoorTigerSandwildlifebkSandbackTigerheadtaileyelegsmouth4shadowI stand at the window and see a house, trees, sky Theoretically I might say there were 327sky. Theoretically I might say there were 327 brightnesses and nuances of colour. Do I have "327"? No. I have sky, house, and trees.No. I have sky, house, and trees.----Max Wertheimer, 1923Max Wertheimer, 19235Perceptual OrganizationGroupingFigure/GroundGroupingFigure/Ground6Key Research Questions in Perceptual OrganizationOrganization • Predictive power– Factors for complex, natural stimuli ?– How do they interact ?• Functional significance– Why should these be useful or confer some evolutionary advantage to a visual organism?evolutionary advantage to a visual organism?• Brain mechanismsHow are these factors implemented given what we–How are these factors implemented given what we know about V1 and higher visual areas?7Attneave’s Cat (1954)Line drawings convey most of the gyinformation8Contours and junctions are fundamental…• Key to recognition, inference of 3D scene properties, visually- guided manipulation and locomotion…• This goes beyond local, V1-like, edge-detection. Contours are the result of perceptual organization grouping and figure/groundorganization, grouping and figure/ground processing9Some computer vision history…py• Local Edge Detection was much studied in the 1970s and early 80s (Sobel, Rosenfeld, Binford-Horn, Marr-Hildreth, Canny …)• Edge linking exploiting curvilinear continuity was studied as well (Rosenfeld, Zucker, Horn, Ullman )Ullman …)• In the 1980s, several authors argued for perceptual organization as a precursor toperceptual organization as a precursor to recognition (Binford, Witkin and Tennebaum, Lowe, Jacobs …)10,)However in the 90s …1. We realized that there was more to images than edges• Biologically inspired filtering approaches (Bergen & Adelson, Malik & Perona..)• Pixel based representations for recognition (Turk & Pentland, √Murase & Nayar, LeCun …)2. We lost faith in the ability of bottom-up vision•Do minimal bottom up processing e g tiled orientationDo minimal bottom up processing , e.g. tiled orientation histograms don’t even assume that linked contours or junctions can be extracted•Matching with memory of previously seen objects then becomes?Matching with memory of previously seen objects then becomes the primary engine for parsing an image.11At Berkeley, we took a contrary view…yy1. Collect Data Set of Human segmented images2. Learn Local Boundary Model for combining brightness, color and texture3. Global framework to capture closure, continuity4. Detect and localize junctions5. Integrate low, mid and high-level information for grouping and figure-ground segmentation12Berkeley Segmentation DataSet [BSDS]13D. Martin, C. Fowlkes, D. Tal, J. Malik. "A Database of Human Segmented Natural Images and its Application to Evaluating Segmentation Algorithms and Measuring Ecological Statistics", ICCV, 200114Contour detection ~19701515Contour detection ~19901616Contour detection ~20041717Contour detection ~2008 (gray)1818Contour detection ~2008 (color)1919Outline1. Collect Data Set of Human segmented images2. Learn Local Boundary Model for combining brightness, color and texture3. Global framework to capture closure, continuity4. Detect and localize junctions5. Integrate low, mid and high-level information for grouping and figure-ground segmentation20Contours can be defined by any of a number of cues (P. Cavanagh)21Cue-Invariant RepresentationsGray level photographsGray level photographsObjects from motionObjects from luminanceObjects from disparityObjects from texture Line drawings22Grill-Spector et al. , Neuron 1998Martin, Fowlkes, Malik PAMI 04PImageBoundary CuesPbBrightnessCue CombinationModelBrightnessColorTextureChallenges: texture cue, cue combinationGll h i b bili f b dGoal: learn the posterior probability of a boundary Pb(x,y,) from local information only23Individual Features• 1976 CIE L*a*b* colorspacer• Brightness Gradient BG(x,y,r,) – Difference of L* distributionsr(x,y)• Color Gradient CG(x,y,r,)– Difference of a*b* distributions• Texture Gradient TG(x,y,r,)–Difference of distributions of V1-like filter responses24These are combined using logistic regressionVarious Cue CombinationsCombinations25Outline1. Collect Data Set of Human segmented images2. Learn Local Boundary Model for combining brightness, color and texture3. Global framework to capture closure, continuity4. Detect and localize junctions5. Integrate low, mid and high-level information for grouping and figure-ground segmentation26Exploiting global constraints:Image Segmentation as Graph PartitioningBuild a weighted graph G=(V,E) from imageV: image pixelsgpE: connections between pairs of nearby pixelspairs of nearby pixelsPartition graph so that similarity within group is large and similarity between groups is small -- Normalized Cuts27ygpz[Shi & Malik 97]Wij small when intervening contour strong, small when weak..Cij = max Pb(x,y) for (x,y) on line segment ij; Wij = exp ( - Cij / 28Eigenvectors carry contour informationgy29We do not try to find regions from the eigenvectors, so we avoid the “broken sky” artifacts of Ncuts …30Key idea – compute edges on ncut eigenvectors, fi t ksum over first k:where is the output of a Gaussian derivative on the jtheigenvectorwhere is the output of a Gaussian derivative on the j-theigenvector of31The Benefits of GlobalizationMaire, Arbelaez, Fowlkes, Malik, CVPR 0832Comparison to other approaches3334Outline1. Collect Data Set of Human segmented images2. Learn Local Boundary Model for combining brightness, color and texture3. Global framework to capture closure, continuity4. Detect and localize junctions5. Integrate low, mid and high-level information for grouping and figure-ground segmentation35Detecting Junctions3637Benchmarking corner detection3839Better object recognition using previous version of