Segmentation using Meta-texture SaliencyYaser Yacoob and Larry DavisComputer Vision Laboratory-UMIACSUniversity of Maryland, College Park, MD 20742yaser/[email protected] address segmentation of an image into patches thathave an underlying salient surface-roughness. Three in-trinsic images are derived: reflectance, shading and meta-texture images. A constructive approach is proposed forcomputing a meta-texture image by preserving, equaliz-ing and enhancing the underlying surface-roughness acrosscolor, brightness and illumination variations. We evaluatethe performance on sample images and illustrate quantita-tively that different patches of the same material, in an im-age, ar e normalized in their statistics despite variations incolor, brightness and illumination. Finally, segmentation byline-based boundary-detection is proposed and results areprovided and compared to known algorithms.1. BackgroundImage segmentation of scenes in which man-made ob-jects are presented in their diversity of appearance is achallenging computer vision problem. For example, whileclothing is, perhaps, the most diverse of such objects, itsbasic components can be simply reduced to: the materialof the thread (i.e., the fiber such as cotton, wool, silk, etc.),the thickness of the thread and the particular weaving pat-tern. Figure 1 shows images we are interested to analyzeand segment, (a) rug on a hardwood floor, (b) woman withcolorful hair, (c) wool sweater laid on a wood surface and(d) textured painting hanging on a wall. Our objective isto delineate regions that have a salient surface texture de-spite significant variations in color, brightness and illumi-nation attributes. Specifically, we seek the separation ofthe rug, hair, sweater and painting. We use the term meta-texture to convey and accentuate the underlying appearanceof rough surfaces in the image plane to differentiate fromtexture that conveys the color patterns on a smooth surfaceand 3D-textons that are tuned to the natural appearance ofmonochromatic surface roughness under viewpoint and il-lumination variations. These images are taken at high res-This work was sponsored by ONR grant N000140610102.(a) (b) (c) (d)Figure 1. A rug on a wood floor, multi-color hair, wool sweater ona wood surface, and textured painting on a wall (respectively).olution (1600x1200 for the face and 4368x2912 otherwise)to capture surface-detail.Color and texture-based segmentation. There is alarge body of research on texture-based segmentation ofcolor images (e.g., [3, 5]). Color and texture features aretypically extracted separately then clustering in a joint spaceis conducted. If applied to the images in Figure 1 they willresult in the identification of multiple regions due to coloror apparent texture variations and despite the similarity ofthe underlying surface texture.Material perception and 3D Textons. Adelson et al.[1] studied the properties of materials under different illumi-nation to determine classification by humans and study sta-tistical texture measures. Leung and Malik [8](also[5, 13])proposed an approach for classifying materials based on 3Dtexture attributes, 3D textons, computed over small patchesto capture local geometric and photometric properties ofmonochromatic images. Recognition of different materi-als under different lighting and viewing conditions wereshown. Note that 3D texton-based approach can be adaptedfor image segmentation of Figure 1(c) since the color tex-tured regions have clear boundaries and large sizes whichin the monochromatic image may be amenable to detec-tion, normalization and analysis. However, Figure 1(a,b,d)pose a challenge since the monochromatic image consists ofsmall regions of constant texture in Figure 1(a) and gradualintensity changes in Figure 1(b,d). Consequently, we viewthe meta-texture image proposed in this paper as a potentialinput for a 3D-texton based recognition process and not asan equivalent approach for processing image texture.Intrinsic Images. Intrinsic images (e.g., [7, 9, 12]) aimto reveal the underlying physical properties of a scene by es-timating the shading (e.g., a function of lighting and surface1normals) and reflectance (e.g., surface color) images. Intrin-sic analysis is suitable for coarse-level images (i.e., whensurface roughness is not visible) but is less suitable for fine-level images where surface roughness is detrimental to thestability of the surface normal over small neighborhoods.Intrinsic images involve scenes which arise from albedo orcolor variations on smooth surfaces while Fig. 1 involvesrough surfaces with complex dependencies on color, view-ing and illumination directions that destabilize the albedoand color variations at fine scales. Measurement of roughsurfaces have been proposed using the Bidirectional TextureFunctions (BTF) [6] and analysis and recognition [5]. Anal-ysis of rough surfaces in real-world images without BTF orlearned models remains open for research.This paper is focused on the diverse appearance of real-world rough-surfaces that defy smooth-surface assumptionsand their surface texture functions (e.g., BTF, 3D Textons)are unknown. The paper’s contributions are characterizingthe problem, proposing a discriminative approach to derivethree intrinsic images from a color image (i.e., shading, re-flectance, and meta-texture), proposing and developing theconcept of salient meta-texture image (MTI) via transform-ing an image into a grey-level image in which the projectedsurface roughness is preserved, equalized and enhancedwhile other properties such as color, brightness variations,etc. are normalized. This MTI is evaluated by consideringimage segmentation by texture-boundary detection.It is important to note that evaluating correctness of thederived three intrinsic images for real-world images cannotbe done (a problem we share with existing work on intrin-sic images). Moreover, the MTI is derived from intertwinedimaging process and attributes of the scene. For example,in Figure 1(c) meta-textures of red regions appear less sharpthan the yellow regions simply due to the imaging processwhile their surface-roughness are presumably equal. Never-theless, we provide empirical results that quantify the per-formance with respect to similarity within an image and ef-fectiveness of segmentation.2. ApproachWe propose that three derived images be computed,shading, reflectance and surface-roughness.TheMTIisa logical extension of intrinsic images [9, 12] to