LETTERSImage statistics and the perception of surfacequalitiesIsamu Motoyoshi1, Shin’ya Nishida1, Lavanya Sharan2& Edward H. Adelson2The world is full of surfaces, and by looking at them we can judgetheir material qualities. Properties such as colour or glossiness canhelp us decide whether a pancake is cooked, or a patch of pavementis icy. Most studies of surface appearance have emphasized tex-tureless matte surfaces1–3, but real-world surfaces, which may havegloss and complex mesostructure, are now receiving increasedattention4–7. Their appearance results from a complex interplayof illumination, reflectance and surface geometry, which are dif-ficult to tease apart given an image. If there were simple imagestatistics that were diagnostic of surface properties it would besensible to use them8–11. Here we show that the skewness of theluminance histogram and the skewness of sub-band filter outputsare correlated with surface gloss and inversely correlated withsurface albedo (diffuse reflectance). We find evidence that humanobservers use skewness, or a similar measure of histogram asym-metry, in making judgements about surfaces. When the image of asurface has positively skewed statistics, it tends to appear darkerand glossier than a similar surface with lower skewness, and this istrue whether the skewness is inherent to the original image or isintroduced by digital manipulation. We also find a visual after-effect based on skewness: adaptation to patterns with skewed stat-istics can alter the apparent lightness and glossiness of surfacesthat are subsequently viewed. We suggest that there are neuralmechanisms sensitive to skewed statistics, and that their outputscan be used in estimating surface properties.Figure 1 shows two renderings of a three-dimensional model ofMichelangelo’s sculpture of St Matthew12. The version on the leftappears darker and glossier than the one on the right. This is trueeven though the two images have been scaled to have the same meanluminance. We are unaware of any theories that will predict thechanges in lightness or gloss that we observe.The image of a surface arises from the combination of the surfacegeometry, the surrounding illumination, and the surface optics. Eachof these components can be complex (for example, the reflectance ateach point is characterized by a four-dimensional function known asthe bidirectional reflectance distribution function13). Each is typicallyunknown, and estimating any one using ‘inverse optics’ requiresknowing the others. To bypass this problem, we have looked forsimple statistical image measurements that can provide informationthat is useful even if not complete. Any two-dimensional image mea-surements that are statistically related to properties of the three-dimensional scene are potentially useful8–11.We made a set of patches of stucco-like material. The values ofalbedo and glossiness were uniform within each patch, but they werevaried systematically from one patch to another by changing paintpigmentation and acrylic media coating, respectively. We photo-graphed these objects, linearized the pixel values and normalizedthe mean luminance by multiplicative scaling. We found that changesin albedo and glossiness were accompanied by characteristic changesin the luminance histogram. Consider the two stucco patches of1Human and Information Science Lab, NTT Communication Science Labs, Nippon Telegraph and Telephone Corporation, 3-1 Morinosato-Wakamiya, Atsugi 243-0198, Japan.2Department of Brain and Cognitive Sciences and Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology, 43 Vassar Street, 46-4115,Cambridge, Massachusetts 02139, USA.Figure 1|These two synthetic images of Michelangelo’s St Matthew sculpture have the same mean luminance. The one on the left looks darker and glossierthan the one on the right.Vol 447|10 May 2007|doi:10.1038/nature05724206Nature ©2007PublishingGroupFig. 2a. In comparison with a light matte surface (left), a dark glossysurface (right) has a long positive tail. In general, as the albedo ofglossy surfaces is decreased, or as the glossiness is increased regardlessof the albedo, the histogram’s skewness tends to increase (Fig. 2b;black circles). These changes make sense given the influence ofspecular and diffuse reflectance on the appearance of specular high-lights. Highlights are stronger and sharper on glossy surfaces, andthey have higher contrast when viewed on darker surfaces, becausethey are seen against a body surface that has a lower luminance.Having observed this physical relationship, we next looked for acorresponding psychophysical relationship. We showed these stuccoimages, one by one, to human observers, presenting them against adark background on a monitor at constant mean luminance, and askedthe observers to rate the lightness (perceived diffuse reflectance) orglossiness of each surface. The judgments were well correlated with thecorresponding physical properties, as shown in Fig. 2b (red circles).Both the lightness and glossiness ratings were also well correlated withthe skewness of the luminance histogram (Fig. 2c) to a degree com-parable with, or even higher than, the correlations with correspondingphysical properties (r 520.87 for correlation with skewness of light-ness ratings, and 0.89 for glossiness ratings, respectively).We next chose a set of images of three materials (stucco, blackcotton fabric and crumpled white paper, all of which were surfacesof uniform albedo and glossiness) and used a lookup table to force theluminance histograms to have specific skewness values. As expected,the lightness rating showed a strong negative dependency on skew-ness, whereas the glossiness rating showed a strong positive depend-ency. This was true for each image class (Fig. 2d). Further tests of awide variety of materials gave similar results, described in Supple-mentary Data A.In addition to the effects of skewness, we found a minor effect ofthe standard deviation of the luminance histogram on both lightnessand glossiness. The mean luminance had a significant effect onlightness1–3, but not on glossiness. We found little, if any, effect ofkurtosis (Supplementary Data B).The above results indicate that skewness or a similar measure ofhistogram asymmetry is useful in estimating surface qualities, andthat humans may indeed use it. How might such statistics be com-puted at the neural level? The early stages of vision are dominated
View Full Document
Unlocking...