EE368 Final Project Report Spring 2007 1 Painting Recognition using Camera phone Images Vincent Gire Sharareh Noorbaloochi Abstract This report discusses the algorithm we implemented to recognize a painting from 33 classes of paintings given to us In addition to outlining the algorithm steps we summarize the results our algorithm achieved on the set of training images given to us in addition to test images we obtained from the Cantor Art Center I INTRODUCTION The goal of this project was to develop a technique which recognizes paintings on display in the Cantor Arts Center based on snapshots taken with a camera phone Nokia N93 This sort of technique could be used as part of an electronic museum guide the user would point her his camera phone at a painting of interest and would hear commentary based on the recognition result This sort of applications implemented on hand held mobile devices is referred to as augmented reality In this project we had to develop the image processing algorithm that would recognize a test painting from the 33 given training paintings available to us The algorithm that we used to recognize the paintings is divided in two main parts 1 Finding the painting in the image and transforming it so that it fits in a 50x50 pixel square 2 Recognizing the painting by extracting its lower dimensional features using eigen images method and classifying it by using Euclidean nearest neighbour classifier As part of our algorithm developing stage we also tried to extract image features using the fisher images technique 2 which gave us worse performance than the eigen image technique We tested our algorithm on 132 newly obtained images from the same paintings and we correctly recognized 99 of them with an average recognition time of 0 5 seconds The paper describes the steps involved in our recognition algorithm II RECOGNITION ALGORITHM 1 Frame Recognition and Flat Projection The first part follows a succession of independent steps to find the painting in the input image rotate and scale it in an appropriate manner so it fits into a 50x50 pixel square 1 Loading and resizing the image 2 Color balancing 3 Color space conversions and thresholding 4 Filtering 5 Region labelling 6 Finding the corners 7 Projecting on a flat square 8 Equalizing 1 1 Load and resize the image The first mandatory step is of course to load the image A time analysis of the whole algorithm loading included reveals that this step is the most time consuming 1 2s in average to load an image with the Matlab function imread to be compared with the 0 4s taken to compute the remaining steps However since the images are quite big we can resize them to reduce the time consumption for each of the following steps The resizing factor is adaptive and chosen such that the resulting surface of the whole image is constant This adaptive resizing factor allows input images of different sizes very useful when we used the new set of testing images taken with a different camera The Matlab function imresize realizes the resizing operation applying first a low pass filter to avoid aliasing effects The low pass filter also reduces extensively the noise of the input image On the other hand resizing the image decreases the definition and changes the histograms of the color components so a trade off has to be found The best analysis to find this trade off is still the human analysis As long as a human can still easily recognize the painting on the input image after resizing the recognition algorithm should be able to do so We this approach we find that input images should have areas not less 250x250 pixels EE368 Final Project Report Spring 2007 2 1 2 Color balancing Even if the camera may have a built in color balancing algorithm it is better to insure that the colors are well balanced This was particularly useful when we used the set of new images taken with a different camera To do so we chose the grey world algorithm seen in class for which the time consumption is negligible Background Painting 1 3 Color space conversions and thresholding The next step is the most critical To find the painting parts in the input image we chose to look for the only black or colored pixels Indeed since all the paintings are always in a museum the background is always the same color and most probably white On the input image the white background appears in different grey levels The grey pixels with intensity in 40 255 are most often background whereas the grey pixels with intensity below 40 are close to black and often belong to paintings This is our first criteria We keep the pixel with intensity bellow 40 To find the colored pixels we change the color space to be independent from the lighting We use the YCbCr space and take the average between the red and blue chroma components as a first measure of colorness We also use the saturation in the HSV Hue Saturation and Value color space as a measure of the intensity of the color Histograms of these two measures have overall the same shape We can model them as the mix of two populations with Gaussian distributions The paintings have a Gaussian distribution which is added to the Gaussian distribution of the background However in our case the paintings always represents a much smaller part of the image than the background and the Gaussian distribution of the paintings can be neglected as a first approximation when computing the mean or variance of our distributions The adaptive threshold is chosen such that we only keep what is outside the main part of the background distribution Since we assume a Gaussian distribution for the background and neglect the effect of the foreground on the distribution this threshold is approximately the mean plus the half the full width at half maximum T mean 2 ln 2 var iance mean 2 35 2 var iance Figure 1 Saturation histogram of background and painting Background Painting Figure 2 Cb and Cr chroma histogram For each color measure average of Cb and Cr and saturation we compute the previous adaptive threshold and only keep the pixel above We end up with one mask for the only black pixels and two masks for the colored pixels that we combined with a OR operations The two masks for the colored pixels are not both necessarily and may be redundant for some pixels but they improve the robustness of the algorithm without increasing a lot the time consumption They may be redundant for some pixel but also complementary for others because the RGB to YCbCr transformation is linear