1A Day at the MuseumCatie Chang, Mary Etezadi-Amoli, and Michelle HewlettI. IntroductionOur objective was to identify 33 different paint-ings from the Cantor Arts Center, based on adatabase of images depicting the paintings at var-ious angles, orientations, and illuminations.We considered two main approaches for solv-ing this image processing problem. The firstused color histograms, which are advantageousbecause they do not change significantly acrossminor variations in scales and angles. Further-more, we found that warping or rotations werenot necessary for achieving high performance.One disadvantage of using this color histogrammethod is that lighting and n oise can greatly af-fect the color content of the images. Our secondapproach was based on scale-invariant feature de-tection . However, the performan ce of the featu redetection was sensitive to the choice of severalparameters, and further optimization over theseparameters may have been necessary to achieveperfect performan ce. In addition, our implemen-tation of feature detection was more computa-tionally expensive than the color histogram ap-proach. After testing both algorithms, the colorhistogram method achieved higher accuracy andfaster runtime.Our color histogram approach consists of fourmain steps (see Figure 1). First, the paintingis segmented from the background. Then, thecolors in this painting are quantized based on acustom color palette, which we designed in or-der to be representative of the colors across allof the training images in our set. Next, the im-age is partitioned into f our quadrants, and nor-malized histograms of color values are computed.Finally, the color histograms of this image arecompared against those of the training images.We use a histogram intersection metric to quan-tify the s imilarity between histograms. The meanof the histogram intersection values across all fourquadrants serves as our final measure of similar-ity.To classify a novel image, we compute its sim-ilarity with every other image in the database.We identify the novel image as the painting cor-respondin g to the most similar training image.We spent a day at the Cantor Arts Center andtook three more pictures of each painting. Ourtraining set was thus comprised of 198 images.Fig. 1. Flowchart of algorithm.II. SegmentationOur first task was to segment the painting fromthe remainder of th e image. Our algorithm con-verts the RGB image to grayscale (Figure 2),and then filters the grayscale image using me-dian filtering to reduce much of the noise presentin the origin al image. We then convert this fil-tered grayscale image to black and white using afixed threshold of 0.45, which worked well giventhe consistent lighting of the museum (Figure 3).We also implemented an adaptive thresholdingmethod, but it was more computationally expen-sive and, for our training set, did not offer signif-2icant improvement over the fixed thr esholding.Fig. 2. Grays c ale imageFig. 3. Black a nd white image, thresholded at 0.45.After obtaining this black and white image andcreating a mask, we perform region labeling onthe w hite regions. Within all of the trainin g im-ages, the painting f alls in the center of the image;therefore, to ensure that we select regions cor-respondin g to the painting, we create a squarearound the center of the mask and choose all re-gions overlap ping with this square. From these,we choose the region having the greatest numberof pixels (Figure 4).We then create a new image that has ones at allFig. 4. Central region with the la rgest number ofpixels.pixels corresponding to this label, and zeros else-where. This captures most of the painting, butcontains a few small holes. To fill in the holes, weinvert this image and perform region labeling onthe background. At this point, we are interestedin finding the region corresponding to the back-ground/wall. In order to do this, we find the la-bels of the four corners of the image, and removethe pixels having these labels. A final mask iscreated by zeroing out these background pixels.Finally, the mask is applied to the original image(Figure 5).Our final segmentation algorithm perf ormsvery well on the training images. The algorithmsuccessfully segments out the wall, other paint-ings, statues, and any other unnecessary objectsthat may appear in the images. Our algorithmalso ensures that the painting we extract fromthe original image lies in the center of the image,which was a valid constraint for our training andtest sets. While shadows around the frames aresometimes retained in the segmented image, thisdid not s eem to affect the overall performance ofour classifier.We tried several different approaches before fi-nalizing our segmentation algorithm . Our first3Fig. 5. Final image after segmentation.approach was to convert the image from RGB toHSV coordinates and use the saturation value tothreshold the image. However, a problem withthis approach was that the saturation values forthe images were very noisy and could not beremedied even with filtering. Secondly, we triededge d etection, but this technique did not per-form very well because several of the frames werequite fancy and did not have clear edges. A thirdapproach, which we refer to as the “wall thresh-olding” method (Figure 6), was to find all of theR, G, and B pixels whose values lie within a win-dow around the average R, G, and B values. Be-cause these are s hades of gray, we assumed thatthese pixels are part of the wall, and thus classi-fied them as background pixels. On most of thetraining images, this method performed beauti-fully and even got rid of the shadows on some ofthe paintings. However, it failed badly on a fewimages, because both lighting and noise tendedto affect the R,G,B values.III. Color Histogram MatchingOverviewWe used a color histogram approach to solvethe paintin g identification problem. The imageis first quantized to 100 colors, and histogramsFig. 6. Wall thre sholding.are computed over four quadrants to provide spa-tial localization. Histogram intersection is usedas the similarity metric. This method d emon-strated the best performance of all the methodswe considered.MethodsBefore settling on our final color histogram al-gorithm, we tried many variations. We initiallytried comparing the 1-D h istograms of the R,G, and B components, using mean square er rorand correlation as performance metrics. This ap-proach did not work well, however, because his-togramming the R , G, and B components sepa-rately