IntroductionBackgroundImage Processing AlgorithmPainting RectificationModified SIFTCorrelation with DatabaseDatabase CreationAlgorithm PerformanceConclusionReferencesAppendixGroup Work LogIdentification of Paintings in Camera-Phone ImagesGabriel M. Hoffmann, Peter W. Kimball, Stephen P. RussellDepartment of Aeronautics and AstronauticsStanford University, Stanford, CA 94305{gabeh,pkimball,sprussell}@stanford.eduAbstract— This paper presents the algorithm developed, forthe EE 368 course project, to identify paintings in query imagestaken with a camera-phone in the European Gallery of the Can-tor Arts Center. The image processing algorithm first extractsthe painting of interest, in the center of the photo. It usesedge detection in saturation space, followed by iterative Radontransforms and region removal to find the smallest boundingbox enclosing the painting. Using the smallest bounding box,the algorithm computes and performs a projective transformto extract and rectify the painting in the query image. Featureidentification is performed on the rectified image using extremain a difference-of-Gaussians pyramid. At each resulting featurelocation, feature descriptors are computed using histogramsof the gradient distribution among neighboring pixels, withoutrotation. The resulting features are correlated with a featuredatabase of paintings with aspect ratios similar to that of thepainting in the query image. A program to produce the featuredatabase was also developed. This program selects a small setof unique, but repeatable, features from the many available foreach painting in the gallery.The image processing algorithm correctly identifies paintingsin all 99 training images provided by the EE368 teaching staff.It performs well on the additional 93 test images taken by thegroup, in difficult configurations, to stress test the algorithm.Details and sensitivities of the algorithm are discussed.I. INTRODUCTIONRecognition of objects in images has practical applicationsacross many disciplines. One potential “Augmented Reality”application is the recognition of paintings in an art exhibit.Gallery visitors could take a picture of a painting with acell phone, and then receive an automated phone call or textmessage containing information about the painting such astitle, artist, historical context, and critical review. This servicerequires rapid identification of the painting in the image,without strong guarantees on lighting, viewing angle, andsize of the painting in the image. Compression artifacts andlow light effects, such as blur and noise, corrupt such imagesto varying degrees. The painting recognition task can beautomated using techniques from the fields of digital imageprocessing and computer vision [1].After consideration and prototyping of several candidatetechniques, it was found that this problem has some uniquecharacteristics. First, the low light conditions cause highvariation in the quality of the images, making the correlationof many feature descriptor types difficult. Second, there maybe multiple paintings, and even other works of art, in theimages. Third, the paintings from the same era showedmany recurring styles and themes, reducing the uniquenessof features. Visually, patches of some paintings can be seenin other paintings.This paper presents the algorithm developed for the EE368 course project, to identify the paintings in the centersof query images taken with a camera-phone. The imagesunder consideration were recorded in the European Galleryof the Cantor Arts Center. The image processing algorithmfirst extracts the painting of interest, in the center of a queryimage. It uses edge detection in saturation space, followedby iterative Radon transforms and region removal to findthe smallest bounding box enclosing the painting. Using thesmallest bounding box, the algorithm computes and performsa projective transform to extract and rectify the painting inthe query image. This step reduces the size of the image forfeature detection, and reduces the requirements for robust-ness of the feature descriptors, such as perspective invariance.Feature identification is performed on the rectified imageusing extrema in a difference-of-Gaussians pyramid. At eachresulting feature location, feature descriptors are computedusing histograms of the gradient distribution among neigh-boring pixels, without rotation. The resulting features arecorrelated with a feature database of paintings with aspectratios similar to that of the painting in the query image. Aprogram to produce the feature database was also developed.This program selects a small set of unique, but repeatable,features from the many available for each painting in thegallery. Because the paintings are rectified, using the knowl-edge that they are rectangular, more general correspondencealgorithms are not required to perform correspondence inaffine or projective spaces.The image processing algorithm, implemented in Matlabscript files, correctly identifies paintings in all 99 trainingimages provided by the EE368 teaching staff, with a meantime of 5.2 seconds per query image on the SCIEN machines.An additional 93 test images were taken by the group inconfigurations expected to be difficult, to stress test the algo-rithm, such as with excessive shaking, extreme perspectives,long distances, and more occlusions. This allowed minormodifications to the parameters so that the algorithm couldperform well with images not considered during develop-ment. Details and sensitivities of the algorithm are discussed.II. BACKGROUNDThere are two major challenges involved in the art recog-nition application described above. First, there must be a wayto extract the region of interest from the image. Second, itis necessary to describe this region in a distinctive way, onethat is repeatable and can be cross referenced to a database ofinformation that uniquely identifies the painting in question.In order to extract a section of an image, it is necessary tohave some information about the region in question. For thisapplication, it is assumed that the painting is in a standard artgallery. That is, the painting is posted on a white wall withsome space between it and adjacent pieces of art. It is fine forother works to be present in the query image, so long as thereis some white space to distinguish different pieces of art.Also, it is assumed that the desired painting is roughly cen-tered in the image frame. More specifically, the center pixelin the image is