AUTOMATIC CLASSIFICATION OF PHOTOGRAPHS AND GRAPHICS 1 Yuanhao Chen1 Zhiwei Li2 Mingjing Li2 Wei Ying Ma2 University of Science and Technology of China Hefei 230026 China 2 Microsoft Research Asia 49 Zhichun Road Beijing 100080 China ABSTRACT In general digital images can be classified into photographs and computer graphics This taxonomy is very useful in many applications such as web image search However there are no effective methods to perform this classification automatically In this paper we manage to solve this problem from two aspects At first we propose some novel low level features that can reveal perceptional differences between photographs and graphics Then we adopt an effective algorithm to perform the classification The experiments conducted on a large scale image database indicate the effectiveness of our algorithm 1 INTRODUCTION According to the ways in which they are generated digital images can be classified into photographs and graphics Photographs are often acquired by cameras and scanners and graphics are generated by computers The taxonomy is very useful in many applications such as web image search desktop search and image processing When searching for images on the web we know both the semantic content and the type of images we want beforehand For example we may want to find cartoon pictures of dogs A helpful step is to limit the search to graphics while filtering out the photographs of dogs Unfortunately current commercial image search engines like Google and Yahoo Image Search do not provide such functionalities These search engines are only based on the textual information such as the surrounding text and the image filename The textual information can describe the semantic content of images to a certain degree but it can rarely distinguish image types Therefore the automatic classification of photographs and graphics can be used to improve the search experience by filtering out the images whose types are improper Even when we do not have prior intensions properly grouping images according to their types can help quickly locate the target images Another important application of the classification is desktop search Personal photograph management is an important component of desktop search The classification of photographs and graphics is needed as the first step of photograph management The classification also plays an important role in the optimization of image processing Photographs and graphics have very different perceptional characteristics Graphics look much simpler than photographs If the characteristics are taken into account the most appropriate method would be adopted to improve the performance of image processing Because of the many potential applications of the taxonomy many methods have been reported for this problem 1 4 5 In 1 Athitsos et al used several features to measure the differences between photographs and graphics The features used include the number of colors most prevalent color farthest neighbor metric saturation metric farthest neighbor histogram metric and a few more An error rate of 9 was reported for distinguishing images encoded by JPEG In 4 Lienhart and Hartmann proposed an algorithm to distinguish actual photos from computer generation realistic looking images such as ray tracing images or screen shots from photo realistic computer games They measured the amount of noise by means of histogram of the absolute difference image between the original and its denoised version Because the computer generated images are less noisy than actual photos this feature can distinguish between actual photos and computer generated images Tian Tsong Ng et al solved the same problem of Lienhart and Hartmann in 5 Motivated by physical image generation process they used a geometry based image model to tackle the problem Although many people have worked on this problem the existing methods are not applicable to web images or large image collections First the computational cost of some algorithms is very high The per image featureextraction time of 5 is more than 50 seconds It is intolerable to web image search engines Second some features used before are not robust enough to noise For example images are usually resized in the web environment Due to the interpolation used in the resizing process the number of unique colors would greatly increase So the performance of the feature using the number of colors would degrade significantly Based on these methods we propose several new features such as the ranked histogram feature and the ranked The work was performed at Microsoft Research Asia 1 4244 0367 7 06 20 00 2006 IEEE 973 ICME 2006 region size feature that can reveal the perceptional differences between photographs and graphics These features exhibit promising performance with low computational cost In Web image retrieval sometimes only the reliable results are needed Therefore we integrate a rejection option in the classification process Ambiguous images such as mixed images have high probability to be rejected And the classification accuracy is improved for the images not rejected The paper is organized in 7 sections An overview of our classification method is presented in Section 2 The difference between photograph and graphics is analyzed in Section 3 Our proposed features as well as other traditional low level features used for classification are illustrated in Section 4 The classifier is described in Section 5 The experimental result and discussions are given in Section 6 Finally we conclude the paper in Section 7 photo generation process z Photographs are often acquired by digital cameras They depict the objects of the real world Because of the texture of the objects and noise in the photo generation process the texture information of photographs is very different from graphics z Certain colors are more likely to appear in graphics than in photographs such as some highly saturated ones 2 ALGORITHM OVERVIEW The ranked histogram feature is used to replace most prevalence color feature proposed in 1 As mentioned above graphics tend to have fewer colors than photographs and the percentage of the pixels of the prevalent colors for graphics is higher So the percentage of pixels having the most prevalent color is used to distinguish the two classes However many graphics have more than one prevalent color In such situation the most prevalent color feature does not work well As it is difficult to define a proper threshold that works well in all