AUTOMATIC CLASSIFICATION OF PHOTOGRAPHS AND GRAPHICS* Yuanhao Chen1, Zhiwei Li2, Mingjing Li2, Wei-Ying Ma2 1University of Science and Technology of China, Hefei 230026, China 2Microsoft Research Asia, 49 Zhichun Road, Beijing 100080, China * The work was performed at Microsoft Research Asia. ABSTRACT In general, digital images can be classified into photographs and computer graphics. This taxonomy is very useful in many applications, such as web image search. However, there are no effective methods to perform this classification automatically. In this paper, we manage to solve this problem from two aspects. At first, we propose some novel low-level features that can reveal perceptional differences between photographs and graphics. Then, we adopt an effective algorithm to perform the classification. The experiments conducted on a large-scale image database indicate the effectiveness of our algorithm. 1. INTRODUCTION According to the ways in which they are generated, digital images can be classified into photographs and graphics. Photographs are often acquired by cameras and scanners, and graphics are generated by computers. The taxonomy is very useful in many applications, such as web image search, desktop search and image processing. When searching for images on the web, we know both the semantic content and the type of images we want beforehand. For example, we may want to find cartoon pictures of dogs. A helpful step is to limit the search to graphics while filtering out the photographs of dogs. Unfortunately, current commercial image search engines, like Google and Yahoo Image Search, do not provide such functionalities. These search engines are only based on the textual information such as the surrounding text and the image filename. The textual information can describe the semantic content of images to a certain degree, but it can rarely distinguish image types. Therefore, the automatic classification of photographs and graphics can be used to improve the search experience by filtering out the images whose types are improper. Even when we do not have prior intensions, properly grouping images according to their types can help quickly locate the target images. Another important application of the classification is desktop search. Personal photograph management is an important component of desktop search. The classification of photographs and graphics is needed as the first step of photograph management. The classification also plays an important role in the optimization of image processing. Photographs and graphics have very different perceptional characteristics. Graphics look much simpler than photographs. If the characteristics are taken into account, the most appropriate method would be adopted to improve the performance of image processing. Because of the many potential applications of the taxonomy, many methods have been reported for this problem [1, 4, 5]. In [1], Athitsos et al, used several features to measure the differences between photographs and graphics. The features used include the number of colors, most prevalent color, farthest neighbor metric, saturation metric, farthest neighbor histogram metric, and a few more. An error rate of 9% was reported for distinguishing images encoded by JPEG. In [4], Lienhart and Hartmann proposed an algorithm to distinguish actual photos from computer-generation realistic-looking images such as ray tracing images or screen shots from photo-realistic computer games. They measured the amount of noise by means of histogram of the absolute difference image between the original and its de-noised version. Because the computer-generated images are less noisy than actual photos, this feature can distinguish between actual photos and computer-generated images. Tian-Tsong Ng et al. solved the same problem of Lienhart and Hartmann in [5]. Motivated by physical image generation process, they used a geometry-based image model to tackle the problem. Although many people have worked on this problem, the existing methods are not applicable to web images or large image collections. First, the computational cost of some algorithms is very high. The per-image feature-extraction time of [5] is more than 50 seconds. It is intolerable to web image search engines. Second, some features used before are not robust enough to noise. For example, images are usually resized in the web environment. Due to the interpolation used in the resizing process, the number of unique colors would greatly increase. So the performance of the feature using the number of colors would degrade significantly. Based on these methods, we propose several new features such as the ranked histogram feature and the ranked 9731424403677/06/$20.00 ©2006 IEEE ICME 2006region size feature that can reveal the perceptional differences between photographs and graphics. These features exhibit promising performance with low computational cost. In Web image retrieval, sometimes only the reliable results are needed. Therefore, we integrate a rejection option in the classification process. Ambiguous images such as mixed images have high probability to be rejected. And the classification accuracy is improved for the images not rejected. The paper is organized in 7 sections. An overview of our classification method is presented in Section 2. The difference between photograph and graphics is analyzed in Section 3. Our proposed features as well as other traditional low-level features used for classification are illustrated in Section 4. The classifier is described in Section 5. The experimental result and discussions are given in Section 6. Finally, we conclude the paper in Section 7. 2. ALGORITHM OVERVIEW We build a demonstration system to discriminate the photographs and graphics. Before the feature extraction, each image is normalized to a predefined size via nearest-neighbor interpolation. We perform this pre-processing for two reasons. One is the consideration of the computational cost. The time used in feature extraction is almost proportional to the image size. So reducing images to small size can greatly decrease the computational cost. The other reason is to facilitate the application in web image search engine. In a typical search engine, images are often stored in the thumbnail format due to copyright problem and disk space cost. Then some low-level features are extracted from each image for classification. Finally,