1In the Eye of the Beholder: A Survey ofModels for Eyes and GazeDan Witzner Hansen, IEEE Member and Qiang Ji, IEEE Senior Member,23rd January 2009 DRAFT2AbstractDespite active research and significant progress in the last 30 years, eye detection and tracking remains challengingdue to the individuality of eyes, occlusion, variability in scale, location, and light conditions. Data on eye locationand details of eye movements have numerous applications, and are essential in face detection, biometric identificationand particular human computer interaction tasks. This paper reviews current progress and state of the art in video-based eye detection and tracking, in order to identify promising techniques as well as issues to be further addressed.We present a detailed review of recent eye models and techniques for eye detection and tracking. We also surveymethods for gaze estimation and compare them based on their geometric properties and reported accuracies. Thisreview shows that despite their apparent simplicity, the development of a general eye detection technique involvesaddressing many challenges, requires further theoretical developments, and is consequently of interest to many otherproblems in computer vision and beyond.Index TermsEye,Eye detection, Eye Tracking, Gaze estimation, review paper, gaze tracking, object detection and tracking,and human computer interaction.I. INTRODUCTIONAs one of the most salient features of the human face, eyes and their movements play an important role inexpressing a person’s desires, needs, cognitive processes, emotional states and interpersonal relations [141]. Theimportance of eye movements to the individual’s perception of and attention to the visual world is implicitlyacknowledged as it is the method through which we gather the information necessary to negotiate our way throughand identify the properties of the visual world. Robust non-intrusive eye detection and tracking is, therefore, crucialfor the development of human computer interaction, attentive user interfaces, and understanding human affectivestates.The unique geometric, photometric, and motion characteristics of the eyes also provide important visual cues forface detection, face recognition, and for understanding facial expressions. For example, one of the primary stagesin the Viola and Jones face detector is a Haar feature corresponding to the eye region [147]. This demonstratesthe importance of the eyes for face detection. Additionally, the distance between the eyes is often utilized forface normalization, for the localization of other facial landmarks, as well as in filtering out structural noise. Gazeestimation and tracking are important for many applications including human attention analysis, human cognitivestate analysis, gaze-based interactive user interfaces, gaze contingent graphical displays, and human factors. A gazetracker is a device for analyzing eye movements. As the eye scans the environment or fixates on particular objectsin the scene, a gaze tracker simultaneously localizes the eye position in the image and tracks its movement overtime to determine the direction of gaze.Research in eye detection and tracking focuses on two areas: eye localization in the image and gaze estimation.There are three aspects of eye detection. One is to detect the existence of eyes, another is to accurately interpreteye positions in the images, and finally, for video images, the detected eyes are tracked from frame to frame. The23rd January 2009 DRAFT3eye position is commonly measured using the pupil or iris center. Gaze estimation is using the detected eyes in theimages to estimate and track where a person is looking in 3D or, alternatively, determining the 3D line of sight. Inthe subsequent discussion, we will use the terms eye detection and gaze tracking to differentiate them, where eyedetection represents eye localization in the image while gaze tracking means estimating gaze paths.This paper focuses on eye detection and gaze tracking in video-based eye trackers (a.k.a video-oculography).A general overview of the components of eye and gaze trackers is shown in figure 1. Video-oculography systemsobtain information from one or more cameras (Image data). The eye location in the image is detected and is eitherused directly in the application or subsequently tracked over frames. Based on the information obtained from the eyeregion and possibly head pose, the direction of gaze can be estimated. This information is then used by gaze-basedapplications e.g. moving the cursor on the screen. The outline of this paper follows the components shown in figure1 and is organized as follows: In section II, we categorize eye models and review eye detection techniques usingthe eye models. An eye model can be used to determine gaze and models for gaze estimation are reviewed insection III. Applications of eye tracking are versatile and a summary is presented in section IV. We summarize andconclude the paper in section V with additional perspectives on eye tracking.DetectionEyeTrackingHead PoseEstimationGazeEstimationEye TrackerImage dataInitialpositionGazecorrdinatesApplicationEyedataHeadpose Eye LocationFig. 1. Components of video-based eye detection and gaze tracking.II. EYE MODELS FOR EYE DETECTIONIn eye detection, it is essential to identify a model of the eye which is sufficiently expressive to take account oflarge variability in the appearance and dynamics, while also sufficiently constrained to be computationally efficient.The appearance of eye regions share commonalities across race, illumination and viewing angle, but, as illustratedin figure 2, even for the same subject, a relatively small variation in viewing angles can cause significant changesin appearance. Despite active research, eye detection and tracking remains a very challenging task due to severalunique issues including occlusion of the eye by the eyelids, eye open/closed, variability in either size, reflectivity23rd January 2009 DRAFT4or head pose, etc. Applications of computer vision, such as people tracking, face detection and various medicalapplications encounter occlusions and shape variations, but rarely of the same order of magnitude and frequencyas seen with eyes.The eye image may be characterized by the intensity distribution of the pupil(s), iris and cornea as well as bytheir shapes. Ethnicity, viewing angle, head pose, color, texture, light conditions, the position of the iris within theeye socket and the state of the