Windows Manager Responsive to User Hand GesturesZhenya Gu & Yunjie MaList of TablesSelectingMovingMinimizingClosing 2 Modules / ImplementationVideo ProcessorWindows Manager 3 Testing and Debugging 4 ConclusionsWindows Manager Responsive to User Hand Gestures Zhenya Gu & Yunjie Ma Massachusetts Institute of Technology 6.111 Digital Electronics Laboratory Final Project Fall 2006 Abstract This project implements a windows manager program responsive to a user’s hand movements. Windows can be opened, moved, minimized, and closed by four different hand gestures. The XVGA display contains two windows, two icons, and a task bar to which both windows can be minimized. The project was developed in two parts, one that analyzed the video input and determined the center of mass of the user’s hand and the action the user is performing, and another that generates the XVGA display seen by the user and performs the actions determined in the other part. During debugging, common problems had to do with windows not responding to the user’s actions. This was due to several factors such as the center of mass going off the screen, logic in the gesture interpreter, and glitching with displaying the BRAM data. These problems were resolved by wiring inputs and outputs to the hex display and observing what actions caused logic problems.iiTable of Contents List of Figures . . . . . . . . iii List of Tables . . . . . . . . . iii 1 Windows Manager Overview . . . . . . 1 1.1 Introduction . . . . . . . 1 1.2 Setup . . . . . . . . 1 1.3 Configuration . . . . . . . 1 1.4 Display . . . . . . . . 3 1.5 Usage . . . . . . . . 3 1.6 Other . . . . . . . . 5 2 Modules / Implementation . . . . . . 6 2.1 Video Decoder . . . . . . . 6 2.2 YCrCb to RGB Converter . . . . . 7 2.3 Write and Read from ZBT RAM . . . . . 7 2.4 Center of Mass Finder . . . . . . 7 2.5 Screen Size Calibration . . . . . . 8 2.6 Hand Shape Detector . . . . . . 9 2.7 Hand Size Calibration . . . . . . 9 2.8 Gesture Interpreter . . . . . . 10 2.9 Display . . . . . . . . 12 3 Testing and Debugging . . . . . . 19 3.1 Read and Write from ZBT RAM . . . . . 19 3.2 Center of Mass Finder . . . . . . 19 3.3 Hand Shape . . . . . . . 20 3.4 Gesture Interpreter . . . . . . 20 3.5 Hand Calibration . . . . . . . 21 3.6 Determining Parameters . . . . . . 21 3.7 Display . . . . . . . . 21 3.8 Putting It Together . . . . . . 22 4 Conclusions . . . . . . . . 24 References . . . . . . . . . 25 Appendices . . . . . . . . . 26 A Screenshots of windows manager display . . . . 26 B Matlab script for generating .coe files . . . . 28 iiList of Figures Figure 1.1 Photo of project setup . . . . . 1 Figure 1.2 Hex display if switch[4] is off . . . . 2 Figure 1.3 Buttons and switches used for configuration . . . 2 Figure 1.4 Hex display if switch[4] is on . . . . 2 Figure 2.1 Block diagram of the video processor . . . 6 Figure 2.2 State transition diagram for the gesture interpreter module . 11 Figure 2.3 Diagram of horizontal and vertical sweep of XVGA . 13 Figure 2.4 Diagram of XVGA pixels . . . . . 13 Figure 2.5 Block diagram of windows manager . . . . 14 Figure 3.1 Screenshot of ModelSim simulation of hand_shape module . 20 List of Tables Table 1.1 Hand size select switches and associated hand gestures . 2 Table 1.2 Cursor color corresponds to hand gesture . . . 3 Table 2.1 Hand size select switches and associated hand gestures . 9 Table 2.2 Legend for state transition diagram . . . . 10 Table 2.3 FSM states, counters, and actions . . . . 12 Table 2.4 Sizes of BRAMs used to store images for XVGA display . 15 Table 2.5 Control signals for display . . . . . 16 Table 2.6 Effect of action on windows properties . . . 17 iii1 Windows Manager Overview 1.1 Introduction For this project, a windows manager that is responsive to hand movements in front of the screen was designed and implemented. The idea originated from the major motion picture, Minority Report, in which Tom Cruise manipulated windows on a screen simply by moving his hands. Our project emulates this concept by recording the movements of the user’s hands with an NTSC camera and translating these movements into commands that change windows displayed on a 1024x768 XVGA screen. 1.2 Setup The camera faces the computer screen. The user manipulates windows with his or hand between the camera and the screen. A picture of the setup can be seen in Figure 1.1. Figure 1.1 Setup – hand between camera and computer screen. 1.3 Configuration To use the windows manager, the user’s hand sizes must first be configured with the system. In order for the windows manager to work, the user must be wearing a black glove. Bright lighting also helps because dark shadows interfere with the video processing. The hand gestures are determined based on hand sizes so calibration of the program to the user’s hand is important. 1There are four hand gestures to calibrate, open, move, minimize, and close. The user’s hand should be in a fist shape for all four actions. The four actions are differentiated by the distance between the hand and the camera. The open gesture should be furthest away from the camera and the close gesture the closest. The four gestures should be a significant distance apart (~ 1.5 inches). Otherwise the windows manager will be difficult to use. To calibrate a hand shape, hold the hand in the desired position and press the calibrate button for a few seconds. Then program this area into the windows manager by using two switches to select the gesture and pressing the reprogram button. This should be repeated for all four gestures. The hand area and area to be reprogrammed into the windows manager are both displayed on the hex display if switch[4] is off. The user can then check if the area to be programmed is the desired area. See Figure 1.2 for a labeled diagram of the hex display. A table of the switches and the corresponding gestures is shown in Table 1.1. Figure 1.3 shows a diagram of the buttons and switches used for configuration. Figure 1.2 Hex display …
View Full Document
Unlocking...