Human Detection and TrackingCS 543 - D.A. ForsythWhy are humans important?•Surveillance•prosecution; intelligence gathering; crime prevention•HCI; architecture;•Synthesis•games; movies;•Safety applications•pedestrian detection•People are interesting•movies; newsWhere you are can suggest you are doing something you shouldn’t beBoult 2001Why are humans important?•Surveillance•prosecution; intelligence gathering; crime prevention•HCI; architecture;•Synthesis•games; movies;•Safety applications•pedestrian detection•People are interesting•movies; newsArikan+Forsyth 02Why are humans important?•Surveillance•prosecution; intelligence gathering; crime prevention•HCI; architecture;•Synthesis•games; movies;•Safety applications•pedestrian detection•People are interesting•movies; newsFrom Dalal+Triggs, 05Why are humans important?•Surveillance•prosecution; intelligence gathering; crime prevention•HCI; architecture;•Synthesis•games; movies;•Safety applications•pedestrian detection•People are interesting•movies; newsPresident George W. Bush makes a statement in the Rose Garden while Secretary of Defense Donald Rumsfeld looks on, July 23, 2003. Rumsfeld said the United States would release graphic photographs of the dead sons of Saddam Hussein to prove they were killed by American troops. Photo by Larry Downing/Reuters News Faces• 5e5 captioned news images• Mainly people “in the wild”• Correspondence problem– some images have many (resp. few) faces, few (resp. many) names (cf. Srihari 95)• Process– Extract proper names– Detect faces (Vogelhuber Schmid 00)– Rectify faces– Kernel PCA rectified faces– Estimate linear discriminants– Now have (face vector; name_1,...., name_k)• Apply a form of modified k-means44773 big face responses34623 properly rectified27742 for k<=4US President George W. Bush (L) makes remarks while Secretary of State Colin Powell (R) listens before signing the US Leadership Against HIV /AIDS , Tuberculosis and Malaria Act of 2003 at the Department of State in Washington, DC. The five-year plan is designed to help prevent and treat AIDS, especially in more than a dozen African and Caribbean nations(AFP/Luke Frazza)German supermodel Claudia Schiffer gave birth to a baby boy by Caesarian section January 30, 2003, her spokeswoman said. The baby is the first child for both Schiffer, 32, and her husband, British film producer Matthew Vaughn, who was at her side for the birth. Schiffer is seen on the German television show 'Bet It...?!' ('Wetten Dass...?!') in Braunschweig, on January 26, 2002. (Alexandra Winkler/Reuters) British director Sam Mendes and his partner actress Kate Winslet arrive at the London premiere of 'The Road to Perdition', September 18, 2002. The films stars Tom Hanks as a Chicago hit man who has a separate family life and co-stars Paul Newman and Jude Law. REUTERS/Dan ChungTopics•Transducing the body•getting some representation out of picture/video•Inferring activity•going from that representation to a description of activityActivity by AppearancePoints•Quite simple features can reveal what people are doing•location•patterns of motion•“appearance”Naming activities•Absence of a canonical vocabulary is a serious problem•strategies•adopt specialized domains (Bobick+Davis 01, Efros et al 03)•guess a vocabulary (Efros et al 03)•match motion to motion and avoid the issue (Efros et al 03)•use vocab useful for synthesis (Ramanan et al 03)Bobick & Davis. PAMI01Surveillance: Where you are can tell what you are doingIntille et al 95, 97Laptev Perez 2007see also Laptev et al 08Movies and captions: Laptev et al 08DatasetsDiscriminative resultsWorks well, depending on task; not rejecting improves thingsmetric learning improves thingsTran + Sorokin 08Youtube videoTran + Sorokin 08IXMAS and AspectThe Effects of AspectFarhadi Kamali 08Motion transductionHuman tracking options•People as blobs (+appearance)•Grimson et al 98; Stauffer et al 00; Haritaoglu et al 98, 00; Okuma et al 04•People as motion fields•Bregler 97;Boyd+Little 98•People as blobs+motion fields•Efros et al 03•Kinematics•Hogg 83; Rohr 93; Deutscher et al 00; Toyama+Blake 02; SidenbladhBlackFleet 00; JuBlackYacoob 96; Song Perona 00; etcWhy is kinematic tracking hard?•It’s hard to detect people •until recently, human trackers were manually started•People move fast, and can move unpredictably•dynamics gives limited constraint on future configuration•appearance changes over time (shading, aspect, etc)•Some body parts are small and tend to have poor contrast•particularly difficult to track •lower arms (small, fast, look like other things); •upper arms (poor contrast)variation in appearancevariation in pose & aspectself-occlusion & clutterFigure from “Efficient Matching of Pictorial Structures,”P. Felzenszwalb and D.P. Huttenlocher, Proc. Computer Vision and Pattern Recognition2000, c 2000, IEEE as used in Forsyth+Ponce, pp 636, 640Build and detect modelstorsoar mle gheadlabelpixelslearnlimbclassifiersgeneralposepictorialstructureunusual posesmall scale"Lola" likelihoodRamanan, Forsyth and Zisserman CVPR05Ramanan, Forsyth and Zisserman CVPR05Ramanan, Forsyth and Zisserman CVPR05Ramanan, Forsyth and Zisserman CVPR05Lifting•Infer 3D configuration from image configuration•Useful for•view independent activity recognition•user interfaces•video motion captureTaylor, 00Ambiguity•Troubled question•lifts are ambiguous (Orthography; Sminchicescu+Triggs 03; etc)•but ambiguities •can be ignored•Taylor 00; Barron+Kakadiaris 00•can be dodged•Ramanan+Forsyth 03; Howe et al 00•Summary+musings in Forsyth etal 06Sminchisescu+Triggs, 03Animating peoplePoints•Some properties of motion, illustrated by animation•motion composes•across time•across the body•motion can be easy to annotate•but good from bad is hard•motion clusters wellCut and Paste works well over time•Motion graph: by analogy with •text synthesis, texture synthesis, video textures•Take measured frames of motion as nodes•from motion capture, given us by our friends•Edge from frame to any that could succeed it•decide by dynamical similarity criterion•see also (Kovar et al 02; Lee et al 02)•A path is a motion•Search with constraints•like root position+orientation, etc.•In various ways•Local (Kovar et al 02)•Lee et al 02;