ECE160 Spring 2011 Lecture 15 1 ECE160 Multimedia Lecture 15: Spring 2011 Image Recognition and RetrievalECE160 Spring 2009 Lecture 15 Image Recognition 2 National Research Priorities  Energy Technologies  Fuel efficient engines  Replacement energy to fossil fuels  Lighter, longer-duration batteries  Bioengineering/Bioinformatics  Genes  disease  Disease  medicine  Search with Multimedia Content  Video surveillance  Photo interpretationECE160 Spring 2009 Lecture 15 Image Recognition 3 Audio Search  Much video already has subtitles  Speech recognition  Then use text searchECE160 Spring 2009 Lecture 15 Image Recognition 4 Multimedia Recognition  Video surveillance  Photo interpretation  Search of photo and video archivesECE160 Spring 2009 Lecture 15 Image Recognition 5 Wide-area Surveillance advertisement of objectvideo.comECE160 Spring 2009 Lecture 15 Image Recognition 6 Surveillance Scenarios (1) Intrusion Detection (2) Passenger Screening (3) Perimeter Monitoring Z Z Use biometric facial recognition to identify individuals of interest through existing closed circuit TV surveillance Monitor and alert on tailgating, loitering, exit/closed entry, other unauthorized access Object tracking and biometric facial recognition to determine vehicles and humans exhibiting suspicious behavior (4) Unattended Baggage Copyright © 2004 Proximex Corp. Identify unattended baggage (or other objects) left for long periods of timeECE160 Spring 2009 Lecture 15 Image Recognition 7 Surveillance in London  45,000+ television cameras in the street  Images recorded for subsequent analysis  Sophisticated software to track a suspect from one camera to the next  Matching of track of suspect to mobile phone records to identify suspectECE160 Spring 2009 Lecture 15 Image Recognition 8 Multimedia Recognition  Video surveillance  Photo interpretation  Search of photo and video archivesECE160 Spring 2009 Lecture 15 Image Recognition 9 How to Organize these Photos?ECE160 Spring 2009 Lecture 15 Image Recognition 10  Keyword-based  Manual labeling is subjective, cumbersome  The aliasing problem  Content-based  Promising for general semantics: outdoor, landscape, flowers, people, etc.  Not enough for wh-queries (where, who, when, or what) Image Organization & RetrievalECE160 Spring 2009 Lecture 15 Image Recognition 11 EXTENTTM = contEXT + contENT  Context  Spatial (location)  Temporal  Social  Others  Content  Perceptual features, such as color, texture, and shape  Holistic features and local featuresECE160 Spring 2009 Lecture 15 Image Recognition 12 EXTENTTM Spatial Temporal Social OthersECE160 Spring 2009 Lecture 15 Image Recognition 13 Augmented Images + Cameraphones with high-quality lens can record location, time, camera parameters, and voice =ECE160 Spring 2009 Lecture 15 Image Recognition 14 Context from Space/Time  GPS or CellID data  Into place names  Time-based grouping  Into meaningful “events”  From place names and time  Time of day  WeatherECE160 Spring 2009 Lecture 15 Image Recognition 15 Example of Using Three Pieces of Information Spatial TemporalECE160 Spring 2009 Lecture 15 Image Recognition 16 Maui Sunsets can be obtained from Space/TimeECE160 Spring 2009 Lecture 15 Image Recognition 17 Use content for verificationECE160 Spring 2009 Lecture 15 Image Recognition 18 Use content to transfer metadataECE160 Spring 2009 Lecture 15 Image Recognition 19 Summarize of the example  Derived from Context  Derive time of the day  Obtain weather  Verify content  Use of Content  Verify context  Transfer context  Much more…ECE160 Spring 2009 Lecture 15 Image Recognition 20 Are They Similar?ECE160 Spring 2009 Lecture 15 Image Recognition 21 Are They Similar?ECE160 Spring 2009 Lecture 15 Image Recognition 22 Are They Similar?  In terms of what?  What is the user’s perception?ECE160 Spring 2009 Lecture 15 Image Recognition 23 Conveying Perception  Image Databases  Conveyed via Examples  Use a sunset picture (or pictures) to find more sunset images  Where does the perfect example come from?ECE160 Spring 2009 Lecture 15 Image Recognition 24 Conveying Perception  Internet Searches  Conveyed via KeywordsECE160 Spring 2009 Lecture 15 Image Recognition 25 Keyword Retrieval  Pros  A user-friendly paradigm  Cons  Annotation is a laborious process  Annotation quality can be subpar  Annotation can be subjective  SynonymsECE160 Spring 2009 Lecture 15 Image Recognition 26 Conveying Perception  Image Databases  Conveyed via Examples  Use a sunset picture (or pictures) to find more sunset images  Where does the perfect example come from?ECE160 Spring 2009 Lecture 15 Image Recognition 27 Are They Similar?ECE160 Spring 2009 Lecture 15 Image Recognition 28 Are They Similar?ECE160 Spring 2009 Lecture 15 Image Recognition 29 Are They Similar?  In terms of what?  What is the user’s perception?ECE160 Spring 2009 Lecture 15 Image Recognition 30 Recogintion of ContentECE160 Spring 2009 Lecture 15 Image Recognition 31 RecognitionECE160 Spring 2009 Lecture 15 Image Recognition 32 RecognitionECE160 Spring 2009 Lecture 15 Image Recognition 33 clouds vs. wavesECE160 Spring 2009 Lecture 15 Image Recognition 34 Web 1.0 vs. Web 2.0 User User User Content Content (Image/Video) Content Content Content Content Content Content (text) Content Content Content Content Content User User User UserECE160 Spring 2009 Lecture 15 Image Recognition 35 Web 2.0  Content + Users + Interactions  Collect rich, organized content  Attract users & interactions  To provide metadata  To provide new content  Improve search quality  With new metadata and data  Via social-network structureECE160 Spring 2009 Lecture 15 Image Recognition 36 User management User management Photo uploading Single/multiple Upload wizard Photo uploading Building social network Social networks Event management Event management Photo search Photo search Metadata collection Metadata fusion Image annotation Metadata collection - contextual - content Metadata fusion Annotate photos External functionalities Internal functionalities