DOC PREVIEW
UCSD CSE 252C - Formulating Semantic Image Annotation

This preview shows page 1-2-3-23-24-25-26-46-47-48 out of 48 pages.

Save
View full document
View full document
Premium Document
Do you want full access? Go Premium and unlock all 48 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 48 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 48 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 48 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 48 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 48 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 48 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 48 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 48 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 48 pages.
Access to all documents
Download any document
Ad free experience
Premium Document
Do you want full access? Go Premium and unlock all 48 pages.
Access to all documents
Download any document
Ad free experience

Unformatted text preview:

Formulating Semantic Image Annotation as a Supervised Learning Problem Gustavo Carneiro and Nuno Vasconcelos CVPR ‘05What is Image Annotation?What is Image Retrieval?Problem: Image Annotation & RetrievalSlide 5Slide 6Slide 7Slide 8Slide 9OutlineSlide 11Notation and Problem StatementSlide 13Slide 14Slide 15Slide 16Slide 17Supervised OVA ModelsSlide 19Slide 20Unsupervised ModelsSlide 22Slide 23Supervised M-ary ModelSlide 25Slide 26Density EstimationSlide 28Slide 29Slide 30Slide 31Slide 32Slide 33Experimental SetupSlide 35Other Annotation SystemsSlide 37New Annotation SystemsAnnotation ResultsSlide 40Slide 41Retrieval ResultsSlide 43Slide 44Automatic Music AnnotationSlide 46Slide 47Work Cited:Formulating Semantic Image Annotation as a Supervised Learning ProblemGustavo Carneiro and Nuno VasconcelosCVPR ‘05Presentation by:Douglas TurnbullCSE Department, UCSDTopic in Vision and LearningNovember 3, 2005What is Image Annotation?Given an image, what are the words that describe the image?What is Image Retrieval?Given a database of images and a query string (e.g. words), what are the images that are described by the words?Query String: “jet”Problem: Image Annotation & RetrievalBased on the low cost of both digital camera and hard disk space, billions of consumer have the ability create and store digital images.There are already billions of digital images stored on personal computers and in commercial databases.How do store images in and retrieve images from a large database?Problem: Image Annotation & RetrievalIn general, people do not spent time labeling, organizing or annotating their personal image collections.Label:•Images are often stored with the name that is produced by the digital camera:–“DSC002861.jpg”•When they are labeled, they are given a vague names that rarely describe the content of the image: –”GoodTimes.jpg”, “China05.txt” Organize:•No standard scheme exists for filing images•Individuals use ad hoc methods: “Chrismas2005Photos” and “Sailing_Photos”•It is hard to merge image collections since the taxonomies (e.g. directory hierarchies) differ from user to user.Problem: Image Annotation & RetrievalIn general, people do not spent time labeling, organizing or annotating their personal image collections.Annotate:•Explicit Annotation: Rarely do we explicitly annotate our images with captions. –An exception is when we are create web galleries •i.e. My wedding photos on www.KodakGallery.com•Implicit Annotation: Sometimes we do implicitly annotate images we imbed images into text (as is the case with webpages.)–Web-based search engines make use of the implicit annotation when they index images.•i.e. Google Image Search, PicsearchProblem: Image Annotation & RetrievalIf we can’t depend on human labeling, organization, or annotation, we will have to resort to “content-based image retrieval”:–We will extract features vectors from each image–Based on these feature vectors, we will use statistical models to characterize the relationship between a query and image features.How do we specify a meaningful query to be able to navigate this image feature space?Problem: Image Annotation & RetrievalContent-Based Image Retrieval: How do we specify a query?Query-by-sketch: Sketch a picture, extract features from the pictures, we the features to find similar images in the database.This requires that1. we have a good drawing interface handy 2. everybody is able to draw3. the quick sketch is able to capture the salient nature of the desired queryNot a very feasible approach.Problem: Image Annotation & RetrievalContent-Based Image Retrieval: How do we specify a query?Query-by-text: Input words into a statistical model that models models the relationship between words and image features.This requires that:1. A keyboard2. A statistical model that can relate words to image features3. Words can be used to capture the salient nature of the desired query.A number of research systems have been develop that find a relationship content-based image features and text for the purpose of image annotation and retrieval.- Mori, Takahashi, Oka (1999)- Daygulu, Barnard, de Freitas (2002)- Blei, Jordan (2003)- Feng, Manmantha, Lavrenko (2004)OutlineNotation and Problem StatementThree General Approaches to Image Annotation1. Supervised One vs. All (OVA) Models2. Unsupervised Models using Latent Variables3. Supervised M-ary ModelEstimating P(image features|words)Experimental Setup and ResultsAutomatic Music AnnotationOutlineNotation and Problem StatementThree General Approaches to Image Annotation1. Supervised One vs. All (OVA) Models2. Unsupervised Models using Latent Variables3. Supervised M-ary ModelEstimating P(image features|words)Experimental Setup and ResultsAutomatic Music AnnotationNotation and Problem StatementNotation and Problem Statementxi = vector of image featuresx = {x1, x2 , … } wi = one wordw = {w1, w2 , … }= vector of feature vectors= vector of wordsImage and Caption Image RegionsNotation and Problem StatementNotation and Problem Statement-Notation and Problem StatementImage RegionsMultiple Instance Learning: this regions has no visual aspect of “jet”Weak Labeling: this image depict sky eventhough the caption does contain “sky”OutlineNotation and Problem StatementThree General Approaches to Image Annotation1. Supervised One vs. All (OVA) Models2. Unsupervised Models using Latent Variables3. Supervised M-ary ModelEstimating P(image features|words)Experimental Setup and ResultsAutomatic Music AnnotationSupervised OVA ModelsEarly research posed the problem as a supervised learning problem: train a classifier for each semantic concept.Binary Classification/Detection Problems:•Holistic Concepts: landscape/cityscape, indoor/outdoor scenes•Object Detection: horses, buildings, trees, etcMuch of the early work focused on feature design and used existing models developed by the machine learning community (SVM, KNN, etc) for classification.Supervised OVA ModelsSupervised OVA ModelsPro:• Easy to implement• Can design features and tune learning algorithm for each classification task• Notion of optimal performance on each task• Data sets represent basis of comparison – OCR data set Con:•Doesn’t scale well with a large vocabulary• Requires train and use L classifier• Hard to compare posterior probabilities output by L classifier• No natural ranking of keywords. • Weak labeling is a problem: • Images not


View Full Document

UCSD CSE 252C - Formulating Semantic Image Annotation

Download Formulating Semantic Image Annotation
Our administrator received your request to download this document. We will send you the file to your email shortly.
Loading Unlocking...
Login

Join to view Formulating Semantic Image Annotation and access 3M+ class-specific study document.

or
We will never post anything without your permission.
Don't have an account?
Sign Up

Join to view Formulating Semantic Image Annotation 2 2 and access 3M+ class-specific study document.

or

By creating an account you agree to our Privacy Policy and Terms Of Use

Already a member?