Slide 1Last LectureNext three lecturesToday: Topic Models for RecognitionSlide 5How many object categories are there?Slide 7DatasetsDatasetsSize of existing datasetsSlide 1114,847 categories, 9,349,136 imagesSlide 13Slide 14How much supervision do you need to learn models of objects?Object label + segmentationObject appears somewhere in the imageImage + text captionImages onlySlide 20Analogy: Discovering topics in text collectionsVisual analogySystem overviewBag of wordsLow-rank matrix factorizationLatent Dirichlet Allocation (LDA)Latent Dirichlet Allocation (LDA)InferenceApply to Caltech 4 + background imagesSlide 31Slide 32Most likely words given topicMost likely words given topicPolysemySlide 36Image clusteringComparison with supervised modelImage as a mixture of topics (objects)Slide 41Slide 42Summary -- SivicLearning Object Categories from contaminated dataSlide 45Slide 46Slide 47Slide 48Proposing bounding boxesComparison between pLSA modelsTraining pLSA models from Google imagesGoogle’s variable search performancePicking the best topicOverall learning schemeMotorbike – pLSAMotorbike – TSI-pLSACar Rear – TSI-pLSASlide 61a chicken and egg problem…Slide 63FrameworkFrameworkNonparametric topic model -Hierarchical Dirichlet Process (HDP)Nonparametric topic model -Hierarchical Dirichlet Process (HDP)ClassificationAnnotationPitfall #1: model driftSlide 71The “cache set”Slide 73ResultSlide 75Slide 76Animals on the WebSlide 78Slide 79Flickr Search - monkey“Animals on the Web” ResultsGeneral ApproachSlide 83Museum and Library CollectionsWeb CollectionsVideoConsumer ProductsPrevious Work - Words & PicturesSlide 89Slide 90Slide 91Select ExemplarsSlide 93Slide 94Slide 97Slide 98Slide 99Slide 100Slide 101Slide 102Summary - BergSlide 104Slide 105Image Sense DisambiguationText contextsLatent Dirichlet allocation (LDA) (Blei et al. ‘03)Latent TopicsWeb Image Sense DictiOnary ModelWISDOM: Using dictionary entries to ground sensesWISDOM: Probabilistic dictionary-based modelEvaluation datasetsISD example resultsISD Results: ROC using each WordNet sense for BASSWISDOM: Removing Abstract SensesResults: Filtering visual sensesResults: Filtering visual sensesLecture SummaryNext LecturesBibliographyAdditional readingSlide 123C280, Computer VisionProf. Trevor [email protected] 13: Topic Models for RecognitionLast Lecture•Scanning window paradigm•GIST•HOG•Boosted Face Detection•Local-feature Alignment; from Roberts to Lowe...•BOW Indexing•Today: learning object categories from the web–LSA and LDA models–Harvesting training data from the web–Exploiting image and text•Tues. Oct. 20th: Generative models–Condensation–ISM–Transformed-HDPs–More Context…•Thurs. Oct. 22nd: Advanced BOW kernels–Pyramid and spatial-pyramid match–Multi-kernel learning–Latent-part SVM modelsNext three lecturesToday: Topic Models for RecognitionGuest lecture by Kate Saenko:•Dataset issues •Topic models for category discovery [Sivic05]•Category discovery from web [Fergus05]•Bootstrapping a category model [Li07]•Using text in addition to image [Berg06]•Learning objects from a dictionary [Saenko08]Topic Models for Object Recognitionslide credit Fei-Fei et. al.How many object categories are there?~10,000 to 30,000Biederman 1987slide credit Fei-Fei et. al.Dataset Issues In Object RecognitionDatasetsCaltech101/256[Fei-Fei et al, 2004][Griffin et al, 2007]PASCAL[Everingham et al, 2009]MSRC[Shotton et al. 2006]slide credit Fei-Fei et. al.ESP[Ahn et al, 2006]LabelMe[ Russell et al, 2005] TinyImageTorralba et al. 2007Lotus Hill[ Yao et al, 2007] Datasetsslide credit Fei-Fei et. al.Size of existing datasetsDatasets # of categories# of imagesper category# of total imagesCollected byCaltech101 101 ~100 ~10K HumanLotus Hill ~300 ~ 500 ~150K HumanLabelMe 183 ~200 ~30K HumanIdeal ~30K >>10^2 A LOT Machineslide credit Fei-Fei et. al.•An ontology of images based on WordNet•Collected using Amazon Mechanical Turk~105+ nodes~108+ imagesshepherd dog, sheep dogGerman shepherdcollieanimalDeng, Dong, Socher, Li & Fei-Fei, CVPR 2009slide credit Fei-Fei et. al.Deng, Wei, Socher, Li, Li, Fei-Fei, CVPR 200914,847 categories, 9,349,136 images•Animals –Fish–Bird–Mammal–Invertebrate•Scenes–Indoors–Geological formations•Sport Activities•Fabric Materials•Instrumentation–Tool–Appliances–…•Plants–…slide credit Fei-Fei et. al.Deng, Wei, Socher, Li, Li, Fei-Fei, CVPR 2009“Cycling”slide credit Fei-Fei et. al.Deng, Wei, Socher, Li, Li, Fei-Fei, CVPR 2009“Drawing room, withdrawing room”slide credit Fei-Fei et. al.How much supervision do you need to learn models of objects?Object label + segmentationAgarwal & Roth ’02, Leibe & Schiele ’03, Torralba et al. ’05LabelMe, PASCAL, TU Darmstadt, MIT scenes and objectsMIT+CMU frontal facesViola & Jones ’01Rowley et al. ’98Object appears somewhere in the imageFergus et al. ’03, Csurka et al. ’04, Dorko & Schmid ’05Caltech 101, PASCAL, MSRCairplanemotorbikefacecarImage + text captionCorel, Flickr, Names+faces, ESP gameBarnard et al. ’03, Berg et al. ’04Images onlyGiven a collection of unlabeled images, discover visual object categories and their segmentation• Which images contain the same object(s) ?• Where is the object in the image?Discovering Objects and Their Location in ImagesPresented at the International Conference on Computer Vision, 2005.J. Sivic, B. C. Russell, A. A. Efros, A. Zisserman, W. T. Freeman.Analogy: Discovering topics in text collectionsBlei, et al. 2003DiscoveredtopicsTextdocumentVisual analogydocumentwordtopics- image- visual word- objectsSystem overviewInput image Compute visual words Discover visual topicsBag of words• LDA model assumes exchangeability• Order of words does not matterStack visual word histogramsas columns in matrixThrow away spatial information!DictionaryHistogramVisual wordsInterest regions22044153518392110612341Low-rank matrix factorization• Latent Semantic Analysis (Deerwester, et al. 1990)• Probabilistic Latent Semantic Analysis (Hofmann 2001)Latent Dirichlet Allocation (LDA)Blei, et al. 2003wij- wordszij- topic assignments- topic mixing weightsθiφk- word mixing weightsLatent Dirichlet Allocation (LDA)Blei, et al. 2003wij- wordszij- topic assignments- topic mixing weightsθiφk- word mixing weightsInferencewij- wordszij- topic assignments- topic mixing weightsθiφk- word mixing weightsUse Gibbs sampler to