Natural Scene Recognition: From Humans to ComputersSlide Number 2Slide Number 3Slide Number 4Slide Number 5Slide Number 6Slide Number 7Slide Number 8Slide Number 9Slide Number 10Slide Number 11Slide Number 12Slide Number 13A feed-forward mechanism?Slide Number 15Slide Number 16Slide Number 17Slide Number 18Slide Number 19Slide Number 20Slide Number 21Slide Number 22Slide Number 23Slide Number 24Slide Number 25Slide Number 26Slide Number 27Slide Number 28Compare to seemingly simpler tasksCompare to seemingly simpler tasksCompare to seemingly simpler tasksCompare to seemingly simpler tasksSlide Number 33Slide Number 34Slide Number 35Slide Number 36Slide Number 37Slide Number 38Slide Number 39Slide Number 40Slide Number 41Slide Number 42Slide Number 43Slide Number 44Rapid Perception of Natural ScenesScene CategoriesSlide Number 47Slide Number 48Behavioral PerformanceSlide Number 50Slide Number 51Pattern RecognitionPattern RecognitionExperimental Setup (fMRI)Voxel SelectionDecoding PerformanceDecoding PerformanceDecoding PerformanceDecoding PerformanceDecoding PerformanceRetinotopic AreasRetinotopic Areas ExcludedPlace Network (PPA + RSC)Slide Number 64Slide Number 65Scene inversion effectScene inversion effectScene inversion effectSlide Number 69Slide Number 70Slide Number 71Slide Number 72Slide Number 731.Feature detection and representation1.Feature detection and representationSlide Number 76Slide Number 77Slide Number 78Analogy to documentsSlide Number 80Slide Number 81Slide Number 82Slide Number 83Slide Number 84Slide Number 85Slide Number 86Slide Number 87Slide Number 88Slide Number 89Slide Number 90Slide Number 91Slide Number 92Slide Number 93Slide Number 94model distance based on topic distributionSlide Number 96Slide Number 97Slide Number 98Slide Number 99Slide Number 100Slide Number 101Slide Number 102Slide Number 103Slide Number 104What’s in a glance?What’s in a glance?Scene levelObject level(Social) Events What, where and who? Classifying events by scene and object recognitionSlide Number 111Slide Number 112Slide Number 113Slide Number 114Slide Number 115Slide Number 116Thank you!Natural Scene Recognition:Natural Scene Recognition: From Humans to ComputersFrom Humans to ComputersLi Fei-Fei1. Computer Science Department 2. Psychology DepartmentA picture is worth a thousand words.--- Confuciusor Printers’ Ink Ad (1921)white and redtextured structurebluegreenruggedelongated shapesbrightgreyporousmonumentbuildingscloudy skytreesmountainspeopleday timewalkingstreetvendors• To understand human visual intelligence by via psychophysical and physiological experiments• To build intelligent visual algorithms for machines and robotsbeachliving roomcityS. Lumet, 1965Potter, Biederman, etc. 1970sBiederman, Science, 1973Biederman, Science, 1973Thorpe, et al. Nature, 1996Thorpe, et al. Nature, 1996150 ms !!150 ms !!150 ms !!150 ms !!Delorme, et al. 1998A feed-forward mechanism?Thorpe, et al. Science, 2001FeatureIntegrationTheoryTreisman et al. 1980Visual Search: find the green-vertical barConjunction of featuresSingle feature Reaction Time # of distractorsTreeRECOGNITIONattentionalmechanismattentional loadmorelessTTTTLTTTTLTTTTLTTTTLTime (ms)0200100 300CentralPeripheralTTTTLFFFFFPeripheral categorization perf. (%)Central discrimination perf. (%)50~75~75interferencePeripheral categorization perf. (%)Central discrimination perf. (%)50~75~75interferencePeripheral categorization perf. (%)Central discrimination perf. (%)50~75~75No interferenceFei-Fei et al. PNAS, 2002Compare to seemingly simpler tasksTTTTLTTTTLCompare to seemingly simpler tasksTTTTLCompare to seemingly simpler tasksTTTTLCompare to seemingly simpler tasksAre animals special?Without color…Fei-Fei et al. Vis. Cog., 2004TTTTLTTTTLTTTTLFei-Fei et al. Vis. Cog., 2004Effect of “meaningful” categoryattentional loadmorelessF.I.T. predicted…Our data shows…attentional loadmorelessLi et al. 2002; Fei-Fei et al. 2005Rapid Perception of Natural ScenesThorpe, et al. Science, 2001-- Where/how does this happen?Where/how does this happen?Scene Categories500 ms 32-45 ms 500 ms < 2000 ms500 ms 32-45 ms 500 ms < 2000 ms6 AFC, N = 4, error bars: s.e.m.Behavioral PerformanceSubjects’ responseViewed image category (ground truth)chance: 0.167PPA: Parahippocampal Place AreaEpstein & Kanwisher, 1998Pattern RecognitionGuess“beach”Testing Image Training ImagesStatisticalPattern RecognitionAlgorithmTrainSelect voxelsTestSelect voxelsPattern RecognitionStatisticalPattern RecognitionAlgorithmSupport Vector Machine (SVM)Gaussian Naïve Bayes (GNB) Neural NetworksExperimental Setup (fMRI)• Passive viewing• 6 blocks per run (all 6 categories)• 12 runs for each subject• Alternating runs feature upright or inverted imagesVoxel SelectionUnivariate Multiple RegressionF-statisticbeachesforestshighwaysbuildingsindustrymountainsN = 4 error bars: s.e.m.Decoding PerformanceDecoding PerformanceN = 4 error bars: s.e.m.Decoding PerformanceDecoding PerformanceClassifier predictionViewed image category (ground truth)chance: 0.167N = 4 error bars: s.e.m.Decoding PerformanceN = 4 error bars: s.e.m.Retinotopic AreasN = 4 error bars: s.e.m.Retinotopic Areas ExcludedN = 4 error bars: s.e.m.Place Network (PPA + RSC)500 ms 32-45 ms 500 ms < 2000 ms500 ms 32-45 ms 500 ms< 2000 msUpright imagesInverted imagesTraining: upright only; Testing: upright & inverted blocks intermixedScene inversion effectN = 4*Scene inversion effectN = 4*Scene inversion effectN = 4***Fei-Fei & Perona, CVPR 2005categorycategorydecisiondecisionlearninglearningfeature detection& representationcodewords dictionarycodewords dictionaryimage representationcategory modelscategory models(and/or) classifiers(and/or) classifiersrecognitionrecognitionRepresentationRepresentationfeature detection& representation1.1.codewords dictionarycodewords dictionary2.2.image representation3.3.1.Feature detection 1.Feature detection and representationand representationextract interest points• DoG• Saliency detector (Kadir and Brady)• grid1.Feature 1.Feature detection anddetection and representationrepresentationrepresent interest points• SIFT (Lowe ’99)• gray scale values2. Codewords dictionary formation2. Codewords dictionary formation3. Image representation3. Image representation…..frequencycodewords3. Image representation3. Image representation…..frequencycodewordsAnalogy to documentsAnalogy to documentsOf all the sensory impressions proceeding to