Vision and visual neuroscience IIThomas Serre & Tomaso PoggioMcGovern Institute for Brain ResearchCenter for Biological and Computational LearningDepartment of Brain & Cognitive Sciences9.520Last class✦Problem of visual recognition✦Historical background✦Neurons and areas in the visual system✦Data and hierarchical feedforward modelsmodified from (Ungerleider and Haxby 1994)modified from (Ungerleider and Haxby 1994)LGNmodified from (Ungerleider and Haxby 1994)modified from (Ungerleider and Haxby 1994)(Hubel & Wiesel 1959)V1modified from (Ungerleider and Haxby 1994)modified from (Ungerleider and Haxby 1994)(Kobatake and Tanaka, 1994)V4modified from (Ungerleider and Haxby 1994)modified from (Ungerleider and Haxby 1994)IT(Desimone, 1984)Rapid categorization(Biederman 1972; Potter 1975; Thorpe 1996)Animalvs.non-animalComplex cellsTuning Simple cellsMAXMain routes Bypass routes PG CortexRostral STSPrefrontalCortexSTPDPVIPLIP7aPPFSTPOV3AMTTPOPGaIPaV3V4PITTFTG3635LIP,VIP,DP,7aV2,V3,V4,MT,MSTPIT, AITAIT,36,35MSTc}V1PGTE46845 1211,13TEaTEmAITV2V1dorsal stream'where' pathwayventral stream'what' pathwayMSTpC1S1S2S3S2bC2classificationunits0.2 - 1.1o0.4 - 1.6o0.6 - 2.4o1.1 - 3.0o0.9 - 4.4o1.2 - 3.2oooooooModel layersRF sizesS47oNum.unitsC2b7oC37o106104107105104107100102103103Increase in complexity (number of subunits), RF size and invarianceUnsupervised task-independent learningSupervised task-dependent learning*Modified from (Gross, 1998)(Riesenhuber & Poggio 1999 2000; Serre Kouh Cadieu Knoblich Kreiman & Poggio 2005; Serre Oliva & Poggio 2007)Receptive field sizesModel Cortex Referencessimple cells 0.2o− 1.1o≈ 0.1o− 1.0o[Schiller et al., 1976e;Hubel and Wiesel, 1965]complex cells 0.4o− 1.6o≈ 0.2o− 2.0oPeak frequencies (cycles /deg)Model Cortex Referencessimple cells range: 1.6 − 9.8 bulk ≈ 1.0 − 4.0 [DeValois et al., 1982a])mean/med: 3.7/ 2.8 mean: ≈ 2.2range: ≈ 0.5 − 8.0complex cells range: 1.8 − 7.8 bulk ≈ 2.0 − 5.6mean/med: 3.9/ 3.2 mean: 3.2range ≈ 0.5 − 8.0Frequency bandwidth at 50% amplitude (cycles / deg)Model Cortex Referencessimple cells range: 1.1 − 1.8 bulk ≈ 1.0 − 1.5 [DeValois et al., 1982a]med: ≈ 1.45 med: ≈ 1.45range ≈ 0.4 − 2.6complex cells range: 1.5 − 2.0 bulk ≈ 1.0 − 2.0med: 1.6 med: 1.6range ≈ 0.4 − 2.6Frequency bandwidth at 71% amplitude (index)Model Cortex Referencessimple cells range: 44 − 58 bulk ≈ 40 − 70 [Schiller et al., 1976d]med: 55complex cells range 40 − 50 bulk ≈ 40 − 60med. 48Orientation bandwidth at 50% amplitude (octaves)Model Cortex Referencessimple cells range: 38o− 49o— [DeValois et al., 1982b ]med: 44ocomplex cells range: 27o− 33obulk ≈ 20o− 90omed: 43omed: 44oOrientation bandwidth at 71% amplitude (octaves)Model Cortex Referencessimple cells range: 27o− 33obulk ≈ 20o− 70o[Schiller et al., 1976c]med: 30ocomplex cells range: 27o− 33obulk ≈ 20o− 90omed: 31o(Serre & Riesenhuber 2004)Example: V150 0 5000.20.40.60.81orientation (in degree)responseoptimal baredgegratingThis classThis class✦Feedforward hierarchical models of the visual cortexThis class✦Feedforward hierarchical models of the visual cortex★Detailed implementation + learningThis class✦Feedforward hierarchical models of the visual cortex★Detailed implementation + learning★Comparison w| neural dataThis class✦Feedforward hierarchical models of the visual cortex★Detailed implementation + learning★Comparison w| neural data ★Agreement with psychophysicsThis class✦Feedforward hierarchical models of the visual cortex★Detailed implementation + learning★Comparison w| neural data ★Agreement with psychophysics★Application to computer visionThis class✦Feedforward hierarchical models of the visual cortex★Detailed implementation + learning★Comparison w| neural data ★Agreement with psychophysics★Application to computer vision✦ Beyond (static) feedforward processingThis class✦Feedforward hierarchical models of the visual cortex★Detailed implementation + learning★Comparison w| neural data ★Agreement with psychophysics★Application to computer vision✦ Beyond (static) feedforward processing★Extension to action recognition in the dorsal streamThis class✦Feedforward hierarchical models of the visual cortex★Detailed implementation + learning★Comparison w| neural data ★Agreement with psychophysics★Application to computer vision✦ Beyond (static) feedforward processing★Extension to action recognition in the dorsal stream ★Attention and cortical feedbacksAnimalvs.non-animalComplex cellsTuning Simple cellsMAXMain routes Bypass routes PG CortexRostral STSPrefrontalCortexSTPDPVIPLIP7aPPFSTPOV3AMTTPOPGaIPaV3V4PITTFTG3635LIP,VIP,DP,7aV2,V3,V4,MT,MSTPIT, AITAIT,36,35MSTc}V1PGTE46845 1211,13TEaTEmAITV2V1dorsal stream'where' pathwayventral stream'what' pathwayMSTpC1S1S2S3S2bC2classificationunits0.2 - 1.1o0.4 - 1.6o0.6 - 2.4o1.1 - 3.0o0.9 - 4.4o1.2 - 3.2oooooooModel layersRF sizesS47oNum.unitsC2b7oC37o106104107105104107100102103103Increase in complexity (number of subunits), RF size and invarianceUnsupervised task-independent learningSupervised task-dependent learning*Modified from (Gross, 1998)(Riesenhuber & Poggio 1999 2000; Serre Kouh Cadieu Knoblich Kreiman & Poggio 2005; Serre Oliva & Poggio 2007)✦Gabor filters✦Parameters fit to V1 data (Serre & Riesenhuber 2004)•17 spatial frequencies (=scales)•4 orientationsAnimalvs.non-animalC1S1S2S3S2bC2classif.unitsS4C2bC3S1 units50 0 5000.20.40.60.81orientation (in degree)responseoptimal baredgegratingAnimalvs.non-animalC1S1S2S3S2bC2classif.unitsS4C2bC3C1 unitsIncrease in tolerance to position (and in RF size)Local max over pool of S1 cellsC1S1Animalvs.non-animalC1S1S2S3S2bC2classif.unitsS4C2bC3C1 unitsIncrease in tolerance to scaleC1Local max over pool of S1 cellsAnimalvs.non-animalC1S1S2S3S2bC2classif.unitsS4C2bC3S2 units✦Features of moderate complexity (n~1,000 types)✦Combination of V1-like complex units at different orientations•Synaptic weights w learned from natural images•5-10 subunits chosen at random from all possible afferents (~100-1,000)stronger facilitationstronger suppressionAnimalvs.non-animalC1S1S2S3S2bC2classif.unitsS4C2bC3C2 units✦Same selectivity as S2 units but increased tolerance to position and size of preferred stimulus✦Local pooling over S2 units with same selectivity but slightly different positions and scales✦S2 units in V2 and C2 in V4?(Hubel &