DOC PREVIEW
CMU CS 10601 - fMRIpredict_4_2_2008

This preview shows page 1-2-3-24-25-26 out of 26 pages.

Save
View full document
View full document
Premium Document
Do you want full access? Go Premium and unlock all 26 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 26 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 26 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 26 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 26 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 26 pages.
Access to all documents
Download any document
Ad free experience
Premium Document
Do you want full access? Go Premium and unlock all 26 pages.
Access to all documents
Download any document
Ad free experience

Unformatted text preview:

Brains Meaning and Machine LearningBrains, Meaning and Machine LearningTom Mitchelland many collaborators Machine Learning DepartmentCarnegie Mellon UniversityApril, 2008Neurosemantics Research TeamProfessional StaffPostdoctoral FellowsMarcel JustTom MitchellVladimir Cherkassky PhD StudentsSvetlana ShinkarevaRob MasonRebecca HutchinsonKai Min Chang Mark Palatucci Indra RustandiAndy CarlsonFrancisco PereirafMRI activation for “bottle”:fMRI activationhighMean activation averaged over 60 different stimuli:average“b ttl ” i ti tibelow average“bottle” minus mean activation:average60 exemplarsCategories ExemplarsBODY PARTS leg arm eye foot handFURNITURE chair table bed desk dresserVEHICLES car airplane train truck bicycleANIMALS horse dog bear cow catKITCHEN UTENSILS glass knife bottle cup spoonTOOLS chisel hammer screwdriver pliers sawBUILDINGS apartment barn house church iglooPART OF A BUILDINGwindowdoorchimneyclosetarchBUILDINGwindowdoorchimneyclosetarchCLOTHING coat dress shirt skirt pantsINSECTS fly ant bee butterfly beetleVEGETABLESlettucetomatocarrotcornceleryVEGETABLESlettucetomatocarrotcornceleryMAN MADE OBJECTS refrigerator key telephone watch bellQti0Question 0:Using fMRI, can we observe brain g,activation representing the meaning of input stimulus? Can we train a classifier to decode the semantic category of stimulus?semantic category of stimulus?Answer:Answer:Yes, for categories such as “tools,” “buildings,” “foods”“body parts”“vehicles”etcfoods, body parts, vehicles, etc.Classification task: is person viewing a “tool” or “building”?1070.80.91uracystatistically significant p<0.05040.50.60.7tion acc010.20.30.4lassificap4 p8 p6 p11 p5 p7 p10 p9 p2 p12 p3 p100.1ParticipantsCParticipantsQuestion 1: Is brain activity common across stimulus modality?stimulus modality? Can we train on word stimuli, then decode picture stimuli?YES: Train on words, then:060.70.80.91accuracy060.70.80.91accuracyTest on words Test on pictures p11 p4 p5 p2 p6 p10 p8 p7 p9 p1 p12 p300.10.20.30.40.50.6Ptii tClassification ap4 p6 p11 p2 p7 p1 p10 p8 p5 p12 p3 p900.10.20.30.40.50.6Ptii tClassification aTherefore, the learned neural activation patterns must ParticipantsParticipantscapture how the brain represents stimulus meaningQuestion 2: Are representations similar across different people?different people? Can we train classifier on data from a collection of people, then ddtilif ?YES: Train on one group of people, and classify fMRI images of new persondecode stimuli for a new person?new personclassify which of 60 itemsTherefore, we can seek a theory of neural representation common to all of us (and of how we vary)What we really want: a generative theoryGenerative theoryfdarbitrary wordpredictedof word representationarbitrary wordpredicted brain activityBut how??Question 3: Can we develop a theory to predict neural encoding for any word?encoding for any word? Pditi tti l dlPredictive computational modelpredictedpredicted activity for “apple”“apple”Statistical features from a trillion-word text corpusMapping learned from fMRI dataApproach: Integrate corpus data and fMRI datadata and fMRI data• Semantic feature i = English word i (e.g., touch)• Value of feature i = co-occurrence frequency of stimulus with i–in trillion-word text collection (tera-word ngram database provided by Google)(gpyg)• Which semantic features? First attempt: 25 sensory/action verbs:–Sensory actions:see hear listen taste touch smell fear–Sensory actions: see, hear, listen, taste, touch, smell, fear, – Motor actions: rub, lift, manipulate, run, push, move, say, eat, – Abstract actions: fill, open, ride, approach, near, enter, drive, wear, break,cleanbreak,clean(why these 25?)(why these 25?)Semantic feature values:“celery”Semantic feature values:“airplane”Semantic feature values: celery0.8368, eat 0.3461, taste0.3153, fillSemantic feature values: airplane0.8673, ride0.2891, see0 2851 say0.3153, fill0.2430, see 0.1145, clean0.0600, open0.2851, say0.1689, near 0.1228, open0.0883, hear0.0586, smell0.0286, touch…0.0883, hear0.0771, run0.0749, lift……0.0000, drive0.0000, wear0 0000 lift…0.0049, smell0.0010, wear0.0000, lift0.0000, break0.0000, ride0.0000, taste0.0000, rub0.0000, manipulatePredicted Activation is Sum of Feature Contributions“t”“t t ”“fill”Predicted Celery =+ 0.350.84“eat”“taste”+ 0.32+ …“fill”c14382,eatfeat(celery)from corpus statisticslearnedstatistics25high∑==1 )(iviivcwfpredictionlowPredicted “Celery”“celery” “airplane”Predicted:fMRI activationhighaverageObserved:below averageaveragePredicted and observed fMRI images for “celery” and “airplane” after training on 58 other words.Evaluating the Computational Model• Train it using 58 of the 60 word stimuli•Apply it to predictfMRIimages for other 2 wordsApply it to predict fMRIimages for other 2 words• Test: show it the observed images for the 2 held-out, and make it predict which is which• Image similarity measured by cosine similarity using only the 500 most“stable”voxels(over training set)only the 500 most stable voxels(over training set)•1770 test pairs in leave-2-out1770 test pairs in leave2out– Random guessing Æ 0.50 prediction accuracy– Accuracy above 0.61 is significant (p<0.05) according empirical distributiondistributionAccuracy predicting images for new words• Accuracy of independently trained models for nine participants (using 500 most stable voxels over the 58 training words):training words):– .85, .83, .82, .78, .78, .76, .73, .72, .68– Mean: .77• Accuracy extrapolating to new categories (when testing on“celery”vs“airplane”leave outall trainingfoods andon celery vsairplane, leave out all training foods and vehicles)– .78, .78, .74, .69, .69, . 68, .67, .64, .64M70–Mean: .70From Q53, 500 most replicable voxelsAccuracy MapSubj left hemisphereinferior j0399Btemporal, fusiform, motor cortexmotor cortex, intraparietal sulcus, if i f linferior frontal, orbital frontal,occipital cortex9 subj meanpWhat are the learned semantic feature activations?predicted “airplane”activity for “airplane”airplaneSemantic features from a trillion-word text corpusMapping learned from fMRI dataLearned Semantic Feature Signatures (P1)‘eat’‘listen’‘touch’tt tifprimarygustatory cortex regions of language processing and primary somatosensory cortexauditionOf the 10,000 most frequent English words which


View Full Document

CMU CS 10601 - fMRIpredict_4_2_2008

Documents in this Course
lecture

lecture

40 pages

Problem

Problem

12 pages

lecture

lecture

36 pages

Lecture

Lecture

31 pages

Review

Review

32 pages

Lecture

Lecture

11 pages

Lecture

Lecture

18 pages

Notes

Notes

10 pages

Boosting

Boosting

21 pages

review

review

21 pages

review

review

28 pages

Lecture

Lecture

31 pages

lecture

lecture

52 pages

Review

Review

26 pages

review

review

29 pages

Lecture

Lecture

37 pages

Lecture

Lecture

35 pages

Boosting

Boosting

17 pages

Review

Review

35 pages

lecture

lecture

32 pages

Lecture

Lecture

28 pages

Lecture

Lecture

30 pages

lecture

lecture

29 pages

leecture

leecture

41 pages

lecture

lecture

34 pages

review

review

38 pages

review

review

31 pages

Lecture

Lecture

41 pages

Lecture

Lecture

15 pages

Lecture

Lecture

21 pages

Lecture

Lecture

38 pages

Notes

Notes

37 pages

lecture

lecture

29 pages

Load more
Download fMRIpredict_4_2_2008
Our administrator received your request to download this document. We will send you the file to your email shortly.
Loading Unlocking...
Login

Join to view fMRIpredict_4_2_2008 and access 3M+ class-specific study document.

or
We will never post anything without your permission.
Don't have an account?
Sign Up

Join to view fMRIpredict_4_2_2008 2 2 and access 3M+ class-specific study document.

or

By creating an account you agree to our Privacy Policy and Terms Of Use

Already a member?