Human and Machine Learning Tom Mitchell Machine Learning DepartmentCarnegie Mellon UniversityApril 23, 20081How can studies of machine (human) learning informmachine (human) learning inform studies of h(hi)li?human (machine) learning?2Outline1. Machine Learning and Human Learning2. Aligning specific results from ML and HL• Learning to predict and achieve rewardsgp• TD learning ↔ Dopamine system in the brain• Value of redundancy in data inputsCiiIddhhi•Cotraining ↔Intersensory redundancy hypothesis3Core questions and conjectures3.Core questions and conjectures3Machine Learning - PracticeSpeech RecognitionObject recognitionMining Databases•Reinforcement learningMining DatabasesControl learning•Reinforcement learning• Supervised learning•Bayesian networksControl learningBayesian networks• Hidden Markov models•Unsupervised clusteringText analysis4pg• Explanation-based learning• ....Machine Learning - TheoryPAC Learning TheoryOther theories for• Reinforcement skill learning# examples (m)• Unsupervised learning• Active student querying(for supervised concept learning)p()representational complexity (H)error rate ()• …error rate (ε)failure probability (δ)… also relating:• # of mistakes during learningprobability (δ)• learner’s query strategy• convergence rate5• asymptotic performance•…ML Has Little to Say About• Learning cumulatively over time • Learning from instruction, lectures, discussions• Role of motivation, forgetting, curiosity, fear, boredomboredom, ...•Implicit (unconscious) versus explicit (deliberate)Implicit (unconscious) versus explicit (deliberate) learning6• ...What We Know About Human Learning*Neural level:Neural level:•Hebbian learning: connection between the pre-synaptic andHebbian learning: connection between the pre-synaptic and post-synaptic neuron increases if pre-synaptic neuron is repeatedly involved in activating post-synaptic2–Biochemistry: NMDA channels, Ca2+, AMPA receptors, ... •Timing matters: strongest effect if pre-synaptic actionTiming matters: strongest effect if presynaptic action potential occurs within 0 - 50msec before postsynaptic firing. • Time constants for synaptic changes are a few minutes. –Can be disrupted by protein inhibitors injected after the training7Can be disrupted by protein inhibitors injected after the training experience* I’m not an expertWhat We Know About HL*St l lSystem level:• In addition to single synapse changes, memory formation involves longer term ‘consolidation’ involving multiple parts of the brain• Time constant for consolidation is hours or days: memory of new ibditdbtiftthiexperiences can be disrupted by events occurring after the experience (e.g., drug interventions, trauma).– E.g., injections in amygdala 24 hours after training can impact recall experience, with no impact on recall within a few hoursexperience, with no impact on recall within a few hours• Consolidation thought to involve regions such as amygdala, hippocampus, frontal cortex. Hippocampus might orchestrate consolidation without itself being home of memories• Dopamine seems to play a role in reward-based learning (and ddi ti )8addictions)* I’m not an expertWhat We Know About HL*Bh i ll lBehavioral level:• Power law of practice: competence vs. training on log-log plot is a straight line across many skill typesstraight line, across many skill types• Role of reasoning and knowledge compilation in learning –chunking, ACT-R, Soarg,,• Timing: Expanded spacing of stimuli aids memory, ...• Theories about role of sleep in learning/consolidation• Implicit and explicit learning. (unaware vs. aware). • Developmental psychology: knows much about sequence of acquired expertise during childhood–Intersensory redundancy hypothesis 9yyyp* I’m not an expertModels of Learning ProcessesMachine Learning: Human Learning:• # of examples• Error rate• # of examples• Error rate• Reinforcement learning• Explanations• Reinforcement learning• Explanations• Learning from examples• Complexity of learner’s representation• Human supervision– Lectures– Question answering• Probability of success• Exploitation / exploration• Prior probabilities• Attention, motivation• Skills vs. Principles•Implicit vs. Explicit learning10• Loss functionspp g• Memory, retention, forgetting1 Learning to predict and achieve rewards1. Learning to predict and achieve rewardsReinforcement learning in MLReinforcement learning in ML↔Dopamine in the brainDopamine in the brain11Reinforcement Learning[Sutton and Barto 1981; Samuel 1957][Sutton and Barto 1981; Samuel 1957]12...]rγr γE[r(s)V2t21tt*+++=++Reinforcement Learning in MLr =1000γ = .9S0S2S1S3V=1000V=72V=81V=90S0S2S1S3...]rγr γE[r)V(s2t21ttt+++=++)V(sγ]E[r)V(s1ttt ++=To learn V use each transition to generate a training signal:To learn V, use each transition to generate a training signal:13Dopamine As Reward Signalt[Schultz et al., Science, 1997]14Dopamine As Reward Signalt[Schultz et al., Science, 1997]15Dopamine As Reward Signalt[Schultz et al., Science, 1997])V(s)V(s γr errort1tt−+=+16RL Models for Human Learning[Seymore et al., Nature 2004]17[Seymore et al., Nature 2004]18Human EEG responses to Pos/Neg Reward from [Nieuwenhuis et al.]Response due to feedback on timing task (press button exactly 1 sec after sound).Neural source appears to be in anteriorto be in anterior cingulate cortex (ACC)Response is abnormal in some subjects with OCD19One Theory of RL in the Brainfrom [Nieuwenhuis et al ]• Basal ganglia monitors events, predict future rewardsfrom [Nieuwenhuis et al.]• When prediction revised upward (downward), causes increase (decrease) in activity of midbrain dopaminergic ifl i ACCneurons, influencing ACC• This dopamine-based activation hltiiithsomehow results in revising the reward prediction function. Possibly through direct influence on Basal ganglia, and via prefrontal cortex20Summary: Temporal Difference ML Model Predicts Dopaminergic Neuron Acitivity during LearningPredicts Dopaminergic Neuron Acitivity during Learning• Evidence now of neural reward signals from g– Direct neural recordings in monkeys– fMRI in humans (1 mm spatial resolution)EEG in humans (110 msec temporal resolution)–EEG in humans (1-10 msec temporal resolution)• Dopaminergic responses track temporal difference error in RLS diff d ff t t fi HL d l•Some differences, and efforts to refine
View Full Document
Unlocking...