1CS 188: Artificial IntelligenceFall 2008Lecture 11: Reinforcement LearningLecture 11: Reinforcement Learning10/2/2008Dan Klein – UC BerkeleyMany slides over the course adapted from either Stuart Russell or Andrew Moore12Reinforcement Learning Reinforcement learning: Still have an MDP: A set of states s ∈ S A set of actions (per state) AA model T(s,a,s’)A model T(s,a,s’) A reward function R(s,a,s’) Still looking for a policy π(s) New twist: don’t know T or R I.e. don’t know which states are good or what the actions do Must actually try actions and states out to learn[DEMO]33Example: Animal Learning RL studied experimentally for more than 60 years in psychology Rewards: food, pain, hunger, drugs, etc. Mechanisms and sophistication debated Example: foraging Bees learn near-optimal foraging plan in field of artificial flowers with controlled nectar supplies Bees have a direct neural connection from nectar intake measurement to motor planning area44Example: Backgammon Reward only for win / loss in terminal states, zero otherwise TD-Gammon learns a function approximation to V(s) using a neural networkV(s) using a neural network Combined with depth 3 search, one of the top 3 players in the world You could imagine training Pacman this way… … but it’s tricky!55Passive Learning Simplified task You don’t know the transitions T(s,a,s’) You don’t know the rewards R(s,a,s’) You are given a policy π(s)Goal: learn the state values(and maybe the model)Goal: learn the state values(and maybe the model) I.e., policy evaluation In this case: Learner “along for the ride” No choice about what actions to take Just execute the policy and learn from experience We’ll get to the active case soon This is NOT offline planning!66Example: Direct Estimation Episodes:y(1,1) up -1(1,2) up -1(1,2) up -1(1,1) up -1(1,2) up -1(1,3) right -1+100-100[DEMO – Optimal Policy]x(1,3) right -1(2,3) right -1(3,3) right -1(3,2) up -1(3,3) right -1(4,3) exit +100(done)(2,3) right -1(3,3) right -1(3,2) up -1(4,2) exit -100(done)V(1,1) ~ (92 + -106) / 2 = -7V(3,3) ~ (99 + 97 + -102) / 3 = 31.3γ = 1, R = -177Model-Based Learning Idea: Learn the model empirically (rather than values) Solve the MDP as if the learned model were correctEmpirical model learningEmpirical model learning Simplest case: Count outcomes for each s,a Normalize to give estimate of T(s,a,s’) Discover R(s,a,s’) the first time we experience (s,a,s’) More complex learners are possible (e.g. if we know that all squares have related action outcomes, e.g. “stationary noise”)88Example: Model-Based Learning Episodes:y+100-100(1,1) up -1(1,2) up -1(1,2) up -1(1,1) up -1(1,2) up -1(1,3) right -1xT(<3,3>, right, <4,3>) = 1 / 3T(<2,3>, right, <3,3>) = 2 / 2γ = 1(1,3) right -1(2,3) right -1(3,3) right -1(3,2) up -1(3,3) right -1(4,3) exit +100(done)(2,3) right -1(3,3) right -1(3,2) up -1(4,2) exit -100 (done)99Recap: Model-Based Policy Evaluation Simplified Bellman updates to calculate V for a fixed policy: New V is expected one-step-look-ahead using current V Unfortunately, need T and Rπ(s)ss, π(s)s,π(s),s’10s,π(s),s’s’10Sample Avg to Replace Expectation? Who needs T and R? Approximate the expectation with samples (drawn from T!)π(s)ss, π(s)11s, π(s)s1’s2’s3’11Model-Free Learning Big idea: why bother learning T? Update V each time we experience a transition Frequent outcomes will contribute more updates (over time) Temporal difference learning (TD)Policy still fixed!π(s)ss, π(s)Policy still fixed! Move values toward value of whatever successor occurs: running average!12s’12Example: TD Policy Evaluation(1,1) up -1(1,2) up -1(1,2) up -1(1,1) up -1(1,2) up -1(1,3) right -1Take γ = 1, α = 0.5(1,3) right -1(2,3) right -1(3,3) right -1(3,2) up -1(3,3) right -1(4,3) exit +100(done)(2,3) right -1(3,3) right -1(3,2) up -1(4,2) exit -100(done)1313Problems with TD Value Learning TD value leaning is model-free for policy evaluation However, if we want to turn our value estimates into a policy, we’re sunk:ass, as,a,s’ Idea: learn Q-values directly Makes action selection model-free too!s,a,s’s’1414Active Learning Full reinforcement learning You don’t know the transitions T(s,a,s’) You don’t know the rewards R(s,a,s’) You can choose any actions you likeGoal: learn the optimal policy (maybe values)Goal: learn the optimal policy (maybe values) In this case: Learner makes choices! Fundamental tradeoff: exploration vs. exploitation This is NOT offline planning!1515Model-Based Learning In general, want to learn the optimal policy, not evaluate a fixed policyIdea: adaptive dynamic programmingIdea: adaptive dynamic programming Learn an initial model of the environment: Solve for the optimal policy for this model (value or policy iteration) Refine model through experience and repeat Crucial: we have to make sure we actually learn about all of the model1616Example: Greedy ADP Imagine we find the lower path to the good exit first Some states will never be visited following this policy from (1,1)? ?from (1,1) We’ll keep re-using this policy because following it never collects the regions of the model we need to learn the optimal policy 1717What Went Wrong? Problem with following optimal policy for current model: Never learn about better regions of the space if current policy neglects themFundamental tradeoff: ? ?Fundamental tradeoff: exploration vs. exploitation Exploration: must take actions with suboptimal estimates to discover new rewards and increase eventual utility Exploitation: once the true optimal policy is learned, exploration reduces utility Systems must explore in the beginning and exploit in the limit1818Q-Value Iteration Value iteration: find successive approx optimal values Start with V0*(s) = 0, which we know is right (why?) Given Vi*, calculate the values for all states for depth i+1: But Q-values are more useful! Start with Q0*(s,a) = 0, which we know is right (why?) Given Qi*, calculate the q-values for all q-states for depth i+1:1919Q-Learning Learn Q*(s,a) values Receive a sample (s,a,s’,r) Consider your old estimate: Consider your new sample estimate:[DEMO – Grid Q’s] Incorporate
View Full Document
Unlocking...