1CS 188: Artificial IntelligenceFall 2006Lecture 13: Advanced Reinforcement Learning10/12/2006Dan Klein – UC BerkeleyToday How advanced reinforcement learning works for large problems Some previews of fundamental ideas we’ll see throughout the rest of the term Next class we’ll start on probabilistic reasoning and reasoning about beliefs2Recap: Q-Learning Learn Q*(s,a) values from samples Receive a sample (s,a,s’,r) On one hand: old estimate of return: But now we have a new estimate for this sample: Nudge the old estimate towards the new sample: Equivalently, average samples over time:Q-Learning Q-learning produces tables of q-values:3Q-Learning In realistic situations, we cannot possibly learn about every single state! Too many states to visit them all in training Too many states to even hold the q-tables in memory Instead, we want to generalize: Learn about some small number of training states from experience Generalize that experience to new, similar states This is a fundamental idea in machine learning, and we’ll see it over and over againExample: Pacman Let’s say we discover through experience that this state is bad: In naïve q-learning, we know nothing about this state or its q-states: Or even this one!4Feature-Based Representations Solution: describe a state using a vector of features Features are functions from states to real numbers (often 0/1) that capture important properties of the state Example features: Distance to closest ghost Distance to closest dot Number of ghosts 1 / (dist to dot)2 Is Pacman in a tunnel? (0/1) …… etc. Can also describe a q-state (s, a) with features (e.g. action moves closer to food)Linear Feature Functions Using a feature representation, we can write a q-function (or value function) for any state using a few weights: Advantage: our experience is summed up in a few powerful numbers Disadvantage: states may share features but be very different in value!5Function Approximation Q-learning with linear q-functions: Intuitive interpretation: Adjust weights of active features E.g. if something unexpectedly bad happens, disprefer all states with that state’s features Formal justification: online least squares (much later)Example: Q-Pacman6Hierarchical Learning Stratagus: Example of a large RL task, from Bhaskara Marthi’sthesis (w/ Stuart Russell) Stratagus is hard for reinforcement learning algorithms > 10100states > 1030actions at each point Time horizon ≈ 104 steps Stratagus is hard for human programmers Typically takes several person-months for game companies to write computer opponent Still, no match for experienced human players Programming involves much trial and error Hierarchical RL Humans supply high-level prior knowledge using partial program Learning algorithm fills in the detailsHierarchical RL7(defun top ()(loop (choose (gather-wood)(gather-gold))))(defun gather-wood ()(with-choice (dest *forest-list*)(nav dest)(action ‘get-wood)(nav *base-loc*)(action ‘dropoff)))(defun gather-gold ()(with-choice (dest *goldmine-list*)(nav dest))(action ‘get-gold)(nav *base-loc*))(action ‘dropoff)))(defun nav (dest)(until (= (pos (get-state)) dest)(with-choice(move ‘(N S E W NOOP))(action move))))Partial “Alisp” ProgramHierarchical RL They then define a hierarchical Q-function which learns a linear feature-based mini-Q-function at each choice point Very good at balancing resources and directing rewards to the right region Still not very good at the strategic elements of these kinds of games (i.e. the Markov game aspect)[DEMO]8Policy SearchPolicy Search Problem: often the feature-based policies that work well aren’t the ones that approximate V / Q best E.g. your value functions from 1.3 were probably horrible estimates of future rewards, but they still produce good decisions We’ll see this distinction between modeling and prediction againlater in the course Solution: learn the policy that maximizes rewards rather than the value that predicts rewards This is the idea behind policy search, such as what controlled the upside-down helicopter9Policy Search Simplest policy search: Start with an initial linear value function or q-function Nudge each feature weight up and down and see if your policy is better than before Problems: How do we tell the policy got better? Need to run many sample episodes! If there are a lot of features, this can be impracticalPolicy Search* Advanced policy search: Write a stochastic (soft) policy: Turns out you can efficiently approximate the derivative of the returns with respect to the parameters w (details in the book, but you don’t have to know them) Take uphill steps, recalculate derivatives, etc.10Take a Deep Breath… We’re done with search and planning! Next, we’ll look at how to reason with probabilities Diagnosis Tracking objects Speech recognition Robot mapping … lots more! Last part of course: machine
View Full Document
Unlocking...