Page 1CS 287: Advanced RoboticsFall 2009Lecture 21: HMMs, Bayes filter, smoother, Kalman filtersPieter AbbeelUC Berkeley EECS Thus far: Optimal control and reinforcement learning We always assumed we got to observe the state at each time and the challenge was to choose a good action Current and next set of lectures The state is not observed Instead, we get some sensory information about the state Challenge: compute a probability distribution over the state which accounts for the sensory information (“evidence”) which we have observed.OverviewPage 2 Helicopter A choice of state: position, orientation, velocity, angular rate Sensors:  GPS : noisy estimate of position (sometimes also velocity) Inertial sensing unit: noisy measurements from (i) 3-axis gyro [=angular rate sensor], (ii) 3-axis accelerometer [=measures acceleration + gravity; e.g., measures (0,0,0) in free-fall], (iii) 3-axis magnetometer Mobile robot inside building A choice of state: position and heading Sensors: Odometry (=sensing motion of actuators): e.g., wheel encoders  Laser range finder: measures time of flight of a laser beam between departure and return (return is typically happening when hitting a surface that reflects the beam back to where it came from)ExamplesFor any random variables X, Y we have: Definition of conditional probability:P(X=x | Y=y) = P(X=x, Y=y) / P(Y=y) Chain rule: (follows directly from the above)P(X=x, Y=y) = P(X=x) P(Y=y | X=x ) = P(Y=y) P(X=x | Y=y) Bayes rule: (really just a re-ordering of terms in the above)P(X=x | Y=y) = P(Y=y | X=x) P(X=x) / P(Y=y) Marginalization:P(X=x) = ∑yP(X=x, Y=y) Note: no assumptions beyond X, Y being random variables are made for any of these to hold true (and when we divide by something, that something is not zero)Probability reviewPage 3For any random variables X, Y, Z, W we have:  Conditional probability: (can condition on a third variable z throughout)P(X=x | Y=y, Z=z) = P(X=x, Y=y | Z=z) / P(Y=y | Z=z) Chain rule:P(X=x, Y=y, Z=z, W=w) = P(X=x) P(Y=y | X=x ) P(Z=z | X=x, Y=y) P(W=w| X=x, Y=y, Z=z) Bayes rule: (can condition on other variable z throughout)P(X=x | Y=y, Z=z) = P(Y=y | X=x, Z=z) P(X=x | Z=z) / P(Y=y | Z=z) Marginalization:P(X=x | W=w) = ∑y,zP(X=x, Y=y, Z=z | W=w) Note: no assumptions beyond X, Y, Z, W being random variables are made for any of these to hold true (and when we divide by something, that something is not zero)Probability reviewIndependence Two random variables X and Y are independent ifffor all x, y : P(X=x, Y=y) = P(X=x) P(Y=y) Representing a probability distribution over a set of random variables X1, X2, …, XTin its most general form can be expensive. E.g., if all Xiare binary valued, then there would be a total of 2Tpossible instantiations and it would require 2T-1 numbers to represent the probability distribution. However, if we assumed the random variables were independent, then we could very compactly represent the joint distribution as follows: P(X1=x1, X2=x2, …, XT=xT) = P(X1=x1) P(X2=x2) … P(XT=xT) Thanks to the independence assumptions, for the binary case, we went from requiring 2T-1 parameters, to only requiring T parameters! Unfortunately independence is often too strong an assumption …Page 4 Two random variables X and Y are conditionally independent given a third random variable Z ifffor all x, y, z : P(X=x, Y=y | Z=z) = P(X=x | Z=z) P(Y=y | Z=z) Chain rule (which holds true for all distributions, no assumptions needed): P(X=x,Y=y,Z=z,W=w) = P(X=x)P(Y=y|X=x)P(Z=z|X=x,Y=y)P(W=w|X=x,Y=y,Z=z) For binary variables the representation requires 1 + 2*1 + 4*1 + 8*1 = 24-1 numbers (just like a full joint probability table) Now assume Z independent of X given Y, and assume W independent of X and Y given Z, then we obtain: P(X=x,Y=y,Z=z,W=w) = P(X=x)P(Y=y|X=x)P(Z=z|Y=y)P(W=w|Z=z) For binary variables the representation requires 1 + 2*1 + 2*1 + 2*1 = 1+(4-1)*2 numbers --- significantly less!! Conditional independenceMarkov Models Models a distribution over a set of random variables X1, X2, …, XTwhere the index is typically associated with some notion of time. Markov models make the assumption: Xtis independent of X1, …, Xt-2when given Xt-1 Chain rule: (always holds true, not just in Markov models!) P(X1= x1, X2= x2, …, XT= xT) = ∏tP(Xt= xt| Xt-1= xt-1, Xt-2= xt-2, …, X1= x1) Now apply the Markov conditional independence assumption: P(X1= x1, X2= x2, …, XT= xT) = ∏tP(Xt= xt| Xt-1= xt-1) (1)  in binary case: 1 + 2*(T-1) numbers required to represent the joint distribution over all variables (vs. 2T– 1) Graphical representation: a variable Xtreceives an arrow from the variables appearing in its conditional probability in the expression for the joint distribution (1) [called a Bayesian network or Bayes net representation]XTX2X1X3X4Page 5Hidden Markov Models Underlying Markov model over states Xt Assumption 1: Xtindependent of X1, …, Xt-2given Xt-1 For each state Xtthere is a random variable Ztwhich is a sensory measurement of Xt Assumption 2: Ztis assumed conditionally independent of the other variables given Xt This gives the following graphical (Bayes net) representation:XTX2Z1X1X3X4Z2Z3Z4ZTHidden Markov Models Chain rule: (no assumptions)P(X1= x1) P(Z1= z1| X1= x1) P(X2= x2| X1= x1, Z1= z1) P(Z2= z2| X1= x1, Z1= z1, X2= x2)… P(XT= xT| X1= x1, Z1= z1, … , XT-1= xT-1, ZT-1= zT-1) P(ZT= zT| X1= x1, Z1= z1, … , XT-1= xT-1, ZT-1= zT-1, XT= xT)XTX2Z1X1X3X4Z2Z3Z4ZT HMM assumptions:P(X1= x1) P(Z1= z1| X1= x1) P(X2= x2| X1= x1) P(Z2= z2| X2= x2) … P(XT= xT| XT-1= xT-1) P(ZT= zT| XT= xT)P(X1=x1, Z1=z1, X2=x2, Z2=z2, …, XT=xT, ZT=zT) =Page 6 What would the graph look like for a Bayesian network with no conditional independence assumptions? Our particular choice of ordering of variables in the chain rule enabled us to easily incorporate the HMM assumptions. What if we had chosen a different ordering in the chain rule expansion?Mini quizExample The HMM is defined by: Initial distribution: Transitions: Observations:Page 7Real HMM Examples Robot localization: Observations are range readings (continuous) States are positions on a map (continuous) Speech recognition HMMs: Observations are acoustic signals (continuous valued) States are specific positions in specific words (so, tens of