1CS 188: Artificial IntelligenceFall 2008Lecture 8: MEU / UtilitiesLecture 8: MEU / Utilities9/23/2008Dan Klein – UC BerkeleyMany slides over the course adapted from either Stuart Russell or Andrew Moore12Maximum Expected Utility Principle of maximum expected utility: A rational agent should chose the action which maximizes its expected utility, given its knowledge Questions:Why not do minimax(worst-case) planning in practice?Why not do minimax(worst-case) planning in practice? If every agent is M’ing it’s EU, why do we need expectations? How do we know that there are utility functions which “work” with expectations (i.e. where expected values can be meaningfully compared)?33Multi-Agent Utilities Similar to minimax: Utilities are now tuples Each player maximizes their own entry their own entry at each node Propagate (or back up) nodes from children Can give rise to cooperation and competition dynamically…1,6,6 7,1,2 6,1,2 7,4,1 5,1,7 1,5,2 7,7,1 5,1,544Expectimax Quantities55Expectimax Pruning?66Expectimax Evaluation For minimax search, terminal utilities are insensitive to monotonic transformations We just want better states to have higher evaluations (get the ordering right) For expectimax, we need the magnitudes to be meaningful as well E.g. must know whether a 50% / 50% lottery between A and B is better than 100% chance of C 100 or -10 vs 0 is different than 10 or -100 vs 0 How do we know such utilities even exist?77Preferences An agent chooses among: Prizes: A, B, etc. Lotteries: situations with uncertain prizes Notation:88Rational Preferences We want some constraints on preferences before we call them rational For example: an agent with intransitive preferences can intransitive preferences can be induced to give away all its money If B > C, then an agent with C would pay (say) 1 cent to get B If A > B, then an agent with B would pay (say) 1 cent to get A If C > A, then an agent with A would pay (say) 1 cent to get C99Rational Preferences Preferences of a rational agent must obey constraints. The axioms of rationality: Theorem: Rational preferences imply behavior describable as maximization of expected utility1010MEU Principle Theorem: [Ramsey, 1931; von Neumann & Morgenstern, 1944] Given any preferences satisfying these constraints, there exists a real-valued function U such that: Maximum expected likelihood (MEU) principle: Choose the action that maximizes expected utility Note: an agent can be entirely rational (consistent with MEU) without ever representing or manipulating utilities and probabilities E.g., a lookup table for perfect tictactoe, reflex vacuum cleaner1111Human Utilities Utilities map states to real numbers. Which numbers? Standard approach to assessment of human utilities: Compare a state A to a standard lottery Lpbetween “best possible prize” u+with probability p“worst possible catastrophe” u-with probability 1-p“worst possible catastrophe” u-with probability 1-p Adjust lottery probability p until A ~ Lp Resulting p is a utility in [0,1]1212Utility Scales Normalized utilities: u+= 1.0, u-= 0.0 Micromorts: one-millionth chance of death, useful for paying to reduce product risks, etc. QALYs: quality-adjusted life years, useful for medical decisions involving substantial riskinvolving substantial risk Note: behavior is invariant under positive linear transformation With deterministic prizes only (no lottery choices), only ordinal utilitycan be determined, i.e., total order on prizes1313Money Money does not behave as a utility function Given a lottery L: Define expected monetary value EMV(L) Usually U(L) < U(EMV(L))I.e., people are risk-averseI.e., people are risk-averse Utility curve: for what probability pam I indifferent between: A prize x A lottery [p,$M; (1-p),$0] for large M? Typical empirical data, extrapolatedwith risk-prone behavior:1414Example: Insurance Consider the lottery [0.5,$1000; 0.5,$0] What is its expected monetary value? ($500) What is its certainty equivalent? Monetary value acceptable in lieu of lottery$400 for most people$400 for most people Difference of $100 is the insurance premium There’s an insurance industry because people will pay to reduce their risk If everyone were risk-prone, no insurance needed!1515Example: Human Rationality? Famous example of Allais (1953) A: [0.8,$4k; 0.2,$0] B: [1.0,$3k; 0.0,$0]C: [0.2,$4k; 0.8,$0]C: [0.2,$4k; 0.8,$0] D: [0.25,$3k; 0.75,$0] Most people prefer B > A, C > D But if U($0) = 0, then B > A ⇒ U($3k) > 0.8 U($4k) C > D ⇒ 0.8 U($4k) > U($3k)1616Reinforcement Learning [DEMOS] Basic idea: Receive feedback in the form of rewardsAgent’s utility is defined by the reward functionAgent’s utility is defined by the reward function Must learn to act so as to maximize expected rewards Change the rewards, change the learned behavior Examples: Playing a game, reward at the end for winning / losing Vacuuming a house, reward for each piece of dirt picked up Automated taxi, reward for each passenger delivered1717Markov Decision Processes An MDP is defined by: A set of states s ∈ S A set of actions a ∈ A A transition function T(s,a,s’) Prob that a from s leads to s’ i.e., P(s’ | s,a)Also called the modelAlso called the model A reward function R(s, a, s’) Sometimes just R(s) or R(s’) A start state (or distribution) Maybe a terminal state MDPs are a family of non-deterministic search problems Reinforcement learning: MDPs where we don’t know the transition or reward functions1818Solving MDPs In deterministic single-agent search problem, want an optimal plan, or sequence of actions, from start to a goal In an MDP, we want an optimal policy π(s) A policy gives an action for each state Optimal policy maximizes expected if followed Defines a reflex agentOptimal policy when R(s, a, s’) = -0.04 for all non-terminals s1919Example Optimal PoliciesR(s) = -2.0R(s) = -0.4R(s) = -0.03R(s) = -0.012020Example: High-Low Three card types: 2, 3, 4 Infinite deck, twice as many 2’s Start with 3 showing After each card, you say “high” or “low”New card is flipped3New card is flipped If you’re right, you win the points shown
View Full Document
Unlocking...