Internal State Predictability as an Evolutionary Precursor of Self-Awareness and AgencyResearch Question: Self-AwarenessEvolution of Self-Awareness and Agency?ApproachMethod (Task): 2D Pole-BalancingMethod: Neuroevolution ControllerInternal State of the ControllerSame Behavior, Different MindSketch of the MethodMethod: Experimental SetupMethod: Measuring PredictabilityResults: Internal State Predictability (ISP)Comparison High ISP and Low ISPResults: Learning TimePerformance and Int. State Dyn.Behavioral PredictabilityExamples of Internal State Dynamics from the High ISP GroupExamples of Internal State Dynamics from the Low ISP GroupExamples of cart x and y position from high ISPExamples of cart x and y position from low ISPRelated WorkConclusionsInternal State Predictability as anEvolutionary Precursor ofSelf-Awareness and AgencySfN 2008 (738.14)November 19, 2008Jaerock Kwon and Yoonsuck ChoeDepartment of Computer ScienceTexas A&M UniversityFor the full paper, see Kwon and Choe (2008).Page 1AbstractWhat is the evolutionary value of self-awareness and agency in intelligentagents? One way to make this problem tractable is to think about the necessaryconditions that lay the foundation for the emergence of agency, and assess theirevolutionary origin. We postulate that one such requirement is the predictabilityof the internal state trajectory. A distinct property of one’s own actions comparedto someone else’s is that one’s own is highly predictable, and this gives the senseof “authorship”. In order to investigate if internal state predictability has anyevolutionary value, we evolved sensorimotor control agents driven by a recurrentneural network in a 2D pole-balancing task. The hidden layer activity of thenetwork was viewed as the internal state of an agent, and the predictability of itstrajectory was measured. We took agents exhibiting equal levels of performancedur ing evolutionary trials, and grouped them into those with high or low internalstate predictability (ISP). The high-ISP group showed better performance thanthe low-ISP group in novel tasks with substantially harder initial conditions. Theseresults indicate that regular ity or predictability of neural activity in internaldynamics of agents can have a positive impact on fitness, and, in turn, can helpus better understand the evolutionary role of self-awareness and agency.Page 2Research Question: Self-AwarenessWhy did self-awareness (or the sense of self) evolve?• Self-awareness is an internal state that may betransparent to the process of evolution (cf.high-performance zombie).• This is a hard question to answer without gettingtangled in philosophical debate.Strategy: Investigate the necessary condition ofself-awareness that may be less controversial.Page 3Evolution of Self-Awareness andAgency?zombieconscious• Performance-wise, conscious agents and zombiescould be indistinguishable (to evolution)!Page 4ApproachIdentify necessary conditions of self-awareness:• Sense of self and agency are closely related.• Authorship is a key ingredient: “I” prescr ibe myactions, and “I” own them.• Important property of authorship: My actions arehighly predictable while others’ are not.Necessary condition identified: Need to be able topredict one’s own internal state (cf. Nolfi et al. 1994).Page 5Method (Task): 2D Pole-Balancingx(x, y)y!y!x• Physical parameters of the pole balancing system:position (x, y); velocity ( ˙x, ˙y); pole angle (θx, θy);angular velocity (˙θx,˙θy).Page 6Method: Neuroevolution ControllerFxFy!x!’x!y!’yxx’yy’Z-1Z-1Z-1X(t)NDPIAA(t)Z-1• Recurrent neural network for 2D pole balancing.• Trained with standard neuroevolution.• Investigate the internal state trajectories.Page 7Internal State of the Controller.....xyztimeyzxxyz• Activation level of hidden units can be seen as theinternal state of the controller agent.Page 8Same Behavior, Different Mind.....yzxxyz.....yzxxyz(a) High Internal State Predictability (a) Low Internal State Predictability• Two controllers with the same level of performancecan have different internal state dynamics!Page 9Sketch of the Methodinternal stateanalysisinternal stateanalysisAll Controllers High−perform.ControllersLow ISPHigh ISPselectionprocessevolutionary1. Evolve controllers to meet a fixed performance criterion (fitnessdoes not measure predictability) in pole-balancing tasks.2. Group high-performance individuals in to high- and low internalstate predictability (ISP) groups.Page 10Method: Experimental Setup• Neuroevolution:– population size 50– mutation rate 0.2; cross over rate 0.7.• 2D pole balancing task:– Pole should be balanced within 15◦within a 3 m× 3 m arena.– Force applied to cart every 0.1 second (= onestep).– Success if pole balanced over 5,000 steps.Page 11Method: Measuring Predictabilityt+1tt−1t−2t−3Neural network predictor (backprop):• Input: hidden unit activation in N steps in the past• Target: current hidden unit activationMeasure how easy it is to learn to predict trajectory.Page 12Results: Internal State Predictability(ISP)• Trained 130 pole balancing agents.• Chose top 10 highest ISP agents and bottom 10 lowest ISP.– high ISPs: µ = 95.61% and σ = 5.55%.– low ISPs: µ = 31.74% and σ = 10.79%.Page 13Comparison High ISP and Low ISP010203040506070809010012345678910Prediction Success Rate (%)Test Case NumberComparison of High and Low PredictabilityHighLow• A comparison of the average predictability from twogroups: high ISP and low ISP.• The predictive success rate of the top 10 and thebottom 10 agents.Page 14Results: Learning Time0200400600800100012001400160012345678910Generation NumberTest Case Number Learning TimeHighLow• No significant difference in learning timePage 15Performance and Int. State Dyn.010002000300040005000600012345678910Number of StepsTest Case NumberPerformance and Internal State DynamicsHighLow• Made the initial conditions in the 2D pole balancingtask harsher.• Performance of high- and low-ISP groups compared.• High-ISP group outperforms the low-ISP group in thechanged environment.Page 16Behavioral Predictability0102030405060708090x posy posx angley anglePrediction Success Rate (%)Behavioral PredictabilityHighLow• Success of high-ISP group may simply be due tosimpler behavioral trajectory.• However, predictability in behavioral predictability isno different between high- and low-ISP groups.Page 17Examples of Internal StateDynamics from