Internal State Predictability as an Evolutionary Precursor of Self-Awareness and AgencyMotivationResearch Question: Self-AwarenessApproachMethod (Task): 2D Pole-BalancingMethod: Neuroevolution ControllerSketch of the MethodMethod: Experimental SetupMethod: Measuring PredictabilityMethod: Experimental SetupResults: Internal State Predictability (ISP)Comparison High ISP and Low ISPResults: Learning TimePerformance and Int. State Dyn.Behavioral PredictabilityExamples of internal state dynamics from the high ISPExamples of internal state dynamics from the low ISPExamples of cart x and y position from high ISPExamples of cart x and y position from low ISPRelated WorkConclusionsInternal State Predictability as anEvolutionary Precursor ofSelf-Awareness and AgencyICDL 2008August 11, 2008Jaerock Kwon and Yoonsuck ChoeDepartment of Computer ScienceTexas A&M UniversityPage 1MotivationThe concept of self (self-awareness, agency) is animportant yet hard subject:• It may lead to consciousness.• It may be necessary for social interaction.• It may play an important role in cognition (Block1995).Page 2Research Question: Self-AwarenessWhy did self-awareness (or the sense of self) evolve?• Self-awareness is an internal state that may betransparent to the process of evolution (cf.high-performance zombie).• This is a hard question to answer without gettingtangled in philosophical debate.Strategy: Investigate the necessary condition ofself-awareness that may be less controversial.Page 3ApproachIdentify necessary conditions of self-awareness:• Sense of self and agency are closely related.• Authorship is a key ingredient: “I” prescribe myactions, and “I” own them.• Important property of authorship: My actions arehighly predictable while others’ are not.Necessary condition identified: Need to be able topredict one’s own internal state (cf. Nolfi et al. 1994).Page 4Method (Task): 2D Pole-Balancingx(x, y)y!y!x• Physical parameters of the pole balancing system:position (x, y); velocity ( ˙x, ˙y); pole angle (θx, θy);angular velocity (˙θx,˙θy).Page 5Method: Neuroevolution ControllerFxFy!x!’x!y!’yxx’yy’Z-1Z-1Z-1X(t)NDPIAA(t)Z-1• Recurrent neural network for 2D pole balancing.• Trained with standard neuroevolution.• Investigate the internal state trajectories.Page 6Sketch of the Methodinternal stateanalysisinternal stateanalysisAll Controllers High−perform.ControllersLow ISPHigh ISPselectionprocessevolutionary1. Evolve controllers to meet a fixed performance criterion (fitnessdoes not measure predictability) in pole-balancing tasks.2. Group high-performance individuals in to high- and low internalstate predictability (ISP) groups.3. Test the two groups in harder tasks.Page 7Method: Experimental Setup• Neuroevolution:– population size 50– mutation rate 0.2; cross over rate 0.7.• 2D pole balancing task:– Pole should be balanced within 15◦within a 3 m× 3 m arena.– Force applied to cart every 0.1 second (= onestep).– Success if pole balanced over 5,000 steps.Page 8Method: Measuring Predictabilityfx(t)x(t−1)x(t−2)x(t−N+1)x(t+1)…ˆx(t + 1) = f (x(t), x(t − 1), x(t − 2), · · · , x(t − N + 1)) .)1( txx(t)x(t−1)x(t−2)x(t−N+1)x(t+1)…• Neural network predictor for a time series.Page 9Method: Experimental Setup• Neural network predictor:– 2,000 training data.– 1,000 test data.– Back-propagation (learning rate 0.2).Page 10Results: Internal State Predictability(ISP)• Trained 130 pole balancing agents.• Chose top 10 highest ISP agents and bottom 10 lowest ISP.– high ISPs: µ = 95.61 % and σ = 5.55%.– low ISPs: µ = 31.74% and σ = 10.79%.Page 11Comparison High ISP and Low ISP010203040506070809010012345678910Prediction Success Rate (%)Test Case NumberComparison of High and Low PredictabilityHighLow• A comparison of the average predictability from twogroups: high ISP and low ISP.• The predictive success rate of the top 10 and thebottom 10 agents.Page 12Results: Learning Time0200400600800100012001400160012345678910Generation NumberTest Case Number Learning TimeHighLow• No significant difference in learning timePage 13Performance and Int. State Dyn.010002000300040005000600012345678910Number of StepsTest Case NumberPerformance and Internal State DynamicsHighLow• Made the initial conditions in the 2D pole balancingtask harsher.• Performance of high- and low-ISP groups compared.• High-ISP group outperforms the low-ISP group in thechanged environment.Page 14Behavioral Predictability0102030405060708090x posy posx angley anglePrediction Success Rate (%)Behavioral PredictabilityHighLow• Success of high-ISP group may simply be due tosimpler behavioral trajectory.• However, predictability in behavioral predictability isno different between high- and low-ISP groups.Page 15Examples of internal state dynamicsfrom the high ISP• Internal state dynamics show smooth trajectories.Page 16Examples of internal state dynamicsfrom the low ISP• Internal state dynamics show abrupt and jitterytrajectories.Page 17Examples of cart x and y positionfrom high ISP-1.6-1.4-1.2-1-0.8-0.6-0.4-0.200.2-0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1 1.2-0.4-0.35-0.3-0.25-0.2-0.15-0.1-0.0500.050.10 0.1 0.2 0.3 0.4 0.5 0.6-0.100.10.20.30.40.50.60.7-0.6 -0.5 -0.4 -0.3 -0.2 -0.1 0 0.100.20.40.60.811.21.41.60 0.2 0.4 0.6 0.8 1 1.2• Behavioral trajectories of x and y positions showcomplex trajectories.Page 18Examples of cart x and y positionfrom low ISP-0.100.10.20.30.40.50.6-0.35 -0.3 -0.25 -0.2 -0.15 -0.1 -0.05 0 0.05-1.2-1-0.8-0.6-0.4-0.200.20.40.6-0.25 -0.2 -0.15 -0.1 -0.05 0 0.05 0.1 0.15 0.2 0.25-1.2-1-0.8-0.6-0.4-0.200.20.4-0.2 -0.15 -0.1 -0.05 0 0.05 0.1-2-1.5-1-0.500.51-0.2 0 0.2 0.4 0.6 0.8 1• Behavioral trajectories of x and y positions showcomplex trajectories.Page 19Related Work• Bayesian self-model (Gold and Scassellati 2007).• Continuous self re-modeling for resilient machines(Bongard et al. 2006).• Autonomous mental development (Weng et al. 2001;Han et al. 2002).• Role of self-awareness in cognition (Block 1995).• Emergence of self-awareness fromself-representation (Menant 2007).Page 20Conclusions• Simpler (more predictable) internal dynamics can achievehigher levels of performance in harsher environmentalconditions.• The increased survival value is not always due tosmoother behavior resulting from the simpler internalstates.• Initially evloution-transparent internal agent properties canaffect external