Near Optimal Combination of Sensory and Motor Uncertainty in Time Duringa Naturalistic Perception-Action TaskA. Aldo Faisal and Daniel M. WolpertDepartment of Engineering, University of Cambridge, Cambridge, United KingdomSubmitted 28 August 2008; accepted in final form 8 December 2008Faisal AA, Wolpert DM. Near optimal combination of sensory andmotor uncertainty in time during a naturalistic perception-action task.J Neurophysiol 101: 1901–1912, 2009. First published December 24,2008; doi:10.1152/jn.90974.2008. Most behavioral tasks have timeconstraints for successful completion, such as catching a ball in flight.Many of these tasks require trading off the time allocated to percep-tion and action, especially when only one of the two is possible at anytime. In general, the longer we perceive, the smaller the uncertainty inperceptual estimates. However, a longer perception phase leaves lesstime for action, which results in less precise movements. Here weexamine subjects catching a virtual ball. Critically, as soon as subjectsbegan to move, the ball became invisible. We study how subjectstrade-off sensory and movement uncertainty by deciding when toinitiate their actions. We formulate this task in a probabilistic frame-work and show that subjects’ decisions when to start moving arestatistically near optimal given their individual sensory and motor uncer-tainties. Moreover, we accurately predict individual subject’s task per-formance. Thus we show that subjects in a natural task are quantitativelyaware of how sensory and motor variability depend on time and act so asto minimize overall task variability.INTRODUCTIONReal world behavior requires the brain to combine a streamof sensory information and motor actions over time. Thisproblem is complicated given that sensory inputs and motoroutputs are subjected to noise and, more generally, uncertainty(Faisal et al. 2008). In the sensory domain, it has previouslybeen shown that human subjects have knowledge about theuncertainty in their sensory modalities and can combine thesemodalities in a statistically optimal fashion to reduce overallsensory uncertainty (van Beers et al. 1996; Ernst and Banks2002; Hillis et al. 2004; Jacobs 1999; Knill 2003; Sober andSabes 2005; van Beers et al. 1999). However, these studiesexamine only synchronous presentation of stimuli and there-fore ignore the role of time in acquiring sensory information. Inthe motor domain, goal-directed movements seem to be con-ducted in such a way as to reduce motor variability (Harris andWolpert 1998) and minimize the task relevant parts of move-ment uncertainty (Todorov and Jordan 2002). Thus in actionand perception tasks, subjects behave in a way to minimize thenegative consequences of uncertainty (Battaglia and Schrater2007). Unlike previous studies, most natural situations involveasynchronous (and possibly overlapping) episodes of sensoryinformation acquisition and motor action, from reaching to anobject that you have previously looked at to using your sideview mirror while driving. Thus successful behavior requires acombination of sensation and action across time. Here, weexamine how subjects choose to allocate time to perception andaction.To do this we used a simple virtual reality experiment,catching a falling ball with a paddle (Fig. 1A). We enforced atrade-off between perception and action phases by making theball invisible once movement is initiated, such that no addi-tional sensory information can be acquired about the ball’strajectory and landing position. Therefore subjects can trade-off their sensory and movement uncertainties by choosing theamount of time they allocate to perception and the amount oftime remaining for action before the ball touches the ground—that is, by making the decision when to switch from perceptionto action. Our approach is to measure independently the timedependence of sensory variability and motor variability and topredict their combined effect on the ball catching task. Notethat the use here of the term variability encompasses manysources. For example, motor variability (here the endpointvariability of the position of the paddle) is constituted bysignal-dependent motor noise (Harris and Wolpert 1998), mo-tor planning variability (Churchland et al. 2006; van Beers etal. 2004), noise in nerve fibers of the CNS and PNS (Faisal andLaughlin 2007), and other sources (Faisal et al. 2008).We investigate whether subjects have knowledge of the timedependence of their uncertainty in both perception and actionby examining whether they choose the optimal switching timeso as to minimize the overall variability of the task andmaximize their chances of catching the ball.We can consider an ideal actor whose aim is to maximize theprobability of catching the ball by minimizing the distancebetween the paddle and ball at touchdown. Both sensory andmotor variability contribute to the overall variability of wherethe paddle is placed relative to the ball (see Fig. 1B forillustration). The longer the actor perceives, the lower thesensory variability about where the ball will land (Fig. 1B,green curve and axes) but the higher the motor variability,because the remaining time for movement decreases (Fig. 1B,blue curve and axes). An ideal actor should therefore choose aswitching time that minimizes the combined effect of sensoryand motor variability (Fig. 1B, black line). To predict theoptimal switching time for each subject, we independentlyquantify in two separate experiments the time dependence ofthe sensory and motor variability (by their variance). Weassume that the two sources of sensory and motor variabilityare independent, and therefore the combined task variability!C2is the sum of the time-dependent sensory variability varianceAddress for reprint requests and other correspondence: A. A. Faisal, Dept.of Engineering, Univ. of Cambridge, Trumpington St., CB2 1PZ Cambridge,UK (E-mail: [email protected]).The costs of publication of this article were defrayed in part by the paymentof page charges. The article must therefore be hereby marked “advertisement”in accordance with 18 U.S.C. Section 1734 solely to indicate this fact.J Neurophysiol 101: 1901–1912, 2009.First published December 24, 2008; doi:10.1152/jn.90974.2008.19010022-3077/09 $8.00 Copyright © 2009 The American Physiological Societywww.jn.org on March 26, 2009 jn.physiology.orgDownloaded from!S2and motor variability!M2Therefore if the total task time isT and subjects switch from
View Full Document
Unlocking...