Abstract | A conspicuous ability of the brain is to seamlessly assimilate and process spatial and temporal features of sensory stimuli. This ability is indispensable for the recognition of natural stimuli. Yet, a general computational framework for processing spatiotemporal stimuli remains elusive. Recent theoretical and experimental work suggests that spatiotemporal processing emerges from the interaction between incoming stimuli and the internal dynamic state of neural networks, including not only their ongoing spiking activity but also their ‘hidden’ neuronal states, such as short-term synaptic plasticity.Box 1 | Spatialization of timeInputs interact with internal statesFigure 1 | Trajectories of active and hidden states. a | A schematic of a neural trajectory. If we consider the firing pattern of two neurons over five time bins, we can visualize the trajectory of this two-neuron network by plotting the number of spikes of each neuron during each time bin on the axes of a two-dimensional plot. The spikes generated by two different hypothetical stimuli are represented in blue and red, and each produces a different neural trajectory (lower plot). Importantly, each point on the trajectory can potentially be used to determine not only which stimulus was presented, but also how long ago the stimulus was presented (colour-coded circles). Thus, the neural trajectory can inherently encode spatial and temporal stimulus features. The coordinates represent the number of spikes of each neuron at each time bin (derived from the upper plot). b | An example of the active trajectory of a population of neurons from the locust antennal lobe. For a large number of neurons it is possible to use mathematical techniques to visualize their trajectory. In this case 87 projection neurons from the locust were recorded during multiple presentations of 2 odours (citral and geraniol). These data were used to calculate the firing rate of each neuron using 50 ms time bins. The 87 vectors were then reduced to 3 dimensions. The resulting three-dimensional plot reveals that each odour produces a different trajectory, and thus different spatiotemporal patterns of activity. The numbers along the trajectory indicate time points (seconds), and the point marked B indicates the resting state of the neuronal population. Part b is modified, with permission, from REF. 24  (2006) Cell Press.Figure 2 | Active and hidden network states. a | An example of short-term plasticity of excitatory postsynaptic potentials (EPSPs) in excitatory synapses between layer 5 pyramidal neurons. Short-term plasticity can take the form of either short-term depression (left) or short-term facilitation (right). The plots show that the strength of synapses can vary dramatically as a function of previous activity, and thus function as a short-lasting memory trace of the recent stimulus history. The traces represent the EPSPs from paired recordings; each presynaptic action potential is marked by a dot. b | Hidden and active states in a network. The spheres represent excitatory (blue) and inhibitory (red) neurons, and the arrows represent a small sample of the potential connections. The baseline state (‘rest’ state, top left panel) is represented as a quiescent state (although in reality background and spontaneous activity must be taken into account). In the presence of a brief stimulus the network response will generate action potentials in a subpopulation of the excitatory and inhibitory neurons (light shades), which defines the active state of the network (top right panel). After the stimulus, the neurons in early cortical areas stop firing. However, as a result of short-term synaptic plasticity (represented by dashed lines) and changes in intrinsic and synaptic currents (represented by different colour shades), the internal state may continue to change for hundreds of milliseconds. Thus, although it is quiescent, the network should be in a different functional state at the time of the next stimulus (at t = 100 ms) — this is referred to as the ‘hidden’ state (bottom left panel). The fact that the network is in a different state implies that it should generate a different response pattern to the next stimulus (bottom right panel), even if the stimulus is identical to the first one (represented as a different pattern of blue spheres). Part a is reproduced, with permission, from REF. 31  (1999) Society for Neuroscience.Decoding neural trajectoriesFigure 3 | Discrimination of complex spatiotemporal patterns. a | A sample spectrogram of the spoken word ‘one’. b | A spatiotemporal pattern of action potentials representing the word ‘one’. Cochlear models can be used to generate a spatiotemporal pattern of spikes generated by the word ‘one’ (left lower panel). This pattern can be reversed (right lower panel) to ensure that the network is discriminating the spatiotemporal patterns of action potentials, as opposed to only the spatial structure. One can perform a principal-component analysis on the spikes of the input patterns, and by plotting the first three dimensions create a visual representation of the input trajectory. The upper panels show that the trajectories are identical except that they flow in opposite temporal directions. Time is represented in colour: note the reverse colour gradient. c | Active states in a cortical microcircuit model. The raster of the recurrent network in response to the forward (blue) and the reverse (red) direction is plotted. The fact that the spatiotemporal spike patterns are no longer simply reverse representations of each other can be seen in the neural trajectories (lower plots). The response shown represents a subsample of the 280 layer-5 neurons of the model described by Haeusler and Maass48. The trajectory calculations plotted the fourth, fifth and sixth dimensions of the principal-component analysis to improve visualization. d | A linear read-out can distinguish between the original speech input and its time reversal at most points in time. A single linear read-out that received synaptic inputs from all neurons in the circuit was trained to produce large output values for any active state that occurred when the word ‘one’ was spoken, but low output values at any time during the time-reversed version of ‘one’. The resulting output values of the read-out are shown for a new trial that included noise injections into the neurons. The fact that this simple linear read-out