Visual Image Reconstruction from Human Brain Activity using a Combination of Multiscale Local Image DecodersIntroductionResultsReconstructed Visual ImagesImage Identification via ReconstructionWeight Distribution on the Cortical SurfaceAdvantage of Multivoxel Pattern DecodersReconstruction using Individual Visual AreasAdvantage of a Multiscale Reconstruction ModelAdvantage of Overlapping Multiscale BasesDiscussionDecoding from Multivoxel PatternsMultiple Scales of Visual RepresentationLinearity of Visual RepresentationModular Decoding and Its ApplicationsExperimental ProceduresSubjectsVisual Stimulus and Experimental DesignMRI AcquisitionMRI Data PreprocessingLabeling of fMRI DataTraining of Local DecodersCombination of Local DecodersEvaluation of PerformanceSupplemental DataAcknowledgmentsReferencesNeuronArticleVisual Image Reconstruction from HumanBrain Activity using a Combination ofMultiscale Local Image DecodersYoichi Miyawaki,1,2,6Hajime Uchida,2,3,6Okito Yamashita,2Masa-aki Sato,2Yusuke Morito,4,5Hiroki C. Tanabe,4,5Norihiro Sadato,4,5and Yukiyasu Kamitani2,3,*1National Institute of Information and Communications Technology, Kyoto, Japan2ATR Computational Neuroscience Laboratories, Kyoto, Japan3Nara Institute of Science and Technology, Nara, Japan4The Graduate University for Advanced Studies, Kanagawa, Japan5National Institute for Physiological Sciences, Aichi, Japan6These authors contributed equally to this work*Correspondence: [email protected] 10.1016/j.neuron.2008.11.004SUMMARYPerceptual experience consists of an enormous num-ber of possible states. Previous fMRI studies havepredicted a perceptual state by classifying brainactivity into prespecified categories. Constraint-freevisual image reconstruction is more challenging, asit is impractical to specify brain activity for all possibleimages. In this study, we recons tructed visual imagesby combining local image bases of multiple scales,whose contrasts were independently decoded fromfMRI activity by automatically selecting relevant vox-els and exploiting their correlated patterns. Binary-contrast, 10 3 10-patch images (2100possible states)were accurately reconstructed without any imageprior on a single trial or volume basis by measuringbrain activity only for several hundred randomimages. Reconstruction was also used to identifythe presented image among millions of candidates.The results suggest that our approach provides an ef-fective means to read out complex perceptual statesfrom brain activity while discovering informationrepresentation in multivoxel patterns.INTRODUCTIONObjective assessment of perceptual experience in terms of brainactivity represents a major challenge in neuroscience. PreviousfMRI studies have shown that visual features, such as orientationand motion direction (Kamitani and Tong, 2005, 2006), and visualobject categories (Cox and Savoy, 2003; Haxby et al., 2001) canbe decoded from fMRI activity patterns by a statistical ‘‘de-coder,’’ which learns the mapping between a brain activitypattern and a stimulus category from a training data set. Further-more, a primitive form of ‘‘mind-reading’’ has been demonstratedby predicting a subjective state under the presentation of an am-biguous stimulus using a decoder trained with unambiguousstimuli (Kamitani and Tong 2005, 2006; Haynes and Rees,2005). However, such a simple classification approach is insuffi-cient to capture the complexity of perceptual experience, sinceour perception consists of numerous possible states, and it is im-practical to measure brain activity for all the states. A recent study(Kay et al., 2008) has demonstrated that a presented image canbe identified among a large number of candidate images usinga receptive field model that predicts fMRI activity for visualimages (see also Mitchell et al., 2008, for a related approach).But the image identification was still constrained by the candidateimage set. Even more challenging is visual image reconstruction,which decodes visual perception into an image, free from theconstraint of categories (see Stanley et al., 1999, for reconstruc-tion using LGN spikes).A possible approach is to utilize the retinotopy in the earlyvisual cortex. The retinotopy associates the specific visual fieldlocation to the active cortical location, or voxel, providing a map-ping from the visual field to the cortical voxels (Engel et al., 1994;Sereno et al., 1995). Thus, one may predict local contrast infor-mation by monitoring the fMRI signals corresponding to theretinotopy map of the target visual field location. The retinotopycan be further elaborated using a voxel receptive-field model. Byinverting the receptive-field model, a presented image can beinferred given the brain activity consistent with the retinotopy(Thirion et al., 2006).However, it may not be optimal to use the retinotopy or theinverse of the receptive field model to predict local contrast inan image. These methods are based on the model of individualvoxel responses given a visual stimulus, and multivoxel patternsare not taken into account for the prediction of the visual stimulus.Recent studies have demonstrated the importance of the activitypattern, in particular the correlation among neurons or corticallocations in the decoding of a stimulus (Averbeck et al., 2006;Chen et al., 2006). Since even a localized small visual stimuluselicits spatially spread activity over multiple cortical voxels (Engelet al., 1997; Shmuel et al., 2007), multivoxel patterns may containinformation useful for predicting the presented stimulus.In addition, a visual image is thought to be represented atmultiple spatial scales in the visual cortex, which may serve toNeuron 60, 915–929, December 11, 2008 ª2008 Elsevier Inc. 915retain the visual sensitivity to fine-to-coarse patterns at a singlevisual field location (Campbell and Robson, 1968; De Valoiset al., 1982). The conventional retinotopy, by contrast, doesnot imply such multiscale representation, as it simply posits alocation-to-location mapping. It may be possible to extract mul-tiscale information from fMRI signals and use it to achieve betterreconstruction.Here, we present an approach to visual image reconstructionusing multivoxel patterns of fMRI signals and multiscale visualrepresentation (Figure 1A). We assume that an image is repre-sented by a linear combination of local image elements of multi-ple scales (colored rectangles). The stimulus state at each localelement (Ci,Cj, .) is predicted by a