IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 27, NO. 4, APRIL 2008 495Brain Anatomical Structure Segmentation by HybridDiscriminative/Generative ModelsZhuowen Tu, Katherine L. Narr, Piotr Dollár, Ivo Dinov, Paul M. Thompson, and Arthur W. Toga*Abstract—In this paper, a hybrid discriminative/generativemodel for brain anatomical structure segmentation is proposed.The learning aspect of the approach is emphasized. In the dis-criminative appearance models, various cues such as intensityand curvatures are combined to locally capture the complexappearances of different anatomical structures. A probabilisticboosting tree (PBT) framework is adopted to learn multiclassdiscriminative models that combine hundreds of features acrossdifferent scales. On the generative model side, both global andlocal shape models are used to capture the shape informationabout each anatomical structure. The parameters to combine thediscriminative appearance and generative shape models are alsoautomatically learned. Thus, low-level and high-level informationis learned and integrated in a hybrid model. Segmentations areobtained by minimizing an energy function associated with theproposed hybrid model. Finally, a grid-face structure is designedto explicitly represent the 3-D region topology. This representa-tion handles an arbitrary number of regions and facilitates fastsurface evolution. Our system was trained and tested on a set of3-D magnetic resonance imaging (MRI) volumes and the resultsobtained are encouraging.Index Terms—Brain anatomical structures, discriminativemodels, generative models, probabilistic boosting tree (PBT),segmentation.I. INTRODUCTIONSEGMENTING subcortical structures from 3-D brain im-ages is of significant practical importance, for example indetecting abnormal brain patterns [1], studying various braindiseases [2] and studying brain growth [3]. Fig. 1 shows anexample 3-D magnetic resonance imaging (MRI) brain volumeand subcortical structures delineated by a neuroanatomist.This subcortical structure segmentation task is very impor-tant but difficult to do even by hand. The various anatomicalstructures have similar intensity patterns (see Fig. 2) makingthese structures difficult to separate based solely on intensity.Furthermore, often there is no clear boundary between theregions. Neuroanatomists often develop and use complicatedprotocols [2] in guiding the manual delineation process andManuscript received May 4, 2007; revised August 31, 2007. This work wassupported by the National Institutes of Health through the NIH Roadmap forMedical Research under Grant U54 RR021813 entitled Center for Computa-tional Biology (CCB). Asterisk indicates corresponding author.Z. Tu, K. L. Narr, I. Dinov, and P. M. Thompson are with the Laboratory ofNeuro Imaging, UCLA Medical School, Los Angeles, CA 90095 USA.P. Dollár is with the Department of Computer Science, University of Cali-fornia, San Diego, La Jolla, CA 92093 USA.*A. W. Toga is with the Laboratory of Neuro Imaging, UCLA MedicalSchool, 635 Charles Young Drive, Los Angeles, CA 90095 USA.Color versions of one or more of the figures in this paper are available onlineat http://ieeexplore.ieee.org.Digital Object Identifier 10.1109/TMI.2007.908121Fig. 1. Illustration of an example 3-D MRI volume. (a) Example MRI volumewith skull stripped. (b) Manually annotated subcortical structures. The goal ofthis work is to design a system that can automatically create such annotations.Fig. 2. Intensity histograms of the eight subcortical structures targeted in thispaper and of the background regions. Note the high degree of overlap betweentheir intensity distributions, which makes these structures difficult to separatebased solely on intensity.those protocols may vary from task to task. A considerableamount of work is required to fully delineate even a single 3-Dbrain volume. Designing algorithms to automatically segmentbrain volumes is challenging in that it is difficult to transfersuch protocols into sound mathematical models or frameworks.In this paper, we use a mathematical model for subcorticalsegmentation that includes both the appearance (voxel intensi-ties) and shape (geometry) of each subcortical region. We use adiscriminative approach to model appearance and a generativemodel to describe shape, and learn and combine them in a prin-cipled manner.We apply our system to the segmentation of eight subcorticalstructures, namely: the left hippocampus (LH), the right hip-pocampus (RH), the left caudate (LC), the right caudate (RC),the left putamen (LP), the right putamen (RP), the left ventricle0278-0062/$25.00 © 2008 IEEE496 IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 27, NO. 4, APRIL 2008TABLE ICOMPARISON OFDIFFERENT3-D SEGMENTATIONALGORITHMS.NOTETHAT ONLY OURWORKCOMBINES ASTRONG GENERATIVESHAPEMODELWITH ADISCRIMINATIVEAPPEARANCE MODEL.IN THEABOVE, SVM REFERS TOSUPPORT VECTORMACHINE(LV), and the right ventricle (RV). We obtained encouraging re-sults. It is worth mentioning that our system is very adaptive andcan be directly used to segment other/more structures.A. Related WorkThere has been considerable recent work on 3-D segmen-tation in medical imaging and some representatives include[4]–[7], [15], [16]. Two systems particularly related to ourapproach are Fischl et al. [4] and Yang et al. [5], with whichwe will compare results. The 3-D segmentation problem isusually tackled in a maximize a posterior (MAP) frameworkin which both appearance models and shape priors are defined.Often, either a generative or a discriminative model is usedfor the appearance model, while the shape models are mostlygenerative based on either local or global geometry. Once anoverall target function is defined, different methods are thenapplied to find the optimal segmentation.Related work can be classified into two broad categories:methods that rely primarily on strong shape models andmethods that rely more on strong appearance models. Table Icompares some representative algorithms (this is not a completelist) for 3-D segmentation based on their appearance models,shape models, inference methods, and specific applications (wegive detailed descriptions below).The first class of methods, including [4]–[8], rely on stronggenerative shape models to perform 3-D segmentation. For theappearance models, each of these methods assumes that thevoxels are drawn from independent and identically-distributed(i.i.d.) Gaussians distributions. Fischl et al. [4] proposed asystem for whole brain