Bayesian Density Estimation and Inference Using MixturesMichael D. Escobar; Mike WestJournal of the American Statistical Association, Vol. 90, No. 430. (Jun., 1995), pp. 577-588.Stable URL:http://links.jstor.org/sici?sici=0162-1459%28199506%2990%3A430%3C577%3ABDEAIU%3E2.0.CO%3B2-8Journal of the American Statistical Association is currently published by American Statistical Association.Your use of the JSTOR archive indicates your acceptance of JSTOR's Terms and Conditions of Use, available athttp://www.jstor.org/about/terms.html. JSTOR's Terms and Conditions of Use provides, in part, that unless you have obtainedprior permission, you may not download an entire issue of a journal or multiple copies of articles, and you may use content inthe JSTOR archive only for your personal, non-commercial use.Please contact the publisher regarding any further use of this work. Publisher contact information may be obtained athttp://www.jstor.org/journals/astata.html.Each copy of any part of a JSTOR transmission must contain the same copyright notice that appears on the screen or printedpage of such transmission.JSTOR is an independent not-for-profit organization dedicated to and preserving a digital archive of scholarly journals. Formore information regarding JSTOR, please contact [email protected]://www.jstor.orgThu Apr 5 11:12:22 2007Bayesian Density Estimation and Inference Using Mixtures Michael D. ESCOBAR and Mike WEST* We describe and illustrate Bayesian inference in models for density estimation using mixtures of Dirichlet processes. These models provide natural settings for density estimation and are exemplified by special cases where data are modeled as a sample from mixtures of normal distributions. Efficient simulation methods are used to approximate various prior, posterior, and predictive distributions. This allows for direct inference on a variety of practical issues, including problems of local versus global smoothing, uncertainty about density estimates, assessment of modality, and the inference on the numbers of components. Also, convergence results are established for a general class of normal mixture models. KEY WORDS: Kernel estimation; Mixtures of Dirichlet processes; Multimodality; Normal mixtures; Posterior sampling; Smoothing parameter estimation. 1. INTRODUCTION Models for uncertain data distributions based on mixtures of standard components, such as normal mixtures, underly mainstream approaches to density estimation, including kernel techniques (Silverman 1986), nonparametric maxi- mum likelihood (Lindsay 1983), and Bayesian approaches using mixtures of Dirichlet processes (Ferguson 1983). The latter provide theoretical bases for more traditional non-parametric methods, such as kernel techniques, and hence a modeling framework within which the various practical problems of local versus global smoothing, smoothing pa- rameter estimation, and the assessment of uncertainty about density estimates may be addressed. In contrast with non- parametric approaches, a formal model allows these prob- lems to be addressed directly via inference about the relevant model parameters. We discuss these issues using data distri- butions derived as normal mixtures in the framework of mixtures of Dirichlet processes, essentially the framework of Ferguson ( 1983). West ( 1990) discussed these models in a special case of the framework studied here. West's paper is concerned with developing approximations to predictive distributions based on a clustering algorithm motivated by the model structure and draws obvious connections with kernel approaches. The current article develops, in a more general framework, a computational method that allows for the evaluation of posterior distributions for all model pa- rameters and direct evaluation of predictive distributions. As a natural by-product, we develop approaches to inference about the numbers of components and modes in a popula- tion distribution. The computational method developed here is a direct ex- tension of the method of Escobar ( 1988, 1994) and is another example of a Gibbs sampler or Markov Chain Monte Carlo method recently been popularized by Gelfand and Smith * Michael D. Escobar is Assistant Professor, Department of Statistics and Department of Preventive Medicine and Biostatistics, University of Toronto, ON M5S 1A8, Canada. Mike West is Professor and Director, Institute of Statistics and Decision Sciences, Duke University, Durham, NC 27708. Mi- chael D. Escobar was partially financed by National Cancer Institute Grant R01-CA54852-01, a National Research Service Award from National In- stitutes of Mental Health Grant MH 15758 and by the National Science and Engineering Research Council of Canada. Mike West was partially financed by National Science Foundation Grants DMS-8903842 and DMS-9024793. The authors would like to thank Hani Doss and Steve MacEachern for helpful discussions. ( 1990). Some of the earlier references on Markov Chain Monte Carlo methods include work of Geman and Geman ( 1984), Hasting ( l970), Metropolis et al. ( 1953), and Tan- ner and Wong ( 1987). Besag and Green ( 1993) and Smith and Roberts ( 1993) recently reviewed Markov Chain Monte Carlo methods. The basic normal mixture model, similar to that of Fer- guson ( 1983) ,is described as follows. Suppose that data Y, , . . . ,Y, are conditionally independent and normally distrib- uted, (Yi I ri) --N(pi, Vi ), with means pi and variances Vi determining the parameters ri = (pi, V,), i = 1, . . . , n. Suppose further that the ri come from some prior distri- bution on 'R X 'R'. Having observed data D, = {y,, . . . , y,) , with yi the observed value of Yi, the distribution of a future case is a mixture of normals; the relevant density function Yn+l --N(P,+~, V,+l) mixed with respect to the posterior predictive distribution for (r,,, 1 D,). If the com- mon prior distribution for the riis uncertain and modeled, in whole or in part, as a Dirichlet process, then the data come from a Dirichlet mixture of normals (Escobar 1988, 1994; Ferguson 1983; West 1990). The important special case in which Vi = V has been studied widely; references were provided by West ( 1990, 1992), who considered the common setup in which the pi have an uncertain prior that is modeled as a Dirichlet process with a normal base measure (see also West and Cao 1993 ) . The connections with kernel estimation techniques are explored in these