Statistics 550 Notes 13 Reading: Section 2.3.Schedule: 1. Take home midterm due Wed. Oct. 25th2. No class next Tuesday due to fall break. We will have class on Thursday.3. The next homework will be assigned next week and due Friday, Nov. 3rd. I. Asymptotic Relative Efficiency (Clarification from last class)Consider two estimators nTand nUand suppose that 2( ) (0, )Lnn T N tq- �and that 2( ) (0, )Lnn U N uq- �.We define the asymptotic relative efficiency of Uto Tby2 2( , ) /ARE U T t u=. For 1, ,nX XK iid ( ,1)N q,1(Sample median,Sample mean) 0.63/ 2AREp= =.The interpretation is that if person A uses the sample median as her estimator of qand person B uses the sample mean as her estimator of q, person B needs a sample size that is only 0.63 as large as person A to obtain the same approximate variance of the estimator. 1Theorem: If ˆnqis the MLE and nq%is any other estimator, then ˆ( , ) 1n nARE q q �%.Thus, the MLE has the smallest asymptotic variance and we say that the MLE is asymptotically efficient and asymptotically optimal.Comments: (1) We will provide an outline of the proof for this theorem when we study the Cramer-Rao (information) inequality in Chapter 3.4; (2) The result is actually more subtle than the stated theorem because it only covers a certain class of well behaved estimators – more details will be study in Stat 552. II. Uniqueness and Existence of the MLEFor a finite sample, when does the MLE exist, when is it unique and how do we find the MLE? If Qis open, ( )l qxis differentiable in qand ˆMLEq exists, then ˆMLEq must satisfy the estimating equation( ) 0l� =qqxThis is known as the likelihood equation. But solving does not necessarily yield the MLE as there may be solutions of that are not maxima, or solutions that are only local maxima. Anomalies of maximum likelihood estimates:2Maximum likelihood estimates are not necessarily unique and do not even have to exist.Nonuniqueness of MLEs example: 1, ,nX XKare iid Uniform(1 1,2 2q q- +).-1 11 if max min( )2 20 otherwisei iX XLq�- < < +�=���xqThus any estimator ˆqthat satisfies1 1ˆmax min2 2i iX Xq- < < + is a maximum likelihood estimator.Nonexistence of maximum likelihood estimator: The likelihood function can be unbounded. An important example is a mixture of normal distributions, which is frequently used in applications.1, ,nX XKiid with density 2 21 22 21 21 2( ) ( )1 1( ) exp (1 ) exp2 22 2x xf x p pm ms sps ps� � � �- - - -= + -� � � �� � � �.This is a mixture of two normal distributions. The unknown parameters are 2 21 2 1 2( , , , , )p m m s s.Let 1 1Xm =. Then as 210s �, 1( )f X � �so that the likelihood function is unbounded. 3Example where the MLE exists and is unique: Normal distribution1, ,nX XKiid 2( , )N ms2211( )1 1( , , ; , ) exp22ninixf x xmmsss p=� �-� �= -� �� �� �� �� ��K2211( , ) log log 2 ( )2 2niinl n Xms s p ms==- - - -�The partials with respect tomand sareSetting the first partial equal to zero and solving for the mle, we obtainˆMLEXm =Setting the second partial equal to zero and substituting the mle for m, we find that the mle for sis211ˆ( )nMLE iiX Xns== -�.To verify that this critical point is a maximum, we need to check the following second derivative conditions:(1) The two second-order partial derivatives are negative:4213 211( )( )niiniilXl nXmm ss ms s=-=�= -��=- + -���22ˆ ˆ,0MLE MLElm m s sm= =�<� and 22ˆ ˆ,0MLE MLElm m s ss= =�<�(2) The Jacobian of the second-order partial derivatives is positive,2 222 22ˆ ˆ, 0 MLE MLEl ll lm m s sm msms s= =� �� ��>� ��� �See attached notes from Casella and Berger for verification of (1) and (2) for normal distribution.Conditions for uniqueness and existence of the MLE: We now provide a general condition under which there is a unique maximum likelihood estimator that is the solution tothe likelihood equation. The condition applies to many exponential families.Boundary of a parameter space: Suppose the parameter space pQ �� is an open set. Let �Q =Q- Qbe the boundary of Q, where Qdenotes the closure of Qin[ , ]p- ��. That is, �Qis the set of points outside of Qthat can be obtained as limits of points in Q, including all points with ��as a coordinate. For instance, for2 2~ ( , ), ( , ) ( , ) (0, )X N ms ms � - �� � �,5{( , ) : ,0 } {( , ) : , {0, }}a b a b a b a b�Q== ��������ΥConvergence of points to boundary: In general, for a sequence { }mqof points from Qopen, we define m� �Qqas m � �to mean that for any subsequence { }mkq, eithermk�q twith �Qtor mkqdiverges with | |mk� �qask � �where | |�denotes the Euclidean norm. Example: In the 2( , )N mscase,1 1( , ),( , ),( , ),( , ),( , )a m m b m b a m m m- --all tend to �Qas m � �.Lemma 2.3.1: Suppose we are given a function :l Q � �where pQ ��is open and lis continuous. Suppose also thatlim { ( ) : }l��� �Q =- �qq qQ.Then there exists ˆ�Qqsuch thatˆ( ) max{ ( ) : }l l= �Qq q q.Proof: Problem 2.3.5.Proposition 2.3.1: Suppose our model is that Xhas pdf or pmf ( | ),p �X q q Q, and that (i) ( )lxqis strictly concave; (ii) ( )l � - �xq as � �Qq. Then the maximum likelihood estimator exists and is unique.6Proof: ( )lxqis continuous because ( )l-xqis convex (see Appendix B.9). By Lemma 2.3.1, ˆMLEqexists. To prove uniqueness, suppose 1ˆq and 2ˆq are distinct maximizers of the likelihood, then( )1 1 2 1 21 1ˆ ˆ ˆ ˆ ˆ( ) ( ) ( ) ( )2 2l l l l� �= + < +� �� �x x x xq q q q qwith the inequality following from the strict concavity of( )lxq; this contradicts 1ˆq being a maximizer of the likelihood.Corollary: If the conditions of Proposition 2.3.1 are satisfied and ( )l qxis differentiable in q, then ˆMLEq is the unique solution to the estimating equation: ( ) 0l� =qqxApplication to Exponential Families: 1. Theorem 1.6.4, Corollary 1.6.5: For a full exponential family, the log likelihood is strictly concave. Consider the exponential family 1( | ) ( ) exp{ ( ) ( )}ki iip h T Ah== -�x x xh hNote that if ( )A his convex, then the log likelihood1log ( | ) log ( ) ( ) ( )ki iip h T Ah== + -�x x xh his concave in h.7Proof that ( )A his convex:Recall that 1( ) log ( )exp[ ( )ki iiA h T dh==���K x x xh. To show that ( )A his convex, we want to show that 1 2 1