CHAPTER 3 LARGE SAMPLE THEORY 1CHAPTER 3: LARGE SAMPLE THEORYCHAPTER 3 LARGE SAMPLE THEORY 2IntroductionCHAPTER 3 LARGE SAMPLE THEORY 3• Why large sample theory– studying small sample property is usually difficultand complicated– large sample theory studies the limit behavior of asequence of random variables, say Xn.– example:¯Xn→ µ,√n(¯Xn− µ)CHAPTER 3 LARGE SAMPLE THEORY 4Modes of ConvergenceCHAPTER 3 LARGE SAMPLE THEORY 5• Convergence almost surelyDefinition 3.1 Xnis said to converge almost surely toX, denoted by Xn→a.s.X, if there exists a set A ⊂ Ωsuch that P (Ac) = 0 and for each ω ∈ A, Xn(ω) → X(ω)in real space.CHAPTER 3 LARGE SAMPLE THEORY 6• Equivalent condition{ω : Xn(ω) → X(ω)}c= ∪ϵ>0∩n{ω : supm≥n|Xm(ω) − X(ω)| > ϵ}⇒ Xn→a.s.X iffP (supm≥n|Xm− X| > ϵ) → 0CHAPTER 3 LARGE SAMPLE THEORY 7• Convergence in probabilityDefinition 3.2 Xnis said to converge in probability toX, denoted by Xn→pX, if for every ϵ > 0,P (|Xn− X| > ϵ) → 0.CHAPTER 3 LARGE SAMPLE THEORY 8• Convergence in moments/meansDefinition 3.3 Xnis said to converge in rth mean to X,denote by Xn→rX, ifE[|Xn−X|r] → 0 as n → ∞ for functions Xn, X ∈ Lr(P ),where X ∈ Lr(P ) means∫|X|rdP < ∞.CHAPTER 3 LARGE SAMPLE THEORY 9• Convergence in distributionDefinition 3.4 Xnis said to converge in distribution ofX, denoted by Xn→dX or Fn→dF (or L(Xn) → L(X)with L referring to the “law” or “distribution”), if thedistribution functions Fnand F of Xnand X satisfyFn(x) → F (x) as n → ∞ for each continuity point x of F .CHAPTER 3 LARGE SAMPLE THEORY 10• Uniform integrabilityDefinition 3.5 A sequence of random variables { Xn} isuniformly integrable iflimλ→∞lim supn→∞E {|Xn|I(|Xn| ≥ λ)} = 0.CHAPTER 3 LARGE SAMPLE THEORY 11• A note– Convergence almost surely and convergence inprobability are the same as we defined in measuretheory.– Two new definitions are∗ convergence in rth mean∗ convergence in distributionCHAPTER 3 LARGE SAMPLE THEORY 12• “convergence in distribution”– is very different from others– example: a sequence X, Y, X, Y, X, Y, .... where X andY are N(0, 1); the sequence converges in distributionto N(0, 1) but the other modes do not hold.– “convergence in distribution” is important forasymptotic statistical inference.CHAPTER 3 LARGE SAMPLE THEORY 13• Relationship among different modesTheorem 3.1 A. If Xn→a.s.X, then Xn→pX.B. If Xn→pX, then Xnk→a.s.X for some subsequenceXnk.C. If Xn→rX, then Xn→pX.D. If Xn→pX and |Xn|ris uniformly integrable, thenXn→rX.E. If Xn→pX and lim supnE|Xn|r≤ E|X|r, thenXn→rX.CHAPTER 3 LARGE SAMPLE THEORY 14F. If Xn→rX, then Xn→r′X for any 0 < r′≤ r.G. If Xn→pX, then Xn→dX.H. Xn→pX if and only if for every subsequence {Xnk}there exists a further subsequence {Xnk,l} such thatXnk,l→a.s.X.I. If Xn→dc for a constant c, then Xn→pc.CHAPTER 3 LARGE SAMPLE THEORY 15CHAPTER 3 LARGE SAMPLE THEORY 16ProofA and B follow from the results in the measure theory.Prove C. Markov inequality: for any increasing function g(·) andrandom variable Y , P (|Y | > ϵ) ≤ E[g(|Y |)g(ϵ)].⇒P (|Xn− X| > ϵ) ≤ E[|Xn−X|rϵr] → 0.CHAPTER 3 LARGE SAMPLE THEORY 17Prove D. It is sufficient to show that for any subsequence of {Xn},there exists a further subsequence {Xnk} such thatE|Xnk− X|r→ 0.For any subsequence of {Xn}, from B, there exists a furthersubsequence {Xnk} such that Xnk→a.s.X. For any ϵ, there existsλ such that lim supnkE[|Xnk|rI(|Xnk|r≥ λ)] < ϵ.Particularly, choose λ such that P (|X|r= λ) = 0⇒ |Xnk|rI(|Xnk|r≥ λ) →a.s.|X|rI(|X|r≥ λ).⇒ By the Fatou’s Lemma,E[|X|rI(|X|r≥ λ)] ≤ lim supnkE[|Xnk|rI(|Xnk|r≥ λ)] < ϵ.CHAPTER 3 LARGE SAMPLE THEORY 18⇒E[|Xnk− X|r]≤ E[|Xnk− X|rI(|Xnk|r< 2λ, |X|r< 2λ)]+E[|Xnk− X|rI(|Xnk|r≥ 2λ, or , |X|r≥ 2λ)]≤ E[|Xnk− X|rI(|Xnk|r< 2λ, |X|r< 2λ)]+2rE[(|Xnk|r+ |X|r)I(|Xnk|r≥ 2λ, or , |X|r≥ 2λ)],where the last inequality follows from the inequality(x + y)r≤ 2r(max(x, y))r≤ 2r(xr+ yr), x ≥ 0, y ≥ 0.When nkis large, the second term is bounded by2 ∗ 2r{E[|Xnk|rI(|Xnk| ≥ λ)] + E[|X|rI(|X| ≥ λ)]} ≤ 2r+1ϵ.⇒ lim supnE[|Xnk− X|r] ≤ 2r+1ϵ.CHAPTER 3 LARGE SAMPLE THEORY 19Prove E. It is sufficient to show that for any subsequence of {Xn},there exists a further subsequence {Xnk} such thatE[|Xnk− X|r] → 0.For any subsequence of {Xn}, there exists a further subsequence{Xnk} such that Xnk→a.s.X. DefineYnk= 2r(|Xnk|r+ |X|r) − |Xnk− X|r≥ 0.⇒ By the Fatou’s Lemma,∫lim infnkYnkdP ≤ lim infnk∫YnkdP.It is equivalent to2r+1E[|X|r] ≤ lim infnk{2rE[|Xnk|r] + 2rE[|X|r] − E[|Xnk− X|r]}.CHAPTER 3 LARGE SAMPLE THEORY 20Prove F. The H¨older inequality:∫|f(x)g(x)|dµ ≤{∫|f(x)|pdµ(x)}1/p{∫|g(x)|pdµ(x)}1/q,1p+1q= 1.Choose µ = P , f = |Xn−X|r′, g ≡ 1 and p = r/r′, q = r/(r −r′) inthe H¨older inequality⇒E[|Xn− X|r′] ≤ E[|Xn− X|r]r′/r→ 0.CHAPTER 3 LARGE SAMPLE THEORY 21Prove G. Xn→pX. If P (X = x) = 0, then for any ϵ > 0,P (|I(Xn≤ x) − I(X ≤ x)| > ϵ)= P (|I(Xn≤ x) − I(X ≤ x)| > ϵ, |X − x| > δ)+P (|I(Xn≤ x) − I(X ≤ x)| > ϵ, |X − x| ≤ δ)≤ P (Xn≤ x, X > x + δ) + P (Xn> x, X < x − δ)+P (|X −x| ≤ δ)≤ P (|Xn− X| > δ) + P (|X − x| ≤ δ).The first term converges to zero since Xn→pX.The second term can be arbitrarily small if δ is small, sincelimδ→0P (|X −x| ≤ δ) = P (X = x) = 0.⇒ I(Xn≤ x) →pI(X ≤ x)⇒ Fn(x) = E[I(Xn≤ x)] → E[I(X ≤ x)] = F (x).CHAPTER 3 LARGE SAMPLE THEORY 22Prove H. One direction follows from B.To prove the other direction, use the contradiction. Suppose thereexists ϵ > 0 such that P (|Xn− X| > ϵ) does not converge to zero.⇒ find a subsequence {Xn′} such hat P (|Xn′− X| > ϵ) > δ forsome δ > 0.However, by the condition, there exists a further subsequence Xn′′such that Xn′′→a.s.X then Xn′′→pX from A. Contradiction!CHAPTER 3 LARGE SAMPLE THEORY 23Prove I. Let X ≡ c.P (|Xn− c| > ϵ ) ≤ 1 − Fn(c + ϵ) + Fn(c − ϵ)→ 1 − FX(c + ϵ) + F (c − ϵ) = 0.CHAPTER 3 LARGE SAMPLE THEORY 24• Some counter-examples(Example 1 ) Suppose that Xnis degenerate at a point1/n; i.e., P (Xn= 1/n) = 1. Then Xnconverges indistribution to zero. Indeed, Xnconverges almost surely.CHAPTER 3 LARGE SAMPLE THEORY 25(Example 2 ) X1, X2, ... are i.i.d with standard