PreliminariesGreat ExpectationsFunctions of a Random VariableJointly Distributed Random VariablesCovariance and CorrelationSome Probability DistributionsLimit TheoremsStatistics TidbitsProbability and Statistics ReviewChristos Alexopoulos and Dave GoldsmanGeorgia Institute of Technology, Atlanta, GA, USA1/31/10Alexopoulos and Goldsman 1/31/10 1 / 45Outline1Preliminaries2Great Expectations3Functions of a Random Variable4Jointly Distributed Random Variables5Covariance and Correlation6Some Probability Distributions7Limit Theorems8Statistics TidbitsAlexopoulos and Goldsman 1/31/10 2 / 45PreliminariesPreliminariesWill assume that you know about sample spaces, events, and thedefinition of probability.Definition: P (A|B) ≡ P (A ∩ B)/P (B) is the conditional probability ofA given B.Example: Toss a fair die. Let A = {1, 2, 3} and B = {3, 4, 5, 6}. ThenP (A|B) =P (A ∩B)P (B)=1/64/6= 1/4. 2Alexopoulos and Goldsman 1/31/10 3 / 45PreliminariesDefinition: If P (A ∩B) = P (A)P (B), then A and B are independentevents.Theorem: If A and B are independent, then P (A|B) = P (A).Example: Toss two dice. Let A = “Sum is 7” and B = “First die is 4”.ThenP (A) = 1/6, P (B) = 1/6, andP (A ∩B) = P ((4, 3)) = 1/36 = P (A)P (B).So A and B are independent. 2Alexopoulos and Goldsman 1/31/10 4 / 45PreliminariesDefinition: A random variable (RV) X is a function from the samplespace Ω to the real line R, i.e., X : Ω → R.Example: Let X be the sum of two dice rolls. Then X((4, 6)) = 10. Inaddition,P (X = x) =1/36 if x = 22/36 if x = 3...1/36 if x = 120 otherwise2Alexopoulos and Goldsman 1/31/10 5 / 45PreliminariesDefinition: If the number of possible values of a RV X is finite orcountably infinite, then X is a discrete RV. Its probability mass function(pmf) is f(x) ≡ P (X = x). Note thatPxf(x) = 1.Example: Flip 2 coins. Let X be the number of heads.f(x) =1/4 if x = 0 or 21/2 if x = 10 otherwise2Examples: Here are some well-known discrete RV’s that you mayknow: Bernoulli(p), Binomial(n, p), Geometric(p), Negative Binomial,Poisson(λ), etc.Alexopoulos and Goldsman 1/31/10 6 / 45PreliminariesDefinition: A continuous RV is one with probability zero at everyindividual point. A RV is continuous if there exists a probability densityfunction (pdf) f (x) such that P (X ∈ A) =RAf(x) dx for every set A.Note thatRxf(x) dx = 1.Example: Pick a random number between 3 and 7. Thenf(x) =(1/4 if 3 ≤ x ≤ 70 otherwise2Examples: Here are some well-known continuous RV’s: Uniform(a, b),Exponential(λ), Normal(µ, σ2), etc.Alexopoulos and Goldsman 1/31/10 7 / 45PreliminariesDefinition: For any RV X (discrete or continuous), the cumulativedistribution function (cdf) isF (x) ≡ P (X ≤ x) =(Py ≤xf(y) if X is discreteRx−∞f(y) dy if X is continuousNote that limx→−∞F (x) = 0 and limx→∞F (x) = 1.Example: Flip 2 coins. Let X be the number of heads.F (x) =0 if x < 01/4 if 0 ≤ x < 13/4 if 1 ≤ x < 21 if x ≥ 22Example: Suppose X ∼ E xp(λ) (i.e., X has the exponentialdistribution with parameter λ). Then f(x) = λe−λx, x ≥ 0, and the cdfis F (x) = 1 − e−λx, x ≥ 0. 2Alexopoulos and Goldsman 1/31/10 8 / 45Great ExpectationsOutline1Preliminaries2Great Expectations3Functions of a Random Variable4Jointly Distributed Random Variables5Covariance and Correlation6Some Probability Distributions7Limit Theorems8Statistics TidbitsAlexopoulos and Goldsman 1/31/10 9 / 45Great ExpectationsGreat ExpectationsDefinition: The expected value (or mean) of a RV X isE[X] ≡(PxxP (X = x) if X is discreteRRxf(x) dx if X is continuousExample: Suppose that X ∼ Bern ou lli(p). ThenX =(1 with prob. p0 with prob. 1 − p (= q)and we have E[X] =Pxxf(x) = p. 2Example: Suppose that X ∼ Uniform(a, b). Thenf(x) =(1b−aif a < x < b0 otherwiseand we have E[X] =RRxf(x) dx = (a + b)/2. 2Alexopoulos and Goldsman 1/31/10 10 / 45Great Expectations“Definition” (the “Law of the Unconscious Statistician”): Suppose thatg(X) is some function of the RV X. ThenE[g(X)] =(Pxg(x)f(x) if X is discreteRRg(x)f(x) dx if X is continuous.Example: Suppose X is the following discrete RV:x 2 3 4f(x) 0.3 0.6 0.1Then E[X3] =Pxx3f(x) = 8(0.3) + 27(0.6) + 64(0.1) = 25. 2Example: Suppose X ∼ U(0, 2). ThenE[Xn] =ZRxnf(x) dx = 2n/(n + 1). 2Alexopoulos and Goldsman 1/31/10 11 / 45Great ExpectationsDefinitions: E[Xn] is the nth moment of X. E[(X − E[X])n] is the nthcentral moment of X.Var(X) ≡ E[(X − E[X])2] = E[X2] −(E[X])2is the variance of X.Example: Suppose X ∼ Bern(p). Recall that E[X] = p. ThenE[X2] =Xxx2f(x) = p andVar(X) = E[X2] −(E[X])2= p(1 − p). 2Example: Suppose X ∼ U(0, 2). By previous examples, E[X] = 1 andE[X2] = 4/3. SoVar(X) = E[X2] −(E[X])2= 1/3. 2Theorem: E[aX + b] = aE[X] + b and Var(aX + b) = a2Var(X).Alexopoulos and Goldsman 1/31/10 12 / 45Functions of a Random VariableOutline1Preliminaries2Great Expectations3Functions of a Random Variable4Jointly Distributed Random Variables5Covariance and Correlation6Some Probability Distributions7Limit Theorems8Statistics TidbitsAlexopoulos and Goldsman 1/31/10 13 / 45Functions of a Random VariableFunctions of a Random VariableProblem: Suppose we have a RV X with pdf/pmf f(x). Let Y = h(X).Find g(y), the pdf/pmf of Y .Discrete Example: Let X denote the number of H’s from two cointosses. We want the pmf for Y = X2− X.x 0 1 2f(x) 1/4 1/2 1/4y = x2− x 0 0 2This implies that g(0) = P (Y = 0) = P (X = 0 or 1) = 3/4 andg(2) = P (Y = 2) = 1/4. In other words,g(y) =(3/4 if y = 01/4 if y = 2. 2Alexopoulos and Goldsman 1/31/10 14 / 45Functions of a Random VariableContinuous Example: Suppose X has pdf f(x) = |x|, −1 ≤ x ≤ 1.Find the pdf of Y = X2.First of all, the cdf of Y isG(y) = P (Y ≤ y)= P (X2≤ y)= P (−√y ≤ X ≤√y)=Z√y−√y|x|dx = y, 0 < y < 1.Thus, the pdf of Y is g(y) = G′(y) = 1, 0 < y < 1, indicating thatY ∼ Unif(0, 1). 2Alexopoulos and Goldsman 1/31/10 15 / 45Functions of a Random VariableInverse Transform Theorem: Suppose X is a continuous randomvariable having cdf F (x). Then, amazingly, F(X) ∼ Unif(0, 1).Proof: Let Y = F (X). Then the cdf of Y isP (Y ≤ y) = P (F (X) ≤ y)= P (X ≤ F−1(y))= F (F−1(y)) = y,which is the cdf of the Unif(0,1). 2This result is of fundamental importance when it comes to generatingrandom variates during a simulation.Alexopoulos and Goldsman 1/31/10 16 / 45Functions of a Random