DefinitionLinksEstimating parametersExampleModel buildingConclusionsSummaryGeneralized linear modelsDouglas BatesNovember 01, 2010Contents1 Definition 12 Links 23 Estimating parameters 54 Example 65 Model building 86 Conclusions 87 Summary 91 Generalized Linear ModelsGeneralized Linear Models• When using linear models (LMs) we assume that the response being modeled is on acontinuous scale.• Sometimes we can bend this assumption a bit if the response is an ordinal response witha moderate to large number of levels. For example, the Scottish secondary school testresults in the mlmRev package are integer values on the scale of 1 to 10 but we analyzethem on a continuous scale.• However, an LM is not suitable for modeling a binary response, an ordinal response withfew levels or a response that represents a count. For these we use generalized linearmodels (GLMs).• To describe GLMs we return to the representation of the response as an n-dimensional,vector-valued, random variable, Y.Parts of LMs carried over to GLMs• Random variables– Y the response variable1• Parameters– β - fixed-effects coefficients– σ - a scale parameter (not always used)• The linear predictor Xβ where– X is the n × p model matrix for βThe probability model• For GLMs we retain some of the properties of the LM probability modelY ∼ NµY, σ2Iwhere µY= XβSpecifically– The distribution of Y depends on β only through the mean, µY= Xβ.– Elements of Y are independent. That is, the distribution of Y is completely specifiedby the univariate distributions, Yi, i = 1, . . . , n.– These univariate, distributions all have the same form. They differ only in theirmeans.• GLMs differ from LMs in the form of the univariate distributions and in how µYdependson the linear predictor, Xβ.2 Specific distributions and linksSome choices of univariate distributions• Typical choices of univariate distributions are:– The Bernoulli distribution for binary (0/1) data, which has probability mass func-tionp(y|µ) = µy(1 − µ)1−y, 0 < µ < 1, y = 0, 1– Several independent binary responses can be represented as a binomial response,but only if all the Bernoulli distributions have the same mean.– The Poisson distribution for count (0, 1, . . . ) data, which has probability mass func-tionp(y|µ) = e−µµyy!, 0 < µ, y = 0, 1, 2, . . .• All of these distributions are completely specified by the mean. This is different from thenormal (or Gaussian) distribution, which also requires the scale parameter, σ.2The link function, g• When the univariate distributions have constraints on µ, such as 0 < µ < 1 (Bernoulli)or 0 < µ (Poisson), we cannot define the mean, µY, to be equal to the linear predictor,Xβ, which is unbounded.• We choose an invertible, univariate link function, g, such that η = g(µ) is unconstrained.The vector-valued link function, g, is defined by applying g component-wise.η = g(µ) where ηi= g(µi), i = 1, . . . , n• We require that g be invertible so that µ = g−1(η) is defined for −∞ < η < ∞ and isin the appropriate range (0 < µ < 1 for the Bernoulli or 0 < µ for the Poisson). Thevector-valued inverse link, g−1, is defined component-wise.“Canonical” link functions• There are many choices of invertible scalar link functions, g, that we could use for a givenset of constraints.• For the Bernoulli and Poisson distributions, however, one link function arises naturallyfrom the definition of the probability mass function. (The same is true for a few other,related but less frequently used, distributions, such as the gamma distribution.)• To derive the canonical link, we consider the logarithm of the probability mass function(or, for continuous distributions, the probability density function).• For distributions in this “exponential” family, the logarithm of the probability mass ordensity can be written as a sum of terms, some of which depend on the response, y, onlyand some of which depend on the mean, µ, only. However, only one term depends onboth y and µ, and this term has the form y · g(µ), where g is the canonical link.The canonical link for the Bernoulli distribution• The logarithm of the probability mass function islog(p(y|µ)) = log(1 − µ) + y logµ1 − µ, 0 < µ < 1, y = 0, 1.• Thus, the canonical link function is the logit linkη = g(µ) = logµ1 − µ.• Because µ = P [Y = 1], the quantity µ/(1 − µ) is the odds ratio (in the range (0, ∞)) andg is the logarithm of the odds ratio, sometimes called “log odds”.• The inverse link isµ = g−1(η) =eη1 + eη=11 + e−η3Plot of canonical link for the Bernoulli distributionµη = log(µ1 − µ)−5050.0 0.2 0.4 0.6 0.8 1.0Plot of inverse canonical link for the Bernoulli distributionηµ =11 + exp(−η)0.00.20.40.60.81.0−5 0 5Evaluating links for the Bernoulli• As a monotone increasing function the maps (−∞, ∞) to (0, 1) the inverse link, g−1, isa cumulative distribution function for a continuous random variable. The link, q, is thecorresponding quantile function.• The canonical link is the quantile function for the logistic distribution (R functions qlogisand plogis).4• In the past the "probit" link was sometimes used instead of the logit link. For thisthe link is the standard normal quantile and the inverse link is the standard normalcumulative, sometimes written Φ(z) (R functions qnorm and pnorm).• As described in R’s help page for the binomial functionthe binomial family (allows) the links "logit", "probit", "cauchit", (cor-responding to logistic, normal and Cauchy CDFs respectively) "log" and"cloglog" (complementary log-log)The canonical link for the Poisson distribution• The logarithm of the probability mass islog(p(y|µ)) = log(y!) − µ + y log(µ)• Thus, the canonical link function for the Poisson is the log linkη = g(µ) = log(µ)• The inverse link isµ = g−1(η) = eηThe canonical link related to the variance• For the canonical link function, the derivative of its inverse is the variance of the response.• For the Bernoulli, the canonical link is the logit and the inverse link is µ = g−1(η) =1/(1 + e−η). Thendµdη=e−η(1 + e−η)2=11 + e−ηe−η1 + e−η= µ(1 − µ) = Var(Y)• For the Poisson, the canonical link is the log and the inverse link is µ = g−1(η) = eη.Thendµdη= eη= µ = Var(Y)3 Estimating parametersEstimating parameters• We determine the maximum likelihood