� � Discrete Choice and Censoring Whitney K. Newey MIT November, 2004 Discrete Choice Multinomial Choice: Data consists of consumer choices of various goods, along char-acteristics. Let there be J choices and y = (y1, . . . , yJ ) where yj = 1 if good j is chosen and yj = 0 otherwise. Let x be o bserved characteristics of the goods and choices. Here a conditional density for y corresponds to conditional choice probabilities P (j|x, β), one for each j, with �Jj=1 P (j|x, β) = 1 for all β and x. Then J ln f(y|x, β) = yj ln P (j|x, β). j=1 Fo r example, the multinmial logit model has x = (x1, . . . , xJ ) and � jβxe P (j|x, β) = . �J� kβxek=1 This model has a random utility interpretation. If the utility of choice j is xj � β + εj where εj are i.i.d. over j with Type I Extreme Va lue distributions (with CDF e−e−ε ), then the probability that j has the highest utility, and is thus chosen, has the form given above. McFadden used this to predict the effect of the introduction of BART on ridership of public and private transportation in the San Fr ancisco Bay area. One problem with this model is that P (j|x, β)/P (k|x, β) = e x� jβ−x� kβ depends only on the characteristics of alternatives j and k (this is called the independence from irrelevant alternatives property, or IIA). Approaches to deal with this include allowing β to be random and allowing εj to be correlated with each other. Allowing β to be random would lead to choice probabilities of the form � jγxe P (j|x, β) = h(γ|β)dγ. �J� kγxek=1 Hard to compute. Multivariate normal εj probabilities (called multinomial probit) also hard to compute. A case with correlated εj that can be computed is nested log it. For y = ln(exp(x1� β/λ) + exp(x2� β/λ)), � jβ/λλye xe P (j|x, β) = . j = 1, 2, � 3� 3β + eλy)ey(exβxeP (3|x, β) = . ex� 3β + eλy Cite as: Whitney Newey, course materials for 14.385 Nonlinear Econometric Analysis, Fall 2007. MIT OpenCourseWare (http://ocw.mit.edu), Massachusetts Institute of Technology. Downloaded on [DD Month YYYY]. 1λ(t, α) =212; Weibull; allows ∂λ> 0,∂λ< 0α ∂t ∂t � � Multinomial logit on branches. Duration Models T : Lifetime or duration (e.g., unemployment, firm lifetimes). x: Regressors (covariates). Goal: Estimate effect of x on T ; also estimate how conditional density of T depends on T . Important general issue is censoring. General parametric model: Let θ be a parameter vector, x regressors, a nd conditional survivor function S(t | x, θ) = Pr(T ≥ t | x, θ) Complete model for conditional distribution of T given x; other ways to describe this model. − df(t | x, θ) = dt S(t | x, θ); conditional pdf λ(t | x, θ) = f(t|x,θ) = − d ln S(t | x, θ); hazard rate S(t|x,θ) dt � tΛ(t | x, θ) = 0 λ(t | x, θ); integrated hazard Relationships: above and S(t|x, θ) = exp(−Λ(t|x, θ)). We use the representat io n that is most convenient for a particular application. Some theories imply things abo ut hazard, e.g. declining reservatio n wage in search theory implies ∂λ (t|x, θ) > 0, when T is the length of an umemployment spell. ∂t Historically important class of models are proportional hazards λ(t | x, θ) = λ(t, α) exp(x �β), θ = α. β Here changes in x just shift hazard up and down, i.e., shape of hazard as function o f t entirely determined by λ(t; α) λ(t | x, θ) λ(t, α) = . λ(˜λ(˜t | x, θ) t, α) Motivation: Convenient starting point, computationally, historically. Also implied by some theoretical models. See Heckman chapter, Handbook of Econometrics, Volume 2. λ(t, α) is called “baseline hazard”. Examples: λ(t, α) = α; constant α tα −1 Cite as: Whitney Newey, course materials for 14.385 Nonlinear Econometric Analysis, Fall 2007. MIT OpenCourseWare (http://ocw.mit.edu), Massachusetts Institute of Technology. Downloaded on [DD Month YYYY]. 2n i � � � � � � � Censoring Have to account for effects of sampling in likelihood. Longer spells will be more likely to appear in data when sampling at fixed points in time. See diagram. Length biased sampling. A general principle illustrated here is that ignoring sampling based on the endogenous variable will lead to inconsistent estimates. Case 1. Random sample of completed spells. Observations are (T1, x1), . . . , (Tn, xn) The MLE maximizes Qˆn(θ) = 1 ln f(Ti|xi, θ) n i Almost never have da t a like this. Case 2. Sample of unemployed people. Sample unemployed, ask them how long they have been unemployed, and then follow them until they are employed again. So, the likelihood must condition on the fact that they are unemployed when surveyed. If we know that they have been unemployed for ti periods, then condition on Ti ≥ ti. (This setting analogo us to that where only observe data when Yi is positive). The MLE maximizes ˆ1 f(Ti|xi, θ)Qn(θ) = ln n S(ti|xi, θ)i 1 = [ln f(Ti|xi, θ) − ln S(ti|xi, θ)] n i Case 3. Right censoring. Do not observe completed spells for everyone. For some we just know duration is greater than ci. For example, we survey unemployed once and find ti, then survey them sometime later, and record when spell ended or whether they are still unemployed. Let di = 1 if complete spell is observed, di = 0 if only know lasts at least to ci. The MLE maximizes 1ˆQn(θ) = [di ln f(Ti|xi, θ) + (1 − di)S(ci|xi, θ) − ln S(ti|xi, θ)] n i Case 4. Discrete data. Do not know any durations. Only know whether spell is shorter or longer than ci. Like when survey the second time we only know whether unemployment has ended. 1ˆQn(θ) = {di ln[S(ti|xi, θ) − S(ci|xi, θ)] + (1 − di) ln S(ci|xi, θ) − ln S(ti|xi, θ)} Cite as: Whitney Newey, course materials for 14.385 Nonlinear Econometric Analysis, Fall 2007. MIT OpenCourseWare (http://ocw.mit.edu), Massachusetts Institute of Technology. Downloaded on [DD Month YYYY]. 3� Discrete data like this is common in applications, e.g. weeks