Today's classNonlinear regression modelsWeight loss dataWhat to do?Delta methodNonlinear regressionNonlinear regression: detailsIteration & DistributionConfidence intervalsWeight loss data- p. 1/11Statistics 203: Introduction to Regressionand Analysis of VarianceNonlinear regressionJonathan Taylor●Today’s class●Nonlinear regression models●Weight loss data●What to do?●Delta method●Nonlinear regression●Nonlinear regression: details●Iteration & Distribution●Confidence intervals●Weight loss data- p. 2/11Today’s class■Nonlinear regression.■Examples in R.●Today’s class●Nonlinear regression models●Weight loss data●What to do?●Delta method●Nonlinear regression●Nonlinear regression: details●Iteration & Distribution●Confidence intervals●Weight loss data- p. 3/11Nonlinear regression models■We have usually assumed regression is of the formYi= β0+p−1Xj=1βjXij+ εi.■Or, the regression functionf(x, β) = β0+p−1Xj=1βjxjis linear in beta.■Many real-life phenomena can be parameterized bynon-linear regression functions. Example:◆Radioactive decay: half-life is a non-linear parameterf(t, θ) = C · 2−t/θ.- p. 4/11Weight loss data0 50 100 150 200 250110 120 130 140 150 160 170 180DaysWeight●Today’s class●Nonlinear regression models●Weight loss data●What to do?●Delta method●Nonlinear regression●Nonlinear regression: details●Iteration & Distribution●Confidence intervals●Weight loss data- p. 5/11What to do?■In some cases, it is possible to linearize to get the originalmodel.■SupposeYi= C · 2−Xi/θεi,thenlog(Yi) = C0− Xi/θ + ε∗i.If ε∗ihave approximately the same variance, than it looks likeoriginal model.■What ifYi= C · 2−Xi/θ+ εiwhere the εi’s have the same variance?●Today’s class●Nonlinear regression models●Weight loss data●What to do?●Delta method●Nonlinear regression●Nonlinear regression: details●Iteration & Distribution●Confidence intervals●Weight loss data- p. 6/11Delta method■The delta method tells us thatVar(f(Yi)) ' f0(E(Yi))2Var(Yi).■In this caseVar(log(Yi)) '1C22−2Xi/θσ2.■Could use weighted least-squares if θ was known.■One possibility: IRLS.◆Starting at some initialbθj, j ≥ 0, regress log(Yi) onto Xisolve forbβj+11, and setbθj+10= −1bβj+11●Today’s class●Nonlinear regression models●Weight loss data●What to do?●Delta method●Nonlinear regression●Nonlinear regression: details●Iteration & Distribution●Confidence intervals●Weight loss data- p. 7/11Nonlinear regression■Alternatively,(bC,bθ) = argmin(C,θ)nXi=1(Yi− C2−Xi/θ)2.■In general, given a possibly non-linear regression functionf(x, θ), θ ∈ Rp: we can try to solvebθ = argminθnXi=1(Yi− f(xi, θ))2where xiis the i-th row of the design matrix.■Techniques: usually use a steepest descent approachinstead of Newton-Raphson.●Today’s class●Nonlinear regression models●Weight loss data●What to do?●Delta method●Nonlinear regression●Nonlinear regression: details●Iteration & Distribution●Confidence intervals●Weight loss data- p. 8/11Nonlinear regression: details■Given an initial valuebθ0f(xi, θ) ' f(xi,bθ0) +pXj=1∂f∂θj(xi,θ)=(xi,bθ0)(θi− θ0i) = η0i(θ).■In matrix formη0(θ) = ω0+ Z0θwhereZ0ij=∂f∂θj(x,θ)=(xi,bθ0)=∂ηi∂θjθ=bθ0ω0i= f(xi, θ0) −pXj=1θ0iZ0ij■The vector η0(θ) is our approximation to the regressionsurface, we want to choose θ to get as close to Y aspossible. Project onto “tangent space.”●Today’s class●Nonlinear regression models●Weight loss data●What to do?●Delta method●Nonlinear regression●Nonlinear regression: details●Iteration & Distribution●Confidence intervals●Weight loss data- p. 9/11Iteration & Distribution■Want to choosebθ1as followsbθ1= argminθnXi=1(Yi− ω0i− Z0θ)2■Solutionbθ1=Z0 tZ0−1Z0 t(Y − ω0).■Givenbθj, setbθj+1=Zj tZj−1Zj t(Y − ωj).■Repeat until convergence.■Approximate distribution ofbθ:bθ ∼ N(θ, bσ2(bZtbZ)−1),wherebσ2=nXi=1(Yi− f(xi,bθ))2/(n − p).●Today’s class●Nonlinear regression models●Weight loss data●What to do?●Delta method●Nonlinear regression●Nonlinear regression: details●Iteration & Distribution●Confidence intervals●Weight loss data- p. 10/11Confidence intervals■If θ is restricted, say θ ≥ 0 the asymptotic confidenceintervals may be inaccurate (may overlap into negativenumbers).■library(MASS) provides another method to obtainconfidence intervals based on “inverting” an F -test.■Basic idea, confidence interval for θ1: for each fixed valueθ1,0we could compute the “extra sum of squares” betweenthe unrestricted model and the model with θ1fixed at θ1,0.F (θ1,0) =SSEθ1=θ1,0(bθ2:p) − SSE(bθ)bσ2∼ F1,n−pat least approximately under H0: θ1= θ1,0.■Or,T (θ1,0) = sign(bθ −bθ1,0)qF (θ1,0) ∼ tn−p.■Confidence interval:{θ1: −tn−p,1−α/2< T (θ1) < tn−p,1−α/2}.- p. 11/11Weight loss data0 50 100 150 200 250110 120 130 140 150 160 170