12.215 Modern NavigationSummary of last classToday’s classConcepts in estimationBasics of parametric estimationObservation modelTaylor series expansionLinearizationEstimationLeast squares estimationWeighted Least SquaresStatistical approach to least squaresData covariance matrixData covariance matrixCovariance matrix of parameter estimatesCovariance matrix of estimated parametersCovariance matrix of post-fit residualsCovariance matrix of post-fit residualsPost-fit residualsExampleSummary12.215 Modern NavigationThomas Herring11/01/2006 12.215 Modern Naviation L13 2Summary of last class• Basic Statistics– Statistical description and parameters• Probability distributions• Descriptions: expectations, variances, moments• Covariances• Estimates of statistical parameters • Propagation of variances– Methods for determining the statistical parameters of quantities derived other statistical variables11/01/2006 12.215 Modern Naviation L13 3Today’s class• Estimation methods– Restrict to basically linear estimation problems (also non-linear problems that are nearly linear)– Restrict to parametric, over determined estimation – Concepts in estimation:• Mathematical models• Statistical models• Least squares and Maximum likelihood estimation• Covariance matrix of estimates parameters• Covariance matrix of post-fit residual11/01/2006 12.215 Modern Naviation L13 4Concepts in estimation• Given multiple observations of a quantity or related to a set ofquantities how do you obtain a “best” estimate.• What do we mean by “best”• How do you quantify of quality of observations and the relationship between errors in observations.• The complete answers to the above questions are complex • We will limit our discussion to parametric estimation mostly forGaussian distributed random errors in measurements.• In parametric estimation, mathematical relationships between observations and parameters that can be used to model the observations is used (e.g., GPS measures pseudoranges to satellites: These observations can be related to the positions of the ground station and satellites plus other parameters that we discuss later).11/01/2006 12.215 Modern Naviation L13 5Basics of parametric estimation• All parametric estimation methods can be broken into a few main steps:– Observation equations: equations that relate the parameters to be estimated to the observed quantities (observables). Mathematical model.• Example: Relationship between pseudorange, receiver position, satellite position (implicit in ρ), clocks, atmospheric and ionosphere delays– Stochastic model: Statistical description that describes the random fluctuations in the measurements and maybe the parameters. In some forms the stochastic model is not explicit.– Inversion that determines the parameters values from the mathematical model consistent with the statistical model.11/01/2006 12.215 Modern Naviation L13 6Observation model• Observation model are equations relating observables to parameters of model:– Observable = function (parameters)– Observables should not appear on right-hand-side of equation– The observed values are the observable plus noise of some stochastic nature• Often function is non-linear and most common method is linearization of function using Taylor series expansion.• Sometimes log linearization for f=a.b.c ie. Products fo parameters11/01/2006 12.215 Modern Naviation L13 7Taylor series expansion• In most common Taylor series approach:• The estimation is made using the difference between the observations and the expected values based on apriori values for the parameters.• The estimation returns adjustments to apriori parameter values• The observations are y+noisey =f(x1,x2,x3,x4)y0+Δy = f (x)x0+∂f (x)∂xΔxx= (x1,x2,x3,x4)11/01/2006 12.215 Modern Naviation L13 8Linearization• Since the linearization is only an approximation, the estimation should be iterated until the adjustments to the parameter values are zero.• For GPS estimation: Convergence rate is 100-1000:1 typically (ie., a 1 meter error in apriori coordinates could results in 1-10 mm of non-linearity error).• To assess, the level on non-linear contribution, the Taylor series expansion is compared to the non-linear evaluation. If the differences are similar in size to the noise in the measurements, then a new Taylor series expansion, about the better estimates of the parameters, is needed.11/01/2006 12.215 Modern Naviation L13 9Estimation• Most common estimation method is “least-squares” in which the parameter estimates are the values that minimize the sum of the squares of the differences between the observations and modeled values based on parameter estimates.• For linear estimation problems, direct matrix formulation for solution• For non-linear problems: Linearization or search technique where parameter space is searched for minimum value• Care with search methods that local minimum is not found (will not treat in this course)11/01/2006 12.215 Modern Naviation L13 10Least squares estimation• Originally formulated by Gauss.• Basic equations: Δy is vector of observations; A is linear matrix relating parameters to observables; Δx is vector of parameters; v is residualΔy = AΔx + vminimize vTv(); superscript T means transposeΔx = (ATA)−1ATΔy11/01/2006 12.215 Modern Naviation L13 11Weighted Least Squares• In standard least squares, nothing is assumed about the residuals v except that they are zero expectation. • One often sees weight-least-squares in which a weight matrix is assigned to the residuals. Residuals with larger elements in W are given more weight.minimize vTWv(); Δx = (ATWA)−1ATWΔy11/01/2006 12.215 Modern Naviation L13 12Statistical approach to least squares• If the weight matrix used in weighted least squares is the inverse of the covariance matrix of the residuals, then weighted least squares is a maximum likelihood estimator for Gaussian distributed random errors.• This choice maximizes the probability density (called a maximum likelihood estimate, MLE)• This latter form of least-squares is most statistically rigorous version.• Sometimes weights are chosen empirically f (v) =1(2π)nVe−12vTV−1v11/01/2006 12.215 Modern Naviation L13 13Data covariance matrix• If we use the inverse of the covariance matrix of the noise in the data, we obtain a MLE if data noise is Gaussian distribution.• How do you obtain data