1© Massachusetts Institute of Technology - Prof. de Weck and Prof. WillcoxEngineering Systems Division and Dept. of Aeronautics and AstronauticsGradient Calculation and Sensitivity AnalysisLecture 9Olivier de WeckKaren WillcoxMultidisciplinary System Design Optimization (MSDO)2© Massachusetts Institute of Technology - Prof. de Weck and Prof. WillcoxEngineering Systems Division and Dept. of Aeronautics and AstronauticsToday‟s Topics• Gradient calculation methods– Analytic and Symbolic– Finite difference– Complex step– Adjoint method– Automatic differentiation• Post-Processing Sensitivity Analysis– effect of changing design variables– effect of changing parameters– effect of changing constraints3© Massachusetts Institute of Technology - Prof. de Weck and Prof. WillcoxEngineering Systems Division and Dept. of Aeronautics and AstronauticsDefinition of the Gradient12nJxJxJxJ“How does the function J value change locally as we change elements of the design vector x?”Compute partial derivativesof J with respect to xiiJxJGradient vector points normalto the tangent hyperplane of J(x)1x2x3x4© Massachusetts Institute of Technology - Prof. de Weck and Prof. WillcoxEngineering Systems Division and Dept. of Aeronautics and Astronautics0 0.5 1 1.5 200.20.40.60.811.21.41.61.82x1x2Contour plot 3.1 3.1 3.13.253.253.253.253.25 3.5 3.5 3.5 3.5 4 4 4 4 5 5Geometry of Gradient vector (2D)Example function:1 2 1 2121,J x x x xxx21 1 222 1 21111Jx x xJJx x xGradient normal to contours5© Massachusetts Institute of Technology - Prof. de Weck and Prof. WillcoxEngineering Systems Division and Dept. of Aeronautics and AstronauticsGeometry of Gradient vector (3D)2221 2 3J x x x123222xJxxincreasingvalues of J1x2x3xTangent plane1 2 32 2 2 6 0x x x111Tox222oTJxGradient vector points to larger values of JJ=3Example6© Massachusetts Institute of Technology - Prof. de Weck and Prof. WillcoxEngineering Systems Division and Dept. of Aeronautics and AstronauticsOther Gradient-Related Quantities• Jacobian: Matrix of derivatives of multiple functions w.r.t. vector of variables• Hessian: Matrix of second-order derivatives121 1 1122 2 212zzzn n nJ J Jx x xJ J Jx x xJ J Jx x xJ12zJJJJz x 1n x z22221222222122122122122nnnnnxJxxJxxJxxJxJxxJxxJxxJxJJHn x nWhy Calculate Gradients• Required by gradient-based optimization algorithms– Normally need gradient of objective function and each constraint w.r.t. design variables at each iteration– Newton methods require Hessians as well• Isoperformance/goal programming• Robust design• Post-processing sensitivity analysis– determine if result is optimal– sensitivity to parameters, constraint values7© Massachusetts Institute of Technology - Prof. de Weck and Prof. WillcoxEngineering Systems Division and Dept. of Aeronautics and Astronautics8© Massachusetts Institute of Technology - Prof. de Weck and Prof. WillcoxEngineering Systems Division and Dept. of Aeronautics and AstronauticsAnalytical SensitivitiesIf the objective function is known in closed form, we can often compute the gradient vector(s) in closed form (analytically):Example:1 2 1 2121,J x x x xxxAnalytical Gradient:21 1 222 1 21111Jx x xJJx x xFor complex systems analytical gradients are rarely availableExamplex1 = x2 =1J(1,1)=30(1,1)0JMinimum9© Massachusetts Institute of Technology - Prof. de Weck and Prof. WillcoxEngineering Systems Division and Dept. of Aeronautics and AstronauticsSymbolic Differentiation• Use symbolic mathematics programs• e.g. MATLAB®, Maple®, Mathematica®» syms x1 x2» J=x1+x2+1/(x1*x2);» dJdx1=diff(J,x1)dJdx1 =1-1/x1^2/x2» dJdx2=diff(J,x2)dJdx2 = 1-1/x1/x2^2construct a symbolic objectdifference operator10© Massachusetts Institute of Technology - Prof. de Weck and Prof. WillcoxEngineering Systems Division and Dept. of Aeronautics and AstronauticsFinite Differences (I)xxoxo+ xxo- xxxfFunction of a single variable f(x)• First-order finite differenceapproximation of gradient:• Second-order finite differenceapproximation of gradient:'ooof x x f xf x O xx'22ooof x x f x xf x O xxForward difference approximation to the derivativeTruncation ErrorCentral difference approximation to the derivativeTruncation Error11© Massachusetts Institute of Technology - Prof. de Weck and Prof. WillcoxEngineering Systems Division and Dept. of Aeronautics and AstronauticsFinite Differences (II)Approximations are derived from Taylor Series expansion:Neglect second order and higher order terms; solve for gradient vector:'ooof x x f xf x O xx''2ooxO x fx x xTruncation Error2' '' 22o o o oxf x x f x xf x f x O x3Forward Difference12© Massachusetts Institute of Technology - Prof. de Weck and Prof. WillcoxEngineering Systems Division and Dept. of Aeronautics and AstronauticsFinite Differences (III)Take Taylor expansion backwards at oxx2' '' 22o o o oxf x x f x xf x f x O x2' '' 22o o o oxf x x f x xf x f x O x(1)(2)(1) - (2) and solve again for derivative'22ooof x x f x xf x O xxCentral Difference22 '''6ooxO x fx x xTruncation Error13© Massachusetts Institute of Technology - Prof. de Weck and Prof. WillcoxEngineering Systems Division and Dept. of Aeronautics and AstronauticsFinite Differences (IV)11 1 1 1 111 1 1 1 1o o ooJ x J x J x x J xJJx x x x xx1J(x)11x1ox11Jx1oJxtrue, analyticalsensitivityfinite differenceapproximation1xJ11x1ox-© Massachusetts Institute of Technology - Prof. de Weck and Prof. WillcoxEngineering Systems Division and Dept. of Aeronautics and AstronauticsFinite Differences (V)xxoxo+ xxo- xxxf• Second-order finite differenceapproximation of second derivative:22''( )o o oof x x f x f x xfxx15© Massachusetts Institute of Technology - Prof. de Weck and Prof. WillcoxEngineering Systems Division and Dept. of Aeronautics and AstronauticsErrors of Finite DifferencingCaution: - Finite differencing always has errors- Very dependent on perturbation size1 2 1 2121,J x x x xxxx1 = x2 =1J(1,1)=30(1,1)0J00.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2012345678910x1x110-910-810-710-610-810-710-610-5Truncation Errors ~ xRoundingErrors ~ 1/ xPerturbation Step Size x1Gradient ErrorChoice of x is criticalJ16© Massachusetts Institute of Technology - Prof. de Weck and Prof. WillcoxEngineering Systems Division and Dept. of Aeronautics and AstronauticsPerturbation Size x Choice• Error Analysis• Machine Precision• Trial and Error – typical value ~ 0.1-1%1/2Axf- Forward difference1/3Axf- Central difference(Gill et al.