DOC PREVIEW
CMU CS 10701 - Recitation

This preview shows page 1-2-3-25-26-27 out of 27 pages.

Save
View full document
View full document
Premium Document
Do you want full access? Go Premium and unlock all 27 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 27 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 27 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 27 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 27 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 27 pages.
Access to all documents
Download any document
Ad free experience
Premium Document
Do you want full access? Go Premium and unlock all 27 pages.
Access to all documents
Download any document
Ad free experience

Unformatted text preview:

Appendix DMatrix calculusFrom too much study, and from extreme passion, comethmadnesse.−Isaac Newton [86,§5]D.1 Directional derivative, Taylor seriesD.1.1 GradientsGradient of a differentiable real function f(x) : RK→R with respect to itsvector domain is defined∇f(x)∆=∂f(x)∂x1∂f(x)∂x2...∂f(x)∂xK∈ RK(1354)while the second-order gradient of the twice differentiable real function withrespect to its vector domain is traditionally called the Hessian ;∇2f(x)∆=∂2f(x)∂2x1∂2f(x)∂x1∂x2···∂2f(x)∂x1∂xK∂2f(x)∂x2∂x1∂2f(x)∂2x2···∂2f(x)∂x2∂xK............∂2f(x)∂xK∂x1∂2f(x)∂xK∂x2···∂2f(x)∂2xK∈ SK(1355)©2001 Jon Dattorro. CO&EDG version 04.18.2006. All rights reserved.Citation: Jon Dattorro, Convex Optimization & Euclidean Distance Geometry,Meboo Publishing USA, 2005.501502 APPENDIX D. MATRIX CALCULUSThe gradient of vector-valued function v(x) : R →RNon real domain isa row-vector∇v(x)∆=h∂v1(x)∂x∂v2(x)∂x···∂vN(x)∂xi∈ RN(1356)while the second-order gradient is∇2v(x)∆=h∂2v1(x)∂x2∂2v2(x)∂x2···∂2vN(x)∂x2i∈ RN(1357)Gradient of vector-valued function h(x) : RK→RNon vector domain is∇h(x)∆=∂h1(x)∂x1∂h2(x)∂x1···∂hN(x)∂x1∂h1(x)∂x2∂h2(x)∂x2···∂hN(x)∂x2.........∂h1(x)∂xK∂h2(x)∂xK···∂hN(x)∂xK= [ ∇h1(x) ∇h2(x) ··· ∇hN(x) ] ∈ RK×N(1358)while the second-order gradient has a three-dimensional representationdubbed cubix ;D.1∇2h(x)∆=∇∂h1(x)∂x1∇∂h2(x)∂x1··· ∇∂hN(x)∂x1∇∂h1(x)∂x2∇∂h2(x)∂x2··· ∇∂hN(x)∂x2.........∇∂h1(x)∂xK∇∂h2(x)∂xK··· ∇∂hN(x)∂xK= [ ∇2h1(x) ∇2h2(x) ··· ∇2hN(x) ] ∈ RK×N ×K(1359)where the gradient of each real entry is with respect to vector x as in (1354).D.1The word matrix comes from the Latin for womb ; related to the prefix matri- derivedfrom mater meaning mother.D.1. DIRECTIONAL DERIVATIVE, TAYLOR SERIES 503The gradient of real function g(X) : RK×L→R on matrix domain is∇g(X)∆=∂g(X)∂X11∂g(X)∂X12···∂g(X)∂X1L∂g(X)∂X21∂g(X)∂X22···∂g(X)∂X2L.........∂g(X)∂XK1∂g(X)∂XK2···∂g(X)∂XKL∈ RK×L=∇X(:,1)g(X)∇X(:,2)g(X)...∇X(:,L)g(X) ∈ RK×1×L(1360)where the gradient ∇X(:,i)is with respect to the ithcolumn of X . Thestrange appearance of (1360) in RK×1×Lis meant to suggest a third dimensionperpendicular to the page (not a diagonal matrix). The second-order gradienthas representation∇2g(X)∆=∇∂g(X)∂X11∇∂g(X)∂X12··· ∇∂g(X)∂X1L∇∂g(X)∂X21∇∂g(X)∂X22··· ∇∂g(X)∂X2L.........∇∂g(X)∂XK1∇∂g(X)∂XK2··· ∇∂g(X)∂XKL∈ RK×L×K×L=∇∇X(:,1)g(X)∇∇X(:,2)g(X)...∇∇X(:,L)g(X) ∈ RK×1×L×K×L(1361)where the gradient ∇ is with respect to matrix X .504 APPENDIX D. MATRIX CALCULUSGradient of vector-valued function g(X) : RK×L→RNon matrix domainis a cubix∇g(X)∆=∇X(:,1)g1(X) ∇X(:,1)g2(X) ··· ∇X(:,1)gN(X)∇X(:,2)g1(X) ∇X(:,2)g2(X) ··· ∇X(:,2)gN(X).........∇X(:,L)g1(X) ∇X(:,L)g2(X) ··· ∇X(:,L)gN(X) = [ ∇g1(X) ∇g2(X) ··· ∇gN(X) ] ∈ RK×N ×L(1362)while the second-order gradient has a five-dimensional representation;∇2g(X)∆=∇∇X(:,1)g1(X) ∇∇X(:,1)g2(X) ··· ∇∇X(:,1)gN(X)∇∇X(:,2)g1(X) ∇∇X(:,2)g2(X) ··· ∇∇X(:,2)gN(X).........∇∇X(:,L)g1(X) ∇∇X(:,L)g2(X) ··· ∇∇X(:,L)gN(X) = [ ∇2g1(X) ∇2g2(X) ··· ∇2gN(X) ] ∈ RK×N ×L×K×L(1363)The gradient of matrix-valued function g(X) : RK×L→RM×Non matrixdomain has a four-dimensional representation called quartix∇g(X)∆=∇g11(X) ∇g12(X) ··· ∇g1N(X)∇g21(X) ∇g22(X) ··· ∇g2N(X).........∇gM1(X) ∇gM2(X) ··· ∇gMN(X)∈ RM×N ×K×L(1364)while the second-order gradient has six-dimensional representation∇2g(X)∆=∇2g11(X) ∇2g12(X) ··· ∇2g1N(X)∇2g21(X) ∇2g22(X) ··· ∇2g2N(X).........∇2gM1(X) ∇2gM2(X) ··· ∇2gMN(X)∈ RM×N ×K×L×K×L(1365)and so on.D.1. DIRECTIONAL DERIVATIVE, TAYLOR SERIES 505D.1.2 Product rules for matrix-functionsGiven dimensionally compatible matrix-valued functions of matrix variablef(X) and g(X)∇Xf(X)Tg(X)= ∇X(f) g + ∇X(g) f (1366)while [35,§8.3] [205]∇Xtrf(X)Tg(X)= ∇Xtrf(X)Tg(Z)+ trg(X) f(Z )TZ←X(1367)These expressions implicitly apply as well to scalar-, vector-, or matrix-valuedfunctions of scalar, vector, or matrix argument s.D.1.2.0.1 Example. Cubix.Suppose f(X) : R2×2→R2= XTa and g(X) : R2×2→R2= Xb . We wishto find∇Xf(X)Tg(X)= ∇XaTX2b (1368)using the product rule. Formula (1366) calls for∇XaTX2b = ∇X(XTa) Xb + ∇X(Xb) XTa (1369)Consider the first of the two terms:∇X(f) g = ∇X(XTa) Xb=∇(XTa)1∇(XTa)2 Xb(1370)The gradient of XTa forms a cubix in R2×2×2.∂(XTa)1∂X11IIIIII∂(XTa)2∂X11IIIIII∂(XTa)1∂X12∂(XTa)2∂X12∂(XTa)1∂X21IIIIII∂(XTa)2∂X21IIIIII∂(XTa)1∂X22∂(XTa)2∂X22∇X(XTa) Xb =(Xb)1(Xb)2∈ R2×1×2(1371)506 APPENDIX D. MATRIX CALCULUSBecause gradient of the product (1368) requires total change with respectto change in each entry of matrix X , the Xb vector must make an innerproduct with each vector in the second dimension of the cubix (indicated bydotted line segments);∇X(XTa) Xb =a100 a1a200 a2b1X11+ b2X12b1X21+ b2X22∈ R2×1×2=a1(b1X11+ b2X12) a1(b1X21+ b2X22)a2(b1X11+ b2X12) a2(b1X21+ b2X22)∈ R2×2= abTXT(1372)where the cubix appears as a complete 2 ×2 ×2 matrix. In like manner forthe second term ∇X(g) f∇X(Xb) XTa =b10b200 b10 b2X11a1+ X21a2X12a1+ X22a2∈ R2×1×2= XTabT∈ R2×2(1373)The solution∇XaTX2b = abTXT+ XTabT(1374)can be found from Table D.2.1 or verified using (1367). 2D.1.2.1 Kronecker productA partial remedy for venturing into hyperdimensional representations, suchas the cubix or quartix, is to first vectorize matrices as in (29). This devicegives rise to the Kronecker product of matrices


View Full Document

CMU CS 10701 - Recitation

Documents in this Course
lecture

lecture

12 pages

lecture

lecture

17 pages

HMMs

HMMs

40 pages

lecture

lecture

15 pages

lecture

lecture

20 pages

Notes

Notes

10 pages

Notes

Notes

15 pages

Lecture

Lecture

22 pages

Lecture

Lecture

13 pages

Lecture

Lecture

24 pages

Lecture9

Lecture9

38 pages

lecture

lecture

26 pages

lecture

lecture

13 pages

Lecture

Lecture

5 pages

lecture

lecture

18 pages

lecture

lecture

22 pages

Boosting

Boosting

11 pages

lecture

lecture

16 pages

lecture

lecture

20 pages

Lecture

Lecture

20 pages

Lecture

Lecture

39 pages

Lecture

Lecture

14 pages

Lecture

Lecture

18 pages

Lecture

Lecture

13 pages

Exam

Exam

10 pages

Lecture

Lecture

27 pages

Lecture

Lecture

15 pages

Lecture

Lecture

24 pages

Lecture

Lecture

16 pages

Lecture

Lecture

23 pages

Lecture6

Lecture6

28 pages

Notes

Notes

34 pages

lecture

lecture

15 pages

Midterm

Midterm

11 pages

lecture

lecture

11 pages

lecture

lecture

23 pages

Boosting

Boosting

35 pages

Lecture

Lecture

49 pages

Lecture

Lecture

22 pages

Lecture

Lecture

16 pages

Lecture

Lecture

18 pages

Lecture

Lecture

35 pages

lecture

lecture

22 pages

lecture

lecture

24 pages

Midterm

Midterm

17 pages

exam

exam

15 pages

Lecture12

Lecture12

32 pages

lecture

lecture

19 pages

Lecture

Lecture

32 pages

boosting

boosting

11 pages

pca-mdps

pca-mdps

56 pages

bns

bns

45 pages

mdps

mdps

42 pages

svms

svms

10 pages

Notes

Notes

12 pages

lecture

lecture

42 pages

lecture

lecture

29 pages

lecture

lecture

15 pages

Lecture

Lecture

12 pages

Lecture

Lecture

24 pages

Lecture

Lecture

22 pages

Midterm

Midterm

5 pages

mdps-rl

mdps-rl

26 pages

Load more
Download Recitation
Our administrator received your request to download this document. We will send you the file to your email shortly.
Loading Unlocking...
Login

Join to view Recitation and access 3M+ class-specific study document.

or
We will never post anything without your permission.
Don't have an account?
Sign Up

Join to view Recitation 2 2 and access 3M+ class-specific study document.

or

By creating an account you agree to our Privacy Policy and Terms Of Use

Already a member?