ECEN 629 Shuguang Cui Lecture 2 Convex functions f Rn R is convex if dom f is convex and for all x y dom f 0 1 f x 1 y f x 1 f y f is concave if f is convex x convex x concave x neither examples on R f x x2 is convex f x log x is concave dom f R f x 1 x is convex dom f R 1 ECEN 629 Shuguang Cui Extended valued extensions for f convex it s convenient to define the extension f x f x x dom f x 6 dom f inequality f x 1 y f x 1 f y holds for all x y Rn 0 1 as an inequality in R we ll use same symbol for f and its extension i e we ll implicitly assume convex functions are extended 2 ECEN 629 Shuguang Cui Epigraph sublevel sets epigraph of a function f is epi f x t x dom f f x t f x epi f x f convex function epi f convex set the sublevel set of f is C x dom f f x f convex sublevel sets are convex converse false 3 ECEN 629 Shuguang Cui Differentiable convex functions gradient of f Rn R f x h f x1 f x2 f xn first order Taylor approximation at x0 iT evaluated at x f x f x0 f x0 T x x0 first order condition for f differentiable f is convex for all x x0 dom f f x T f x f x0 f x0 x x0 i e 1st order approx is a global underestimator x0 x f x0 f x0 T x x0 4 ECEN 629 Shuguang Cui epigraph interpretation for all x t epi f f x0 1 T x x0 t f x0 0 i e f x0 1 defines supporting hyperplane to epi f at x0 f x0 f x f x0 1 5 ECEN 629 Shuguang Cui Hessian of a twice differentiable function 2 2 f x x2 x1 2f x1 x2 2f x2 2 2f xn x1 2f xn x2 f x2 1 2f 2f x1 xn 2f x2 xn 2f x2 n evaluated at x 2nd order Taylor series expansion around x0 1 f x f x0 f x0 T x x0 x x0 T 2f x0 x x0 2 second order condition for f twice differentiable f is convex for all x dom f 2f x 0 6 ECEN 629 Shuguang Cui Simple examples linear and affine functions are convex and concave quadratic function f x xT P x 2q T x r convex P 0 concave P 0 P P T any norm is convex examples on R x is convex on R for 1 0 concave for 0 1 log x is concave on R x log x is convex on R e x is convex x max 0 x max 0 x are convex Rx 2 log e t dt is concave 7 ECEN 629 Shuguang Cui Elementary properties a function is convex iff it is convex on all lines f convex f x0 th convex in t for all x0 h positive multiple of convex function is convex f convex 0 f convex sum of convex functions is convex f1 f2 convex f1 f2 convex extends to infinite sums integrals g x y convex in x Z g x y dy convex 8 ECEN 629 Shuguang Cui pointwise maximum f1 f2 convex max f1 x f2 x convex corresponds to intersection of epigraphs f2 x epi max f1 f2 f1 x x pointwise supremum f convex sup f convex A affine transformation of domain f convex f Ax b convex 9 ECEN 629 Shuguang Cui More examples piecewise linear functions f x maxi aTi x bi is convex in x epi f is polyhedron max distance to any set sups S kx sk is convex in x f x x 1 x 2 x 3 is convex on Rn x i is the ith largest xj Q 1 n is concave on Rn f x i xi Pm T 1 f x is convex i 1 log bi ai x dom f x aTi x bi i 1 m least squares cost as functions of weights f w inf x X i wi aTi x bi 2 is concave in w 10 ECEN 629 Shuguang Cui Convex functions of matrices T Tr A X P i j Aij Xij is linear in X on Rn n log det X 1 is convex on X Sn X 0 1 2 1 2 proof let i be the eigenvalues of X0 HX0 f t log det X0 tH 1 log det X0 log det I tX0 X 1 log det X0 log 1 t i 1 2 1 1 2 1 HX0 i is a convex function of t det X 1 n is concave on X Sn X 0 max X is convex on Sn proof max X supkyk2 1 y T Xy kXk2 1 X max X T X 1 2 is convex on Rm n proof kXk2 supkuk2 1 kXuk2 11 ECEN 629 Shuguang Cui Minimizing over some variables if h x y is convex in x and y then f x inf h x y y is convex in x corresponds to projection of epigraph x y t x t h x y f x y x 12 ECEN 629 Shuguang Cui examples if S Rn is convex then min distance to S dist x S inf kx sk s S is convex in x if g is convex then f y inf g x Ax y is convex in y proof assume A Rm n has rank m find B s t R B N A then Ax y iff x AT AAT 1 y Bz for some z and hence f y inf g AT AAT 1y Bz z 13 ECEN 629 Shuguang Cui Composition one dimensional case f x h g x g Rn R h R R is convex if g convex h convex nondecreasing g concave h convex nonincreasing proof differentiable functions x R 2 f h g g h examples f x f x f x f x exp g x is convex if g is convex 1 g x is convex if g is concave positive g x p p 1 is convex if g x convex positive P i log fi x is convex on x fi x 0 if fi are convex 14 ECEN 629 Shuguang Cui Composition k dimensional case f x h g1 x gk x with h Rk R gi Rn R is convex if h convex nondecreasing in each arg gi convex h convex nonincreasing in each arg gi concave etc proof differentiable functions n 1 g1 f hT gk examples T g1 2h gk g1 gk f x maxi gi x is convex if each gi is P f x log i exp gi x is convex if each gi is 15 ECEN 629 Shuguang Cui Jensen s inequality f Rn R convex two points 1 2 1 …