Elastic and plastic splines: some experimentalcomparisonsRoger Koenker and Ivan MizeraAbstract. We give some empirical comparisons between two nonparametricregression methods based on regularization: the elastic or thin-plate splines,and plastic splines based on total variation penalties.1. Nonparametric regression with elastic and plastic splinesWe seek f to fit the dependence of response points zi∈ R on two-dimensionalcovariates (xi, yi) living in the domain Ω. To this end, we employ regularization:f is obtained as a minimizer of(1)nXi=1ρ(zi− f(xi, yi)) + λJ(f).The first term is traditionally called (in)fidelity, since it measures the overall lackof fit of f (xi, yi) to zi. The second penalty term shrinks the solution towards a moreplausible or desirable alternative, the extent of this shrinkage being controlled bythe regularization parameter λ. See Green and Silverman (1994), Eubank (1999)or Wahba (1990).1.1. Elastic splinesWe coin the term elastic splines for what are usually called thin-plate splines(defined on the idealized domain Ω = R2). They arise as solutions of (1) with thepenalty(2) J2(f) =ZZR2f2xx(x, y) + 2f2xy(x, y) + f2y y(x, y) dx dy .This penalty can be considered a natural extension of the easily interpretable one-dimensional prototypeR(f00)2. The only unnatural feature is the fact that thepenalty is evaluated over all of R2instead of over a more realistic bounded domainΩ, for instance the convex hull of the (xi, yi) points. The latter alternative wasconsidered, among others, by Green and Silverman (1994) who coined the namefinite-window thin-plate splines. However, the algorithm for the finite-window al-ternative is more involved, though not necessarily slower—and, mostly important,2 Roger Koenker and Ivan Mizeranot available to us at the present moment, therefore this version of elastic splineswill not be considered further here.The name “elastic splines” comes from the quite well-known physical modelunderlying the whole setting, in which the penalty is interpreted as the potentialenergy of a displacement, from the horizontal position, of an elastic thin (metal)plate, the displacement that mimicks the form of the fitted function interpolatingthe data points. As usual in physical theories, idealizations are inevitable. Thedisplacement should be small, we may say infinitesimal, thus rather in the form²f(x, y) than f (x, y). The material aspects of the plate are rather limiting withrespect to its physical reality: it is thin, that is, we may abstract from its thirddimension; it is elastic, hence it does not deform, only bend, and it is a plate, nota membrane, which means it is stiff—its behavior is rather that of steel than thatof gum.Despite its simplifications, such a physical analogy serves as a very usefulhint in the world of otherwise potentially endless possibilities.Figure 1. Chicago data: elastic fit. The large triangles at rearare artifacts of the visualization method.Elastic and plastic splines 31.2. Plastic splinesPlastic splines arise as solutions of another instance of the regularization scheme(1), when the penalty is(3) J22,k·k(f, Ω) =_Ω∇2f =ZZΩk∇2f(x, y)k dx dy,where ∇2f(x, y) denotes the Hessian of f at (x, y). Formulas like this should beread with some caution here: the derivatives are not only the classical ones appliedto classical smo oth functions, but also the generalized ones applying to certainSchwartzian distributions. Probably the easiest way to apprehend this is to thinkabout the right-hand side of (3) as about a definition for smooth functions, whichis subsequently extended, by continuity or rather lower semicontinuity, to all func-tions whose gradient has bounded variation (the property essentially equivalentto the bounded area of the graphs of the components). The similar extension ex-ercise with the quadratic penalty (2) would not yield anything new, but here itconsiderably broadens the scope and adds also functions with sharp edges andspikes.The necessity of choosing a matrix norm means that it is more appropriateto refer to (3) as to a family of penalties. Plastic penalties are always consideredover bounded Ω; an obvious choice of the matrix norm is the `2(Hilbert, Schmidt,Frobenius, Schur) norm k·k2. It establishes a parallel between thin-plate and plasticpenalties—compare (2) with(4) J22,2(f, Ω) =ZZΩqf2xx(x, y) + 2f2xy(x, y) + f2y y(x, y) dx dy .Despite its appealing simplicity, there are also other, and not irrelevant, normchoices possible. We require that all norms are orthogonal-similarity invariant, toensure that the resulting penalties are coordinate-free, as is the thin-plate penalty(2). For further motivation, theory and examples, see Koenker and Mizera (2001)and Mizera (2002).Any plastic penalty, regardless of the choice of the norm, can be considered anatural extension of the one-dimensional penalty equal to the total variation of thederivative, that is,R|f00| for smooth functions. The latter penalty was introducedby Koenker, Ng and Portnoy (1994), who were motivated by the quantile regressioninfidelity ρ(u) = ρτ(u) = u(τ − I[u < 0]). Another way to justify the `1is oncomputational grounds, or by scale equivariance considerations as those given byKoenker and Mizera (2001).Given all of this, and also the form of the plastic penalty, it can be said thatplastic splines are an `1alternative to the `2elastic ones. The adjective “plastic”comes here from the fact that, again under certain idealization, the penalty canbe interpreted as the deformation energy, the work done by the stress in thecourse of deformation, of the plate displacement mimicking the interpolated shape.The theory in which is this interpretation possible, the deformation theory of aperfectly plastic rigid-plastic body, is a kind of a limit case of various other physical4 Roger Koenker and Ivan Mizeraapproaches to plasticity—nevertheless, it is extensively used in engineering despiteall its simplifications. Various yield criteria (a notion considered by the theory)then lead to various matrix norms for plastic penalties. We may again think of areal-world material model such as a metal, but now formable—cast into artisticshapes in the copp er foundry.Another link, connected to the mathematical expression of the penalty, leadsto the so-called total-variation based denoising of Rudin, Osher and Fatemi (1992),motivated by a desire to recover edges, extrema, and other “sharp” features, whilenot p