On smoothing potentially non-stationary climate time seriesMichael E. MannDepartment of Environmental Sciences, University of Virginia, Charlottesville, Virginia, USAReceived 23 January 2004; revised 10 March 2004; accepted 18 March 2004; published 15 April 2004.[1] A simple approach to the smoothing of a potentiallynon-stationary time series is presented which providesan optimal choice among three alternative, readilymotivated and easily implemented boundary constraints.This method is applied to the smoothing of the instrumentalNorthern Hemisphere (NH) annual mean and cold-season North Atlantic Oscillation (NAO) time series,yielding an objective estimate of the smoothed decadal-scale variations in these series including long-termtrends.INDEX TERMS: 3309 Meteorology and AtmosphericDynamics: Climatology (1620); 3394 Meteorolog y andAtmospheric Dynamics: Instruments and techniques; 4294Oceanography: General: Instruments and techniques; 4594Oceanography: Physical: Instruments and techniques.Citation: Mann, M. E. (2004), On smoothing potentially non-stationary climate time series, Geophys. Res. Lett., 31, L07214,doi:10.1029/2004GL019569.1. Introduction[2] Proper smoothing of climate time series, particularlythose exhibiting non-stationary behavior (e.g., substantialtrends late in the series) is essential for placing recent trendsin the context of past variability. The smoothing of a timeseries can be posed as an inverse problem with non-uniqueboundary constraints [Park, 1992], for which additionalobjective considerations must be made to determine thebehavior near the boundaries. Various different boundaryconstraints have recently been employed, for example, inthe smoothing of the Northern Hemisphere mean tempera-ture series [Folland et al., 2001; Mann and Jones, 2003;Soon et al., 2004]. The approach used by Mann and Jones[2003], as noted therein, employed a smoothing boundaryconstraint optimized to resolve the non-stationary latebehavior of the time series in comparison with previouslyemployed constraints involving e.g., the padding of theseries with mean values after the boundary [Folland et al.,2001; Mann, 2002; Mann et al., 2003]. The approach usedby Mann and Jones [2003] (which is incorrectly assumedby Soon et al. [2004] to be a ‘wavelet’ approach), isdescribed in more detail in this study.[3] The three lowest order boundary constraints that canbe applied to a smooth [see, e.g., Park, 1992] involve theminimization near the boundaries of either: (1) the zerothderivative of the smooth (yielding the ‘smallest’ or ‘mini-mum norm’ solution), (2) the 1st derivative of the smooth(yielding the ‘minimum slope’ constraint), and (3) the 2ndderivative of the smooth (yielding the smoothest or ‘mini-mum roughness’ solution). Application of constra int(1) favors the tendency of the smooth to approach the meanvalue (i.e., ‘climatology’) near the boundaries. Applicationof (2) favors the tendency of the smooth to approach aconstant local value near the boundary. Application of (3)favors the tendency of the smooth to approach the boundarywith a constant slope. The first two approaches will under-estimate the behavior of the time series near the boundariesin the presence of a long-term trend, but the 3rd approachmayleadtoanextrapolationerrorinthepresenceofleverage by outliers near the boundaries. Without additionalconsiderations, none of these thre e con straints can befavored on a priori grounds. An objective choice, nonethe-less, can be motivated as that particular constraint of thethree which minimizes some measure of misfit of thesmooth with respect to the original time series.[4] In this paper, we describe both time-domain andfrequency-domain a pproaches to im plementing each ofthese three alternative boundary conditions, and employan objective measure of the quality of fit of the variouscandidate smooths.[5] We note that while our focus here is on smoothing oftime series, similar considerations can be applied to alter-native statistical time series modeling such as change pointanalysis [Tome´ and Miranda, 2004]. We provide applica-tions to two relevant instrumental climate time series, theNorthern Hemisphere (NH) annual mean series from 1856 –2003 of Jones et al. [1999], and the cold-season NorthAtlantic Oscillation (NAO) times series of Jones et al.[1997] from 1825/26 to 1999/2000.2. Method[6] While constraints (1)–(3) can be applied explicitly inthe frequency domain [e.g., Park, 1992; Ghil et al., 2002], itis possible to implement reasonable approximations to theseconstraints in a si mple manner in the time domain asfollows: To approximate the ‘minimum norm’ constraint,one pads the series with the long-term mean beyond theboundaries (up to at least one filter width) prior to smooth-ing. To approximate the ‘minimum slope’ constraint, onepads the series with the values within one filter width of theboundary reflected about the time boundary. This leads thesmooth towards zero slope as it approaches the boundary.Finally, to approximate the ‘minimum roughness’ con-straint, one pads the series with the values within one filterwidth of the boundary reflected about the time boundary,and reflected vertically (i.e., about the ‘‘y’’ axis) relative tothe final value. This tends to impose a point of inflection atthe boundary, and leads the smooth towards the boundarywith constant slope. Alternative approximate implementa-tions of constraints (1)– (3) are of course possible.[7] We first make use of a routine that we have written inthe ‘Matlab’ programming language which implementsconstraints (1) –(3), as described above, making use of a10 point ‘‘Butterworth’’ low-pass filter for smoothing; otherGEOPHYSICAL RESEARCH LETTERS, VOL. 31, L07214, doi:10.1029/2004GL019569, 2004Copyright 2004 by the American Geophysical Union.0094-8276/04/2004GL019569$05.00L07214 1of4filters can be substituted yielding similar results. OurMatlab routine is provided here: ftp://holocene.evsc.virginia.edu/pub/mann/Filter/lowpass.m [note that this rou-tine requires access to the ‘Matlab’ time series analysis(‘signal’) toolbox]. For each of the three alternativesmooths, the resulting mean-square error (‘MSE’) of thesmooth is calculated as a fraction of the total variance inthe series resolved by the smooth. That constraint providingthe minimum MSE is a rguably the optimal constraintamong the three tested. MSE, which penalizes the mean-squared deviations,