More on Forecasting Vladas Pipiras STOR UNC CH March 2022 What is this about We have looked at a few forecasting methods e g based on ARIMA models Here A few other forecasting methods including more na ve How does one think about forecast accuracy Which method to use And perhaps not to use Some simple forecasting methods yT h T y y1 yT T yT h T yT yT h T yT yt yt 1 yT h h T 1 T X t 2 cid 18 yT y1 T 1 cid 19 Average method R meanf y h Na ve method R naive y h or rwf y h Drift method R rwf y h drift TRUE Example library fpp2 beer2 window ausbeer start 1992 end c 2007 4 beerfit1 meanf beer2 h 10 beerfit2 rwf beer2 h 10 beerfit3 snaive beer2 h 10 autoplot window ausbeer start 1992 autolayer beerfit1 series Mean PI FALSE autolayer beerfit2 series Na ve PI FALSE autolayer beerfit3 series Seasonal na ve PI FALSE xlab Year ylab Megalitres ggtitle Forecasts for quarterly beer production guides colour guide legend title Forecast 1 Evaluating forecast accuracy Training and test sets The accuracy of forecasts can be determined by considering not model residuals but only how well a model performs on new data that were not used when tting the model A model which ts the training data well will not necessarily forecast well A perfect t can always be obtained by using a model with enough parameters Over tting a model to data is just as bad as failing to identify a systematic pattern in the data Measures of forecast accuracy are based on Forecast errors eT h yT h yT h T Scale dependent errors Mean absolute error MAE mean et Root mean squared error RMSE q mean e2 t Percentage errors The percentage error is given by pt 100et yt Mean absolute percentage error MAPE mean pt Scaled errors where for a non seasonal timme series MASE mean qj qj ej 1 T 1 T X t 2 yt yt 1 2 4004505001995200020052010YearMegalitresForecastMeanNa veSeasonal na veForecasts for quarterly beer production and for a seasonal time series qj ej T X t m 1 1 T m yt yt m Example beer3 window ausbeer start 2008 accuracy beerfit1 beer3 Training set Test set Training set Test set ME MAE RMSE ACF1 MPE 0 000 43 62858 35 23438 0 9365102 7 886776 2 463942 0 10915105 13 775 38 44724 34 82500 3 9698659 8 283390 2 435315 0 06905715 Theil s U NA 0 801254 MASE MAPE MAE 0 4761905 65 31511 54 73016 MASE 0 9162496 12 16415 3 827284 51 4000000 62 69290 57 40000 12 9549160 14 18442 4 013986 MAPE RMSE MPE accuracy beerfit2 beer3 ME Training set Test set Training set 0 24098292 0 06905715 Test set accuracy beerfit3 beer3 ACF1 Theil s U NA 1 254009 ME RMSE ACF1 Training set 2 133333 16 78193 14 3 0 5537713 3 313685 1 0000000 0 2876333 Test set 0 1318407 Training set Test set Theil s U NA 0 298728 1 1475536 3 168503 0 9370629 5 200000 14 31084 13 4 MAPE MASE MAE MPE Another example googfc1 meanf goog200 h 40 googfc2 rwf goog200 h 40 googfc3 rwf goog200 drift TRUE h 40 autoplot subset goog end 240 autolayer googfc1 PI FALSE series Mean autolayer googfc2 PI FALSE series Na ve autolayer googfc3 PI FALSE series Drift xlab Day ylab Closing Price US ggtitle Google stock price daily ending 6 Dec 13 guides colour guide legend title Forecast 3 googtest window goog start 201 end 240 accuracy googfc1 googtest ME Training set 4 296286e 15 Test set Training set 0 9668981 Test set ACF1 Theil s U NA 0 8104340 13 92142 accuracy googfc2 googtest ME Training set 0 6967249 Test set Training set 0 06038617 0 81043397 Test set accuracy googfc3 googtest ACF1 Theil s U NA 3 451903 MASE 7 182995 1 132697e 02 114 21375 113 26971 20 3222979 20 32230 30 280376 MPE 26 86941 0 6596884 RMSE 36 91961 MAPE 5 95376 MAE MASE 3 740697 0 1426616 0 8437137 1 000000 24 3677328 28 434837 24 593517 4 3171356 4 3599811 6 574582 RMSE 6 208148 MAPE MPE MAE ME Training set 5 998536e 15 Test set Training set 0 06038617 0 64732736 Test set ACF1 Theil s U NA 1 709275 RMSE 6 168928 1 008487e 01 14 077291 11 667241 Time series cross validation MAE MASE 3 824406 0 01570676 0 8630093 1 022378 1 77566103 2 0700918 3 119002 MAPE MPE A more sophisticated version of training test sets is time series cross validation 4 400450500550050100150200250DayClosing Price US ForecastDriftMeanNa veGoogle stock price daily ending 6 Dec 13 The forecast accuracy is computed by averaging over the test sets e tsCV goog200 rwf drift TRUE h 1 sqrt mean e 2 na rm TRUE sqrt mean residuals rwf goog200 drift TRUE 2 na rm TRUE 1 6 233245 1 6 168928 A good way to choose the best forecasting model is to nd the model with the smallest RMSE computed using time series cross validation More on cross validation Variants of prequential approaches 5 Variants of cross validation Some observations from Cerqueira et al 2020 Empirical experiments suggest that blocked cross validation can be applied to stationary time series When the time series are non stationary the most accurate estimates are produced by out of sample methods particularly the holdout approach repeated in multiple testing periods Some other forecasting methods Exponential smoothing Simple exponential smoothing SES For forecasting data with no clear trend or seasonal pattern yT 1 T yT 1 yT 1 1 2yT 2 where 0 1 is the smoothing parameter This can be rewritten as yT 1 T yT 1 yT T 1 and more generally as This is also expressed in a component form as yt 1 t yt 1 yt t 1 Forecast equation Smoothing equation yt h t t t yt 1 t 1 The smoothing parameter and the starting value 0 are chosen to minimize SSE yt yt t 1 2 T X t 1 T X t 1 e2 t 6 Example oildata window oil start 1996 Estimate parameters fc ses oildata h 5 fc model alpha 0 8339 Smoothing parameters Simple exponential smoothing Call ses y oildata h 5 BIC 178 1430 179 8573 180 8141 fc Initial states l 446 5868 sigma 29 8282 AICc AIC 2014 2015 2016 2017 2018 Point Forecast Lo 95 Hi 80 Lo 80 Hi 95 542 6806 504 4541 580 9070 484 2183 601 1429 542 6806 492 9073 592 4539 466 5589 618 8023 542 6806 483 5747 601 7864 452 2860 633 0752 542 6806 475 5269 609 8343 439 9778 645 3834 542 6806 468 3452 617 0159 428 9945 656 3667 Accuracy of one step ahead training errors round accuracy fc 2 ACF1 Training set 6 4 28 12 22 26 1 1 4 61 0 93 0 03 MAE MPE MAPE MASE RMSE ME autoplot fc autolayer fitted fc series Fitted ylab Oil millions of tonnes xlab …