A THREE-STEP METHOD FOR CHOOSING THE NUMBER OF BOOTSTRAP REPETITIONS BY DONALD W. K. ANDREWS and MOSHE BUSHINSKY COWLES FOUNDATION PAPER NO. 1001 COWLES FOUNDATION FOR RESEARCH IN ECONOMICS YALE UNIVERSITY Box 208281 New Haven, Connecticut 06520-8281 2000 http://cowles.econ.yale.edu/Ž.Econometrica, Vol. 68, No. 1 January, 2000 , 23᎐51A THREE-STEP METHOD FOR CHOOSING THENUMBER OF BOOTSTRAP REPETITIONSBY DONALD W. K. ANDREWS AND MOSHE BUCHINSKY1This paper considers the problem of choosing the number of bootstrap repetitions Bfor bootstrap standard errors, confidence intervals, confidence regions, hypothesis tests,p-values, and bias correction. For each of these problems, the paper provides a three-stepmethod for choosing B to achieve a desired level of accuracy. Accuracy is measured bythe percentage deviation of the bootstrap standard error estimate, confidence intervallength, test’s critical value, test’s p-value, or bias-corrected estimate based on B bootstrapsimulations from the corresponding ideal bootstrap quantities for which Bs⬁.The results apply quite generally to parametric, semiparametric, and nonparametricmodels with independent and dependent data. The results apply to the standard nonpara-metric iid bootstrap, moving block bootstraps for time series data, parametric andsemiparametric bootstraps, and bootstraps for regression models based on bootstrappingresiduals.Monte Carlo simulations show that the proposed methods work very well.KEYWORDS: Bias correction, bootstrap, bootstrap repetitions, confidence interval, hy-pothesis test, p-value, simulation, standard error estimate.1. INTRODUCTIONBOOTSTRAP METHODS HAVE GAINEDa great deal of popularity in empiricalresearch. Although the methods are easy to apply, determining the number ofbootstrap repetitions, B,toemploy is a common problem in the existingliterature. Typically, this number is determined in a somewhat ad hoc manner.This is problematic, because one can obtain a ‘‘different answer’’ from the samedata merely by using different simulation draws if B is chosen to be too small.On the other hand, it is expensive to compute the bootstrap statistics of interest,if B is chosen to be extremely large. Thus, it is desirable to be able to determinea value of B that obtains a suitable level of accuracy for a given problem athand. This paper addresses this issue in the context of the three main branchesof statistical inference, viz., point estimation, interval and region estimation, andhypothesis testing.We provide methods for determining B to attain specified levels of accuracyfor bootstrap standard error estimates, confidence intervals, confidence regions,hypothesis tests, and bias correction. The basic strategy is the same in each case.1The authors thank Ariel Pakes, three referees, and the co-editor for helpful comments; GlenaAmes for typing the original manuscript; and Rosemarie Lewis for proofreading the manuscript. Thefirst author acknowledges the research support of the National Science Foundation via GrantNumbers SBR-9410975 and SBR-9730277. The second author acknowledges the research support ofthe National Science Foundation via Grant Number SBR-9320386 and the Alfred P. Sloan Founda-tion via a Research Fellowship.23D. W. K. ANDREWS AND M. BUCHINSKY24We approximate the distribution of the appropriate bootstrap statistic by itsasymptotic distribution as B ª ⬁. Here we are referring to the distribution ofthe statistic with respect to the simulation randomness conditional on thesample. We replace unknown parameters in the asymptotic distribution byconsistent estimates. Then, we determine a formula for how large B needs to beto attain a desired level of accuracy based on the asymptotic approximation. Athree-step method for choosing B is proposed for each case. Three steps arerequired because one needs to estimate unknown parameters in the initial twosteps before one can determine a suitable choice of B in the third step.The measure of accuracy employed is the percentage deviation of the boot-strap quantity of interest based on B repetitions from the ideal bootstrapquantity, for which B s⬁.Inparticular, in the different applications considered,accuracy is measured by the percentage deviation of a bootstrap standard errorestimate, confidence interval ‘‘length,’’ critical value of a test, p-value, orbias-corrected estimate based on B repetitions from its ideal value based onBs⬁. For a symmetric two-sided confidence interval, the ‘‘length’’ is just thedistance between the lower and upper bounds of the interval. For a one-sidedconfidence interval, the interval has an infinite length. In this case, the ‘‘length’’that we consider is the lower or upper length of the interval depending uponwhether the one-sided interval provides a lower bound or an upper bound. Bydefinition, the lower length of a confidence interval for a parameter␪based on aˆparameter estimate␪is the distance between the lower endpoint of theˆconfidence interval and the parameter estimate␪. The upper length is definedanalogously. For two-sided equal-tailed confidence intervals, we consider boththe lower and upper lengths of the confidence interval.The accuracy obtained by a given choice of B is stochastic, because thebootstrap simulations are random. To determine a suitable value of B,wespecify a bound on the relevant percentage deviation, denoted pdb, and werequire that the actual percentage deviation be less than this bound with aspecified probability, 1 y␶, close to one. The three-step method takes pdb and␶as given and specifies a data-dependent method of determining a value of B,denoted B*, such that the desired level of accuracy is obtained. For example,Ž.Ž.one might take pdb,␶s 10, .05 . Then, the three-step method yields a valueB* such that the relevant percentage deviation is less than 10% with approxi-mate probability .95.The three-step methods are applicable in parametric, semiparametric, andŽ.nonparametric models with independent and identically distributed iid data,Ž.independent and nonidentically distributed inid data, and time series data. Themethods are applicable when the bootstrap employed is the standard nonpara-metric iid bootstrap, a moving block bootstrap for time series, a parametric orsemiparametric bootstrap, or a bootstrap for regression models that is based onbootstrapping residuals. The methods are applicable to statistics that havenormal and