Graduate Lectures and Problems in QualityControl and Engineering Statistics:Theory and MethodsTo AccompanyStatistical Quality Assurance Methods for EngineersbyVardeman and JobeStephen B. VardemanV2.0: January 2001c° Stephen Vardeman 2001. Permission to copy for educationalpurposes granted by the author, subject to the requirement thatthis title page be a¢xed to each copy (full or partial) produced.Chapter 3An Introduction to DiscreteStochastic ControlTheory/Minimum VarianceControlSection 3.6 of V&J provides an elementary introduction to the topic of Engi-neering Control and contrasts this adjustment methodology with (the processmonitoring methodology of) control charting. The last item under the En-gineering Control heading of Table 3.10 of V&J makes reference to “optimalstochastic control” theory. The object of this theory is to model system behav-ior using probability tools and let the consequences of the model assumptionshelp guide one in the choice of e¤ective control/adjustment algorithms. Thischapter provides a very brief introduction to this theory.3.1 General ExpositionLetf: : : ; Z(¡1); Z(0); Z(1); Z(2); : : :gstand for observations on a process assuming that no control actions are taken.One …rst needs a stochastic/probabilistic model for the sequence fZ(t)g, andwe will letFstand for such a model. F is a joint distribution for the Z’s and might, forexample, be:1. a simple random walk model speci…ed by the equation Z(t) = Z(t ¡ 1) +²(t), where the ²’s are iid normal (0; ¾2) random variables,3738CHAPTER 3. AN INTRODUCTION TO DISCRETE STOCHASTIC CONTROL THEORY/MIN2. a random walk model with drift speci…ed by the equation Z(t) = Z(t ¡1)+d+²(t), where d is a constant and the ²’s are iid normal (0; ¾2) randomvariables, or3. some Box-Jenkins ARIMA model for the fZ(t)g sequence.Then leta(t)stand for a control action taken at time t, after observing the process. Oneneeds notation for the current impact of control actions taken in past periods,so we will further letA(a; s)stand for the current impact on the process of a control action a taken s periodsago. In many systems, the control actions, a, are numerical, and A(a; s) = ah(s)where h(s) is the so-called “impulse response function” giving the impact of aunit control action taken s periods previous. A(a; s) might, for example, be:1. given by A(a; s) = a for s ¸ 1 in a machine tool control problem where “a”means “move the cutting tool out a units” (and the controlled variable isa measured dimension of a work piece),2. given by A(a; s) = 0 for s · u and by A(a; s) = a for s > u in a machinetool control problem where “a” means “move the cutting tool out a units”and there are u periods of dead time, or3. given by A(a; s) =¡1 ¡ exp¡¡sh¿¢¢a for s ¸ 1 in a chemical processcontrol problem with time constant ¿ and control period h seconds.We will then assume that what one actually observes for (controlled) processbehavior at time t ¸ 1 isY (t) = Z(t) +t¡1Xs=0A(a(s); t ¡ s) ;which is the sum of what would have been observed with no control and all ofthe current e¤ects of previous control actions. For t ¸ 0, a(t) will be chosenbased onf: : : ; Z(¡1); Z(0); Y (1); Y (2); : : : ; Y (t)g :A common objective in this context is to choose the actions so as to minimizeEF(Y (t) ¡ T (t))2ortXs=1EF(Y (s) ¡ T (s))23.1. GENERAL EXPOSITION 39for some (possibly time-dependent) target value T (s). The problem of choosingof control actions to accomplish this goal is called the “minimum variance“(MV) control problem, and it has a solution that can be described in fairly(deceptively, perhaps) simple terms.Note …rst that given f: : : ; Z(¡1); Z(0); Y (1); Y (2); : : : ; Y (t)g one can recoverf: : : ; Z(¡1); Z(0); Z(1); Z(2); : : : ; Z(t)g. This is becauseZ(s) = Y (s) ¡s¡1Xr=0A(a(r); s ¡ r)i.e., to get Z(s), one simply subtracts the (known) e¤ects of previous controlactions from Y (s).Then the model F (at least in theory) provides one a conditional distributionfor Z(t + 1); Z(t + 2); Z(t + 3); : : : given the observed Z’s through time t. Theconditional distribution for Z(t + 1); Z(t + 2); Z(t + 3) : : : given what one canobserve through time t, namely f: : : ; Z(¡1); Z(0); Y (1); Y (2); : : : ; Y (t)g, is thenthe conditional distribution one gets for Z(t +1); Z(t + 2); Z(t +3); : : : from themodel F after recovering Z(1); Z(2); : : : ; Z(t) from the corresponding Y ’s. Thenfor s ¸ t + 1, letEF[Z(s)j : : : ; Z(¡1); Z(0); Z(1); Z(2); : : : ; Z(t)] or just EF[Z(s)jZt]stand for the mean of this conditional distribution of Z(s) available at time t.Suppose that there are u ¸ 0 periods of dead time (u could be 0). Thenthe earliest Y that one can hope to in‡uence by choice of a(t) is Y (t + u + 1).Notice then that if one takes action a(t) at time t, one’s most natural projectionof Y (t + u + 1) at time t isbY (t + u + 1jt):= EF[Z(t + u + 1)jZt] +t¡1Xs=0A(a(s); t + u + 1 ¡ s) + A(a(t); u + 1)It is then natural (and in fact turns out to give the MV control strategy) to tryto choose a(t) so thatbY (t + u + 1jt) = T (t + u + 1) :That is, the MV strategy is to try to choose a(t) so thatA(a(t); u+1) = T (t+u+1)¡(EF[Z(t + u + 1)jZt] +t¡1Xs=0A(a(s); t + u + 1 ¡ s)):A caveat here is that in practice MV control tends to be “ragged.” Thatis, in order to exactly optimize the mean squared error, constant tweaking (andoften fairly large adjustments are required). By changing one’s control objectivesomewhat it is possible to produce “smoother” optimal control policies that are40CHAPTER 3. AN INTRODUCTION TO DISCRETE STOCHASTIC CONTROL THEORY/MINnearly as e¤ective as MV algorithms in terms of keeping a process on target.That is, instead of trying to optimizeEFtXs=1(Y (s) ¡ T (s))2;in a situation where the a’s are numerical (a = 0 indicating “no adjustment”and the “size” of adjustments increasing with jaj) one might for a constant ¸ > 0set out to minimize the alternative criterionEFÃtXs=1(Y (s) ¡ T (s))2+ ¸t¡1Xs=0(a(s))2!:Doing so will “smooth” the MV algorithm.3.2 An ExampleTo illustrate the meaning of the preceding formalism, consider the model (F)speci…ed byZ(t) = W (t) + ²(t) for t ¸ 0and W (t) = W (t ¡ 1) + d + º(t) for t ¸ 1¾(3.1)for d a (known) constant, the ²’s normal (0; ¾2²), the º’s normal (0; ¾2º) andall the ²’s and º’s independent. (Z(t) is a random walk with drift observedwith error.) Under this model and an appropriate