Markov Processes© Arizona State University, Department of Mathematics and Statistics 1 of 4Markov ChainsSo what is a Markov process or Markov chain? You might call it a short cut. When we worked the previousproblems where we had chains of probability we had to build tree diagrams or extensive tables to calculate theprobability after a number of steps. With the Markov process, we can get it all done through matrix arithmetic. Formally, we define a Markov Chain to be a sequence of experiments, each of which results in one of a finitenumber of states (1, 2, 3, …m). There is a similarity here to a Bernoulli trial, but these are not the same crittersas you will see from the examples.The first step in working with these chains is to create a Transition Matrix (P). This is a square matrix whoseentries represent the probability of moving from one state to another. We let pij represents the probability ofmoving from state i to state j in one observation (step).For example, look at the matrix to the right. It tells us all of the following:If we start in state 1 (s1), after one step, the probability of staying in s1 is .85, andthe probability of moving to s2 is .15. If we start in state 2 (s2) then, after one step, the probability of staying there is .55,and the probability of moving to s1 is .45.Notice that each row in a transition matrix is a probability distribution, so the sum ofthe entries in the row is 1.We can demonstrate the same information by a tree process, but once we get past about two steps through theprocess, it becomes a forest. Here’s how it would look in a video demonstration: Markov Maps.In the video, you saw the question marks, this signifies the initial probability distribution, . This is a(0) v row vector (if there are n states) whose entries represent the probability of starting in each respective1n×state. For example, if there were two states, and we had an equal chance of starting in either one, then the initialprobability distribution would be . If we were guaranteed to start in state 1, it would be(0) 0.5,0.5v =. If we were guaranteed to start in state 2, it would be . (0) 1,0v =(0) 0,1v =The probability distribution after k steps, observations, or stages is denoted as , and can be calculated as( ) kvfollows: where P is the transition Matrix.( ) ( 1) (0) k k kv v P v P−= =Let’s set up a couple of transition matrices, then manipulate them to see what the future will be. The Markovprocess is a great predictive tool for market analysis or psychological evaluation when we have reliablestatistics about the probabilities involved.1 21 .85 .152 .45 .55s sss   Markov Processes© Arizona State University, Department of Mathematics and Statistics2 of 4Example 1: There are two insurance companies in town, Big Mega Inc. and Mom's Insurance. Everyone intown has one or the other. Every year, 5% of Big Mega's customers switch to Mom's, and 3% ofMom's customers move to Big Mega. Create the Transition matrix,P.The matrix is to the right. I chose to label the states to remind me of wherethey came from. Generically we will use s1, s2, ..., Sn in homework. Noticethat we have complementary situations provided for the switchers. If 5% movethen 95% remain with Big Mega.So where does the initial probability distribution come into this? Supposeright now Big Mega has 94% of the customers in town, and Mom's has therest, find the initial probability distribution.Again the result is to the right. 1. So after one year, what percentage of the customers will each have?The calculation is easy, especially with a calculator. Just find.(1) (0) 0.8948,0.1052kv v P= =2. After 5 years, what percentage will each have? Still easy. Just find(5) (0) 5 0.747381... ,0.252618...v v P= =Notice that these are extended decimal forms. They are repeating but good luck finding thedenominator easily. We will let the calculator do the work and reflect at least 6 decimal places.Our anser is that about 74.7381% will be insured by Mig Mega while about 25.2618 % will beinsured by Mom’s Insurance. Notice that I used “about.” The two results do not add to 100%since we are not seeing all the significant digits in the calculation. Example 2: A company sells three models of cars, call them A, B, and C. They have found that the transitionmatrix below describes how customers change from onemodel to the other annually.Notice that this is a valid transition matrix. We can interpret it to saythat customer satisfaction with model A is low since only 25% arewilling to stay with it. Of the remainder, 35% move up to B, and 40%move to C.Customers are more satisfied with model B since 65% stick with it.Only 10% would fall back to A while 25% move up to C. It does have the curious feature that once someone buys model C, theynever leave it! We’ll talk more about this in a later lesson..95 .05.03 .97BM MIBMMI   (0) 0.94,0.06v =.25 .35 .40.10 .65 .250 0 1A B CABC     Markov Processes© Arizona State University, Department of Mathematics and Statistics 3 of 41. What is the probability that someone will own model B after three years if the initial probabilitydistribution is uniform.Recall that uniform means that all states are equally likely. Hence each has . 13p =That gives us (0)1 1 1 , ,3 3 3v =We need to calculate (3) (0) 3(3)3.25 .35 .401 1 1, , .10 .65 .253 3 30 0 1331 757 1231, ,8000 8000 8000v v Pv  = =   =Again notice that this time I got the denominator! I used the TI-83/84 Math >> Frac operator and itfound it for me. Whenever possible, use this option. It will simplify your results and give an exact resultwhen it works. I did try it on the previous example. It failed me.Example 3: A company produces 4 products, call them A, B, C, D. It discovers that customers prefer them inthe weightings of 2:3:5:10 if they start in with product A. However, starting with product B theweightings change to 5:3:2:10 . Starting with product C, we see 4:4:10:2. And finally when theystart with product D, the find this weightings: 2:2:2:14. 1. Create the transition matrix.Since the weightings provided for the first row is 2:3:5:10, thiscreates probabilities of for Product3 5 102 , , ,20 20 20 20  A. These were reduced to decimal values as shown in the table.The other rows were created similarly, 2. Suppose the initial probability distribution is has all customersinitially owning product A. What