1 “My Weekend” Example (by Simon Colton, http://www.doc.ic.ac.uk/~sgc/teaching/v231/lecture11.html) Imagine you only ever do one of the following four things for any weekend: • go shopping • watch a movie • play tennis • just stay in What you do depends on three factors: 1. weather (windy, rainy or sunny) 2. how much money you have (rich or poor) 3. whether your parents are visiting (yes or no) You say to yourself: if my parents are visiting, we'll go to the cinema. If they're not visiting and it's sunny, then I'll play tennis, and so on. Suppose we have the following instances in our (training) dataset: Weekend (Example) Weather Parents Money Decision (Category) W1 Sunny Yes Rich Cinema W2 Sunny No Rich Tennis W3 Windy Yes Rich Cinema W4 Rainy Yes Poor Cinema W5 Rainy No Rich Stay in W6 Rainy Yes Poor Cinema W7 Windy No Poor Cinema W8 Windy No Rich Shopping W9 Windy Yes Rich Cinema W10 Sunny No Rich Tennis Apply the ID3 algorithm on this dataset to induce a decision tree. Show all your work. SolutioThe first our tree: Entropy(= -(6/10)= -(6/10)= 0.4422and we nGain(S, w(|Srain|/10= 1.571 -= 1.571 -Gain(S, p= 1.571 -Gain(S, m= 1.571 -This meaconvinceentropy mFrom therainy: Now we default caTennis annode hereLooking do not alsame situon thing we neeither weath(S) = -pcinema ) * log2(6/10) * -0.737 -(2 + 0.4644 + need to determweather) = 10)*Entropy(S- (0.3)*Entro- (0.3)*(0.91parents) = 1.- (0.5) * 0 - (money) = 1.5- (0.7) * (1.8ans that the fe yourself whmeans and loe weather nolook at the fategorisationnd Tennis ree. Hence weat the seconl belong to tuation happeed to do is wher, parents olog2(pcinema)) -(2/10) * lo2/10) * -2.320.3322 + 0.3mine the bes.571 - (|SsunSrain) opy(Ssun) - (08) - (0.4)*(0571 - (|Syes|/(0.5) * 1.922571 - (|Srich|/842) - (0.3) *first node in hy this scoreook at the wade, we drawfirst branch. n leaf node hespectively. Ae put an attribnd branch, Swthe same clasens with the twork out whior money. T) -ptennis log2(og2(2/10) -(122 -(1/10) * 3322 = 1.57st of: |/10)*Entrop0.4)*Entropy0.81125) - (0/10)*Entropy2 = 1.571 - 0/10)*Entropy* 0 = 1.571 -the decisioned (slightly) ay informatiw a branch foSsunny = {Where. The catAs these arebute node hewindy = {W3, ss, so we putthird branchich attribute o do this, we(ptennis) -pshop1/10) * log2(-3.322 -(1/11 py(Ssun) - (|Swy(Swind) - (0.0.3)*(0.918) y(Syes) - (|Sno0.961 = 0.61 y(Srich) - (|Sp 1.2894 = 0.n tree will behigher than ion gain is caor the values 1, W2, W10tegorisations not all the sere, which wW7, W8, Wt an attributeh, hence our a will be put e need to calpping log2(psh(1/10) -(1/100) * -3.322 wind|/10)*En3)*Entropy(= 0.70 o|/10)*Entro poor|/10)*Entr.2816 e the weathethe parents aalculated. that weathe 0}. This is nos of W1, W2same, we canwe will leaveW9}. Again, te node here, amended treinto the nodlculate: hopping) -pstay_i0) * log2(1/1ntropy(Swind) (Srain) opy(Sno) ropy(Spoor) er attribute. Aattribute - rer can take: sot empty, so 2 and W10 annot put a cae blank for ththis is not emleft blank foee looks like de at the top oin log2(pstay_in0) - As an exerciemember whsunny, windywe do not pare Cinema, ategorisationhe time beingmpty, and thor now. The this: 2 of n) ise, hat y and put a n leaf g. hey Now we we've alrfor Gain(to be SsunexamplesHence wGain(Ssun= 0.918 -Gain(Ssun= 0.918 -Notice thare all in same catechoose atGiven ouyes and nreplaced Hence, thSno contaends herehave to fill iready remov(Ssunny, parennny = {W1,Ws). In effect, Weekene can calculanny, parents) - (1/3)*0 - (2nny, money) =- (3/3)*0.918hat Entropy(Sthe same caegory (tennittributes to pur calculationno, and we wthe set S by he branch foains W2 and e at a categoin the choiceed that fromnts) and GainW2,W10} (anwe are internd (ExamplW1 W2 W10 ate: = 0.918 - (|S2/3)*0 = 0.91= 0.918 - (|S8 - (0/3)*0 =Syes) and Entategory (cineis). This shouput in nodes.ns, attribute will draw a bthe set SSunnr yes stops aW10, but thrisation leafe of attributem the list of an(Ssunny, monnd, for this prested only inle) WeatherSunny Sunny Sunny Syes|/|S|)*Ent18 Srich|/|S|)*Ent= 0.918 - 0.9tropy(Sno) wema), and Snuld make it m. A should bebranch from tny, looking aat a categorishese are in thf. Hence our e A, which wattributes to uney). Firstlyart of the bran this part ofrParentsMYes RNo RNo Rtropy(Syes) -tropy(Srich) -18 = 0 were both zerno similarly cmore obvioue taken as pathe node forat Syes, we sesation leaf, whe same cateupgraded trewe know canuse. So, we ny, Entropy(Ssanch, we wif the table: MoneyDecisRich Rich Rich (|Sno|/|S|)*E (|Spoor|/|S|)*ro, because Scontains examus why we uarents. The tr each of these that the onwith the category (Tenniee looks like nnot be weathneed to calcusunny) = 0.918ill ignore all sion (CategoCinema Tennis Tennis ntropy(Sno) *Entropy(SpoSyes containsmples whichuse informatitwo values fse. Remembnly example egory being Cis). Hence the this: her, becauseulate the val8. Next, we the other ory) oor) s examples wh are all in thion gain to from parentsbering that wof this is WCinema. Alshe branch for3 e lues set S which he s are we 1. so, r no … And the fWeather| Pare| PareWeather| Pare| Pare| | M| | MWeather| Pare| Pare final tree loor = Sunny ents = Yesents = No:r = Windy ents = Yesents = No Money = RiMoney = Por = Rainy ents = Yesents = No:oks like: s: Cinema: Tennis s: Cinemaich: Shopoor: Cines: Cinema: Stay_in a a pping ema a n 4 Prunin“Pruning aand assign(p. 69 in T(0) Origin (1) ChoosDoing so rand shoppcommon But since the tree is ng a decision noning it the moTom Mitchell’al fully growne the subtreeresults in the ping (1 instanin the entire the classificas further modode consists oost common c’s textbook) n tree e under ‘Monfollowing trence each in thdataset). tion under thdified as: of removing thclassificationey’ to prune,ee (where thehe subtree, Whe ‘Parent’ nohe subtree roof the trainin because thate ‘most commW7 and W8; ciode above becooted at that ng examples at’s the lowestmon classificatnema was secomes the sanode, makingaffiliated witht subtree. tion’ was a tielected becauame for both g it a leaf nodh that node.”e between cise it was morof its branche5 de, nema re es, (2) Next cThere are‘cinema’ b [Nb3 ch(3) Next cThere are‘tennis’ be choose the sue 3 instances wbecause that’Note that theoth subtrees instances hahosen