CMSC 411Computer Systems ArchitectureLecture 19Lecture 19Storage Systems, concl. &MultiprocessorsAlan Sussmanl@ d [email protected]• Homework #5 posted, due April 28•Exam #2 next Thursday April 23•Exam #2 next Thursday, April 23– practice exam posted later today• Cache simulator project questions?CMSC 411 - 19 (some from Patterson, Sussman, others)2Designing an I/O SystemDesigning an I/O SystemDesigning an I/O System• Price, performance, and capacity issues• Need to choose– which I/O devices to connect– how to connect them•Example: The CPU is seldom the limiting factor•Example: The CPU is seldom the limiting factor for I/O performance• Suppose the CPU can handle 10,000 I/O i d (IOPS)operations per second (IOPS)• And suppose the average I/O size is 16 KBCMSC 411 - 19 (some from Patterson, Sussman, others)4I/O Systems• The other links in the I/O chain are:– the I/O controller - suppose it adds 1 ms overhead per I/O tioperation– the I/O bus - suppose it is a bus that can transfer 20 MB/sec = 20 KB/ms–thedisk-suppose it rotates at 7200 RPM with 8 ms average–the disk-suppose it rotates at 7200 RPM, with 8 ms average seek time and 6 MB/sec transfer rateCMSC 411 - 19 (some from Patterson, Sussman, others)5I/O System Performance• Consider the disk time first:– 7200 RPM = 7200/(60*103) = .12 revolutions per ms– 6 MB/sec = 6 KB/ms– So the average disk time is seek + rotational latency + transfer =8 ms + .5 / .12 ms + 16 / 6 = 14.9 ms•So the average time per transfer is•So the average time per transfer is– I/O controller time + bus time + disk time = 1 ms + 16 / 20 ms + 14.9 ms = 16.7 ms• So with one controller, one bus, and one disk, can do at ,, ,most– 1/(16.7*10-3) = 60 IOPS• If this is not good enough, should analyze to see whether it is better to add more controllers, more buses, or more disks• Another, more complex, performance analysis in Section 6.7, for the Internet Archive ClusterCMSC 411 - 19 (some from Patterson, Sussman, others)6Storage Example: Internet Archive• Goal of making a historical record of the Internet– Internet Archive began in 1996– Wayback Machine interface performs time travel to see whatthe website at a URL looked like in the past• It contains over a petabyte (1015bytes), and is12growing by 20 terabytes (1012bytes) ofnew dataper month•Inadditiontostoringthehistoricalrecord,theInadditiontostoringthehistoricalrecord,thesame hardware is used to crawl the Web everyfew months to get snapshots of the Internet4/16/2009(some from Patterson, Sussman, others)7Internet Archive Cluster• 1U storage node PetaBox GB2000 fromCapricornTechnologiesCapricornTechnologies– Contains 4 500 GB Parallel ATA (PATA) disk drives,512 MB of DDR266 DRAM, one 10/100/1000Ethernetinterfaceanda1GHC3ProcessorfromEthernetinterface,anda1GHzC3ProcessorfromVIA (80x86).– Node dissipates ≈ 80 watts• 40 GB2000s in a standard VME rack,⇒ 80 TB of raw storage capacity• 40 nodes are connected with a 48-port10/100 or 10/100/1000 Ethernet switch1PtBt12k4/16/2009(some from Patterson, Sussman, others)8•1PetaByte=12racksEstimated Cost• VIA processor, 512 MB of DDR266 DRAM, ATA disk controller, power supply, fans, and l $500enclosure = $500• 7200 RPM Parallel ATA drive holds 500 GB = $375. $• 48-port 10/100/1000 Ethernet switch and all cables for a rack = $3000.C$8400f80TBk•Cost$84,500for a80-TBrack.• 160 Disks are ≈ 60% of the cost4/16/2009(some from Patterson, Sussman, others)9Estimated Performance• 7200 RPM Parallel ATA drive holds 500 GB, has an average time seek of 8.5 ms, transfers at 50 MB/second from the disk The PATA link speed is 133 MB/seconddisk. The PATA link speed is 133 MB/second.– performance of the VIA processor is 1000 MIPS.– operating system uses 50,000 CPU instructions for a disk I/O.–network protocol stack uses 100,000 CPU instructions to transmit a p,data block between the cluster and the external world• ATA controller overhead is 0.1 ms to perform a disk I/O.• Average I/O size is 16 KB for accesses to the historical record via theWaybackinterface and 50 KB whenrecord via the Waybackinterface, and 50 KB when collecting a new snapshot• Disks are limit: ≈ 75 I/Os/s per disk, 300/s per node, 12000/sper rack, or about 200 to 600 Mbytes / sec Bandwidth perkrack• Switch needs to support 1.6 to 3.8 Gbits/second over 40Gbit/sec links4/16/2009(some from Patterson, Sussman, others)10Estimated Reliability• CPU/memory/enclosure MTTF is 1,000,000 hours (x 40)(x 40)• PATA Disk MTTF is 125,000 hours (x 160)• PATA controller MTTF is 500,000 hours (x 40)()• Ethernet Switch MTTF is 500,000 hours (x 1)• Power supply MTTF is 200,000 hours (x 40)• Fan MTTF is 200,000 hours (x 40)• PATA cable MTTF is 1,000,000 hours (x 40)MTTFfthti531h(3k)•MTTFforthe systemis531hours(≈3weeks)• 70% of time failures are disks•20%oftimefailuresarefansorpowersupplies4/16/2009(some from Patterson, Sussman, others)11•20%oftimefailuresarefansorpowersuppliesConclusions - Fallacies• Disks never fail– a mean time to failure (MTTF) for one disk of 1.2M hours, or 140 t d b i th d f di k f f140 years, computed by running thousands of disks for a few months, then counting the number that failed– but a more useful measure is the % of disks that fail in a given time period (e.g., 5 years), computed as #failed disks/total p(g,y),p#disks» where # failed disks = #disks * (#hours/disk )/MTTFCMSC 411 - 19 (some from Patterson, Sussman, others)12Conclusions - Fallacies• Computer systems can achieve 99.999% availabilityavailability– that’s 5 minutes per year downtime, and highly unlikely in your environment–in 2001 well managed servers typically available 99% to 99 9%–in 2001, well managed servers typically available 99% to 99.9% of time• DRAM will replace disks in desktop and server machinesmachines– disk manufacturers have pushed the rate of technology improvement in disks to match or exceed that of DRAM–instead of DRAMs killing disks, disks are killing tapesinstead of DRAMs killing disks, disks are killing tapesCMSC 411 - 19 (some from Patterson, Sussman, others)13Conclusions - Fallacies• Average disk seek is for a seek of 1/3 of the li dcylinders– just a rule of thumb for seeking from one random location to another random location on a different cylinder, assuming a large number of cylinderslarge number of cylinders– problems with that rule are that» seek time is not