In Proceedings of the ACM International Symposium on Computer Architecture, San Diego, CA, June 2007Power Provisioning for a Warehouse-sized ComputerXiaobo Fan Wolf-Dietrich Weber Luiz André BarrosoGoogle Inc.1600 Amphitheatre PkwyMountain View, CA 94043{xiaobo,wolf,luiz}@google.comABSTRACTLarge-scale Internet services require a computing infrastructure thatcan be appropriately described as a warehouse-sized computingsystem. The cost of building datacenter facilities capable of de-livering a given power capacity to such a computer can rival the re-curring energy consumption costs themselves. Therefore, there arestrong economic incentives to operate facilities as close as possibleto maximum capacity, so that the non-recurring facility costs can bebest amortized. That is difficult to achieve in practice because ofuncertainties in equipment power ratings and because power con-sumption tends to vary significantly with the actual computing ac-tivity. Effective power provisioning strategies are needed to deter-mine how much computing equipment can be safely and efficientlyhosted within a given power budget.In this paper we present the aggregate power usage character-istics of large collections of servers (up to 15 thousand) for dif-ferent classes of applications over a period of approximately sixmonths. Those observations allow us to evaluate opportunities formaximizing the use of the deployed power capacity of datacenters,and assess the risks of over-subscribing it. We find that even inwell-tuned applications there is a noticeable gap (7 - 16%) betweenachieved and theoretical aggregate peak power usage at the clusterlevel (thousands of servers). The gap grows to almost 40% in wholedatacenters. This headroom can be used to deploy additional com-pute equipment within the same power budget with minimal riskof exceeding it. We use our modeling framework to estimate thepotential of power management schemes to reduce peak power andenergy usage. We find that the opportunities for power and energysavings are significant, but greater at the cluster-level (thousands ofservers) than at the rack-level (tens). Finally we argue that systemsneed to be power efficient across the activity range, and not only atpeak performance levels.Categories and Subject Descriptors: C.0 [Computer Systems Or-ganization]: General - System architectures; C.4 [Computer Sys-tems Organization]: Performance of Systems - Design studies, Mea-surement techniques, Modeling techniques.General Terms: Measurement, Experimentation.Keywords: Power modeling, power provisioning, energy efficiency.Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.ISCA’07, June 9–13, 2007, San Diego, California, USA.Copyright 2007 ACM 978-1-59593-706-3/07/0006 ...$5.00.1. INTRODUCTIONWith the onset of large-scale Internet services and the massivelyparallel computing infrastructure that is required to support them,the job of a computer architect has expanded to include the designof warehouse-sized computing systems, made up of thousands ofcomputing nodes, their associated storage hierarchy and intercon-nection infrastructure [3]. Power and energy are first-order con-cerns in the design of these new computers, as the cost of poweringserver systems has been steadily rising with higher performing sys-tems, while the cost of hardware has remained relatively stable.Barroso [2] argued that if these trends were to continue the cost ofthe energy consumed by a server during its lifetime could surpassthe cost of the equipment itself. By comparison, another energy-related cost factor has yet to receive significant attention: the costof building a datacenter facility capable of providing power to agroup of servers.Typical datacenter building costs fall between $10 and $20 perdeployed Watt of peak critical power (power for computing equip-ment only, excluding cooling and other ancillary loads) [25], whileelectricity costs in the U.S. are approximately $0.80/Watt-year (lessthan that in areas where large datacenters tend to be deployed). Un-like energy costs that vary with actual usage, the cost of building adatacenter is fixed for a given peak power delivery capacity. Con-sequently, the more under-utilized a facility, the more expensiveit becomes as a fraction the total cost of ownership. For exam-ple, if a facility operates at 85% of its peak capacity on average,the cost of building the facility will still be higher than all elec-tricity expenses for ten years of operation1. Maximizing usage ofthe available power budget is also important for existing facilities,since it can allow the computing infrastructure to grow or to en-able upgrades without requiring the acquisition of new datacentercapacity, which can take years if it involves new construction.The incentive to fully utilize the power budget of a datacenteris offset by the business risk of exceeding its maximum capacity,which could result in outages or costly violations of service agree-ments.Determining the right deployment and power management strate-gies requires understanding the simultaneous power usage charac-teristics of groups of hundreds or thousands of machines, over time.This is complicated by three important factors: the rated maxi-mum power (or nameplate value) of computing equipment is usu-ally overly conservative and therefore of limited usefulness; actualconsumed power of servers varies significantly with the amount ofactivity, making it hard to predict; different applications exerciselarge-scale systems differently. Consequently only the monitoring1Assumes typical Tier-2 [25] datacenter costs of $11/Watt of crit-ical power and a 50% energy overhead for cooling and conversionlosses.of real large-scale workloads can yield insight into the aggregateload at the datacenter level.In this paper we present the power usage characteristics of threelarge-scale workloads as well as a workload mix from an actualdatacenter, each using up to several thousand servers, over a periodof about six months. We focus on critical power, and examine howpower usage varies over time and over different aggregation levels(from individual racks to an entire cluster).