An FPGA-based Simulator for Datacenter NetworksZhangxi TanComputer Science DivisionUC Berkeley, [email protected] Asanovi´cComputer Science DivisionUC Berkeley, [email protected] PattersonComputer Science DivisionUC Berkeley, [email protected] describe an FPGA-based datacenter network simulatorfor researchers to rapidly experiment with O(10,000) nodedatacenter network architectures. Our simulation approachconfigures the FPGA hardware to implement abstract mod-els of key datacenter building blocks, including all levels ofswitches and servers. We model servers using a completeSPARC v8 ISA implementation, enabling each node to runreal node software, such as LAMP and Hadoop. Our ini-tial implementation simulates a 64-server system and hassuccessfully reproduced the TCP incast throughput collapseproblem. When running a modern parallel benchmark, sim-ulation performance is two-orders of magnitude faster thana popular full-system software simulator. We plan to scaleup our testbed to run on multiple BEE3 FPGA boards,where each board is capable of simulating 1500 servers withswitches.1. INTRODUCTIONIn recent years, datacenters have been growing rapidly toscales of 10,000 to 100,000 servers [18]. Many key technolo-gies make such incredible scaling possible, including modu-larized container-based datacenter construction and servervirtualization. Traditionally, datacenter networks employa fat-tree-like three-tier hierarchy containing thousands ofswitches at all levels: rack level, aggregate level, and corelevel [1].As observed in [13], the network infrastructure is one of themost vital optimizations in a datacenter. First, network-ing infrastructure has a significant impact on server utiliza-tion, which is an important factor in datacenter power con-sumption. Second, network infrastructure is crucial for sup-porting data intensive Map-Reduce jobs. Finally, networkinfrastructure accounts for 18% of the monthly datacentercosts, which is the third largest contributing factor. In ad-dition, existing large commercial switches and routers com-mand very healthy margins, despite being relatively unreli-able [26]. Sometimes, correlated failures are found in repli-cated million-dollar units [26]. Therefore, many researchershave proposed novel datacenter network architectures [14,15, 17, 22, 25, 26] with most of them focusing on new switchdesigns. There are also several new network products em-phasizing low latency and simple switch designs [3, 4].When comparing these new network architectures, we founda wide variety of design choices in almost every aspect of thedesign space, such as switch designs, network topology, pro-tocols, and applications. For example, there is an ongoingdebate between low-radix and high-radix switch design. Webelieve these basic disagreements about fundamental designdecisions are due to the different observations and assump-tions taken from various existing datacenter infrastructuresand applications, and the lack of a sound methodology toevaluate new options. Most proposed designs have onlybeen tested with a very small testbed running unrealistic mi-crobenchmarks, as it is very difficult to evaluate network ar-chitecture innovations at scale without first building a largedatacenter.To address the above issue, we propose using Field-Program-mable Gate Arrays (FPGAs) to build a reconfigurable sim-ulation testbed at the scale of O(10,000) nodes. Each nodein the testbed is capable of running real datacenter applica-tions. Furthermore, network elements in our testbed pro-vide detailed visibility so that we can examine the com-plex network behavior that administrators see when deploy-ing equivalently scaled datacenter software. We built thetestbed on top of a cost-efficient FPGA-based full-systemmanycore simulator, RAMP Gold [24]. Instead of mappingthe real target hardware directly, we build several abstractedmodels of key datacenter components and compose themtogether in FPGAs. We can then construct a 10,000-nodesystem from a rack of multi-FPGA boards, e.g., the BEE3[10] system. To the best of our knowledge, our approachwill probably be the first to simulate datacenter hardwarealong with real software at such a scale. The testbed alsoprovides an excellent environment to quantitatively analyzeand compare existing network architecture proposals.We show that although the simulation performance is slowerthan prototyping a datacenter using real hardware, abstractFPGA models allow flexible parameterization and are stilltwo orders of magnitude faster than software simulators atthe equivalent level of detail. As a proof of concept, webuilt a prototype of our simulator in a single Xilinx Virtex 5LX110 FPGA simulating 64 servers connecting to a 64-portrack switch. Employing this testbed, we have successfullyreproduced the TCP Incast throughput collapse effect [27],which occurs in real datacenters. We also show the impor-tance of simulating real node software when studying theTCP Incast problem.Network Architecture Testbed Scale WorkloadPolicy away switching layer [17] Click software router Single switch MicrobenchmarkDCell [16] Commercial hardware ∼20 nodes Synthetic workloadPortland (v1) [6] Virtual machine+commercial switch 20 switches+16 servers MicrobenchmarkPortland (v2) [22] Virtual machine+NetFPGA 20 switches+16 servers Synthetic workloadBCube [15] Commercial hardware+NetFPGA 8 switches+16 servers MicrobenchmarkVL2 [14] Commercial hardware 10 servers+10 switches MicrobenchmarkThacker’s container network [26] Prototyping with FPGA boards - -Table 1: Datacenter network architecture proposals and their evaluations2. EVALUATING DATACENTER NETWORKSWe begin by identifying the key issues in evaluating data-center networks. Several recent novel network architecturesemploy a simple, low-latency, supercomputer-like intercon-nect. For example, the Sun Infiniband datacenter switch [3]has a 300 ns port-port latency as opposed to the 7–8 µs ofcommon Gigabit Ethernet switches. As a result, evaluatingdatacenter network architectures really requires simulatinga computer system with the following three features.1. Scale: Datacenters contains O(10,000) servers or more.2. Performance: Large datacenter switches have 48/96ports, and are massively parallel. Each port has 1–4 Kflow tables and several input/output packet buffers. Inthe worst case, there are ∼200 concurrent events everyclock cycle.3. Accuracy: A datacenter network operates at