Supporting Systolic and Memory Communication in iWarp Shekhar Borkar, Robert Cohn, George Cox, Thomas Gross, H. T. Kung. Monica Lam, Margie Levine, Brian Moore. Wire Moore, Craig Peterson, Jim Susman, Jii Sutton, John Urbanski, and Jon Webb School of Computer Science Intel Corporation, CO4-01 Carnegie Mellon University 5200 N.E. Elam Young Pkwy Pittsburgh, Pennsylvania 15213 Hillsboro. Oregon 97124 Abstract iWarp is a parallel architecture developed jointly by Car- negie Mellon University and Intel Corporation. The iWarp communication system supports two widely used interproces- sor communication styles: memory comnaunication and systolic communication. This paper describes the rationale, architecture, and implementation for the iWarp communica- tion system. The sending or receiving processor of a message can per- form either memory or systolic communication. In memory communication, the entire message is buffered in the local memory of the processor before it is transmitted or after it is received. Therefore communication begins or terminates at the local memory. For conventional message passing methods, both sending and receiving processors use memory communication. In systolic communication, individual data items are transferred as they are produced, or are used as they are received, by the program running at the processor. Memory communication is flexible and well suited for general computing; whereas systolic cortummication is ef- ficient and well suited for speed critical applications. A major achievement of the iWarp effort is the derivation of a common design to satisfy the requirements of both sys- tolic and memory uxnmunication styles. This is made pos- sible by two important innovations in communication: (1) program access to communication and (2) logical channels. The former allows programs to access data as they are trans- mitted and to redirect portions of messages to different des- tinations efficiently. The latter increases the connectivity between the processors and guarantees communication bandwidth for classes of messages. These innovations have provided a focus for the iWarp architecture. The result is a communication system that provides a total bandwidth of 320 MBytes/xc and that is integrated on a single VLSI com- ponent with a 20 MFLOPS plus 20 MIPS long instruction word computation engine. ‘llte research was supported in part by Defense Advanced Research Projects Agency (DOD) monitored by the Space and Naval Warfare Systems Command under Contrad NOOO39-87-C-0251. Authors’ affiliations: R. Cohn, T. Gross, H. T. Kung, and J. Webb are with Carnegie Mellon University; S. Borkar, G. Cox, M. Levine, B. Moore, W. Moore. C. Petersen, I. Susman, J. Sutton, and J. Urbanski are. with Intel; M. Lam, who was a Ph.D. student at Carnegie Mellon University, is now with Computer Systems Laboratory. Stanford University, Stanford, CA 94305 1. Introduction iWarp [5] is a distributed parallel computing system under joint development by Carnegie Mellon University and Intel Corporation since 1986. The architecture is derived from the original Warp architecture developed by Carnegie Mellon [2]. The building block of an iWarp system is the iwarp ceN, made out of a single-chip iWurp processor (or iWarp component) connected to a local mcxnory. Parallel systems of different scales and topologies can be built cost-effectively by simply Iinking together iWarp cells. Figure 1 illustrates one possible configuration. Parallel System iWarp Component Figure 1. iWarp cell: a building block for parallel systems The iWarp processor integrates both a high-speed computa- tion and communication capability in a single component. The processor is a powerful computation engine that employs instruction-level parallelism to allow simultaneous operation of multiple functional units. What makes iWarp unique, however, is its interprocessor communication capability. An iWarp processor can simultaneously communicate with a number of other iWarp processors at very high speeds. More importantly, the iWarp processor Ihas a highly flexible com- munication mechanism that can support different program- ming models, including the tightly coupled computing found in systolic arrays and the message passing style of computa- tion found in distributed memory machines. These com- munication capabilities allow the effective use of iWarp for a wide range of applications. CH2887-8/90/0000/0070$01.00O1990 IEEE 70The iWarp component consists of three autonomous subsys- tems, as depicted in Figure 1. The computation agent, which executes programs, can deliver 20 (or 10) MFLOPS for single (or double) precision calculations plus 20 MIPS for integer/logic operations. The corrmuuI ication agent, which implements the iwarp’s communication system, can sustain an aggregate intercell communication bandwidth of 320 MBytes/set by using four input and four outPut busses. The memory agent, which provides a high-bandwidth interface to the local memory, can transfer streams of data into or out of the communication agent at a rate of 160 MBytes/set. The first silicon of the iWarp component was fabricated in December 1989. It consists of approximately 650,OOCl transis- tors and measures about 1.4cm (551mil) on a side. Figure 2 shows a photo of the component, together with a floor plan that highlights the major units. The iWarp component operates at a frequency of 20 MHz, with the exception that the data is transferred between processors at twice that frequency (40 MHz). Three iWarp demonstration systems will be delivered to Carnegie Mellon by the Fall of 1990. Each of these systems consists of an 8x8 torus of iWarp cells, deliver- ing more than 1.2 GFLOPS. The system can be readily expanded to include up to 1,024 cells for an aggregate com- puting power of over 20 GFLOPS and communication bandwidth of 160 GBytes/sec. The software for