1An Evaluation of the DEC Memory ChannelCase Studies in Reflective Memory and Cooperative SchedulingAndrew Geweke and Frederick WongUniversity of California, Berkeley{geweke,fredwong}@cs.berkeley.edu21 May 1999AbstractWith the rise of clusters as a vehicle for very-high-performance computing, an increasing emphasis isbeing placed upon the communication interfacebetween each processor and the underlyingnetwork. And while many studies have exploredthe design of both traditional send/receivenetwork interfaces and shared-memory designs, athird alternative — reflective memory — offers asignificantly different design space and provides itsown, separate benefits and challenges.In this paper, we investigate Memory Channel, ahigh-speed reflective-memory network, byimplementing MPI above Memory Channel. Ourresults show that Memory Channel is a powerfulsubstrate for message-passing programs, but thatsignificant obstacles remain to building truly large-scale message-passing software above this network.Finally, we investigate time-sharing of a MemoryChannel-based cluster and demonstrate the needfor event support at the hardware layer.1 IntroductionWith the advent of fast communication interfacesfor commodity workstations [1], powerfulmassively parallel processors (MPPs) can bequickly and cheaply constructed from commodityhardware. Such networks as Myrinet [4], GigabitEthernet, and others adapt easily to the standardPCI peripheral bus and provide low-latency, very-high-bandwidth access to the entire cluster from asingle processor, creating a system-area network(SAN). This combination has proved to beextremely powerful, allowing clusters to supplanttraditional supercomputers in many areas.The semantics presented by these networkinterfaces vary significantly. Most popularcommercial network interfaces, to date, provideone of two models to the processor: shared-memoryor send-receive. By imitating a shared-memorymultiprocessor (SMP), shared-memory networkinterfaces provide a well-known programmingmodel with implicit communication to software.Send-receive network interfaces, on the otherhand, provide a well-known programming modelwith explicit communication: in many ways, theyimitate much-higher-performance versions oftypical Ethernet and/or Token Ring LANs.A third sort of communication interface, however,provides a separate but equally interesting designpoint. Reflective-memory[16,17] interfaces are a sortof hybrid between explicit send-receive interfacesand implicit shared-memory models. Like ashared-memory interface, communication isperformed using the processor’s native load andstore instructions; like a send-receive interface,however, there is no single global view of “onelarge memory”.We have constructed a traditional message-passinglibrary, MPI[13], on top of Compaq’s MemoryChannnel[15] reflective-memory network interface.Doing so provides a wealth of information aboutthe usability and performance of reflective-memory interfaces under message-passingworkloads.2This paper is organized as follows. Section 2describes the underlying Memory Channelinterface and its characteristics. Section 3 coversour multi-protocol implementation of MPI overshared memory and Memory Channel, and themicro-benchmarks performance. We then comparethis data to the performance of MPI over trueshared memory and the raw Memory Channelinterface. Section 4 explores the implementationissues under a time-sharing workload. Section 5then considers some of the factors affectingperformance under shared and non-sharedworkloads, and some of the challenges poised byconstructing a message-passing interface over areflective-memory substrate. Finally, section 6draws some conclusions and suggests possibleimprovements to hardware and software designsfor large-scale reflective-memory designs.2 BackgroundThe first half of this section describes reflectivememory in general and the Memory Channelhardware and software used; the second half thencontinues on to describe the MPICH softwareused to implement MPI.2.1 Reflective Memory and MemoryChannelReflective memory is a powerful communicationabstraction, but one that also differs significantlyfrom most currently widely-used models.Reflective memory provides a process a write-onlymemory “window” onto another process’s addressspace. Once the “window” has been set up, allprocessor writes into that “window” go not tomemory, but instead directly into the address spaceof the destination process. This propagation ofwrites into another process is the basis ofreflective-memory communication.Reflective memory differs from shared memory intwo important ways: first, the “window” is write-only, and attempts to read from it produceunpredictable results or a processor trap; second,the “window” is one-way: the second process mustset up an entirely separate “window” of its ownback into the first process if it wishes to returncommunication.These distinctions are quite fundamental toreflective memory and are what makes the designspace of reflective-memory communicationinteresting.Memory Channel is an implementation ofreflective memory; it consists of a switchedsystem-area network, composed of a PCI network-interface adapter card, 32-bit-wide parallel coppercables, and (optionally) switches.The raw hardware has an end-to-end latencyclaimed to be under five microseconds; indeed, wemeasured an end-to-end latency of approximately2.7 microseconds for a four-byte message (thesmallest possible). Bandwidth is claimed to bebetween 35 MByte/s and 70 MByte/s; ourexperiments produced a maximum bandwidth ofapproximately 62.5 MByte/s. The hardware hasbuilt-in error detection (though not correction);however, as the error rate is extremely low, wecould not test the claimed rate of approximatelythree errors per year of continuous operation.Our testbed consisted of a pair of CompaqAlphaServer 2100s, each of which contains two533MHz DECchip Alpha 21164s, 256 MBytes ofRAM, and a single Memory Channel PCI interfacecard. We connected the two machines directly(back-to-back) with a single Memory Channelcable.The Memory Channel software introduces theconcept of “Memory Channel address space”. Thehardware does not maintain any addressable stateitself; instead, this “address space” is anabstraction (in reality simply a layer of indirection)provided to simplify Memory Channelprogramming.“Address space” is allocated and named using theimc_asalloc() function; this function