Colyseus: A Distributed Architecture for Online Multiplayer GamesAshwin BharambeCarnegie Mellon [email protected] PangCarnegie Mellon [email protected] SeshanCarnegie Mellon [email protected] paper presents the design, implementation, andevaluation of Colyseus, a distributed architecture forinteractive multiplayer games. Colyseus takes advan-tage of a game’s tolerance for weakly consistent stateand predictable workload to meet the tight latency con-straints of game-play and maintain scalable communi-cation costs. In addition, it provides a rich distributedquery interface and effective pre-fetching subsystem tohelp locate and replicate objects before they are accessedat a node. We have implemented Colyseus and mod-ified Quake II, a popular first person shooter game, touse it. Our measurements of Quake II and our ownColyseus-based game with hundreds of players showsthat Colyseus effectively distributes game traffic acrossthe participating nodes, allowing Colyseus to supportlow-latency game-play for an order of magnitude moreplayers than existing single server designs, with similarper-node bandwidth costs.1 IntroductionNetworked games are rapidly evolving from small 4-8person, one-time play games to large-scale games in-volving thousands of participants and persistent gameworlds. Almost all networked games, however, arecentralized — players send control messages to a cen-tral server and the server sends relevant state updatesto all active players. This approach suffers from thewell known robustness and scalability problems of sin-gle server designs. For example, high update ratesprevent even well provisioned servers from supportingmore than several tens of players in first person shooter(FPS) games. Further, client-server game designs oftenforce players to rely on infrastructure provided by thegame manufacturers. These infrastructures are some-times not well provisioned nor long-lived; thus, theyeither provide poor performance or prevent users fromplaying their game long after their purchase.A distributed design can potentially address the aboveshortcomings. However, architecting a distributed ap-plication is difficult due to the challenges of partition-ing the application’s state (e.g., game objects) and ex-ecution (e.g., the logic to simulate player and game AIactions) among the participating nodes. Distributing anetworked game is even more difficult due to the per-formance demands of real-time game-play. In addition,since the game-play of an individual player translates toupdates to the shared state of the game application, thereis much more write traffic and write-sharing than mostdistributed applications.Fortunately, we can take advantage of two fundamen-tal properties of games to address these challenges. First,games tolerate weak consistency in the application state.For example, current client-server implementations min-imize interactive response time by presenting a weaklyconsistent view of the game world to players. Second,game-play is usually governed by a strict set of rulesthat make the reads and writes to the shared state highlypredictable. For example, most reads and writes by aplayer occur upon objects that are physically close tothat player in the game world. The challenge, then, isto arrive at a scalable and efficient state and logic par-titioning that enables reasonably consistent, low-latencygame-play. This paper presents the design, implemen-tation, and evaluation of Colyseus, a novel distributedarchitecture for interactive multiplayer games designedto achieve the above goals.In Colyseus, any node may create read-only repli-cas of any game object. However, objects in Colyseusfollow a single-copy consistency model — i.e., all up-dates to an object are serialized through exactly one pri-mary copy in the system. This approach mirrors theconsistency model of existing client-server architectureson a per object basis. Although replicas are only keptweakly consistent with the primary copy, they enablethe low-latency read access needed to keep game execu-tion timely. The challenge is for each node to determinethe set of replicas it needs in advance of executing anygame logic. Colyseus provides a rich query interfaceover the system-wide collection of objects to identifyand fetch required objects. We have implemented thisquery interface on both a randomized distributed hash ta-ble (DHT) [28] and a dynamically load balanced, range-based DHT [3]. However, lookups in DHTs can be tooslow for finding required replicas in games. To hide thislookup latency, Colyseus uses locality and predictabil-ity in data access patterns to speculatively pre-fetch ob-jects. This mechanism is only used to discover relevantobjects; updates are propagated from primary copies toNSDI ’06: 3rd Symposium on Networked Systems Design & ImplementationUSENIX Association155replicas directly. We show that the combination of allthese techniques is critical to enabling interactive game-play.Colyseus enables games to efficiently use widely dis-tributed servers to support a large community of users.We have modified Quake II [22], a popular server-based First Person Shooter (FPS) game, to run on ourimplementation of Colyseus, and have also used mea-surements of Quake III [23] game-play to develop ourown Colyseus-based game with players that mimic theQuake III workload. These concrete case studies illus-trate the practicality of using our architecture to dis-tribute existing game implementations. Our measure-ments on an Emulab testbed with hundreds of playersshow that Colyseus is effective at distributing game traf-fic and workload across the participating nodes, whileproviding servers and players with low-latency and con-sistent views of the game world. In the following sec-tions, we provide background about general game designas well as the design and evaluation of Colyseus.2 BackgroundIn this section, we survey the requirements of onlinemultiplayer games and demonstrate the fundamentallimitations of existing client-server implementations. Inaddition, we provide evidence that resources exist fordistributed deployments of multiplayer games.2.1 Contemporary Game DesignTo determine the requirements of multiplayer games, westudied the source code of several popular and publiclyreleased engines for virtual reality games, includingQuake II [22], Quake III [23 ], and the Torque Network-ing Library [30]. In these games, each player (gameclient)