OutlineTopic: Paging to Remote MemoryMotivation: How to exploit remote memory?Slide 4Global Memory Service (GMS)What does it take to page out to remote memory (and back in)?Global Memory System (GMS) in a NutshellGMS Page Fetch (getpage)ScenariosGlobal ReplacementGMS Putpage/MovepageTopic: Energy Efficiency of MemoryMotivation: What can the OS do about the energy consumption of memory?Energy Efficiency MetricsPhysics BasicsTerminology and SymbolsRelationshipsPower Management (RAMBUS memory)Slide 23Slide 24RDRAM as a Memory HierarchyTwo Dimensions to Control EnergyDual-state (Static) HW Power State PoliciesQuad-state (Dynamic) HW PoliciesSki Rental AnalogyPage Allocation PolicesDual-state + Random Allocation*Quad-state HW + Sequential Allocation*The Design SpaceSpin-down Disk ModelSlide 35Reducing Energy ConsumptionPower SpecsSlide 38Spin-Down PoliciesOutline•Objective for Lecture–Finish up memory issues:•Remote memory in the role of backing store•Power-aware memory (Duke research)Topic: Paging to Remote MemoryMotivation: How to exploit remote memory?P$Memory“I/O bottleneck”VM and file cachingRemoteMemoryLow latency networks make retrieving data across the network faster than local diskDistributed Shared Memory (DSM)Motivation: How to exploit remote memory?P$Memory“I/O bottleneck”VM and file cachingRemoteMemoryLow latency networks make retrieving data across the network faster than local diskGlobal Memory Service (GMS)–GMS kernel module for Digital Unix and FreeBSD –remote paging, cooperative caching“Idle” workstations or memory serversM P M P M P M PM PCompute nodesSwitched LANM PM PPageout/pageinPageout/pagein-peer-to-peer styleWhat does it take to page out to remote memory (and back in)?•How to uniquely name and identify pages/blocks?•128-bit Unique Identifiers (UIDs) Global Page Numbers (GPN).•How to maintain consistent information about the location of pages in the network?•Each node keeps a Page Frame Directory of pages cached locally.•A distributed Global Cache Directory identifies the caching site(s) for each page or block.•Hash on page ID’s to get GCD site. •How to keep a consistent view of the membership in the cluster and the status of participating nodes?•How to globally coordinate memory usage for the page cache?•“Think globally, act locally.” – choice of replacement algorithmGlobal Memory System (GMS) in a Nutshell•Basic operations: putpage, getpage, movepage, discardpage.•Directory site is self for private pages, hashed for sharable.•Caching site selected locally based on global information.getpageDirectory Site (GCD)M PCaching Site (PFD)M PGMS ClientM PDirectory Site (GCD)M PCaching Site (PFD)M PGMS ClientM PputpagepagerequestupdatepagerequestGCD: Global Cache Directory, PFD: Page Frame Directory, P: Processor, M: MemoryGMS Page Fetch (getpage) directory sitecaching sitegetpage forward(delegated RPC)getpagerequestgetpagereplyDirectory site delegates requestto the correct caching site.Page fetches are requested with a getpageRPC call to page directory site.Getpage can result from ademand fault or a prefetchpolicy decision (asynchronous).Caching site sends pagedirectly to requesting node.ScenariosNode ANode BCase 1: Page in another node’s Global memory.Case 2: case 1, but B has no Global pages.Case 3: Page is on disk -- going active . . .Case 4: Shared page in local memory of node A.In each case Node B faultson some Virtual Addr.Global -- LocalFaulted pageLocal LRUGlobal LRU Or copy OrGlobal Replacement•Fantasy: on a page fetch, choose the globally “coldest” page in the cluster to replace (i.e., global LRU).–may demote a local page to global•Reality: approximate global LRU by making replacement choices across epochs of (at most) M evictions in T seconds.–trade off accuracy for performance–nodes send summary of page values to initiator at start of epoch–initiator determines which nodes will yield the epoch’s evictions–favor local pages over global pages–victims are chosen probabilisticallyGMS Putpage/Movepagecaching site #1 caching site #2directory sitemovepageupdateupdate putpageDirectory (GCD) site is selectedby a hash on the global pagenumber; every node is GCD sitefor its own private pages.Caching site is selected locallybased on global informationdistributed each epoch.Page/block evictions trigger an asynchonous putpagerequest from the client; the page is sent as apayload piggybacked on the message.Topic: Energy Efficiency of MemoryMotivation: What can the OS do about the energy consumption of memory?MemoryOtherMemoryOtherLaptop Power Budget9 Watt ProcessorHandheld Power Budget1 Watt ProcessorEnergy Efficiency Metrics•Power consumption in watts (mW).•Battery lifetime in hours (seconds, months).•Energy consumption in Joules (mJ).•Energy  Delay•Watt per megabytePhysics Basics•Voltage is amount of energy transferred from a power source (e.g., battery) by the charge flowing through the circuit.•Power is the rate of energy transfer•Current is the rate of flow of charge in circuitTerminology and SymbolsConcept Symbol Unit Abbrev.Current I amperes ACharge Q coulombs CVoltage V volts VEnergy E joules JPower P watts WResistance R ohms RelationshipsEnergy (Joules) = Power (watts) * Time (sec)E = P * tPower (watts) = Voltage (volts) * Current (amps) P = V * ICurrent (amps) = Voltage (volts) / Resistance (ohms)I = V / RPower Management (RAMBUS memory)activePowereddownstandbynapRead or write300mW180 mW30mW3mW2 chips+60ns+6ns+6sv.a.  p.a mappingcould exploit page coloring ideas60nsPower Management (RAMBUS memory)activePowereddownstandbynapRead or write300mW180 mW30mW3mW2 chips+60ns+6ns+6sv.a.  p.a mappingcould exploit page coloring ideas60nsPower Management (RAMBUS memory)activePowereddownstandbynapRead or write300mW180 mW30mW3mW2 chips+60ns+6ns+6sv.a.  p.a mappingcould exploit page coloring ideas60nsRDRAM as a Memory Hierarchy•Each chip can be independently put into appropriate power mode•Number of chips at each “level” of the hierarchy can vary dynamically.ActiveNapPolicy choices–initial page placement in an “appropriate” chip–dynamic movement of page from one chip to another–transitioning of power state of chip containing pageActiveCPU/$OSPage MappingAllocationChip 1Chip 2Chip nPower DownStandbyActivectrl ctrl ctrlHardware controlSoftware controlTwo