Distributed Storage and ConsistencyDistributed Storage and ConsistencyStorage moves into the netStorage moves into the netStorage capacity/volumeStorage capacity/volumeAdministrative costAdministrative costNetwork bandwidthNetwork bandwidthNetwork delaysNetwork delaysNetwork costNetwork costShared storage with scalable bandwidth and capacity.Shared storage with scalable bandwidth and capacity.Consolidate Consolidate ——multiplex multiplex ——decentralize decentralize ——replicate.replicate.Reconfigure to mixReconfigure to mix--andand--match loads and resources.match loads and resources.Storage as a serviceStorage as a serviceSSP SSP ASPASPStorage Service ProviderStorage Service ProviderApplication Service ProviderApplication Service ProviderOutsourcing: storage and/or applications as a Outsourcing: storage and/or applications as a serviceservice..For ASPs (e.g., Web services), storage is just a component.For ASPs (e.g., Web services), storage is just a component.Storage AbstractionsStorage Abstractions• relational database (IBM and Oracle)tables, transactions, query language• file systemhierarchical name space of files with ACLsEach file is a linear space of fixed-size blocks.• block storageSAN, Petal, RAID-in-a-box (e.g., EMC)Each logical unit (LU) or volume is a linear space of fixed-size blocks.• object storageobject == file, with a flat name space: NASD, DDS, PorcupineVarying views of the object size: NASD/OSD/Slice objects may act as large-ish “buckets” that aggregate file system state.• persistent objectspointer structures, requires transactions: OODB, ObjectStoreNetwork Block StorageNetwork Block StorageOne approach to scalable storage is to attach raw block storage to a network.• Abstraction: OS addresses storage by <volume, sector>.iSCSI, Petal, FibreChannel: access through special device driver• Dedicated Storage Area Network or general-purpose network.FibreChannel (FC) vs. Ethernet• Volume-based administrative toolsbackup, volume replication, remote sharing• Called “raw” or “block”, “storage volumes” or just “SAN”.• Least common denominator for any file system or database.“NAS vs. SAN”“NAS vs. SAN”In the commercial sector there is a raging debate today about “NAS vs. SAN”.• Network-Attached Storage has been the dominant approach to shared storage since NFS.NAS == NFS or CIFS: named files over Ethernet/Internet.E.g., Network Appliance “filers”• Proponents of FibreChannel SANs market them as a fundamentally faster way to access shared storage.no “indirection through a file server” (“SAD”)lower overhead on clientsnetwork is better/faster (if not cheaper) and dedicated/trustedBrocade, HP, Emulex are some big players.NAS vs. SAN: Cutting through the BSNAS vs. SAN: Cutting through the BS• FibreChannel a high-end technology incorporating NIC enhancements to reduce host overhead.......but bogged down in interoperability problems.• Ethernet is getting faster faster than FibreChannel.gigabit, 10-gigabit, + smarter NICs, + smarter/faster switches• Future battleground is Ethernet vs. Infiniband.• The choice of network is fundamentally orthogonal to storage service design.Well, almost: flow control, RDMA, user-level access (DAFS/VI)• The fundamental questions are really about abstractions.shared raw volume vs. shared file volume vs. private disksStorage ArchitectureStorage ArchitectureAny of these abstractions can be built using any, some, or all of the others.Use the “right” abstraction for your application.Basic operations: create/remove, open/close, read/write.The fundamental questions are:• What is the best way to build the abstraction you want?division of function between device, network, server, and client• What level of the system should implement the features and properties you want?IP LANIP LANMed Med CtrCtrDuke Mass Storage Duke Mass Storage TestbedTestbedCampus FC netCampus FC netIBM Shark/HSMIBM Shark/HSMIP LANIP LANBrain LabBrain LabGoalGoal: managed storage on : managed storage on demand for crossdemand for cross--disciplinary disciplinary research.research.Direct SAN access for “power Direct SAN access for “power clients” and NAS clients” and NAS PoPsPoPs; other ; other clients access through NAS.clients access through NAS.ProblemsProblemspoor interoperability• Must have a common volume layout across heterogeneous SAN clients. poor sharing control• The granularity of access control is an entire volume.• SAN clients must be trusted.• SAN clients must coordinate their access.$$$Med Med CtrCtrDuke Storage Duke Storage TestbedTestbed, v2.0, v2.0Campus FC netCampus FC netIBM Shark/HSMIBM Shark/HSMCampus IP netCampus IP netBrain LabBrain LabEach SAN volume is managed Each SAN volume is managed by a single NAS by a single NAS PoPPoP..All access to each volume is All access to each volume is mediated by its NAS mediated by its NAS PoPPoP..Testbed Testbed v2.0: pro and conv2.0: pro and conSupports resource sharing and data sharing.Does not leverage Fibre Channel investment.Does not scale access to individual volumes.Prone to load imbalances.Data crosses campus IP network in the clear.Identities and authentication must be centrally administered.It’s only as good as the NAS clients, which tend to be fair at best.Sharing Network StorageSharing Network StorageHow can we control sharing to a space of files or blocks?• Access control etc.• Data model and storage abstraction• Caching• Optimistic replicationConsistency• One-copy consistency vs. weak consistency• Read-only (immutable) files?• Read-mostly files with weak consistency?• Write-anywhere files?File/Block Cache ConsistencyFile/Block Cache Consistency•Basic write-ownership protocol.Distributed shared memory (software DSM)• Timestamp validation (NFS).Timestamp each cache entry, and periodically query the server: “has this file changed since time t?”; invalidate cache if stale.• Callback invalidation (AFS, Sprite, Spritely NFS).Request notification (callback) from the server if the file changes; invalidate cache and/or disable caching on callback.• Leases (NQ-NFS, NFSv4, DAFS)[Gray&Cheriton89,Macklem93]Software DSM 101Software DSM 101Software-based distributed shared memory (DSM) provides an illusion of shared memory on a cluster.• remote-fork the same program on each node• data resides in common virtual address spacelibrary/kernel collude to make the shared VAS appear consistent•