Distributed File Systems Andy Wang COP 5611 Advanced Operating Systems Outline Basic concepts NFS Andrew File System Replicated file systems Ficus Coda Serverless file systems Basic Distributed FS Concepts You are here the file s there what do you do about it Important questions What files can I access How do I name them How do I get the data How do I synchronize with others What files can be accessed Several possible choices Every file in the world Every file stored in this kind of system Every file in my local installation Selected volumes Selected individual files What dictates the proper choice Why not make every file available Naming issues Scaling issues Local autonomy Security Network traffic Naming Files in a Distributed System How much transparency Does every user machine sub network need its own namespace How do I find a site that stores the file that I name Is it implicit in the name Can my naming scheme scale Must everyone agree on my scheme How do I get data for non local files Fetch it over the network How much caching Replication What security is required for data transport Synchronization and Consistency Will there be trouble if multiple sites want to update a file Can I get any guarantee that I always see consistent versions of data i e will I ever see old data after new How soon do I see new data NFS Networked file system Provide distributed filing by remote access With a high degree of transparency Method of providing highly transparent access to remote files Developed by Sun NFS Characteristics Volume level access RPC based Stateless remote file access Uses XDR Location not name transparent Implementation for many systems All interoperate even non Unix ones Currently based on VFS VFS Vnode Review VFS Virtual File System Common interface allowing multiple file system implementations on one system Plugged in below user level Files represented by vnodes NFS Diagram NFS Client NFS Server tmp x mnt y home foo bin bar File Handles On the client site files are represented by vnodes The client NFS implementation internally represents remote files as handles Opaque to client But meaningful to server To name remote file provide handle to server NFS Handle Diagram Client side Server side User process file descriptor handle NFS server VFS level vnode vnode VFS level NFS level handle inode UFS How to make this work Could integrate it into the kernel Non portable non distributable Instead use existing features to do the work VFS for common interface RPC for data transport Using RPC for NFS Must have some process at server that answers the RPC requests Continuously running daemon process Somehow must perform mounts over machine boundaries A second daemon process for this NFS Processes nfsd daemons server daemons that accept RPC calls for NFS rpc mountd daemons server daemons that handle mount requests biod daemons optional client daemons that can improve performance NFS from the Client s Side User issues a normal file operation Like read Passes through vnode interface to clientside NFS implementation Client side NFS implementation formats and sends an RPC packet to perform operation Single client blocks until NFS RPC returns NFS RPC Procedures 16 RPC procedures to implement NFS Lookup is the key operation Some for files some for file systems Including directory ops link ops read write etc Because it fetches handles Other NFS file operations use the handle Mount Operations Must mount an NFS file system on the client before you can use it Requires local and remote operations Local operations indicate mount point has an NFS type VFS at that point in hierarchy Remote operations go to remote rpc mountd Mount provides primal file handle NFS on the Server Side The server side is represented by the local VFS actually storing the data Plus rpc mountd and nfsd daemons NFS is stateless servers do not keep track of clients Each NFS operation must be selfcontained From server s point of view Implications of Statelessness NFS RPC requests must completely describe operations NFS requests should be idempotent NFS should use a stateless transport protocol e g UDP Servers don t worry about client crashes Server crashes won t leave junk lying around An Important Implication of Statelessness Servers don t know what files clients think are open Unlike in UFS LFS most local VFS file systems Makes it much harder to provide certain semantics Also scales nicely though Preserving UNIX File Operation Semantics NFS works hard to provide identical semantics to local UFS operations Some of this is tricky Especially given statelessness of server E g how do you avoid discarding pages of unlinked file a client has open Sleazy NFS Tricks Used to provide desired semantics despite statelessness of the server E g if client unlinks open file send rename to server rather than remove Perform actual remove when file is closed Won t work if file removed on server Won t work with cooperating clients File Handles Method clients use to identify files Created by the server on the file lookup Must be unique mappings of server file identifier to universal identifier File handles become invalid when server frees or reuses inode Inode generation number in handle shows when stale NFS Daemon Processes nfsd daemon biod daemon rpc mount daemon rpc lockd daemon rpc statd daemon nfsd Daemon Server daemon to handle incoming RPC requests Often multiple nfsd daemons per site Incoming NFS RPC requests go to one nfsd daemon Which makes a kernel call to do the real work Using daemons allows multiple threads biod Daemon Most client NFS operations go from VFS NFS implementation to the server biod daemon does readahead for clients To make use of kernel file buffer cache Only improves performance NFS works correctly without biod daemon Also flushes buffered writes for clients rpc mount Daemon Runs on server to handle VFS level operations for NFS Particularly remote mount requests Provides initial file handle for a remote volume Also checks that incoming requests are from privileged ports in UDP IP packet source address rpc lockd Daemon NFS server is stateless so it does not handle file locking rpc lockd provides locking Runs on both client and server Client side catches request forwards to sever daemon rpc lockd handles lock recovery when server crashes rpc statd Daemon Also runs on both client and server Used to check status of a machine Server s rpc lockd asks rpc statd to store permanent lock information in file system And to monitor
View Full Document
Unlocking...