1Performance of the IBM General Parallel File SystemTerry Jones, Alice Koniges, R. Kim YatesLawrence Livermore LaboratoryPresented by Michael BlackCMSC714, Fall 2005Purpose of paperIBM has developed the GPFS.How well does it really perform?• Bandwidth?• Scalability?• Problems to be solved in future development?2Presentation Outline• What is the General Parallel File System?• How does it work?• How well does it work?What is the GPFS?Parallel file system:• hundreds of computers (nodes)– massive scalability!• nodes can:– run applications– manage RAID disks• each node has access to all files on all disks3Features of GPFS• Single files can span several disks on several nodes– All processes can write output to same file– No need for hundreds of output files• Processes can write to different parts of same file at same time– High bandwidth for concurrent access• File access uses standard Unix POSIX callsComparable File Systems• Intel PFS:– Nonstandard interface– Poor performance on concurrent access to same file•SGI XFS:– Standard interface, good performance, butonly works on shared memory architectures4Platform for GPFSIBM’s RS/6000:• scales to thousands of processors• autonomous nodes run Unix kernel• proprietary interconnect - “The Switch”– provides 83 MB/s one-way– allows all nodes to talk in pairs simultaneously– uniform access timeHow is GPFS Implemented?• Set of services located on some nodes– services provided by mmfsd - a GPFS daemon• mmfsd provides:– file system access (for mounting GPFS)– metanode service (file permissions & attributes)– stripe group manager (info on the disks)– token manager server (synchronizes file access)– configuration manager (ensures last 2 services are working)5Layers on each node:• Application or Disk• mmfsd daemon– allows node to mount file system– performs reads/writes on that node’s disks• Virtual Shared Disk– handles reads/writes to remote disks– contains pagepool - ~50 MB disk cache• allows “write-behind” - application need not wait for actual write•IP layerConsistency• Maintained by token manager server• mmfsd determines if node has write access to file– if not, must acquire token– gets list of who has tokens from token manager– talks to node with token, tries to acquire it• tokens guard bytes in files, not whole files– allows consistent concurrent access to different parts of same file6Steps in remote writeWrite steps continued• Application calls mmfsd with pointer to buffer• mmfsd acquires write token• mmfsd checks metanode to see where disk is• mmfsd copies data to pagepool (application can now continue)• VSD copies from pagepool to IP, breaks into packets• Data copied through Switch to VSD receive buffer• VSD server reassembles data to buddy buffer• VSD releases receive buffer, writes to disk device driver• VSD releases buddy buffer, sends ack to client• Client releases pagepool7Architecture issues with GPFS v1.2• clients can send faster than server can save to disk– exponential backoff used to slow down client• data copied twice in client– memory -> pagepool– pagepool -> IP bufferAnalyzing performance:How much bandwidth is needed?• Rule of thumb:– At peak, 1 byte every 500 flops– --> 7.3 GB/s on RS/6000• Rule of thumb:– Store half of memory every hour on average– Should take 5 minutes ideally– --> 4.4 GB/s on RS/6000• Also must take varying I/O access patterns and reliability into account8Experimental objectivesLooking at multiple tasks writing to 1 file:• How does throughput vary based on:– # clients– amount of data being transferred• How does GPFS scale?• How do I/O access characteristics affect performance?– large sequential writes to same file– small interleaved writes to same fileMethodology•Benchmark: ileave_or_random– written in C, uses MPI• Measuring throughput:– time for all processes to write fixed amount of data to single file• Effect of I/O access patterns:– benchmarks are adjusted for highly sequential (“segmented”) or highly interleaved (“strided”)9Test: Vary client:server ratioOptimal client:server ratio is 4:1why?• when too low, server starves• when too high, server buffers overflowTest: Vary transfer block size• block size makes no difference• # clients makes no difference up to 25610Test: Round-robin file access• GPFS performs poorly with strided access if block size < 256kB• Reflects token management overheadTest: Scalability with constant client:server ratio• Linear up to 58 servers11Conclusions• Good throughput as long as client:server ratio is less than 6– could be increased if data flow is improved• Programs must use segmented access patterns– suggested improvement: allow token management to be turned off (user must manage
View Full Document
Unlocking...