COSC 6374 Parallel Computation Parallel I O III MPI I O Edgar Gabriel Spring 2009 Edgar Gabriel MPI I O abstractions MPI I O relies on two abstractions from MPI 1 2 Define groups of processes having access to a file communicators Define complex access patterns derived datatypes Analogy between message passing and file access Both move data from one address space to another COSC 6374 Parallel Computation Edgar Gabriel 1 MPI I O definitions File view portion of a file visible to a process Processes can share a common view Views can overlap or be disjoint Views can be changed during runtime A process can have multiple instances of a file open using different file views Process 0 Process 1 Process 2 Process 3 File COSC 6374 Parallel Computation Edgar Gabriel MPI I O definitions cont File type datatype used to construct the file view File types must not have overlapping regions if used in write operations Displacements must increase monotonically Elementary type etype basic unit of the data accessed by the program A file type consists logically of a series of etypes Default file view displacement 0 etype file type MPI Byte File type etype etype etype COSC 6374 Parallel Computation Edgar Gabriel 2 General file manipulation functions MPI File open MPI Comm comm char filename int amode MPI Info info MPI File fh Collective operation All processes have to provide the same amode comm must be an intra communicator Values for amode MPI MODE RDONLY MPI MODE WRONLY MPI MODE RDWR MPI MODE CREATE MPI MODE APPEND You can combine several amodes e g C MPI MODE CREATE MPI MODE WRONLY Fortran MPI MODE CREATE MPI MODE WRONLY COSC 6374 Parallel Computation Edgar Gabriel General file manipulation functions MPI File close MPI File fh MPI File delete char filename MPI Info info MPI File get size MPI File fh MPI Offset size MPI File get group MPI File fh MPI Group group MPI File get amode MPI File fh int amode MPI File preallocate MPI File fh int size Guessed what these functions are doing Close a file Delete a file Determine the size of a file multiple of etypes Determine the group of processes used to open the file Determine the amode used when opening the file For a new file try to pre allocate the following size COSC 6374 Parallel Computation Edgar Gabriel 3 Setting a file view MPI File set view MPI File fh MPI Offset disp MPI Datatype etype MPI Datatype filetype char datarep MPI Info info The argument list disp start of the file view etype and filetype as discussed previously datarep data representation used discussed later info hints to the MPI library discussed later Collective operation datarep and extent of etype have to be identical on all processes filetype disp and info might vary Resets file pointers to zero COSC 6374 Parallel Computation Edgar Gabriel Usage example int buf 2 MPI File fh MPI Datatype etype datatype ftype MPI File open MPI COMM WORLD myfile txt MPI MODE RDONLY fh etype datatype MPI INT MPI Type vector 2 1 4 etype ftype MPI File set view fh disp etype ftype native MPI INFO NULL MPI File read fh buf 2 datatype status disp and ftype define the data stream read from file datatype defines the data layout in memory and has to be a multiple of etype File File view disp Data layout in memory filetype etype datatype COSC 6374 Parallel Computation Edgar Gabriel 4 MPI I O non collective functions Positioning Synchronism Function Individual file pointers Blocking MPI File read MPI File write MPI File iread Non blocking MPI File iwrite Shared file pointer MPI File read shared Blocking MPI File write shared MPI File iread shared Non blocking MPI File iwrite shared Explicit offset MPI File read at Blocking MPI File write at MPI File iread at Non blocking MPI File iwrite at COSC 6374 Parallel Computation Edgar Gabriel Example reading a matrix in parallel Assuming row major ordering C style Basic datatype etype MPI DOUBLE 4 processes 0 1 2 3 Each process reads 4 MPI DOUBLEs at once 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 COSC 6374 Parallel Computation Edgar Gabriel 5 Example using individual file pointers create etype num elements 4 MPI Type contiguous num elements MPI DOUBLE etype MPI Type commit etype create filetype for i 0 i 4 i displ i rank i size multiples of etype blocklength i 1 MPI Type indexed 4 blocklength displ etype ftype MPI Type commit ftype MPI File open MPI COMM WORLD myfile txt MPI MODE RDONLY MPI INFO NULL fh MPI File set view fh 0 etype ftype native MPI INFO NULL for i 0 i 4 i MPI File read fh buf num elements i 1 etype status do something with the data COSC 6374 Parallel Computation Edgar Gabriel The same example using explicit offsets create etype MPI Type contiguous num elements MPI DOUBLE etype MPI Type commit etype MPI File open MPI COMM WORLD myfile txt MPI MODE RDONLY MPI INFO NULL fh MPI File set view fh 0 etype etype native MPI INFO NULL for i 0 i 4 i offset rank i size MPI File read at fh offset buf num elements i 1 etype status do something with the data Simpler code than individual file pointers No prefetching of data possible COSC 6374 Parallel Computation Edgar Gabriel 6 using non blocking functions create etype MPI Type contiguous num elements MPI DOUBLE etype MPI Type commit etype MPI File open MPI COMM WORLD myfile txt MPI MODE RDONLY MPI INFO NULL fh MPI File set view fh 0 etype etype native MPI INFO NULL for i 0 i 4 i offset rank i size MPI File iread at fh offset buf num elements i 1 etype req i MPI Waitall 4 req array of statuses do something with the data COSC 6374 Parallel Computation Edgar Gabriel Consistency of file operation MPI does not provide sequential consistency across all processes per default Write on one process is initially just visible on the same process Two possibilities to change this behavior MPI File set atomicity MPI File fh int flag If flag true all write operations are atomic Collective operation MPI File sync MPI File fh Flushes all write operations on the calling process COSC 6374 Parallel Computation Edgar Gabriel 7 File pointers associated with a file Each file has two file pointers A shared file pointer An individual file pointer Determining the position of a file pointer MPI File get position MPI File fh MPI Offset offset MPI File get position shared MPI File fh MPI Offset offset COSC 6374 Parallel Computation Edgar Gabriel Setting the position of the file pointer MPI File seek MPI