mpirun -np 6 n1-4 ./multi_mmap -w regular -r mmap -i 1000 -b 4096 /ocfs2/mmap_test1 Tail of test log: ca-test28 (rank 3): Write via write block 3 of 'w' ca-test27 (rank 2): Expect via mmap block 3 of 'w' ca-test26 (rank 1): Expect via mmap block 3 of 'w' ca-test26 (rank 5): Expect via mmap block 3 of 'w' ca-test25 (rank 0): Expect via mmap block 3 of 'w' ca-test25 (rank 4): Expect via mmap block 3 of 'w' ca-test25 (rank 4): Write via write block 4 of 'x' ca-test27 (rank 2): Expect via mmap block 4 of 'x' ca-test26 (rank 1): Expect via mmap block 4 of 'x' ca-test26 (rank 5): Expect via mmap block 4 of 'x' ca-test28 (rank 3): Expect via mmap block 4 of 'x' ca-test25 (rank 0): Expect via mmap block 4 of 'x' ca-test26 (rank 5): Write via write block 5 of 'y' ca-test27 (rank 2): Expect via mmap block 5 of 'y' ca-test28 (rank 3): Expect via mmap block 5 of 'y' ca-test26 (rank 1): Expect via mmap block 5 of 'y' ca-test25 (rank 0): Expect via mmap block 5 of 'y' ca-test25 (rank 4): Expect via mmap block 5 of 'y' ca-test25 (rank 0): Write via write block 0 of 'u' ca-test27 (rank 2): Expect via mmap block 0 of 'u' ca-test28 (rank 3): Expect via mmap block 0 of 'u' ca-test26 (rank 1): Expect via mmap block 0 of 'u' ca-test26 (rank 5): Expect via mmap block 0 of 'u' ca-test25 (rank 4): Expect via mmap block 0 of 'u' ca-test26 (rank 1): Write via write block 1 of 'v' ca-test27 (rank 2): Expect via mmap block 1 of 'v' ca-test28 (rank 3): Expect via mmap block 1 of 'v' ca-test26 (rank 5): Expect via mmap block 1 of 'v' ca-test25 (rank 0): Expect via mmap block 1 of 'v' ca-test25 (rank 4): Expect via mmap block 1 of 'v' ca-test27 (rank 2): Write via write block 2 of 'w' ca-test26 (rank 1): Expect via mmap block 2 of 'w' ca-test26 (rank 5): Expect via mmap block 2 of 'w' ca-test28 (rank 3): Expect via mmap block 2 of 'w' ca-test25 (rank 0): Expect via mmap block 2 of 'w' ca-test25 (rank 4): Verification failed for block 2. ca-test25 (rank 4): First 10 expected chars: "wwwwwwwwww" ca-test25 (rank 4): First 10 actual chars: "vvvvvvvvvv" ca-test25 (rank 4): Expect via mmap block 2 of 'w' ca-test28 (rank 3): Write via write block 3 of 'x' MPI_Recv: process in local group is dead (rank 2, comm 3) Rank (2, MPI_COMM_WORLD): Call stack within LAM: Rank (2, MPI_COMM_WORLD): - MPI_Recv() Rank (2, MPI_COMM_WORLD): - MPI_Barrier() Rank (2, MPI_COMM_WORLD): - MPI_Barrier() MPI_Recv: process in local group is dead (rank 3, comm 3) Rank (3, MPI_COMM_WORLD): Call stack within LAM: Rank (3, MPI_COMM_WORLD): - MPI_Recv() Rank (3, MPI_COMM_WORLD): - MPI_Barrier() Rank (3, MPI_COMM_WORLD): - MPI_Barrier() Rank (2, MPI_COMM_WORLD): - main() Rank (3, MPI_COMM_WORLD): - main() MPI_Recv: process in local group is dead (rank 1, SSI:coll:smp:local comm for CID 0) Rank (5, MPI_COMM_WORLD): Call stack within LAM: Rank (5, MPI_COMM_WORLD): - MPI_Recv() Rank (5, MPI_COMM_WORLD): - MPI_Bcast() Rank (5, MPI_COMM_WORLD): - MPI_Barrier() Rank (5, MPI_COMM_WORLD): - main() ----------------------------------------------------------------------------- One of the processes started by mpirun has exited with a nonzero exit code. This typically indicates that the process finished in error. If your process did not finish in error, be sure to include a "return 0" or "exit(0)" in your C code before exiting the application. PID 5835 failed on node n1 (139.185.49.25) with exit status 1. ----------------------------------------------------------------------------- dmesg from ca-test25: ocfs2: dumping nopage trace buffer (curr = 8) (5839,1):print_trace:109 ocfs2: [9] Inode 6258143, index 4 (5839,1):print_trace:109 ocfs2: [10] Inode 6258143, index 5 (5839,1):print_trace:109 ocfs2: [11] Inode 6258143, index 5 (5839,1):print_trace:109 ocfs2: [12] Inode 6258143, index 0 (5839,1):print_trace:109 ocfs2: [13] Inode 6258143, index 1 (5839,1):print_trace:109 ocfs2: [14] Inode 6258143, index 1 (5839,1):print_trace:109 ocfs2: [15] Inode 6258143, index 2 (5839,1):print_trace:109 ocfs2: [16] Inode 6258143, index 2 (5839,1):print_trace:109 ocfs2: [17] Inode 6258143, index 3 (5839,1):print_trace:109 ocfs2: [18] Inode 6258143, index 3 (5839,1):print_trace:109 ocfs2: [19] Inode 6258143, index 4 (5839,1):print_trace:109 ocfs2: [20] Inode 6258143, index 5 (5839,1):print_trace:109 ocfs2: [21] Inode 6258143, index 5 (5839,1):print_trace:109 ocfs2: [22] Inode 6258143, index 0 (5839,1):print_trace:109 ocfs2: [23] Inode 6258143, index 1 (5839,1):print_trace:109 ocfs2: [24] Inode 6258143, index 1 (5839,1):print_trace:109 ocfs2: [25] Inode 6258143, index 2 (5839,1):print_trace:109 ocfs2: [26] Inode 6258143, index 2 (5839,1):print_trace:109 ocfs2: [27] Inode 6258143, index 3 (5839,1):print_trace:109 ocfs2: [28] Inode 6258143, index 3 (5839,1):print_trace:109 ocfs2: [29] Inode 6258143, index 4 (5839,1):print_trace:109 ocfs2: [30] Inode 6258143, index 5 (5839,1):print_trace:109 ocfs2: [31] Inode 6258143, index 5 (5839,1):print_trace:109 ocfs2: [32] Inode 6258143, index 0 (5839,1):print_trace:109 ocfs2: [33] Inode 6258143, index 1 (5839,1):print_trace:109 ocfs2: [34] Inode 6258143, index 1 (5839,1):print_trace:109 ocfs2: [35] Inode 6258143, index 2 (5839,1):print_trace:109 ocfs2: [36] Inode 6258143, index 2 (5839,1):print_trace:109 ocfs2: [37] Inode 6258143, index 3 (5839,1):print_trace:109 ocfs2: [38] Inode 6258143, index 3 (5839,1):print_trace:109 ocfs2: [39] Inode 6258143, index 4 (5839,1):print_trace:109 ocfs2: [40] Inode 6258143, index 5 (5839,1):print_trace:109 ocfs2: [41] Inode 6258143, index 5 (5839,1):print_trace:109 ocfs2: [42] Inode 6258143, index 0 (5839,1):print_trace:109 ocfs2: [43] Inode 6258143, index 1 (5839,1):print_trace:109 ocfs2: [44] Inode 6258143, index 1 (5839,1):print_trace:109 ocfs2: [45] Inode 6258143, index 2 (5839,1):print_trace:109 ocfs2: [46] Inode 6258143, index 2 (5839,1):print_trace:109 ocfs2: [47] Inode 6258143, index 2 (5839,1):print_trace:109 ocfs2: [48] Inode 6258143, index 2 (5839,1):print_trace:109 ocfs2: [49] Inode 6258143, index 3 (5839,1):print_trace:109 ocfs2: [0] Inode 6258143, index 3 (5839,1):print_trace:109 ocfs2: [1] Inode 6258143, index 4 (5839,1):print_trace:109 ocfs2: [2] Inode 6258143, index 5 (5839,1):print_trace:109 ocfs2: [3] Inode 6258143, index 5 (5839,1):print_trace:109 ocfs2: [4] Inode 6258143, index 0 (5839,1):print_trace:109 ocfs2: [5] Inode 6258143, index 1 (5839,1):print_trace:109 ocfs2: [6] Inode 6258143, index 1 (5839,1):print_trace:109 ocfs2: [7] Inode 6258143, index 2 (5839,1):print_trace:109 ocfs2: [8] Inode 6258143, index 3 ocfs2: last page data: wwwwwwwwwwwwwww