[mfasheh@ca-build8 multi_node_mmap]$ mpirun -np 6 n1-4 ./multi_mmap -w regular -r mmap -i 1000 -b 4096 /ocfs2/mmap_test9 ca-test25 (rank 0): Expect via mmap block 3 of 'f'. My pid = 6609 ca-test25 (rank 4): Expect via mmap block 3 of 'f'. My pid = 6610 ca-test25 (rank 4): Write via write block 4 of 'g'. My pid = 6610 ca-test27 (rank 2): Expect via mmap block 4 of 'g'. My pid = 7635 ca-test25 (rank 0): Expect via mmap block 4 of 'g'. My pid = 6609 ca-test28 (rank 3): Expect via mmap block 4 of 'g'. My pid = 7727 ca-test26 (rank 1): Expect via mmap block 4 of 'g'. My pid = 7230 ca-test26 (rank 5): Expect via mmap block 4 of 'g'. My pid = 7231 ca-test26 (rank 5): Write via write block 5 of 'h'. My pid = 7231 ca-test26 (rank 1): Expect via mmap block 5 of 'h'. My pid = 7230 ca-test27 (rank 2): Expect via mmap block 5 of 'h'. My pid = 7635 ca-test25 (rank 0): Expect via mmap block 5 of 'h'. My pid = 6609 ca-test28 (rank 3): Expect via mmap block 5 of 'h'. My pid = 7727 ca-test25 (rank 4): Expect via mmap block 5 of 'h'. My pid = 6610 ca-test25 (rank 0): Write via write block 0 of 'd'. My pid = 6609 ca-test25 (rank 4): Expect via mmap block 0 of 'd'. My pid = 6610 ca-test28 (rank 3): Expect via mmap block 0 of 'd'. My pid = 7727 ca-test27 (rank 2): Expect via mmap block 0 of 'd'. My pid = 7635 ca-test26 (rank 1): Expect via mmap block 0 of 'd'. My pid = 7230 ca-test26 (rank 5): Expect via mmap block 0 of 'd'. My pid = 7231 ca-test26 (rank 1): Write via write block 1 of 'e'. My pid = 7230 ca-test26 (rank 5): Expect via mmap block 1 of 'e'. My pid = 7231 ca-test27 (rank 2): Expect via mmap block 1 of 'e'. My pid = 7635 ca-test28 (rank 3): Expect via mmap block 1 of 'e'. My pid = 7727 ca-test25 (rank 0): Expect via mmap block 1 of 'e'. My pid = 6609 ca-test25 (rank 4): Expect via mmap block 1 of 'e'. My pid = 6610 ca-test27 (rank 2): Write via write block 2 of 'f'. My pid = 7635 ca-test26 (rank 1): Expect via mmap block 2 of 'f'. My pid = 7230 ca-test28 (rank 3): Expect via mmap block 2 of 'f'. My pid = 7727 ca-test26 (rank 5): Expect via mmap block 2 of 'f'. My pid = 7231 ca-test25 (rank 4): Expect via mmap block 2 of 'f'. My pid = 6610 ca-test25 (rank 0): Expect via mmap block 2 of 'f'. My pid = 6609 ca-test28 (rank 3): Write via write block 3 of 'g'. My pid = 7727 ca-test26 (rank 1): Expect via mmap block 3 of 'g'. My pid = 7230 ca-test27 (rank 2): Expect via mmap block 3 of 'g'. My pid = 7635 ca-test26 (rank 5): Expect via mmap block 3 of 'g'. My pid = 7231 ca-test25 (rank 0): Expect via mmap block 3 of 'g'. My pid = 6609 ca-test25 (rank 4): Expect via mmap block 3 of 'g'. My pid = 6610 ca-test25 (rank 4): Write via write block 4 of 'h'. My pid = 6610 ca-test28 (rank 3): Expect via mmap block 4 of 'h'. My pid = 7727 ca-test27 (rank 2): Expect via mmap block 4 of 'h'. My pid = 7635 ca-test26 (rank 1): Expect via mmap block 4 of 'h'. My pid = 7230 ca-test26 (rank 5): Expect via mmap block 4 of 'h'. My pid = 7231 ca-test25 (rank 0): Expect via mmap block 4 of 'h'. My pid = 6609 ca-test26 (rank 5): Write via write block 5 of 'i'. My pid = 7231 ca-test28 (rank 3): Expect via mmap block 5 of 'i'. My pid = 7727 ca-test27 (rank 2): Expect via mmap block 5 of 'i'. My pid = 7635 ca-test26 (rank 1): Expect via mmap block 5 of 'i'. My pid = 7230 ca-test25 (rank 0): Expect via mmap block 5 of 'i'. My pid = 6609 ca-test25 (rank 4): Expect via mmap block 5 of 'i'. My pid = 6610 ca-test25 (rank 0): Write via write block 0 of 'e'. My pid = 6609 ca-test28 (rank 3): Expect via mmap block 0 of 'e'. My pid = 7727 ca-test27 (rank 2): Expect via mmap block 0 of 'e'. My pid = 7635 ca-test25 (rank 4): Expect via mmap block 0 of 'e'. My pid = 6610 ca-test26 (rank 1): Expect via mmap block 0 of 'e'. My pid = 7230 ca-test26 (rank 5): Expect via mmap block 0 of 'e'. My pid = 7231 ca-test26 (rank 1): Write via write block 1 of 'f'. My pid = 7230 ca-test26 (rank 5): Expect via mmap block 1 of 'f'. My pid = 7231 ca-test28 (rank 3): Expect via mmap block 1 of 'f'. My pid = 7727 ca-test27 (rank 2): Expect via mmap block 1 of 'f'. My pid = 7635 ca-test25 (rank 0): Expect via mmap block 1 of 'f'. My pid = 6609 ca-test25 (rank 4): Verification failed for block 1. ca-test25 (rank 4): First 10 expected chars: "ffffffffff" ca-test25 (rank 4): First 10 actual chars: "eeeeeeeeee" ca-test25 (rank 4): Expect via mmap block 1 of 'f'. My pid = 6610 ca-test27 (rank 2): Write via write block 2 of 'g'. My pid = 7635 MPI_Recv: process in local group is dead (rank 2, comm 3) Rank (2, MPI_COMM_WORLD): Call stack within LAM: Rank (2, MPI_COMM_WORLD): - MPI_Recv() Rank (2, MPI_COMM_WORLD): - MPI_Barrier() Rank (2, MPI_COMM_WORLD): - MPI_Barrier() MPI_Recv: process in local group is dead (rank 3, comm 3) Rank (3, MPI_COMM_WORLD): Call stack within LAM: Rank (3, MPI_COMM_WORLD): - MPI_Recv() Rank (3, MPI_COMM_WORLD): - MPI_Barrier() Rank (3, MPI_COMM_WORLD): - MPI_Barrier() Rank (3, MPI_COMM_WORLD): - main() Rank (2, MPI_COMM_WORLD): - main() ----------------------------------------------------------------------------- One of the processes started by mpirun has exited with a nonzero exit code. This typically indicates that the process finished in error. If your process did not finish in error, be sure to include a "return 0" or "exit(0)" in your C code before exiting the application. PID 6609 failed on node n1 (139.185.49.25) with exit status 1. ----------------------------------------------------------------------------- MPI_Recv: process in local group is dead (rank 1, SSI:coll:smp:local comm for CID 0) Rank (5, MPI_COMM_WORLD): Call stack within LAM: Rank (5, MPI_COMM_WORLD): - MPI_Recv() Rank (5, MPI_COMM_WORLD): - MPI_Bcast() Rank (5, MPI_COMM_WORLD): - MPI_Barrier() Rank (5, MPI_COMM_WORLD): - main() dmesg on ca-test25: ocfs2: dumping nopage trace buffer (curr = 11) (6612,1):print_dlm_trace:142 ocfs2: [12] Inode 6258151, pid: 6594, location 0, downconvert to Protected Read (6612,1):print_dlm_trace:142 ocfs2: [13] Inode 6258151, pid: 6594, location 0, downconvert to No Lock (6612,1):print_nopage_trace:130 ocfs2: [14] Inode 6258151, index 5,pid: 6610, data fffffffffffffff (6612,1):print_nopage_trace:130 ocfs2: [15] Inode 6258151, index 5,pid: 6609, data fffffffffffffff (6612,1):print_nopage_trace:130 ocfs2: [16] Inode 6258151, index 0,pid: 6610, data bbbbbbbbbbbbbbb (6612,1):print_dlm_trace:142 ocfs2: [17] Inode 6258151, pid: 6594, location 0, downconvert to Protected Read (6612,1):print_dlm_trace:142 ocfs2: [18] Inode 6258151, pid: 6594, location 0, downconvert to No Lock (6612,1):print_nopage_trace:130 ocfs2: [19] Inode 6258151, index 1,pid: 6610, data ccccccccccccccc (6612,1):print_nopage_trace:130 ocfs2: [20] Inode 6258151, index 1,pid: 6609, data ccccccccccccccc (6612,1):print_dlm_trace:142 ocfs2: [21] Inode 6258151, pid: 6594, location 0, downconvert to No Lock (6612,1):print_nopage_trace:130 ocfs2: [22] Inode 6258151, index 2,pid: 6609, data ddddddddddddddd (6612,1):print_nopage_trace:130 ocfs2: [23] Inode 6258151, index 2,pid: 6610, data ddddddddddddddd (6612,1):print_dlm_trace:142 ocfs2: [24] Inode 6258151, pid: 6594, location 0, downconvert to No Lock (6612,1):print_nopage_trace:130 ocfs2: [25] Inode 6258151, index 3,pid: 6609, data eeeeeeeeeeeeeee (6612,1):print_nopage_trace:130 ocfs2: [26] Inode 6258151, index 3,pid: 6610, data eeeeeeeeeeeeeee (6612,1):print_nopage_trace:130 ocfs2: [27] Inode 6258151, index 4,pid: 6609, data fffffffffffffff (6612,1):print_dlm_trace:142 ocfs2: [28] Inode 6258151, pid: 6594, location 0, downconvert to Protected Read (6612,1):print_dlm_trace:142 ocfs2: [29] Inode 6258151, pid: 6594, location 0, downconvert to No Lock (6612,1):print_nopage_trace:130 ocfs2: [30] Inode 6258151, index 5,pid: 6609, data ggggggggggggggg (6612,1):print_nopage_trace:130 ocfs2: [31] Inode 6258151, index 5,pid: 6610, data ggggggggggggggg (6612,1):print_nopage_trace:130 ocfs2: [32] Inode 6258151, index 0,pid: 6610, data ccccccccccccccc (6612,1):print_dlm_trace:142 ocfs2: [33] Inode 6258151, pid: 6594, location 0, downconvert to Protected Read (6612,1):print_dlm_trace:142 ocfs2: [34] Inode 6258151, pid: 6594, location 0, downconvert to No Lock (6612,1):print_nopage_trace:130 ocfs2: [35] Inode 6258151, index 1,pid: 6609, data ddddddddddddddd (6612,1):print_nopage_trace:130 ocfs2: [36] Inode 6258151, index 1,pid: 6610, data ddddddddddddddd (6612,1):print_dlm_trace:142 ocfs2: [37] Inode 6258151, pid: 6594, location 0, downconvert to No Lock (6612,1):print_nopage_trace:130 ocfs2: [38] Inode 6258151, index 2,pid: 6609, data eeeeeeeeeeeeeee (6612,1):print_nopage_trace:130 ocfs2: [39] Inode 6258151, index 2,pid: 6610, data eeeeeeeeeeeeeee (6612,1):print_dlm_trace:142 ocfs2: [40] Inode 6258151, pid: 6594, location 0, downconvert to No Lock (6612,1):print_nopage_trace:130 ocfs2: [41] Inode 6258151, index 3,pid: 6609, data fffffffffffffff (6612,1):print_nopage_trace:130 ocfs2: [42] Inode 6258151, index 3,pid: 6610, data fffffffffffffff (6612,1):print_nopage_trace:130 ocfs2: [43] Inode 6258151, index 4,pid: 6609, data ggggggggggggggg (6612,1):print_dlm_trace:142 ocfs2: [44] Inode 6258151, pid: 6594, location 0, downconvert to Protected Read (6612,1):print_dlm_trace:142 ocfs2: [45] Inode 6258151, pid: 6594, location 0, downconvert to No Lock (6612,1):print_nopage_trace:130 ocfs2: [46] Inode 6258151, index 5,pid: 6609, data hhhhhhhhhhhhhhh (6612,1):print_nopage_trace:130 ocfs2: [47] Inode 6258151, index 5,pid: 6610, data hhhhhhhhhhhhhhh (6612,1):print_nopage_trace:130 ocfs2: [48] Inode 6258151, index 0,pid: 6610, data ddddddddddddddd (6612,1):print_dlm_trace:142 ocfs2: [49] Inode 6258151, pid: 6594, location 0, downconvert to Protected Read (6612,1):print_dlm_trace:142 ocfs2: [50] Inode 6258151, pid: 6594, location 0, downconvert to No Lock (6612,1):print_dlm_trace:142 ocfs2: [51] Inode 6258151, pid: 6594, location 0, downconvert to No Lock (6612,1):print_nopage_trace:130 ocfs2: [52] Inode 6258151, index 1,pid: 6610, data eeeeeeeeeeeeeee (6612,1):print_nopage_trace:130 ocfs2: [53] Inode 6258151, index 1,pid: 6610, data eeeeeeeeeeeeeee (6612,1):print_nopage_trace:130 ocfs2: [54] Inode 6258151, index 1,pid: 6609, data eeeeeeeeeeeeeee (6612,1):print_nopage_trace:130 ocfs2: [55] Inode 6258151, index 1,pid: 6609, data eeeeeeeeeeeeeee (6612,1):print_nopage_trace:130 ocfs2: [56] Inode 6258151, index 2,pid: 6610, data fffffffffffffff (6612,1):print_nopage_trace:130 ocfs2: [57] Inode 6258151, index 2,pid: 6609, data fffffffffffffff (6612,1):print_dlm_trace:142 ocfs2: [58] Inode 6258151, pid: 6594, location 0, downconvert to No Lock (6612,1):print_nopage_trace:130 ocfs2: [59] Inode 6258151, index 3,pid: 6609, data ggggggggggggggg (6612,1):print_nopage_trace:130 ocfs2: [0] Inode 6258151, index 3,pid: 6610, data ggggggggggggggg (6612,1):print_nopage_trace:130 ocfs2: [1] Inode 6258151, index 4,pid: 6609, data hhhhhhhhhhhhhhh (6612,1):print_dlm_trace:142 ocfs2: [2] Inode 6258151, pid: 6594, location 0, downconvert to Protected Read (6612,1):print_dlm_trace:142 ocfs2: [3] Inode 6258151, pid: 6594, location 0, downconvert to No Lock (6612,1):print_nopage_trace:130 ocfs2: [4] Inode 6258151, index 5,pid: 6609, data iiiiiiiiiiiiiii (6612,1):print_nopage_trace:130 ocfs2: [5] Inode 6258151, index 5,pid: 6610, data iiiiiiiiiiiiiii (6612,1):print_nopage_trace:130 ocfs2: [6] Inode 6258151, index 0,pid: 6610, data eeeeeeeeeeeeeee (6612,1):print_dlm_trace:142 ocfs2: [7] Inode 6258151, pid: 6594, location 0, downconvert to Protected Read (6612,1):print_dlm_trace:142 ocfs2: [8] Inode 6258151, pid: 6594, location 0, downconvert to No Lock (6612,1):print_nopage_trace:130 ocfs2: [9] Inode 6258151, index 1,pid: 6609, data fffffffffffffff (6612,1):print_dlm_trace:142 ocfs2: [10] Inode 6258151, pid: 6594, location 0, downconvert to No Lock (6612,1):print_nopage_trace:130 ocfs2: [11] Inode 6258151, index 4,pid: 6609, data eeeeeeeeeeeeeee