MPI調試–出錯資訊整理

如果是用FORTRAN寫程式，建議加上implicit
none，特別是代碼比較多時，可以檢查出編譯過程中的很多問題。

1、

• [root@c0108 parallel]# mpiexec -n 5 ./simple
• aborting job:
• Fatal error in MPI_Irecv: Invalid rank, error stack:
• MPI_Irecv(143): MPI_Irecv(buf=0x25dab60, count=0, MPI_DOUBLE_PRECISION, src=5, tag=99, MPI_COMM_WORLD, request=0x7fffa02ca86c) failed
• MPI_Irecv(95): Invalid rank has value 5 but must be nonnegative and less than 5
• rank 4 in job 5  c0108_52041   caused collective abort of all ranks
•   exit status of rank 4: return code 13

上面的意思是，進程號為5的無效，因為[root@c0108 parallel]# mpiexec -n 5 ./simple啟動並執行時候，開了5個進程：0 1 2 3 4，所以一定是代碼本身的問題，但不一定是某個進程號本身，也有可能是某個參數傳遞未成功等，MPI總會出現許多莫名的錯誤。。。

My Code中MPI_Irecv語句有限，於是通過添加print語句的方法進行調試，找出錯誤碼所在的行，如下


print *, myid+1,'111111111111111111'!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!


call MPI_Irecv(P(1,1,location),IMAX*JMAX*MIN(ITSP, ke-myke),

     &MPI_DOUBLE_PRECISION,MYID+1,RELY,MPI_COMM_WORLD,REQ,IERR)

2、

• [root@c0109 test]# mpiexec -n 5 ./simple
• rank 3 in job 22  c0109_51164   caused collective abort of all ranks
•   exit status of rank 3: killed by signal 11
• [root@c0109 test]#
• 其中signal 11是段錯誤。Signal 11, or officially know as "segmentation fault", means that the program accessed a memory location that was not assigned. That's usually
  a bug in the program.

3、

• [root@c0108 test]# mpirun -np 4 ./simple
• aborting job:
• Fatal error in MPI_Wait: Invalid MPI_Request, error stack:
• MPI_Wait(139): MPI_Wait(request=0x7fff1f675228, status0x7fff1f675218) failed
• MPI_Wait(75): Invalid MPI_Request
• rank 2 in job 24  c0108_52041   caused collective abort of all ranks
•   exit status of rank 2: return code 13

solution:

generally it's because MPI_Test of MPI_Wait is supplied a request thatis unknown to MPICH (the request wasn't the one returned by MPICH
whenyou made the Isend/Irecv/send_init/recv_init)就是說MPI_Irecv沒有和MPI_Wait(req,status,IERR)對應,控制代碼對錯號了。。如果MPI_Wait()函數有很多，可以採用注釋的方法一個個鎖定錯誤。。。另外：如果是FORTRAN程式，請首先檢查一下status變數定義：integer
req,status(MPI_STATUS_SIZE),ierr


4、

aborting job: Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(195): Initialization failed MPID_Init(170): failure during portals initialization MPIDI_Portals_Init(321): progress_init failed MPIDI_PortalsI_Progress_init(653):
Out of memory   



There is not enough memory on the nodes for the program plus MPI buffers to fit.



You can decrease the amount of memory that MPI is using for buffers by using MPICH_UNEX_BUFFER_SIZE environment variable.

歡迎批評指正，多多交流，謝謝！