Newsgroups: comp.parallel.mpi,comp.programming.threads
From: Alison White <alisonw@us.ibm.com>
Subject: Re: Mixing MPI and Threads
Organization: IBM
Date: Wed, 07 Jan 1998 14:53:02 -0500
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Message-ID: <34B3DD1E.41C6@us.ibm.com>

Alexandre Carissimi wrote:
> 
> Hi,
> 
>         I have a problem mixing MPI-LAM and threads, and I would
> like with somebody had this kind of problem, and  if you know if
> there is some problem on LAM MPI or if my problem is due a bug on
> my source code.
> 
>         My application is very simple: It is a ring composed by
> "n" nodes, where "n" is the number of MPI process. Each node has
> two threads: a thread to read messages arriving on left side and
> sends to the right side, and another one that read messages
> arriving on right side and sends to the left side.
> 
>           -----                       -----
>          |  t1 | ------------------> |t1   |
>          |  t2 | <-----------------  |t2   |
>           -----                       -----
>            | ^                         | ^
>            | |                         | |
>            V |                         V |
>           -----                       -----
>          |   t1| ------------------> |t1   |
>          |   t2| <-----------------  |t2   |
>           -----                       -----
> 
>         To avoid problems due to fact that MPI isn't thread
> safe I did some equivalent functions protected by a global
> mutex, like:
> 
> int Irecv(void *buf, int count, MPI_Datatype datatype, int dest,
>             int tag, MPI_Comm comm, MPI_Request *request)
> {
>       int r;
> 
>       pthread_mutex_lock(&mutex);
>       r=MPI_Irecv(buf,count,datatype,dest,tag,comm,request);
>       pthread_mutex_unlock(&mutex);
>       return r;
> }
> 
>         I use to communicate only non-blocking versions to
> avoid blocking on MPI. So I have only MPI_Irecv, MPI_Isend
> and MPI_Itest calls (all protected by the global mutex).
> 
>         THE PROBLEM IS:
> 
>         When I execute this program using the -C2C option
> on mpirun command line I have a error message like:
> 
> % mpirun -c 3 -c2c -w -nger /home/asc/threads/RingMT
> MPI_Irecv: internal MPI error: Invalid argument (rank 0, comm 0)
> MPI_Irecv: internal MPI error: Invalid argument (rank 1, comm 0)
> MPI_Irecv: internal MPI error: Invalid argument (rank 2, comm 0)
> 7543 exited with status 1
> 7542 exited with status 22
> 7544 exited with status 1
> 
>         But, if I execute the program without the -C2C option
> it works well. My QUESTION is WHY THIS HAPPEN? Somebody knows
> why bypassing LAM deamon I have this problem? It is possible
> to solve this.
> 
> Thanks in advance.
> Regards.
> 
> ASC
> 
> __________________________________________________________________
> CARISSIMI, Alexandre                  Alexandre.Carissimi@imag.fr
> Laboratoire de Modelisation et Calcul - LMC
> Institut National Polytechnique de Grenoble - INPG
> 100, Rue des Mathematiques               Fone: (33) 04.76.51.46.03
> 38041 - Grenoble - Cedex 9  FRANCE       Fax : (33) 04.76.51.48.46
>                             (International calls removes 0 from 04)
> 
> "Le seul moteur pour gagner c'est innover" Renault (F1 - 1996)
> ________________________________________________________________

Hi,

I don't know anything about MPI-LAM, but, I think you have a problem 
in your program. You have protected the MPI calls with a mutex, but the
the calls are non-blocking.  So, control returns to your thread 
immediately, but MPI continues processing the non-blocking calls, so 
MPI work may still be done on multiple threads.  I think your approach
would be valid if you used only blocking MPI calls.

Alison White