Newsgroups: comp.parallel.mpi,comp.programming.threads
From: Alexandre Carissimi <Alexandre.Carissimi@imag.fr>
Subject: Re: Mixing MPI and Threads
Organization: LMC-Laboratoire de Modelisation et Calcul
Date: Fri, 09 Jan 1998 12:22:52 +0100
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Message-ID: <34B6088C.3B2D@imag.fr>

Alison White wrote:
 > 
 > Alexandre Carissimi wrote:
 > >
 > > Hi,
 > >
 > >         I have a problem mixing MPI-LAM and threads, and I would
 > > like with somebody had this kind of problem, and  if you know if
 > > there is some problem on LAM MPI or if my problem is due a bug on
 > > my source code.
 > >
 > >         My application is very simple: It is a ring composed by
 > > "n" nodes, where "n" is the number of MPI process. Each node has
 > > two threads: a thread to read messages arriving on left side and
 > > sends to the right side, and another one that read messages
 > > arriving on right side and sends to the left side.
 > >
 > >           -----                       -----
 > >          |  t1 | ------------------> |t1   |
 > >          |  t2 | <-----------------  |t2   |
 > >           -----                       -----
 > >            | ^                         | ^
 > >            | |                         | |
 > >            V |                         V |
 > >           -----                       -----
> >          |   t1| ------------------> |t1   |
> >          |   t2| <-----------------  |t2   |
> >           -----                       -----
> >
> >         To avoid problems due to fact that MPI isn't thread
> > safe I did some equivalent functions protected by a global
> > mutex, like:
> >
> > int Irecv(void *buf, int count, MPI_Datatype datatype, int dest,
> >             int tag, MPI_Comm comm, MPI_Request *request)
> > {
> >       int r;
> >
> >       pthread_mutex_lock(&mutex);
> >       r=MPI_Irecv(buf,count,datatype,dest,tag,comm,request);
> >       pthread_mutex_unlock(&mutex);
> >       return r;
> > }
> >
> >         I use to communicate only non-blocking versions to
> > avoid blocking on MPI. So I have only MPI_Irecv, MPI_Isend
> > and MPI_Itest calls (all protected by the global mutex).
> >
> >         THE PROBLEM IS:
> >
> >         When I execute this program using the -C2C option
> > on mpirun command line I have a error message like:
> >
> > % mpirun -c 3 -c2c -w -nger /home/asc/threads/RingMT
> > MPI_Irecv: internal MPI error: Invalid argument (rank 0, comm 0)
> > MPI_Irecv: internal MPI error: Invalid argument (rank 1, comm 0)
> > MPI_Irecv: internal MPI error: Invalid argument (rank 2, comm 0)
> > 7543 exited with status 1
> > 7542 exited with status 22
> > 7544 exited with status 1
> >
> >         But, if I execute the program without the -C2C option
> > it works well. My QUESTION is WHY THIS HAPPEN? Somebody knows
> > why bypassing LAM deamon I have this problem? It is possible
> > to solve this.
> >
> > Thanks in advance.
> > Regards.
> >
> > ASC

> 
> I don't know anything about MPI-LAM, but, I think you have a problem
> in your program. You have protected the MPI calls with a mutex, but the
> the calls are non-blocking.  So, control returns to your thread
> immediately, but MPI continues processing the non-blocking calls, so
> MPI work may still be done on multiple threads.  I think your approach
> would be valid if you used only blocking MPI calls.
> 
> Alison White

	This is the goal. Suppose one blocking receive is done 
before the send is done. You  get the mutex on calling receive
which wait by a send that never will be done because the mutex
to MPI acces is granted by receive. If I use blocking call I'll
have deadlocks. Ok?

	I did another version using IPROBE which has the same
problem. If I run with -c2c option it hangs up saying:

MPI_Iprobe: internal MPI error: Invalid argument (rank 2, comm 0)

and if I run without -c2c option the program works.

	I would like to use the -c2c because we measured performance
using a ping-pong program and the version with -c2c is TWICE faster
than version ran without -c2c option.

Regards
ASC


__________________________________________________________________
CARISSIMI, Alexandre                  Alexandre.Carissimi@imag.fr
Laboratoire de Modelisation et Calcul - LMC
Institut National Polytechnique de Grenoble - INPG
100, Rue des Mathematiques               Fone: (33) 04.76.51.46.03
38041 - Grenoble - Cedex 9  FRANCE       Fax : (33) 04.76.51.48.46
                            (International calls removes 0 from 04)

"Le seul moteur pour gagner c'est innover" Renault (F1 - 1996)
________________________________________________________________