From: montjoy@ra.ececs.uc.edu (Robert Montjoy) Newsgroups: comp.parallel.mpi,comp.unix.solaris Subject: Re: MPICH on Solaris (ch_shmem) Date: 3 Dec 1998 19:57:17 GMT Organization: University of Cincinnati, ECE/CS Dept Distribution: inet Message-Id: <746qet$j0q$1@news.ececs.uc.edu> References: <74515u$nd6$1@news.usf.edu> Xref: ukc comp.parallel.mpi:4395 comp.unix.solaris:164313 Hi.. I think that this is using the SYS V shared memory mechism and this has one big lock around the IPC calls. Essentially all Shared memory access is serialized. I think Caspar posted about this a while back (2 or 3 months ago). It would probably be better to use normal ch_p4 device. From sunsolve: Bug Id: 1180075 Category: kernel Subcategory: other State: committed Release summary: s494_fcs, 5.5, s494_fcs_d, s495_beta Synopsis: SYS V message facilities are protected by a single global mutex Integrated in releases: Patch id: Description: The SYS V message facilities are protected by a single mutex. This causes horrible scaling for multi-threaded apps on an MP system. This should be changed to a mutex per message queue. The description field as copied from bug report 1239172 follows: LADOK is a swedish student grade application widely used within the university world in Sweden. It runs on top of a swedish database called MIMER, sold by a company called Sysdeco. We have installed this initial SUN based customer system at Chalmers, and are experiencing very bad performance. Running virtual adrian during two large database searches shows that disks, CPU, memory etc are very little utilised, but we have "serious mutex contention". We've been in contact with Adrian Cockcroft, and we've run kgmon on the system while loaded. His conclusion is: "I think the problem is that we have a single threaded sysV message queue implementation, that is probably the cause of the contention. You also have a lot of idle time, so you must be waiting for messages all the time." ------ In article <74515u$nd6$1@news.usf.edu>, David Rabson (PHY) wrote: > >How should I tune shared memory for MPICH on a two-processor Ultra 60 >with loads (>1GB) of memory? I generally use MPI indirectly through >SCALAPACK and BLACS, so I haven't much control over the low-level calls. > >My current program (a matrix diagonalization) runs on four processes >with MPICH (version 1.1.1) compiled with ch_shmem. For a run that goes >on for half an hour, profiling (-xpg, gprof) shows that the program spends >98% of its time in > > MPID_SHMEM_ReadControl(). > >Even a tiny problem (only a few bytes that need to be shared among >processes) runs for several minutes stuck in this routine, so I >suspect that my shared-memory tuning is to blame. > > >1. Should I get Sun's MPI instead of the one I compiled myself? >2. Are there MPI-compilation options I need to know about? >3. Should I change any of the shared-memory parameters in /etc/system? -- Rob Montjoy - Systems Engineer - University of Cincinnati - DEPT OF ECECS E-Mail: Rob_Montjoy@ececs.uc.EDU - UNIX Consultant or Personal E-Mail Sun-FAQ@ececs.uc.edu - Comp.sys.sun.admin FAQ related E-Mail -- One seldom sees a monument to a committee.