Message-Id: <3601B1EB.1E3719FE@sprint.ca> From: Tomasz Korycki Reply-To: tkorycki@sprint.ca Organization: flowchART Enterprises Mime-Version: 1.0 Newsgroups: comp.parallel.mpi Subject: Re: Latency and MPI_Probe on SP2 References: <6tcupe$32t$1@news.kren.nm.kr> <35FB5FED.456D33F0@sprint.ca> <6ti00q$eus$1@news.kren.nm.kr> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Date: Thu, 17 Sep 1998 21:05:47 -0400 J Kim wrote: > > >What model switch sits in Your frame? > > I don't know what kind of switch is used. > Are there various switch models in SP2 ? Yes, I know of at least three. You would find that info in the original manuals: there should be one for a switch. You can also use perspectives or spmon to look for a "model" I am forwarding two messages I received from Todd of Mobil, who thought I was asking Your question. He's generally right, I just want to vallidate Your data, to make sure Todd's reasons look like only ones. >>>>>>>>>>>>>>>>>>>>>>Message 1: Tomasz, I read your post in comp.parallel.mpi and may be able to help you a little. 1.) You can get the theoretical hardware latency times out of IBM if you press them. Again, these are theoretical. They don't appear to have many 'real world' tests. 2.) I haven't noticed the inconsistancy you describe, but I can see how it can be inconsistant for small test cases. The first case may also result in buffer allocation and initialization so that might be accounting for the differences. 3.) The difference in what you observed and what you expected is the difference between theory and reality. MPI provides an interface to send and receive messages asynchronously. Theoretically, an implementation could take advantage of it. In reality, the SP/2 uses the main system processors to send and receive messages. There is no "I/O controller" type of functionality for parallel message passing as there is for disk access. So, the reality is that the main cpus perform your computational logic, AND send and receive the messages. Asynchronously sending and receiving data doesn't exist. Computation and messaging compete with each other. With this in mind, your observations make perfect sense. 4.) I fought with MPI_IProbe and the "asynchronous" interfaces for quite awhile. I'll save you the time, testing, and frustration. It's slow and buggy on the SP/2. MPI_IProbe doesn't always work, and as you noted, it is slow. So, now you probably want to know the fastest way to send/receive messages on the SP/2. Here it is. You have to use the standard (synchronous) send and receive at the same time. Additionally, since the SP/2 switch will allow each processor to send at full switch bandwidth while it receives at full bandwidth, it is advisable to have the processors send in a 'ring' so that each processor is sending to and receiving from exactly one other processor. The send to and receive from processor does not need to be the same one, but don't try to have more than one processor send to the same destination as the collision will hurt your transfer speed. Use barriers between the send/receive pairs to keep the processors synchronized as all the data is transferred. The SP/2 messaging system functions very well when all the processors are kept in 'lock-step' with each other. An example for four CPU's: P[0-3] = Processor 0 to 3 st = send to rf = receive from P0 P1 P2 P3 -------------------barrier all processors------------- st:P1, rf:P3 st:P2, rf:P0 st:P3, rf:P1 st:P0, rf:P2 -------------------barrier all processors------------- st:P2, rf:P2 st:P3, rfP3 st:P0, rf:P0 st:P1, rfP1 -------------------barrier all processors------------- st:P3, rf:P1 st:P0, rfP2 st:P1, rf:P3 st:P2, rfP0 -------------------barrier all processors------------- Using this synchronzied, tightly-coupled, sending/receiving technique you'll get as much bandwidth as available, without any competition with your own messages. Remember, asynchronous messaging is a myth on the SP/2. Do your computation, then send the data, then repeat. If you have any further questions, just ask. Regards, Todd Williams todd_e_williams@email.mobil.com >>>>>>>>>>>>>>>>>>>Message 2: Tomasz, A note of clarification. In my previous email when I mentioned using the synchronous messaging interface, I actually meant the blocking interface, ie. MPI_Send and MPI_Recv. Sorry for any confusion. Regards, Todd todd_e_williams@email.mobil.com