-*- text -*-

Copyright 1997, University of Notre Dame.
Authors: Andrew Lumsdaine, Michael P. McNally, Jeremy G. Siek,
         Jeffery M. Squyres.

This file is part of the Notre Dame C++ bindings for MPI

You should have received a copy of the License Agreement for the
Notre Dame C++ bindings for MPI along with the software;  see the
file LICENSE.  If not, contact Office of Research, University of Notre
Dame, Notre Dame, IN  46556.

Permission to modify the code and to distribute modified code is
granted, provided the text of this NOTICE is retained, a notice that
the code was modified is included with the above COPYRIGHT NOTICE and
with the COPYRIGHT NOTICE in the LICENSE file, and that the LICENSE
file is distributed with the modified code.

LICENSOR MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR IMPLIED.
By way of example, but not limitation, Licensor MAKES NO
REPRESENTATIONS OR WARRANTIES OF MERCHANTABILITY OR FITNESS FOR ANY
PARTICULAR PURPOSE OR THAT THE USE OF THE LICENSED SOFTWARE COMPONENTS
OR DOCUMENTATION WILL NOT INFRINGE ANY PATENTS, COPYRIGHTS, TRADEMARKS
OR OTHER RIGHTS.

-----------------------------------------------------------------------

                       MPI 1.2 C++ Bindings

I. INTRODUCTION
---------------

This implementation of the MPI 1.2 C++ bindings is not a standalone
C++ MPI implementation; it is a layer on top of the MPI 1.1/1.2 C
bindings for MPI.  You must already have an MPI implementation in
order to use this package.

The MPI standards documents are available from http://www.mpi-forum.org/.

The bindings have been compiled and tested on the combinations of
architectures and MPI implementations listed in Table 1.

These MPI C++ bindings are available from their main distribution site:

	http://www.cse.nd.edu/~lsc/research/mpi2c++/


Hardware	Operating System	Compilers	MPI Implementations
=============== ======================= =============== ======================
Sun SPARC	Solaris 2.5.1		Workshop CC	MPICH 1.1.0, LAM 6.1
					v4.2,
					g++ v2.7.2.3
--------------- ----------------------- --------------- ----------------------
SGI		IRIX 6.2		CC v7.10,v7.20	Native v3.0,
							MPICH 1.1.0, LAM 6.1
--------------- ----------------------- --------------- ----------------------
IBM RS/6000	AIX 4.1.4		xlC v3.1.4.0	Native, POE v2.1.0.18,
and SP							MPICH 1.1.0, LAM 6.1
--------------- ----------------------- --------------- ----------------------
HP		HP-UX 10.20		aCC A.01.05	Native v1.2,v1.3.1,
					aCC A.01.06	MPICH 1.1.0, LAM 6.1
--------------- ----------------------- --------------- ----------------------
CRAY T3E	UNICOS/mk 1.6.2.40	Cray C++ v3.0.0.0 Native v1.1.0.4
--------------- ----------------------- --------------- ----------------------
	Table 1: Supported hardware/OSs/Compilers/MPI implementations


II. USING THE LIBRARY
---------------------

Include files

Users should include <mpi++.h> in order to use the MPI C++
bindings.  This is not conformant with the MPI-2 standard, which says
the header for the C++ bindings should be in <mpi.h>. The name mpi++.h
was used because as a layer on top of existing MPI
implementations. we do not have access to the underlying <mpi.h>.


Libraries

After following the instructions in the INSTALL file, libmpi++.a will
be created.  This library needs to be linked with all user MPI C++
applications, in addition to any other libraries that the underlying
MPI library requires.  This link step is automatically handled for the
user by the "balky" compiler wrapper script (see below).


Best Automated Language Kompiler Yet (balky)

balky is a wrapper compiler script that will include all necessary
compiler switches for compiling and linking MPI C++ user programs.
After a successful "make install", it will be installed into
$prefix/bin.  A sample C++ MPI program is shown below, followed by the
balky command to compile it. Note that you do not need to specify
were the <mpi++.h> file is found, as balky automatically takes care
of this.

foo.cc:
-----
#include <iostream.h>
#include <mpi++.h>

int
main(int argc, char* argv[])
{
  MPI::Init(argc, argv);
  int rank = MPI::COMM_WORLD.Get_rank();
  int size = MPI::COMM_WORLD.Get_size();
  cout << "Hello world, I am " << rank << " of " << size << endl;
  MPI::Finalize();
  return 0;
}
-----

Compilation of foo.cc:
-----
unix% balky foo.cc -O -o foo
unix% 
-----


III. IMPLEMENTATION ISSUES
--------------------------

Namespaces

The MPI-2 standard requires that the C++ bindings reside in an MPI
namespace. Most C++ compilers do not yet support the namespace
construct, although some do. Compilers that fully support namespaces
will utilize this C++ feature when compiling the bindings.  Compilers
that do not implement namespaces will use nested classes instead.

The configure script will attempt to determine if your C++ compiler
supports namespace.  If the configure script decides incorrectly, you
can override the results of the namespace test with command line switches
(see the INSTALL file).


Profiling

The C++ bindings can be configured in two ways: profilable and not
profilable. With the profilable version, it is posible to link in an
intermediate MPI:: layer in order to profile MPI functionality.  

The profilable C++ bindings have two layers.  The upper layer consists
of classes and functions in the MPI namespace that can be replaced by user-
or vendor-supplied functions.  The lower layer resides in the PMPI
namespace and does the actual work (usually by just calling the
underlying MPI C bindings). 

The non-profilable version of the C++ bindings is a single layer; the
MPI:: classes and functions which call the C bindings.  

The non-profilable version is fully inlined so that it adds very
little overhead to the C bindings.  The bottom layer of the profilable
bindings is also fully inlined, so there is only the cost of one extra
function call (from the MPI:: layer to the PMPI:: layer), which is
necessary to allow for profiling.


MPI-2 Bindings Names

Names for MPI-1 functions were not always consistent; some are of the
form MPI_Object_action, while others are of the form
MPI_Object_subset.  Verb usage was not consistent either, such as
MPI_ERRHANDLER_SET and MPI_ATTR_PUT.  

The MPI Forum decided to use a completely consistent naming scheme for
MPI-2, and decided that the C++ bindings for MPI-1.2 would follow this
consistent naming scheme.  As such, many of the C++ bindings have
slightly different names than their C and Fortran counterparts.  For
example, MPI_COMM_RANK became MPI::Comm::Get_rank().  A table that
cross references the MPI language neutral names to their C++ bindings
counterparts is provided in Annex B of the MPI-2 standard (see
http://www.mpi-forum.org/).


MPI-2 Deprecated Functions

The MPI Forum decided to deprecate several functions in MPI-1; the use
of these functions is discouraged in future MPI programs.  But since
there is probably some existing code that uses these functions, the
MPI Forum did not remove them from the 1.2 standard.  However, since
the C++ bindings are new, there is no code that will break by the
removal of these functions.  Therefore, the MPI 1.2 C++ function names
do not include bindings for the deprecated functions.

Most of the deprecated names were replaced with new names, some of
which have different semantics than the original name.  The C++
bindings for these functions are included in this library because the
new semantics could be replicated based upon the old MPI-1 C
functions.  Table 2 lists the deprecated MPI-1 names and the
corresponding new C++ names.

MPI 1.1 Function	C++ Binding
=======================	=====================================
MPI_ADDRESS		MPI::Get_address
MPI_TYPE_HINDEXED	MPI::Datatype::Create_hindexed
MPI_TYPE_HVECTOR	MPI::Datatype::Create_hvector
MPI_TYPE_STRUCT		MPI::Datatype::Create_struct

MPI_TYPE_EXTENT		MPI::Datatype::Get_extent
MPI_TYPE_UB		MPI::Datatype::Get_extent
MPI_TYPE_LB		MPI::Datatype::Get_extent
MPI_LB			No corresponding MPI-2 Binding
MPI_UB			No corresponding MPI-2 Binding
=======================	=====================================
	   Table 2: Deprecated MPI-1 functions


New MPI 1.2 Functions

There is one new function specified in the standard for MPI 1.2,
MPI_GET_VERSION.  Since this library is built upon existing MPI 1.1
implementations, it was not possible to implement this function while
still remaining a layer completely independant to the underlying
implementation.



File Organization

The "real" functionality of the C++ bindings can be found in the src/
subdirectory in files suffixed with "_inln.h".  The functions in these
files contain the calls to the underlying C bindings.  Preprocessor
macros "assign" these functions to either the MPI:: classes or the
PMPI:: classes, depending on whether the profiling option was selected
in the configure step (see the INSTALL file). These files comprise the
bottom layer of the profilable C++ bindings, and the only layer of the
non-profilable bindings.

The upper layer of the profilable bindings consist of the files with
the ".cc" extension.  They contain the functions in the MPI::
namespace.


Implementors: How to Merge <mpi++.h> and <mpi.h>

1. Put extern "C" { } around everything in <mpi.h>
2. Put a preprocesor macro around the extern "C" to ensure that it is
only used when the preprocessor macro "_cplus_plus" is defined. 
3. Insert the contents of <mpi++.h> at the end of <mpi.h> (You will
still need all the other header files that are included in <mpi++.h>,
but they can be collapsed into one file if necessary).

Figure 1 shows an example of these steps.

----------------------------------------------------------------------
#ifdef _cplus_plus
extern "C" {
#endif

/* The contents of <mpi.h> */

#ifdef _cplus_plus
}

/* The contents of <mpi++.h> */

#endif
----------------------------------------------------------------------
    Figure 1: Example of how to merge <mpi++.h> and <mpi.h>


Miscellanious Implementation Notes

COMM_NULL: The MPI-2 standard does not specify exactly how
MPI::COMM_NULL should be implemented. It only specifies that
MPI::COMM_NULL must be comparible with and assignable to instances of
MPI::Comm's derived classes.  This package implements COMM_NULL
as a singleton object of a class named Comm_Null (which is a base
class of MPI::Comm).

MPI::Status: There are several functions in the C++ bindings with two
versions, one that takes a MPI::Status object and one that does not.
Since the underlying MPI C function requires an MPI_Status handle in
its parameter list, in the cases where there is no MPI::Status object
being passed to the C++ function, the private MPI:Status instance
"ignored_status" is used.  ignored_status is declared as a static
object in each of the MPI classes that need to use it.  That is, there
is only one ignored_status for each type of MPI class that requires
it.

Profiling: When the library is compiled with profiling enabled, the
MPI:: functions only call their corresponding PMPI:: functions.  The
one exception to this rule is the Clone method.  This method currently
performs some actions in the MPI layer before invoking the underlying
PMPI:: Clone function.  It is recomended that users do not attempt to
profile this function.


Using Untested Compilers

The C++ bindings make extensize use of many of the new features of the
C++ language.  The compilers that have been used so far implement
these "newer" features with varying degrees of reliability.  Much
care has gone into making the bindings work with the compilers listed
in Table 1.  Until C++ compilers become conformant, it may be
difficult to use the bindings with other compilers.  In the meantime,
the configure script will try its best to figure out the capabilities
of your compiler and act accordingly.


IV. CONTACT INFORMATION
-----------------------

These MPI C++ bindings are available from their main distribution site:

	http://www.cse.nd.edu/~lsc/research/mpi2c++/

This distribution includes:

	- Source code for the MPI 1.2 C++ bindings (src/ subdirectory)
	- Several example programs (contrib/examples/ subdirectory)
	- All the examples from chapter 10 of the MPI-2 standard
	  (contrib/examples/chapter_10_mpi2.cc)
        - A test suite for the MPI 1.2 C++ bindings
	  (contrib/test_suite/ )

See the INSTALL file in this directory for how to build the C++
bindings library.  The INSTALL file also includes information on how
to use the bindings.

Bug reports should be sent to mpi2c++-bugs@mpi.nd.edu.  Please include
the output from configure, make, make install, and the program that
you are having problems with.

Questions, comments, suggestions, and requests for additional
information should be sent to mpi2c++@mpi.nd.edu



