Copyright (c) 2003-2020 High Performance Computing Center Stuttgart,
                        University of Stuttgart.  All rights reserved.
Copyright (c) 2005-2009 The University of Tennessee and The University
                        of Tennessee Research Foundation.  All rights
                        reserved.
Copyright (c) 2007      The Trustees of Indiana University and Indiana
                        University Research and Technology
                        Corporation.  All rights reserved.
Copyright (c) 2009      Cisco Systems, Inc.  All rights reserved.

$COPYRIGHT$

Additional copyrights may follow

$HEADER$

===========================================================================

           Documentation (beta ,-]) for the MPI-Testsuite
           ----------------------------------------------

Overview
========

1.) Introduction
2.) Getting Started
2.1.) Compiling the MPI-Testsuite
2.2.) Running the MPI-Testsuite
2.3.) MPI-implementations already tested
2.4.) Tests failing on specific platforms
3.) Extending the testsuite
3.1.) Adding new tests


Introduction
============
This MPI-testsuite was initially developed for the use with PACX-MPI and
has been extended within the Open MPI project.

The main focus is on:
- High degree of code coverage through combinations of tests.
- Easy maintainability,
- Easy integration of new tests,
- Rich underlying functionality for flexible tests
  (convenience functions for datatypes, comms and checking),
- Only a single binary (for single, since expensive
  MPI_Init/MPI_Finalize) to make it as quick and easy as possible to
  run automatically.



Getting Started
===============

Prerequisites
-------------
Required dependencies to build and run the MPI test suite:
- MPI library with c compiler support
- make

Optional dependencies for developers:
- gengetopt (used for the generation of the command line parser code)
- autoconf, automake


Compiling the MPI-Testsuite
---------------------------
The MPI-Testsuite uses an autotools based build system.
To configure provide the MPI c compiler with

>>> ./configure CC=mpicc

After a successful configure-run, one may compile with make.

>>> make


Running the MPI-Testsuite
-------------------------
The MPI-Testsuite may be run with an arbitrary number of processes.
It runs a variety of P2P and Collective tests with varying
datatypes and preset communicators. Each test specifies, which kind of
datatypes, e.g. struct-datatypes and comms, e.g. MPI_COMM_SELF,
intra-comms and alike it may run.

Per default, ALL tests will be run with ALL combinations of datatypes
and ALL generated communicators! Only failed tests are shown afterwards.

Through command-line switches, the user may influence and reduce the
number of tests with the following commands, first the showing the help:

>>> mpirun -np 1 ./mpi_test_suite -h

Usage: mpi_test_suite [OPTION]...

  -h, --help                   Print help and exit
  -V, --version                Print version and exit
  -t, --test=STRING            tests or test-classes  (default=`all')
  -c, --comm=STRING            communicators or commicator-classes
                                 (default=`all')
  -d, --datatype=STRING        datatypes of datatype-classes  (default=`all')
  -n, --num-values=STRING      number of values to communicate in tests
                                 (default=`1000')

All multiple test-/comm-/datatype-names and num-values must be comma-separated.
Names are not case-sensitive, due to spaces in names, propper quoting should be
used. The special name 'all' can be used to select all tests/comms/datatypes.
To exclude a test/comm/datatype prefix it with '^' but be aware, that the
selection will happen in order, so use 'all,^exclude'.

  -a, --atomic-io              enable atomicity for files in I/O for all tests
                                 that support it
  -j, --num-threads=INT        number of additional threads to execute the
                                 tests  (default=`0')
  -r, --report=STRING          level of detail for test report  (possible
                                 values="summary", "run", "full"
                                 default=`summary')
  -x, --execution-mode=STRING  level of correctness testing  (possible
                                 values="disabled", "strict", "relaxed"
                                 default=`relaxed')
  -v, --verbose                enable verbose output for debugging purpose
  -l, --list                   list all available tests, communicators,
                                 datatypes and corresponding classes


The following command lists all available tests/test-classes, comms/comm-classes and
type/type-classes:

>>> mpirun -np 1 ./mpi_test_suite -l

Showing all the individual tests, comms and types would be too long, however,
all of the tests, comms and types are grouped in classes, all of which are
listed with (first the class, followed by number, then a free distinct name):

Num Tests : 100
Environment test:0 Status
Environment test:1 Request_Null
P2P test:2 Ring
...
P2P test:27 Alltoall with topo comm
Collective test:28 Bcast
...
Collective test:46 Alltoall
One-sided test:47 get_fence
...
One-sided test:58 Ring with Put
IO test:59 file simple
...
IO test:99 io_with_hole2
Test-Class:0 Unspecified
Test-Class:1 Environment
Test-Class:2 P2P
Test-Class:3 Collective
Test-Class:4 One-sided
Test-Class:5 Dynamic
Test-Class:6 IO
Test-Class:7 Threaded
Communicator:0 MPI_COMM_WORLD
...
Communicator:12 Intracomm merged of the Halved Intercomm
Communicator-Class:0 COMM_SELF
Communicator-Class:1 COMM_NULL
Communicator-Class:2 INTRA_COMM
Communicator-Class:3 INTER_COMM
Communicator-Class:4 CART_COMM
Communicator-Class:5 TOPO_COMM
Datatype:0 MPI_CHAR
...
Datatype:30 function MPI_Type_dup() n/a
Datatype-Class:0 STANDARD_C_INT_TYPES
Datatype-Class:1 STANDARD_C_FLOAT_TYPES
Datatype-Class:2 STANDARD_C_TYPES
Datatype-Class:3 STRUCT_C_TYPES
Datatype-Class:4 ALL_C_TYPES
Report:0 Summary
Report:1 Run
Report:2 Full
Test mode:0 disabled
Test mode:1 strict
Test mode:2 relaxed


For example, the command

>>> mpirun -np 32 ./mpi_test_suite -t "Ring\ Bsend,Collective" -c COMM_SELF,INTER_COMM -d MPI_CHAR

will run the test called "Ring Bsend" and ALL test-cases belonging to
test-class Collective with the communicator MPI_COMM_SELF and
ALL communicators which are belonging to communicator-class INTER_COMM
(e.g. the comm called "Zero-and-Rest Intercommunicator") *but only* with
the datatype MPI_CHAR.
(However, You may also just specify the corresponding number ,-]).

If no options are specified, ALL tests are run with all applicable
communicators and all applicable datatypes. Of course, if a test is not
applicable for a certain combination (e.g. "Ring" doesn't support
MPI_COMM_NULL) it is not being run.


MPI-implementations already tested
----------------------------------
We have run the testsuite successfully on
  Linux/IA32 using MPIch-1.2.7
  Linux/IA32 using LAM-7.0
  Linux/IA32 using PACX-MPI-5.0 on top of MPIch-1.2.5
  Linux/EM64t using MPIch-1.2.4
  NEC SX6 using MPIsx
  NEC SX8 using MPIsx
  Hitachi
  SGI Origin


Tests failing on specific platforms
-----------------------------------
On some platforms, we found, that some tests fail:
- With MPIch-1.2.5 and the ch_shmem-device, we trigger a bug in "Ring Ssend", which seems to be known:

  P2P tests Alltoall - Issend (9/14), comm Zero-and-Rest Intercommunicator (8/10), type MPI_CHAR (1/18)
  [0] MPI Internal Aborting program Nested call to GetSendPkt
  [0] Nested call to GetSendPkt

  The source of MPIch-1.2.5/mpid/ch_shmem/shmempriv.c, line 590 explains here:
                /* There is an implementation bug in the flow control
                   code that can cause MPID_DeviceCheck to call a
                   routine that calls this routine. When that happens,
                   we'll quickly drop into the same code, so we prefer
                   to abort.  The test is here because if we find
                   a free packet, it is ok to enter this routine,
                   just not ok to enter and then call DeviceCheck */
                if (nest_count++ > 0) {
                    MPID_Abort( 0, 1, "MPI Internal",
                                "Nested call to GetSendPkt" );
                }



Extending the testsuite
=======================

Adding new tests
----------------

In order to integrate a new test, the programmer should
  - write three functions in a new file with a descriptive name, like:
      tst_p2p_simple_ring_bsend.c
    containing:
      tst_p2p_simple_ring_bsend_init
      tst_p2p_simple_ring_bsend_run
      tst_p2p_simple_ring_bsend_cleanup
    where all names start with "tst_",
    and the kind of communication calls tested (actually a TST_CLASS)
    and the communication pattern (here a simple ring using MPI_Bsend).
 - Add the new file to the list of sources for the mpi_test_suite target in Makefile.am
   and update the build system with
   >>> autoreconf -vif

 - Add the new functions to mpi_test_suite.h in a sorted manner
 - Add the test-description to the tst_tests-array in tst_test.c:
   Here, the
   struct tst_test {
     int class;
     char * description;
     int run_with_comm;
     tst_int64 run_with_type;
     int needs_sync;
     int (*tst_init_func) (const struct tst_env * env);
     int (*tst_run_func) (const struct tst_env * env);
     int (*tst_cleanup_func) (const struct tst_env * env);
   };
   describes with
     class:            Which kind of test is being run
                       (one of TST_CLASS_ENV, TST_CLASS_P2P, TST_CLASS_COLL, TST_CLASS_THREADED),
     description:      A short notion of what is being done,
     run_with_comm:    An OR-ed list of which communicators may be used within the test
                       (several of TST_MPI_COMM_SELF, TST_MPI_COMM_NULL, TST_MPI_INTRA_COMM,
                       TST_MPI_INTER_COMM, TST_MPI_CART_COMM, TST_MPI_TOPO_COMM).
     run_with_type:    Similar to the above, a OR-ed list of type being one/several of:
                       TST_MPI_CHAR, TST_MPI_UNSIGNED_CHAR, ..., TST_MPI_INT, ... or even
                       sets like TST_MPI_STANDARD_C_INT_TYPES or TST_MPI_STRUCT_FORTRAN_TYPES
     needs_sync:       If non-zero a MPI_Barrier is done before and after the test, if the
                       test's communication may not intermingle with other (or own) test due
                       to MPI_ANY_SOURCE or MPI_ANY_TAG or alike.
     tst_init_func:    The initialization function run once before the test to allocate arrays.
     tst_run_func:     The test-function itself, run once or many times!
     tst_cleanup_func: The clean-up function run once after the test to release arrays and buffers.


   For example the Buffered-Send test using a Ring-topology of communication is declared like this:
    {TST_CLASS_P2P, "Ring Bsend",
     TST_MPI_INTRA_COMM,
     TST_MPI_ALL_C_TYPES,
     TST_NONE,
     &tst_p2p_simple_ring_bsend_init, &tst_p2p_simple_ring_bsend_run, &tst_p2p_simple_ring_bsend_cleanup},


   These functions are passed a struct tst_env, which contains:
   struct tst_env {
     int comm;
     int type;
     int test;
     int values_num;
   };

   An identifier for the communicator, type, test and the number of values to be communicated.
   The respective MPI-objects may be retrieved with the corresponding functions, e.g.
     mpi_comm = tst_comm_getcomm (env->comm);



These tests must adhere to the following rules:

1.) They must run with an arbitrary number of processes!
    Even if it means, that some processes are just hanging in the MPI_Barrier if it needs_sync!

2.) They MUST adhere to their specification in tst_tests.c!

3.) They should run with as many communicators as possible (like declaring run_with_comm:
    (TST_MPI_COMM_SELF | TST_MPI_COMM_NULL | TST_MPI_INTRA_COMM | TST_MPI_INTER_COMM) or more ,-]

4.) They should run with as many types as possible (see above).

5.) Every call to MPI-functions should be wrapped with the MPI_CHECK macro, which
    checks for the return value... This doesn't buy much, but anyway.

6.) Every communicated data should be initialized with the tst_type_setstandardarray
    and checked with tst_type_checkstandardarray!