Index: trunk/anuga_core/source/pypar_dist/README
===================================================================
--- trunk/anuga_core/source/pypar_dist/README	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/README	(revision 8494)
@@ -0,0 +1,400 @@
+------------------------------------------------
+PyPAR - Parallel Python, efficient and scalable
+parallelism using the message passing interface (MPI).
+
+Author:  Ole Nielsen (2001 -)
+Email:   Ole.Nielsen@anu.edu.au
+Version: See pypar.__version__
+Date:    See pypar.__date__
+
+Major contributions by
+    Gian Paolo Ciceri (gp.ciceri@acm.org)
+    Prabhu Ramachandran (prabhu@aero.iitm.ernet.in)
+
+Minor but important contributions by
+    Doug Orr (doug@arbor.net)
+    Michal Kaukic (mike@frcatel.fri.utc.sk)
+
+------------------------------------------------
+
+The python module pypar.py and the C-extension mpi.c
+implements scalable parallelism on distributed and shared
+memory architectures using the essential subset of
+the Message Passing Interface (MPI) standard.
+
+FEATURES
+
+- Python interpreter is not modified:
+  Parallel python programs need only import the pypar module.
+
+- Easy installation: This is essentially about compiling and linking
+  the C-extension with the local MPI installation. A distutils setup file
+  is included.
+
+- Flexibility: Pypar allows communication of general Python objects
+  of any type.
+
+- Intuitive API:
+  The user need only specify what to send and to which processor.
+  Pypar takes care of details about
+  data types and MPI specifics such as tags, communicators and buffers.
+  Receiving is analogous.
+
+- Efficiency:
+  Full bandwidth of C-MPI programs is achieved for consecutive Numerical
+  arrays. Latency is less than twice that of pure C-MPI programs.
+  Test programs to verify this are included (pytiming, ctiming.c)
+
+- Lightweight:
+  Pypar consists of just two files: mpiext.c and pypar.py
+
+See the DOC file for instructions on how to program with pypar.
+
+
+
+PRE-REQUISITES (on all nodes)
+
+  Python 2.0 or later
+  Numeric Python (incl RandomArray) matching the Python installation
+  Native MPI C library
+  Native C compiler
+
+  Pypar has been tested on the following platforms
+    OPENMPI with Linux on Opteron 64 bit
+    OPENMPI with Ubuntu Linux (32 bit)
+    MPI on DEC Alpha
+    LAM/MPI (6.5.6, 7.0) on Linux (Debian Woody/Cid and Red Hat 7.1)
+    MPICH and LAM 7.0 on Solaris (Sun Enterprise)
+    MPICH on Linux
+    MPICH on Windows (NT/2000)
+    MPICH for Darwin (Mac OS)
+    LAM/MPI with Linux on Opteron 64 bit
+
+  There are currently issues with MPICH2, so we recommend using OPENMPI
+  On Ubuntu systems you need to install the following 2 packages:
+  apt-get install openmpi-bin
+  apt-get install libopenmpi-dev
+
+
+INSTALL
+
+  UNIX PLATFORMS
+    Type
+      python setup.py install
+
+    This should install pypar and its extension in the default
+    site-packages directory on your system.
+
+    If you wish to install it in your home directory use
+
+      python setup.py install --prefix=~
+
+  WINDOWS (NT/2000, MPICH)
+    To build, you need:
+      1) MPICH (http://www-unix.mcs.anl.gov/mpi/mpich/)
+      2) MinGW (http://www.mingw.org). Tested with GCC v 2.95-3.6.
+      3) Set MPICH_DIR to the appropriate directory
+      4) Build using 'python setup.py build --compiler=mingw32'
+      5) Install using 'python setup.py install'.
+
+  ---
+  If you encountered any installation problems, read on.
+  Do not worry about messages about unresolved symbols. This is normal
+  for shared libraries on some machines.
+
+  To compile C-extension mpi.c requires either a mpi-aware c compiler
+  such as mpicc or a standard c compiler with mpi libraries linked in.
+  See local MPI installation for details, possibly edit Makefile and
+  type make.
+  See installation notes below about compiling under mpi.
+
+
+
+COMPILE PYPAR LOCALLY
+
+  If you don't want to install pypar using distutils or if you encounter
+  problems doing so, you can compile it locally using the the script
+  compile_pypar_locally.py (located in the pypar source directory)
+  is a non-distutils way of compiling the extension and will put the
+  mpiext.so file in the current working dir.
+  With a PATH to that dir, you can run your pypar programs using the
+  local installation.
+
+
+
+TESTING
+  Pypar comes with a number of tests and demos available in the
+  examples directory - please try these to verify the installation.
+
+  See also mandelbrot examples (in demos/mandelbrot_example) using different parallel
+  approaches to compute the well-known fractal in parallel.
+  Before running any of these you must compile their C extensions. This is done as follows:
+
+  python compile_mandelbrot_c_extensions.py
+
+
+  In particular the script mandel_parallel_cyclic.py should produce almost perfect speedup.
+  To run it on 2 processors, write
+
+  mpirun -np 2 python mandel_parallel_dynamic.py
+
+
+
+
+RUNNING PYTHON MPI JOBS
+
+  Pypar runs in exactly the same way as MPI programs written in
+  C or Fortran.
+
+  E.g. to run the enclosed demo script (demo.py) on 4 processors,
+  enter a command similar to
+
+    mpirun -np 4 python demo.py
+
+  Consult your MPI distribution for exact syntax of mpirun.
+  Sometimes it is called prun and often parallel jobs will be
+  submitted in batch mode.
+
+  You can also execute demo.py as a stand-alone executable python script
+
+    mpirun -np 4 demo.py
+
+
+  Enclosed is a script to estimate the communication speed of your system.
+  Execute
+
+    mpirun -np 4 pytiming
+
+  Care has been taken in pypar to achieve the same bandwidth and almost as
+  good communication latency as corresponding C/MPI programs.
+  Please compile and run ctiming.c to see the reference values:
+
+    make ctiming
+    mpirun -np 4 ctiming
+
+  Note that timings might fluctuate from run to run due to variable
+  system load.
+
+  An example of a master-slave program using pypar is available in demo2.py.
+
+
+INSTALLATION NOTES (If all else fails)
+
+Most MPI implementations provide a script or an executable called
+"mpicc" which compiles C programs using MPI and does
+not require any explicitly mentioned libraries.
+If such a script exists, but with a different name, change
+the name in the beginning of compile.py. If no such script exists, put
+the name of your C compiler in that place and add all required linking
+options yourself.
+
+For example, on an Alpha server it would look something like
+
+ cc -c mpi.c -I/opt/Python-2.1/include/python2.1/
+ cc -shared mpi.o -o mpi.so -lmpi -lelan
+
+ or using the wrapper
+
+ mpicc -c mpi.c -I/opt/Python-2.1/include/python2.1/
+ mpicc -shared mpi.o -o mpi.so
+
+On Linux (using LAM-MPI) it is
+
+ mpicc -c mpi.c -I/opt/Python-2.1/include/python2.1/
+ mpicc -shared mpi.o -o mpi.so
+
+ Start processors using
+ lamboot -v lamhosts
+
+On Linux (using OPENMPI) the compilation and executions steps are as with LAM-MPI, but
+there is no need for lamboot anymore. With OPENMPI you may need to uncomment the line
+
+  btl = ^openib
+
+in /etc/openmpi/openmpi-mca-params.conf to get rid of the warning "OpenIB on host ... was unable to find any HCAs." Another warning, "unable to open osc pt2pt: file not found (ignored)", is a known openmpi problem with an unknown solution.
+See http://code.google.com/p/petsc4py/wiki/InstallationNotes for more details.
+
+
+
+HOW DOES PYPAR WORK?
+
+Pypar works as follows
+    1  mpirun starts P python processes as specified by mpirun -np P
+    2  each python process imports pypar which in turn imports a shared
+       library (python extension mpiext.so) that has been statically linked
+       to the C MPI libraries using e.g. mpicc (or cc -lmpi)
+    3  The Python extension proceeds to call MPI_init with any commandline
+       parameters passed in by mpirun (As far as I remember MPICH uses this
+       to identify rank and size, whereas LAM doesn't)
+    4  The supported MPI calls are made available to Python through the
+       pypar interface
+
+    If  pypar is invoked sequentially it supports a rudimentary interface
+    with e.g. rank() == 0 and size() == 1
+
+
+
+DOCUMENTATION
+
+  See the file DOC for an introduction to pypar.
+  See also examples demo.py and pytiming
+  as well as documentation in pypar.py.
+
+
+HISTORY
+
+See release_notes.txt for newer releases!
+
+version 2.0.1_alpha (11/7/7) Setup patch by jk Nilsen and removal
+of obsolete code
+version 2.0alpha (3/7/7) Migration to numpy and patches to setup
+Also started Subversion repository on Sourceforge.
+
+version 1.9.3 (24 April 2007)
+  added bypass form for reduce
+version 1.9.2
+  First version under Subversion
+version 1.9.1 (15 Dec 2003)
+  Consistent naming (all lower case, no abbreviations)
+version 1.9 (3 Dec 2003)
+  Obsoleted raw forms of communication as they were confusing.
+  The semantics of send, receive, scatter, gather etc is now that
+  one can optionally specify a buffer to use if desired.
+  Bypass forms added.
+
+version 1.8.2 (16 November 2003)
+  Fixed scatter, gather and reduce calls to automatically
+  figure out buffer lengths. Also tested scatter and gather for
+  complex and multidimensional data.
+version 1.8.1 (13 November 2003)
+  Added direct support (i.e. not using the vanilla protocol)
+  for multidimensional Numeric arrays of type Complex.
+version 1.8 (11 November 2003)
+  Changed status object to be (optionally) returned from receive and
+  raw_receive rather maintaining a global state where one instance of
+  the status object is modified.
+  This was suggested by the audience at presentation at Department of
+  Computer Science, Australian National University, 18 June 2003.
+
+version 1.7 (7 November 2003)
+  Added support for Numerical arrays of arbitrary dimension
+  in send, receive and bcast.
+  The need for this obvious functionality was pointed by Michal Kaukic.
+
+version 1.6.5 (30 August 2003)
+  Added NULL commandline argument as required by LAM 7.0 and
+  identified by Doug Orr and Dave Reed.
+version 1.6.4 (10 Jan 2003)
+  Comments and review of installation
+version 1.6.3 (10 Jan 2003)
+  Minor issues and clean-up
+version 1.6.2 (29 Oct 2002)
+  Added Windows platform to installation as contributed by
+  Simon Frost
+version 1.6.0 (18 Oct 2002)
+  Changed installation to distutils as contributed by
+  Prabhu Ramachandran
+
+version 1.5 (30 April 2002)
+  Got pypar to work with MPICH/Linux and cleaned up initialisation
+version 1.4 (4 March 2002)
+  Fixed-up and ran testpypar on 22 processors on Sun
+version 1.3 (21 February 2002)
+  Added gather and reduce fixed up testpypar.py
+
+version 1.2.2, 1.2.3 (17 February 2002)
+  Minor fixes in distribution
+version 1.2.1 (16 February 2002)
+  Status block, MPI_ANY_TAG, MPI_ANY_SOURCE exported
+Version 1.2 (15 February 2002)
+  Scatter added by Gian Paolo Ciceri
+Version 1.1 (14 February 2002)
+  Bcast added by Gian Paolo Ciceri
+Version 1.0.2 (10 February 2002)
+  Modified by Gian Paulo Ciceri to allow pypar run under Python 2.2
+Version 1.0.1 (8 February 2002)
+  Modified to install on SUN enterprise systems under Mpich
+Version 1.0 (7 February 2002)
+  First public release for Python 2.1 (OMN)
+
+
+KNOWN BUGS
+  Scatter for the moment works only properly when the amount of data is a
+  multiple of the number of processors (as does the underlying MPI_Scatter).
+  I am working on a more general scatter (and gather) which will
+  distribute data as evenly as possible for all amounts of data.
+
+
+LICENSE
+    This program is free software; you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation; either version 2 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License (http://www.gnu.org/copyleft/gpl.html)
+    for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with this program; if not, write to the Free Software
+    Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307
+
+
+ Contact address: Ole.Nielsen@anu.edu.au
+
+ =============================================================================
+
+ACKNOWLEDGEMENTS
+  This work was supported by School of Mathematical Sciences at
+  the Australian National University and funded by Australian
+  Partnership for Advanced Computing (APAC).
+  Many thanks go to
+
+    Gian Paolo Ciceri (gp.ciceri@acm.org)
+        for fixing pypar to run under Python 2.2 and for adding all the
+        collective communication stuff from version 1.1 and onwards.
+    Prabhu Ramachandran (prabhu@aero.iitm.ernet.in)
+        for making a proper distutils installation procedure
+    Simon D. W. Frost (sdfrost@ucsd.edu) and Vladimir Lazunin (lazunin@gmail.com)
+        for testing pypar under Windows and creating the installation procedure
+    Jakob Schiotz (schiotz@fysik.dtu.dk) and Steven Farcy (steven@artabel.net)
+        for pointing out initial problems with pypar and MPICH.
+    Markus Hegland (Markus.Hegland@anu.edu.au)
+        for supporting the work.
+    Doug Orr (doug@arbor.net)
+        for identifying and fixing problem with LAM 7.0 requiring
+	last commandline argumnet to be NULL.
+    Dave Reed (dreed@capital.edu)
+        for pointing out problem with LAM 7.0 and for testing the fix.
+    Matthew Wakefield (matthew.wakefield@anu.edu.au)
+    	for pointing out and fixing support for Mac OS X (darwin)
+    David Brown (David.L.Brown@kla-tencor.com)
+    	for contributing to the Windows/Cygwin installation procedure
+    Michal Kaukic (mike@frcatel.fri.utc.sk)
+	for pointing out the need for multi dimensional arrays and
+	for suggesting an implementation in the 2D case.
+    J K Nilsen for support and compiler patches
+    Ross Wilson for designed the pypar logo in January 2009
+    Andrew Rowe (mr_andrew_d_rowe@hotmail.com) for fixing windows installation
+        in March 2011.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Index: trunk/anuga_core/source/pypar_dist/TODO
===================================================================
--- trunk/anuga_core/source/pypar_dist/TODO	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/TODO	(revision 8494)
@@ -0,0 +1,74 @@
+PYPAR TODO LIST:
+----------------
+	
+Check that code for numpy migration is OK (Vladimir and Prabhu?) 	
+
+Check that pypar works properly with numpy on a variety of platforms (Everyone)
+
+Update setup.py to work with new structure (Prabhu?)
+
+Update create_release.py to make distros for sourceforge (Ole)	
+	
+Find a TRAC server to manage this list (and point to the sourceforg SVN)
+
+Remove obsoleted code (e.g. raw forms)
+	
+Fix up reduce to deal with complex as well as ND.	
+	
+New tests using unit testing framework.
+
+Better documentation (LaTeX files in pypar/documentation/manuals 
+should provide a good starting point).
+
+Use proper Python exceptions in mpiext.c
+
+Create Debian and RPM packages (maybe even a Windows installer)
+
+Make list of projects using pypar, e.g. http://sourceforge.net/projects/pyfdm/	
+
+
+Incorporate pypar in Parallel Knoppix (Michael Creel)
+	
+Add examples from Summerschool 2005 to pypar distribution.
+		
+Re-introduce optional length if one wants to communicate less 
+than given array (very low priotity)
+
+max_tag is set to 32767. This works for Linux/LAM      
+I couldn't use MPI_TAG_UB as it returned 0.
+I would like a general solution.
+
+
+Scatter needs to send buffer specified on all processes
+even though it is ignored by all non-root processes.
+How could we overcome that?
+Similar problem for gather
+  
+Gather and scatter: One should be able to specify along which axis
+arrays should be concatenated.
+
+
+
+REMEMBER
+
+mpirun -np 3 python -i -c "import pypar"
+Remember to put pypar.pth in site-packages automatically..
+
+
+DONE:
+-----
+
+Migration to numpy
+Address 2D arays as per mails (dir: 2D)	- DONE version 1.7
+Fix up status object to be returned (depending on optional flag arg)	
+
+Get pypar to work under redhat 7.1 with LAM as per suggestion by Doug Orr
+(LAM assumes that last commandline arg is None) See fra_DOUG.husk3.
+ALso acknowledge Dave Reed for pointing out the problem. DONE.
+Wait for dave reed to acknowledge that it also works under solaris.
+DONE in 1.6.5
+
+
+Also add to doc that pypar needs Numeric (and Python 2.3?) (Already in there)
+
+Fix confusing errmsg when Numeric isn't installed (David Brown)
Index: trunk/anuga_core/source/pypar_dist/anuga_setup.py
===================================================================
--- trunk/anuga_core/source/pypar_dist/anuga_setup.py	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/anuga_setup.py	(revision 8494)
@@ -0,0 +1,12 @@
+
+import os
+anuga_source = os.path.split(os.path.abspath(os.getcwd()))[0]
+
+
+cmd = 'python setup.py install --install-purelib=%s --install-platlib=%s '%( anuga_source , anuga_source)
+
+print cmd
+
+os.chdir('source')
+os.system (cmd)
+
Index: trunk/anuga_core/source/pypar_dist/create_release.py
===================================================================
--- trunk/anuga_core/source/pypar_dist/create_release.py	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/create_release.py	(revision 8494)
@@ -0,0 +1,257 @@
+"""Create a release of pypar
+
+The release name will be compose of a major release code obtained from
+pypar.py as well as the subversion version number.
+For example
+
+pypar-2.0.2_39
+
+refers to pypar version 2.0.2 with subversion number 39.
+
+The release files will be store locally in ~ (home) and also
+copied to sourceforge where the release manager must log in
+and create the Release using the File Releases interface.
+
+This script assumes a Linux platform
+
+"""
+
+from os import sep, system, remove, popen, chdir, getcwd, listdir
+from os.path import join
+from tempfile import mktemp
+from sys import platform, stdout
+
+
+
+def get_revision_number():
+    """Get the version number of the SVN
+    NOTE: This requires that the command svn is on the system PATH
+    (simply aliasing svn to the binary will not work)
+    """
+
+    # Error msg
+    msg = 'Command "svn" is not '
+    msg += 'recognised on the system PATH.\n\n'
+    msg += 'Try to obtain the version info '
+    msg += 'by using the command: "svn info".\n'
+    msg += 'In this case, make sure svn is accessible on the system path. '
+    msg += 'Simply aliasing svn to the binary will not work. '
+      
+    try:
+        # The null stuff is so this section fails quitly.
+        # This could cause the svn info command to fail due to
+        # the redirection being bad on some platforms.
+        # If that occurs then change this code.
+        if platform[0:3] == 'win':
+            system('svn up')
+            fid = popen('svn info 2> null')
+        else:
+            system('svn up')            
+            fid = popen('svn info 2>/dev/null')
+	
+    except:
+        raise Exception(msg)
+    else:
+        #print 'Got version from svn'            
+        version_info = fid.read()
+      
+        if version_info == '':
+          raise Exception(msg)    
+        else:
+          pass
+          print 'Got version from file'
+
+            
+    for line in version_info.split('\n'):
+        if line.startswith('Revision:'):
+            break
+
+    fields = line.split(':')
+    msg = 'Keyword "Revision" was not found anywhere in text: %s'\
+          %version_info
+    assert fields[0].startswith('Revision'), msg            
+
+    try:
+        revision_number = int(fields[1])
+    except:
+        msg = 'Revision number must be an integer. I got %s' %fields[1]
+        msg += 'Check that the command svn is on the system path' 
+        raise Exception(msg)                
+        
+    return revision_number
+
+
+
+if __name__ == '__main__':
+  
+    if platform == 'win32':
+        msg = 'This script is not written for Windows.'+\
+              'Please run it on a Unix platform'
+        raise Exception, msg
+
+    # Get version number from file __metadat__
+    # This can't be imported because source is a package
+    # and mpiext needs to be compiled before anything can
+    # be imported.
+    # Instead we read it manually (until a better solution is found)
+
+    fid = open('source/__metadata__.py')
+    major_revision = None
+    for line in fid.readlines():
+        if line.startswith('__version__'):
+            i = line.find('=')
+            major_revision = str(line[i+1:].strip())[1:-1]
+            
+    if major_revision is None:
+        raise 'No version was found'
+
+    
+
+    # line separator 
+    lsep = '----------------------------------------------------------------------'
+
+    # Get svn revision number and create
+    # file with version info for release.
+    # This will mean that the version currently checked out is
+    # the one which will be released.
+
+    svn_revision = get_revision_number()
+    revision = '%s_%s' %(major_revision, svn_revision)
+    print 'Creating pypar revision %s' %revision
+
+    distro_filename = 'pypar-%s.tgz' %revision
+
+    
+    
+    # Create area directory
+    release_name = 'pypar_%s' %revision 
+    release_dir = '~/%s' %release_name 
+    s = 'mkdir %s' %release_dir
+    try:
+        print s    
+        system(s)
+    except:
+        pass
+
+
+    # Export a clean directory tree from the working copy to a temporary dir
+    tmp_dir = mktemp()
+    s = 'mkdir %s' %tmp_dir
+    print s    
+    system(s)
+    
+    distro_dir = join(tmp_dir, release_name) 
+    s = 'mkdir %s' %distro_dir
+    print s    
+    system(s)    
+
+    
+    
+
+    #-----------------------------
+    # Get pypar source
+    s = 'svn export -r %d source %s/source' %(svn_revision,
+                                              distro_dir) 
+    print s
+    system(s)
+    
+    #-----------------------------
+    # Copy license file to top dir   
+    s = 'cp %s/source/LICENSE %s' %(distro_dir, distro_dir)     
+    print s
+    system(s)    
+   
+    
+    #-----------------------------
+    # Get demos
+    s = 'svn export -r %d demos %s/demos' %(svn_revision,
+                                            distro_dir) 
+    print s
+    system(s)
+
+    
+    #-----------------------------
+    # Get documentation
+    s = 'svn export -r %d documentation %s/documentation' %(svn_revision,
+                                                            distro_dir) 
+    print s
+    system(s)
+    
+
+
+    # Zip everything up
+    s = 'cd %s; tar cvfz %s *' %(tmp_dir, distro_filename)
+    print s
+    system(s)
+
+    # Move distro to release area
+    s = '/bin/mv %s/*.tgz %s' %(tmp_dir, release_dir) 
+    print s
+    system(s)
+
+    # Clean up
+    s = '/bin/rm -rf %s/pypar' %(tmp_dir) 
+    print s
+    system(s)
+
+
+    #-----------------------------
+
+    print 'Done'
+    print
+    print
+    print lsep
+    print 'The release files are in %s:' %release_dir
+    system('ls -la %s' %release_dir)
+    print lsep
+    print
+    print
+
+    answer = raw_input('Do you want to upload this to sourceforge? Y/N [Y]')
+    if answer.lower() != 'n':
+        
+        #------------------------------
+        print 'Uploading to sourceforge'
+
+
+        import os, os.path
+        release_dir = os.path.expanduser(release_dir)
+        os.chdir(release_dir)
+        print 'Reading from', os.getcwd()
+
+        s = 'rsync -avP -e ssh *.tgz uniomni@frs.sourceforge.net:uploads/'
+        print s
+        os.system(s)
+
+        #from ftplib import FTP
+        #ftp = FTP('upload.sourceforge.net')
+        #print ftp.login() # Anonymous
+        #print ftp.cwd('incoming')
+        #
+        #for filename in os.listdir('.'):
+        #    print 'Uploading %s... ' %filename,
+        #    stdout.flush()
+        #
+        #    fid=open(filename, 'rb')
+        #    print ftp.storbinary('STOR %s' %filename, fid)
+        #    fid.close()
+        #
+        #print 'FTP done'
+        #print ftp.quit()
+
+        print
+        print lsep
+        print '    ********************* NOTE *************************'
+        print lsep
+        print 'To complete this release you must log into'
+        print 'http://sourceforge.net/projects/pypar as admin'
+        print 'and complete the process by selecting File Releases '
+        print 'in the admin menu there.'
+        print lsep
+        print
+        print
+        
+
+
+
+
Index: trunk/anuga_core/source/pypar_dist/demos/bsend_demo.py
===================================================================
--- trunk/anuga_core/source/pypar_dist/demos/bsend_demo.py	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/demos/bsend_demo.py	(revision 8494)
@@ -0,0 +1,412 @@
+# -*- coding: iso-8859-1 -*-
+
+#########################################################################
+#########################################################################
+
+__title__ = "bsend examples"
+__author__ = "Constantinos Makassikis"
+__date__ = "18/09/2007"
+
+import numpy
+import string
+import time
+import sys, os
+try:
+	import pypar
+except:
+	raise 'Could not find module pypar'
+
+#
+# Hereafter, can be found a set of functions implementing a broadcast.
+#
+# Besides being used for testing purposes, they illustrate different
+# ways of using 'bsend()' function. w/o and w/ buffers.
+# 
+# Understanding how 'MPI_Bsend()' is used in MPI might be useful. 
+#
+
+#
+# In short, when one wants to send data using 'MPI_Bsend()' he has to 
+# allocate a buffer big enough to host all the data that needs to be
+# send.
+#
+# buffer = malloc();
+# MPI_Attach(buffer);
+# MPI_Bsend(buffer, data1, ...);
+# MPI_Bsend(buffer, data2, ...);
+# MPI_Detach();
+# free(buffer);
+#
+
+def simple_broadcast_v1(x, root, myid, numprocs):
+	""" Broadcast implementation using bsend() w/o explicitly defining 
+	receive buffers.
+
+	Input parameters:
+	 - x: data to broadcast.
+	 - root: rank of the process that initiates the broadcast.
+	 - myid: rank of the process calling the function.
+
+	Return value: the broadcasted data.
+	"""
+
+	list = range(0, root) + range(root + 1, numprocs)
+
+	# The process of rank 'root' sends 'x' to all the other processes.
+	if root == myid:
+		for i in list:
+
+			# Determine automatically the size of bsend's buffer.
+			pypar.push_for_alloc(x)						
+
+			# Allocate and attach bsend's buffer.	
+			pypar.alloc_and_attach()							
+
+			# Send data to process of rank 'i'.
+			pypar.bsend(x, i)				
+
+		 	# Detach and deallocate bsend's buffer.
+			pypar.detach_and_dealloc()							
+
+		x_ = x
+
+	# All processes with rank distinct from 'root' start receiving.
+	else:
+		x_ = pypar.receive(root)
+		
+	return x_
+
+def simple_broadcast_v2(x, root, myid, numprocs):
+	""" Optimized version of 'simple_broadcast_v1()'.
+
+	Since every bsend needed to achieve the broadcast requires a buffer
+	of the same size, it is sufficient to allocate that buffer once 
+	and to use it for all bsend calls.
+
+	This illustrates the use of functions alloc(), attach(), dealloc()
+	detach() instead of mpi_alloc_and_attach() and 
+	mpi_detach_and_dealloc(). As it can be seen the former alternatives
+	give a finer control.
+
+	Note: a call to 
+	
+	...
+	alloc_and_attach()
+	...
+
+	is equivalent to:
+
+	...
+	alloc()
+	attach()
+	...
+
+	Input parameters:
+	 - x: data to broadcast.
+	 - root: rank of the process that initiates the broadcast.
+	 - myid: rank of the process calling the function.
+
+	Return value: the broadcasted data.
+	"""
+	list = range(0, root) + range(root + 1, numprocs)
+
+	# The process of rank 'root' sends 'x' to all the other processes.
+	if root == myid:
+		
+		# Determine automatically the size of bsend's buffer.
+		pypar.push_for_alloc(x)						
+
+		# Allocate bsend's buffer.
+		pypar.alloc()							
+
+		for i in list:
+
+			# Attach bsend's buffer.
+			pypar.attach()							
+
+			# Send data to process of rank 'i'.
+			pypar.bsend(x, i)				
+
+		 	# Detach bsend's buffer.
+			pypar.detach()							
+
+		x_ = x
+
+		# Deallocate bsend's buffer.
+		pypar.dealloc()							
+
+	# All processes with rank distinct from 'root' start receiving.
+	else:
+		x_ = pypar.receive(root)
+		
+	return x_
+
+def simple_broadcast_v3(x, root, myid, numprocs):
+	""" Alternative implementation to 'simple_broadcast_v1()'.
+
+	Make all bsend calls using a single buffer and a
+	single call to 'mpi_alloc_and_attach()' and 'mpi_detach_and_dealloc()'.
+
+	This implementation illustrates how to use 'push_for_alloc()' in order
+	to allocate a single buffer, big enough, to handle all sends within
+	one call to 'mpi_alloc_and_attach()' and 'mpi_detach_and_dealloc()'.
+
+	Input parameters:
+	 - x: data to broadcast.
+	 - root: rank of the process that initiates the broadcast.
+	 - myid: rank of the process calling the function.
+
+	Return value: the broadcasted data.
+	"""
+
+	list = range(0, root) + range(root + 1, numprocs)
+
+	# The process of rank 'root' sends 'x' to all other processes.
+	if root == myid:
+		# Determine the size of bsend's buffer.
+		for i in list:
+			pypar.push_for_alloc(x)						
+
+		# Allocate and attach bsend's buffer.	
+		pypar.alloc_and_attach()							
+
+		# Make all sends.
+		for i in list:
+			pypar.bsend(x, i)				
+
+		# Deallocate and detach bsend's buffer.
+		pypar.detach_and_dealloc()							
+
+		x_ = x
+
+	# All processes with rank distinct from 'root' start receiving.
+	else:
+		x_ = pypar.receive(root)
+		
+	return x_
+
+def simple_broadcast_v4(x, root, myid, numprocs, buffer):
+	""" Broadcast implementation using bsend() w/ explicit definition of
+	receive buffers.
+
+	Input parameters:
+	 - x: data to broadcast.
+	 - root: rank of the process that initiates the broadcast.
+	 - myid: rank of the process calling the function.
+	 - buffer: well-dimensioned user-defined buffer for receiving 'x'.
+
+	Return value: the broadcasted data.
+	"""
+	list = range(0, root) + range(root + 1, numprocs)
+
+	# The process of rank 'root' sends 'x' to all the other processes.
+	if root == myid:
+		for i in list:
+
+			# Determine automatically the size of bsend's buffer.
+			pypar.push_for_alloc(x, use_buffer=True)
+
+			# Allocate and attach bsend's buffer.	
+			pypar.alloc_and_attach()							
+
+			# Send data to process of rank 'i'.
+			pypar.bsend(x, i, use_buffer=True)				
+
+		 	# Detach and deallocate bsend's buffer.
+			pypar.detach_and_dealloc()							
+
+		buffer = x
+
+	# All processes with rank distinct from 'root' start receiving.
+	else:
+		buffer = pypar.receive(root, buffer)
+		
+	return buffer
+
+def simple_broadcast_v5(x, root, myid, numprocs, buffer):
+	""" Same as simple_broadcast_v4 except that it uses Pypar's
+	bypass mode.
+
+	The use of bypass mode implies that the programmer has to define his 
+	own buffers on the receiving side (same as when 'use_buffer' is True) 
+	and that he is limited to send numpy arrays and nothing else !
+
+	Hence, this function works only with numpy arrays.
+
+	Input parameters:
+	 - x: data to broadcast.
+	 - root: rank of the process that initiates the broadcast.
+	 - myid: rank of the process calling the function.
+	 - buffer: well-dimensioned user-defined buffer for receiving 'x'.
+
+	Return value: the broadcasted data.
+	"""
+	list = range(0, root) + range(root + 1, numprocs)
+
+	# The process of rank 'root' sends 'x' to all the other processes.
+	if root == myid:
+		for i in list:
+
+			# Determine automatically the size of bsend's buffer.
+			pypar.push_for_alloc(x, bypass=True)						
+
+			# Allocate and attach bsend's buffer.	
+			pypar.alloc_and_attach()							
+
+			# Send data to process of rank 'i'.
+			pypar.bsend(x, i, bypass=True)				
+
+		 	# Detach and deallocate bsend's buffer.
+			pypar.detach_and_dealloc()							
+
+		buffer = x
+
+	# All processes with rank distinct from 'root' start receiving.
+	else:
+		buffer = pypar.receive(root, buffer, bypass=True)
+		
+	return buffer
+
+#
+# Main
+#
+
+if __name__ == "__main__":
+
+	datetime = time.ctime(time.time())
+
+	numprocs = pypar.size()
+	myid = pypar.rank()
+	hostname = pypar.get_processor_name()
+
+	#print "%s - I am proc %d of %d on %s" %(datetime, myid, numprocs, hostname)
+
+	# Testing broadcast of an array - v1.
+	if myid == 0:
+		snd_tab = numpy.array([[1, 2, 3, 4], [5, 6, 7, 8]])
+	else:
+		snd_tab = numpy.array([[0, 0, 0, 0], [0, 0, 0, 0]])
+
+	rcv_tab = simple_broadcast_v1(snd_tab, 0, myid, numprocs) 
+
+	print "v1 - rank " + str(myid) + " received: " + str(rcv_tab)
+
+	pypar.barrier()
+
+	# Testing broadcast of an array - v2.
+	if myid == 0:
+		snd_tab = numpy.array([[1, 2, 3, 4], [5, 6, 7, 8]])
+	else:
+		snd_tab = numpy.array([[0, 0, 0, 0], [0, 0, 0, 0]])
+
+	rcv_tab = simple_broadcast_v2(snd_tab, 0, myid, numprocs) 
+
+	print "v2 - rank " + str(myid) + " received: " + str(rcv_tab)
+
+	pypar.barrier()
+
+	# Testing broadcast of an array - v3.
+	if myid == 0:
+		snd_tab = numpy.array([[1, 2, 3, 4], [5, 6, 7, 8]])
+	else:
+		snd_tab = numpy.array([[0, 0, 0, 0], [0, 0, 0, 0]])
+
+	rcv_tab = simple_broadcast_v3(snd_tab, 0, myid, numprocs) 
+
+	print "v3 - rank " + str(myid) + " received: " + str(rcv_tab)
+
+	pypar.barrier()
+
+	# Testing broadcast of an array - v4.
+	if myid == 0:
+		snd_tab = numpy.array([[1, 2, 3, 4], [5, 6, 7, 8]])
+	else:
+		snd_tab = numpy.array([[0, 0, 0, 0], [0, 0, 0, 0]])
+
+	rcv_tab = numpy.array([[0, 0, 0, 0], [0, 0, 0, 0]])
+	rcv_tab = simple_broadcast_v4(snd_tab, 0, myid, numprocs, rcv_tab) 
+
+	print "v4 - rank " + str(myid) + " received: " + str(rcv_tab)
+
+	pypar.barrier()
+
+	# Testing broadcast of an array - v5.
+	if myid == 0:
+		snd_tab = numpy.array([[1, 2, 3, 4], [5, 6, 7, 8]])
+	else:
+		snd_tab = numpy.array([[0, 0, 0, 0], [0, 0, 0, 0]])
+
+	rcv_tab = numpy.array([[0, 0, 0, 0], [0, 0, 0, 0]])
+	rcv_tab = simple_broadcast_v5(snd_tab, 0, myid, numprocs, rcv_tab) 
+
+	print "v5 - rank " + str(myid) + " received: " + str(rcv_tab)
+
+	pypar.barrier()
+
+	# Testing broadcast of string - v1.
+	if myid == 0:
+		snd_str = "<< Mes sama in corpore sano. >>"
+	else:
+		snd_str = "Failed !!!"
+		
+	rcv_str = simple_broadcast_v1(snd_str, 0, myid, numprocs) 
+
+	print "v1 - rank " + str(myid) + " received: " + str(rcv_str)
+
+	pypar.barrier()
+
+	# Testing broadcast of string - v2.
+	if myid == 0:
+		snd_str = "<< Mes sama in corpore sano. >>"
+	else:
+		snd_str = "Failed !!!"
+		
+	rcv_str = simple_broadcast_v2(snd_str, 0, myid, numprocs) 
+
+	print "v2 - rank " + str(myid) + " received: " + str(rcv_str)
+
+	pypar.barrier()
+
+	# Testing broadcast of string - v3.
+	if myid == 0:
+		snd_str = "<< Mes sama in corpore sano. >>"
+	else:
+		snd_str = "Failed !!!"
+		
+	rcv_str = simple_broadcast_v3(snd_str, 0, myid, numprocs) 
+
+	print "v3 - rank " + str(myid) + " received: " + str(rcv_str)
+
+	pypar.barrier()
+
+	# Testing broadcast of a string - v4.
+	if myid == 0:
+		snd_str = "<< Mes sama in corpore sano. >>"
+	else:
+		snd_str = "Failed !!!"
+		
+	rcv_str = "0000000000000000000000000000000"
+	rcv_str = simple_broadcast_v4(snd_str, 0, myid, numprocs, rcv_str) 
+
+	print "v4 - rank " + str(myid) + " received: " + str(rcv_str)
+
+	pypar.barrier()
+
+	"""
+	# For those who want anyway to try out sending something different
+	# from numpy arrays with bypass mode enabled.
+	# Testing broadcast of a string - v5.
+	if myid == 0:
+		snd_str = "<< Mes sama in corpore sano. >>"
+	else:
+		snd_str = "Failed !!!"
+		
+	rcv_str = "0000000000000000000000000000000"
+	rcv_str = simple_broadcast_v5(snd_str, 0, myid, numprocs, rcv_str) 
+
+	print "v5 - rank " + str(myid) + " received: " + str(rcv_str)
+	"""
+
+	pypar.barrier()
+	pypar.finalize()
+
Index: trunk/anuga_core/source/pypar_dist/demos/demo.py
===================================================================
--- trunk/anuga_core/source/pypar_dist/demos/demo.py	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/demos/demo.py	(revision 8494)
@@ -0,0 +1,61 @@
+#########################################################
+#   
+#  Example: Running Python in parallel using pypar (MPI).
+#   
+#  Author:  Ole Nielsen,  SMS, ANU, Jan. 2002. 
+#
+#########################################################
+#
+# The purpose of this code is to demonstrate how Python can be
+# used to communicate among processes using pypar.
+#
+# This demo passes messages on in a ring from processor n-1 to n starting
+# and ending with processor 0.
+# Each processor adds some text to the message before passing it on
+# and writes a log statement to the screen.
+#
+# To execute:
+#
+#   mpirun -np 4 demo.py
+
+
+import pypar    # The Python-MPI interface 
+
+numproc = pypar.size()
+myid =    pypar.rank()
+node =    pypar.get_processor_name()
+
+print "I am proc %d of %d on node %s" %(myid, numproc, node)
+
+
+if numproc < 2:
+  print "Demo must run on at least 2 processors to continue"      
+  pypar.abort()
+  
+if myid == 0:
+  msg = "MSGP0"  
+  
+  print 'Processor 0 sending message "%s" to processor %d' %(msg, 1)
+  pypar.send(msg, 1)
+
+  msg, status = pypar.receive(numproc-1, return_status=True)
+  print 'Processor 0 received message "%s" from processor %d' %(msg, numproc-1)
+  print 'Size of msg was %d bytes' %(status.bytes())
+
+else:
+  source = myid-1
+  destination = (myid+1)%numproc
+  
+  msg, status = pypar.receive(source, return_status=True)
+  print 'Processor %d received message "%s" from processor %d'\
+        %(myid, msg, source)
+  print 'Size of msg was %d bytes' %(status.bytes())  
+
+  msg = msg + '->P' + str(myid) #Update message     
+  print 'Processor %d sending msg "%s" to %d' %(myid, msg, destination)
+  pypar.send(msg, destination)
+
+pypar.finalize()
+
+
+
Index: trunk/anuga_core/source/pypar_dist/demos/demo2.py
===================================================================
--- trunk/anuga_core/source/pypar_dist/demos/demo2.py	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/demos/demo2.py	(revision 8494)
@@ -0,0 +1,106 @@
+#########################################################
+#
+#  Example: Distributing work in parallel using pypar (MPI).
+#
+#  Author:  Ole Nielsen, ANU, March 2002.
+#
+#########################################################
+#
+"""Skeleton for using pypar to distribute work across processors.
+   It is assumed that the function work knows how to perform different
+   tasks (or the same task on different data) based on values of
+   myid and numprocs.
+"""
+
+
+
+def work(myid, numprocs, data):
+  """This simple example function that slices up the data
+     based on values of numproc and myid.
+  """
+
+  import numpy
+
+  # Identify local slice and process it
+  #
+  interval = len(data)
+  myinterval = interval/numprocs
+
+  mylower = myid*myinterval
+  if myid == numprocs-1:
+    myupper = interval+1
+  else:
+    myupper = mylower + myinterval
+
+  mydata = data[mylower:myupper]
+
+  # Computation (average)
+  #
+  myavg = float(numpy.sum(mydata))/len(mydata)
+  print "P%d: %s Local avg=%.4f" %(myid, str(mydata), myavg)
+
+  return myavg*len(mydata)
+
+
+
+
+###################################################
+# Main program - communication takes place here
+#
+import pypar, numpy
+
+
+# Get data. Here it is just generated but it could be read
+# from file or given as an input parameter.
+#
+lower = 100
+upper = 121
+data = numpy.array(range(lower,upper))
+
+#
+# Get parallel parameters
+#
+numprocs = pypar.size()    # Number of processors
+myid = pypar.rank()        # Id of this processor
+node = pypar.get_processor_name()
+
+print "I am proc %d of %d on node %s" %(myid, numprocs, node)
+
+
+
+#
+# Do work in parallel
+#
+x = work(myid, numprocs, data)   #Do work on all processors
+print "Proc %d finished working" %myid
+
+#
+# Communication
+#
+if numprocs > 1:
+  #
+  # Processor 0 gathers all results and merge them
+  #
+  if myid == 0:
+    for id in range(1,numprocs):
+      print "P%d receving from P%d" %(0, id)
+      x = x + pypar.receive(id)  #Add up (would be more complex in general)
+
+  # All other processors send their results back to processor 0
+  #
+  else:
+    print "P%d sending to P%d" %(myid, 0)
+    pypar.send(x, 0)
+
+print "Proc %d after communication" %myid
+#
+# Compute overall average and report
+#
+
+if myid == 0:
+  avg = x/len(data)
+  print "Global average is %.4f" %avg
+
+pypar.finalize()
+
+
Index: trunk/anuga_core/source/pypar_dist/demos/demo3.py
===================================================================
--- trunk/anuga_core/source/pypar_dist/demos/demo3.py	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/demos/demo3.py	(revision 8494)
@@ -0,0 +1,127 @@
+"""
+Master/Slave Parallel decomposition sample
+
+Run as
+   python demo3.py
+or
+   mpirun -np 2 demo3.py
+  (perhaps try number of processors more than 2)
+  
+
+OMN, GPC FEB 2002
+"""
+
+import sys
+
+try:
+    import numpy
+except:
+    raise Exception, 'Module numpy must be present to run pypar'
+  
+try:
+    import pypar
+except:
+    raise Exception, 'Module pypar must be present to run parallel'
+
+
+print 'Modules numpy, pypar imported OK'
+
+WORKTAG = 1
+DIETAG =  2
+
+
+def master():
+    numCompleted = 0
+
+    print '[MASTER]: I am processor %d of %d on node %s'\
+          %(MPI_myid, MPI_numproc, MPI_node)
+
+    # start slaves distributing the first work slot
+    for i in range(1, min(MPI_numproc, numWorks)):
+        work = workList[i]
+        pypar.send(work, destination=i, tag=WORKTAG)
+        print '[MASTER]: sent work "%s" to node %d' %(work, i)
+
+    # dispatch the remaining work slots on dynamic load-balancing policy
+    # the quicker to do the job, the more jobs it takes
+    for work in workList[MPI_numproc:]:
+        result, status = pypar.receive(source=pypar.any_source, tag=WORKTAG,
+                                       return_status=True)
+
+        print '[MASTER]: received result "%s" from node %d'\
+              %(result, status.source)
+
+        numCompleted += 1
+        pypar.send(work, destination=status.source, tag=WORKTAG)
+        print '[MASTER]: sent work "%s" to node %d' %(work, status.source)
+
+    # all works have been dispatched out
+    print '[MASTER]: toDo : %d' %numWorks
+    print '[MASTER]: done : %d' %numCompleted
+
+    # I've still to take into the remaining completions
+    while (numCompleted < numWorks):
+        result, status = pypar.receive(source=pypar.any_source,
+                                       tag=WORKTAG,
+                                       return_status=True)
+        print '[MASTER]: received (final) result "%s" from node %d'\
+                  %(result, status.source)
+        numCompleted += 1
+        print '[MASTER]: %d completed' %numCompleted
+
+    print '[MASTER]: about to terminate slaves'
+
+    # Tell slaves to stop working
+    for i in range(1, MPI_numproc):
+        pypar.send('#', destination=i, tag=DIETAG)
+        print '[MASTER]: sent termination signal to node %d' %(i, )
+
+    return
+
+def slave():
+
+    print '[SLAVE %d]: I am processor %d of %d on node %s'\
+                     %(MPI_myid, MPI_myid, MPI_numproc, MPI_node)
+
+    while True:
+        result, status = pypar.receive(source=0,
+                                       tag=pypar.any_tag,
+                                       return_status=True)
+        print '[SLAVE %d]: received work "%s" with tag %d from node %d'\
+                  %(MPI_myid, result, status.tag, status.source)
+
+        if (status.tag == DIETAG):
+            print '[SLAVE %d]: received termination from node %d'\
+                             %(MPI_myid, 0)
+            return
+        else:
+            result = 'X'+result
+            pypar.send(result, destination=0, tag=WORKTAG)
+            print '[SLAVE %d]: sent result "%s" to node %d'\
+            %(MPI_myid, result, 0)
+
+
+
+if __name__ == '__main__':
+    MPI_myid =    pypar.rank()
+    MPI_numproc = pypar.size()
+    MPI_node =    pypar.get_processor_name()
+
+    workList = ('_dummy_', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j')
+    numWorks = len(workList) - 1
+
+    #FIXME, better control here
+    if MPI_numproc > numWorks or MPI_numproc < 2:
+        pypar.finalize()
+        if MPI_myid == 0:
+              print 'ERROR: Number of processors must be in the interval [2,%d].'%numWorks
+              sys.exit(-1)
+
+    if MPI_myid == 0:
+        master()
+    else:
+        slave()
+
+    pypar.finalize()
+    print 'MPI environment finalized.'
+    
Index: trunk/anuga_core/source/pypar_dist/demos/demo4.py
===================================================================
--- trunk/anuga_core/source/pypar_dist/demos/demo4.py	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/demos/demo4.py	(revision 8494)
@@ -0,0 +1,65 @@
+#########################################################
+#
+#  Example: Running Python in parallel using pypar (MPI).
+#
+#  Author:  Ole Nielsen,  SMS, ANU, Jan. 2002.
+#
+#########################################################
+#
+# The purpose of this code is to demonstrate how Python can be
+# used to communicate among processes using pypar.
+#
+# This demo passes messages on in a ring from processor n-1 to n starting
+# and ending with processor 0.
+# Each processor adds some text to the message before passing it on
+# and writes a log statement to the screen.
+#
+# To execute:
+#
+#   mpirun -np 4 demo.py
+
+
+import pypar    # The Python-MPI interface
+import numpy
+typ = 'd'  #'i', 'l', 'f', 'd'
+
+numproc = pypar.size()
+myid =    pypar.rank()
+node =    pypar.Get_processor_name()
+
+print "I am proc %d of %d on node %s" %(myid, numproc, node)
+
+if numproc < 2:
+  print "Demo must run on at least 2 processors to continue"
+  pypar.abort()
+
+if myid == 0:
+  msg = numpy.array([0], typ)
+
+  print 'Processor 0 sending message "%s" to processor %d' %(str(msg), 1)
+  pypar.send(msg, 1)
+
+  msg, status = pypar.receive(numproc-1, return_status=True)
+  print 'Processor 0 received message "%s" from processor %d' %(str(msg), numproc-1)
+  print 'Size of msg was %d bytes' %(status.bytes())
+
+else:
+  source = myid-1
+  destination = (myid+1)%numproc
+
+  msg, status = pypar.receive(source, return_status=True)
+  print 'Processor %d received message "%s" from processor %d'\
+        %(myid, str(msg), source)
+  print 'Size of msg was %d bytes' %(status.bytes())
+
+  msg = list(msg)
+  msg.append(myid)
+  print msg
+  msg = numpy.array(msg, typ)  #Update message
+  print 'Processor %d sending msg "%s" to %d' %(myid, msg, destination)
+  pypar.send(msg, destination)
+
+pypar.finalize()
+
+
+
Index: trunk/anuga_core/source/pypar_dist/demos/mandelbrot_example/README
===================================================================
--- trunk/anuga_core/source/pypar_dist/demos/mandelbrot_example/README	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/demos/mandelbrot_example/README	(revision 8494)
@@ -0,0 +1,18 @@
+Files in this directory gives examples of different parallel strategies using the mandbrot fractal as example. The codes need PIL (Python Imaging Library) to
+show the results, but the computational bit will still work without PIL.
+It also tries to show images using either xv of eog (EyeOfGnome).
+
+This material was developed as part of a teaching module and is documented 
+further at 
+
+http://datamining.anu.edu.au/~ole/work/teaching/summerschool2005/
+
+
+To compile C extensions in this directory
+
+python compile_mandelbrot_extensions.py
+
+or
+python compile.py mandel_ext.c
+python compile.py mandelplot_ext.c
+
Index: trunk/anuga_core/source/pypar_dist/demos/mandelbrot_example/compile.py
===================================================================
--- trunk/anuga_core/source/pypar_dist/demos/mandelbrot_example/compile.py	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/demos/mandelbrot_example/compile.py	(revision 8494)
@@ -0,0 +1,207 @@
+"""compile.py - compile Python C-extension
+
+   Commandline usage: 
+     python compile.py <filename>
+
+   Usage from within Python:
+     import compile
+     compile.compile(<filename>,..)
+
+   Ole Nielsen, Oct 2001      
+"""   
+
+import numpy
+def compile(FNs=None, CC=None, LD = None, SFLAG = None, verbose = 1):
+  """compile(FNs=None, CC=None, LD = None, SFLAG = None):
+  
+     Compile FN(s) using compiler CC (e.g. mpicc), 
+     Loader LD and shared flag SFLAG.
+     If CC is absent use default compiler dependent on platform
+     if LD is absent CC is used.
+     if SFLAG is absent platform default is used
+     FNs can be either one filename or a list of filenames
+     In the latter case, the first will be used to name so file.
+  """
+  import os, string, sys, types
+  
+  # Input check
+  #
+  assert not FNs is None, "No filename provided"
+
+  if not type(FNs) == types.ListType:
+    FNs = [FNs]
+
+  # Determine platform and compiler
+  #
+  if sys.platform == 'sunos5':  #Solaris
+    if CC:
+      compiler = CC
+    else:  
+      compiler = 'gcc'
+    if LD:
+      loader = LD
+    else:  
+      loader = compiler
+    if SFLAG:
+      sharedflag = SFLAG
+    else:  
+      sharedflag = 'G'
+      
+  elif sys.platform == 'osf1V5':  #Compaq AlphaServer
+    if CC:
+      compiler = CC
+    else:  
+      compiler = 'cc'
+    if LD:
+      loader = LD
+    else:  
+      loader = compiler
+    if SFLAG:
+      sharedflag = SFLAG
+    else:  
+      sharedflag = 'shared'    
+      
+  elif sys.platform == 'linux2':  #Linux
+    if CC:
+      compiler = CC
+    else:  
+      compiler = 'gcc'
+    if LD:
+      loader = LD
+    else:  
+      loader = compiler
+    if SFLAG:
+      sharedflag = SFLAG
+    else:  
+      sharedflag = 'shared'    
+      
+  elif sys.platform == 'darwin':  #Mac OS X:
+    if CC:
+      compiler = CC
+    else:  
+      compiler = 'cc'
+    if LD:
+      loader = LD
+    else:  
+      loader = compiler
+    if SFLAG:
+      sharedflag = SFLAG
+    else:  
+      sharedflag = 'bundle -flat_namespace -undefined suppress'
+      
+  else:
+    if verbose: print "Unrecognised platform %s - revert to default"\
+                %sys.platform
+    
+    if CC:
+      compiler = CC
+    else:  
+      compiler = 'cc'
+    if LD:
+      loader = LD
+    else:  
+      loader = 'ld'
+    if SFLAG:
+      sharedflag = SFLAG
+    else:  
+      sharedflag = 'G'
+
+            
+  # Find location of include files
+  #
+  python_include = os.path.join(os.path.join(sys.exec_prefix, 'include'),
+                                'python'+sys.version[:3])
+  
+  
+  # Check existence of Python.h
+  #
+  headerfile = python_include + '/Python.h'
+  try:
+    open(headerfile, 'r')
+  except:
+    raise """Did not find Python header file %s.
+    Make sure files for Python C-extensions are installed. 
+    In debian linux, for example, you need to install a
+    package called something like python2.1-dev""" %headerfile
+
+
+  # Get numpy include
+  numpy_include = numpy.get_include()
+  
+  
+  # Check filename(s)
+  #
+  object_files = ''
+  for FN in FNs:        
+    root, ext = os.path.splitext(FN)
+    if ext == '':
+      FN = FN + '.c'
+    elif ext.lower() != '.c':
+      raise Exception, "Unrecognised extension: " + FN
+    
+    try:
+      open(FN,'r')
+    except:    
+      raise Exception, "Could not open: " + FN
+
+    if not object_files: root1 = root  #Remember first filename        
+    object_files += root + '.o '  
+  
+  
+    # Compile
+    #
+    s = "%s -c %s -I%s -I%s -o %s.o -Wall" %(compiler, FN,
+                                       python_include, numpy_include,
+                                       root)
+    if os.name == 'posix' and os.uname()[4] == 'x86_64':
+      #Extra flags for 64 bit architectures
+      #s += ' -fPIC -m64' #gcc
+      #s += ' -fPIC -tp amd64' #pgcc
+      s += ' -fPIC' # Position independent code is enought for 64 bit archs
+      
+    
+    if verbose:
+      print s
+    else:
+      s = s + ' 2> /dev/null' #Suppress errors
+  
+    try:
+      os.system(s)
+    except:
+      raise 'Could not compile %s - please try manually' %FN  
+
+  
+  # Make shared library (*.so)
+  s = "%s -%s %s -o %s.so" %(loader, sharedflag, object_files, root1)
+
+  if os.name == 'posix' and os.uname()[4] == 'x86_64':
+      pass
+      #Extra flags for 64 bit architectures using Portland compilers
+      s += ' -mcmodel=medium'
+  
+  if verbose:
+    print s
+  else:
+    s = s + ' 2> /dev/null' #Suppress warnings
+  
+  try:  
+    err=os.system(s)
+  except:
+    raise 'Could not link %s - please try manually' %root1
+    
+
+
+
+if __name__ == '__main__':
+
+  import sys
+  
+  # Get file to compile
+  #
+  if len(sys.argv) < 2:
+    print "Usage:"
+    print "  python compile.py <filename>"
+    sys.exit()
+
+  compile(sys.argv[1], verbose=1)
+
Index: trunk/anuga_core/source/pypar_dist/demos/mandelbrot_example/compile_mandelbrot_c_extensions.py
===================================================================
--- trunk/anuga_core/source/pypar_dist/demos/mandelbrot_example/compile_mandelbrot_c_extensions.py	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/demos/mandelbrot_example/compile_mandelbrot_c_extensions.py	(revision 8494)
@@ -0,0 +1,12 @@
+"""This module will compile the necessary c-extensions for the mandelbrot demos
+"""
+
+import os
+
+s = 'python compile.py mandel_ext.c'
+print
+os.system(s)
+
+s = 'python compile.py mandelplot_ext.c'
+print
+os.system(s)
Index: trunk/anuga_core/source/pypar_dist/demos/mandelbrot_example/mandel_ext.c
===================================================================
--- trunk/anuga_core/source/pypar_dist/demos/mandelbrot_example/mandel_ext.c	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/demos/mandelbrot_example/mandel_ext.c	(revision 8494)
@@ -0,0 +1,69 @@
+/* Python - C extension for fast computation of the Mandelbrot iteration
+//
+// To compile:
+//  python compile.py mandel_ext.c
+//
+// Example of how to use within Python:
+//
+// from mandel_ext import calculate_point
+//
+// kmax = 256
+// c = complex(0.5, 0.5)
+// k = calculate_point(c, kmax)
+//
+// Ole Nielsen, SUT 2003 */
+	
+#include "Python.h"
+
+/* Computational function */
+int _calculate_point(Py_complex c, int kmax) {
+  int count;
+  double temp, lengthsq = 0.0;  
+  Py_complex z;
+  
+  z.real = 0.0; z.imag = 0.0; 
+  count = 0;
+
+  while (count < kmax && lengthsq <= 4) {
+       temp = z.real * z.real - z.imag * z.imag + c.real;
+       z.imag = 2.0 * z.real * z.imag + c.imag;
+       z.real = temp;
+       lengthsq = z.real * z.real + z.imag * z.imag;
+       count++;    
+  }
+
+  return count;
+}
+
+
+/* Interface to Python */
+PyObject *calculate_point(PyObject *self, PyObject *args) {
+  PyComplexObject *C;  /* Python Complex object */
+  Py_complex c;        /* C Complex structure */
+  int kmax, count;
+
+  /* Convert Python arguments to C  */
+  if (!PyArg_ParseTuple(args, "Oi", &C, &kmax))
+    return NULL;
+  c = PyComplex_AsCComplex((PyObject*) C);   
+  
+  /* Call underlying routine */
+  count = _calculate_point(c, kmax); 
+
+  /* Return results as a Python object */
+  return Py_BuildValue("i", count);
+}
+
+
+/* Method table for python module */
+static struct PyMethodDef MethodTable[] = {
+  {"calculate_point", calculate_point, METH_VARARGS},  
+  {NULL, NULL}
+};
+
+
+/* Module initialisation  */
+void initmandel_ext(void){
+  Py_InitModule("mandel_ext", MethodTable);
+}
+
Index: trunk/anuga_core/source/pypar_dist/demos/mandelbrot_example/mandel_parallel_blockwise.py
===================================================================
--- trunk/anuga_core/source/pypar_dist/demos/mandelbrot_example/mandel_parallel_blockwise.py	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/demos/mandelbrot_example/mandel_parallel_blockwise.py	(revision 8494)
@@ -0,0 +1,64 @@
+"""Parallel program computing the Mandelbrot set using static partitioned 
+load balancing.
+
+   Due to the uneven work load of different regions,
+   this strategy will *not* generally lead to linear speedup
+
+   See mandel_parallel_dynamic.py for a better approach.
+
+   Ole Nielsen, SUT 2003
+"""
+
+from mandelbrot import calculate_region, balance
+from mandelplot import plot
+import pypar
+
+# User definable parameters
+kmax = 2**15   # Maximal number of iterations (=number of colors)
+M = N = 700    # width = height = N (200 or 700)
+
+# Region in complex plane
+real_min = -2.0
+real_max =  1.0
+imag_min = -1.5
+imag_max =  1.5
+
+#Initialise
+t = pypar.time()
+P = pypar.size()
+p = pypar.rank()
+processor_name = pypar.get_processor_name()
+
+print 'Processor %d initialised on node %s' %(p,processor_name)
+
+
+# Balanced work partitioning (row wise)
+Mlo, Mhi = pypar.balance(M, P, p)
+print 'p%d: [%d, %d], Interval length=%d' %(p, Mlo, Mhi, Mhi-Mlo)
+
+# Parallel computation 
+A = calculate_region(real_min, real_max, imag_min, imag_max, kmax,
+                     M, N, Mlo = Mlo, Mhi = Mhi)
+
+print 'Processor %d: time = %.2f' %(p, pypar.time() - t)
+
+
+# Communication phase
+if p == 0:
+    for d in range(1, P):
+        A += pypar.receive(source=d)
+
+    print 'Computed region in %.2f seconds' %(pypar.time()-t)
+    try:
+        plot(A, kmax)
+    except:
+        pass    
+else:
+    pypar.send(A, destination=0)
+
+pypar.finalize()                
+
+
+
+
+
Index: trunk/anuga_core/source/pypar_dist/demos/mandelbrot_example/mandel_parallel_cyclic.py
===================================================================
--- trunk/anuga_core/source/pypar_dist/demos/mandelbrot_example/mandel_parallel_cyclic.py	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/demos/mandelbrot_example/mandel_parallel_cyclic.py	(revision 8494)
@@ -0,0 +1,62 @@
+"""Parallel program computing the Mandelbrot set using static,
+   cyclic load balancing.
+
+   This is probably the best approach for this problem.
+
+   Ole Nielsen, SUT 2003
+"""
+
+from mandelbrot import calculate_region_cyclic
+from mandelplot import plot
+import pypar
+
+# User definable parameters
+kmax = 2**15  # Maximal number of iterations (=number of colors)
+M = N = 700   # width = height = N (200 or 700)
+
+# Region in complex plane [-2:2]
+real_min = -2.0
+real_max =  1.0
+imag_min = -1.5
+imag_max =  1.5
+
+#Initialise
+t = pypar.time()
+P = pypar.size()
+p = pypar.rank()
+processor_name = pypar.get_processor_name()
+
+print 'Processor %d initialised on node %s' %(p, processor_name)
+
+
+# Parallel computation
+A = calculate_region_cyclic(real_min, real_max, imag_min,
+                            imag_max, kmax,
+                            M, N, p, P)
+
+print 'Processor %d: time = %.2f' %(p, pypar.time() - t)
+
+
+# Communication phase
+if p == 0:
+    for d in range(1, P):
+        print 'Proc 0 receiving from %i' % d
+        A += pypar.receive(source=d)
+        print 'Received from %i' % d
+
+    print 'Computed region in %.2f seconds' %(pypar.time() - t)
+    try:
+        plot(A, kmax)
+    except:
+        pass
+
+else:
+    print 'Sending to 0'
+    pypar.send(A, destination=0)
+
+pypar.finalize()
+
+
+
+
+
Index: trunk/anuga_core/source/pypar_dist/demos/mandelbrot_example/mandel_parallel_dynamic.py
===================================================================
--- trunk/anuga_core/source/pypar_dist/demos/mandelbrot_example/mandel_parallel_dynamic.py	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/demos/mandelbrot_example/mandel_parallel_dynamic.py	(revision 8494)
@@ -0,0 +1,104 @@
+"""Parallel program computing the Mandelbrot set using dynamic load balancing.
+   Simplified version for use with exercise 15.
+
+   To keep master process busy, run with P+1 processes
+   where P is the number of physical processors.
+   
+   Ole Nielsen, SUT 2003
+"""
+
+from mandelbrot import calculate_region, balance
+from mandelplot import plot
+import pypar, numpy
+
+
+# User definable parameters
+kmax = 2**15  # Maximal number of iterations (=number of colors)
+M = N = 700   # width = height = N
+B = 24        # Number of blocks (first dim)
+
+
+# Region in complex plane [-2:2]
+real_min = -2.0
+real_max =  1.0
+imag_min = -1.5
+imag_max =  1.5
+
+# MPI controls
+work_tag = 0
+result_tag = 1
+
+#Initialise
+t = pypar.time()
+P = pypar.size()
+p = pypar.rank()
+processor_name = pypar.get_processor_name()
+
+print 'Processor %d initialised on node %s' %(p,processor_name)
+
+assert P > 1, 'Must have at least one slave'
+assert B > P-1, 'Must have more work packets than slaves'
+
+
+A = numpy.zeros((M,N), dtype='i')
+if p == 0:
+    # Create work pool (B blocks)
+    # using balanced work partitioning
+    workpool = []
+    for i in range(B):
+        Mlo, Mhi = balance(M, B, i)    
+        workpool.append( (Mlo, Mhi) )
+
+
+    # Distribute initial work to slaves
+    w = 0 
+    for d in range(1, P):
+        pypar.send(workpool[w], destination=d, tag=work_tag)
+        w += 1
+
+    #Receive computed work and distribute more
+    terminated = 0
+    while(terminated < P-1):
+        R, status = pypar.receive(pypar.any_source, tag=result_tag,
+                                  return_status=True)
+        A += R            #Aggregate data        
+        d = status.source #Id of slave that just finished        
+
+        if w < len(workpool):
+            #Send new work to slave d
+            pypar.send(workpool[w], destination=d, tag=work_tag)
+            w += 1
+        else:
+            #Tell slave d to terminate
+            pypar.send(None, destination=d, tag=work_tag) 
+            terminated += 1
+        
+    print 'Computed region in %.2f seconds' %(pypar.time()-t)
+    try:
+        plot(A, kmax)
+    except:
+        pass    
+else:
+    while(True):
+        #Receive work (or None)
+        W = pypar.receive(source=0, tag=work_tag)
+        
+        if W is None:
+            print 'Slave p%d finished: time = %.2f' %(p, pypar.time() - t)
+            break
+
+        #Compute allocated work
+        A = calculate_region(real_min, real_max, imag_min, imag_max, kmax,
+                             M, N, Mlo = W[0], Mhi = W[1])
+
+        #Return result
+        pypar.send(A, destination=0, tag=result_tag)
+
+pypar.finalize()
+
+
+
+
+
+
+
Index: trunk/anuga_core/source/pypar_dist/demos/mandelbrot_example/mandel_sequential.py
===================================================================
--- trunk/anuga_core/source/pypar_dist/demos/mandelbrot_example/mandel_sequential.py	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/demos/mandelbrot_example/mandel_sequential.py	(revision 8494)
@@ -0,0 +1,32 @@
+"""Sequential program computing the Mandelbrot set.
+
+Ole Nielsen, SUT 2003
+"""
+
+from mandelbrot import calculate_region
+from mandelplot import plot
+import time
+
+# User definable parameters
+kmax = 2**15  # Maximal number of iterations (=number of colors)
+M = N = 700   # width = height = N (200, 400, 600, 700 are good)
+#M = N = 400   # width = height = N (200, 400, 600, 700 are good)
+
+# Region in complex plane
+real_min = -2.0
+real_max =  1.0
+imag_min = -1.5
+imag_max =  1.5
+
+# Compute Mandelbrot set
+t0 = time.time()
+A = calculate_region(real_min, real_max, imag_min, imag_max, kmax, M, N)
+print 'Computed region in %.2f seconds' %(time.time()-t0)
+
+# Plot result
+try:
+    plot(A, kmax)
+except:
+    pass    
+
+
Index: trunk/anuga_core/source/pypar_dist/demos/mandelbrot_example/mandelbrot.py
===================================================================
--- trunk/anuga_core/source/pypar_dist/demos/mandelbrot_example/mandelbrot.py	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/demos/mandelbrot_example/mandelbrot.py	(revision 8494)
@@ -0,0 +1,97 @@
+"""Fundamental routines for computing the Mandelbrot set
+
+   Ole Nielsen, SUT 2003
+"""
+
+def balance(N, P, p):
+    """Compute p'th interval when N is distributed over P bins.
+    """
+
+    from math import floor
+
+    L = int(floor(float(N)/P))
+    K = N - P*L
+    if p < K:
+        Nlo = p*L + p
+        Nhi = Nlo + L + 1
+    else:
+        Nlo = p*L + K
+        Nhi = Nlo + L
+
+    return Nlo, Nhi
+
+
+def calculate_point(c, kmax):
+    """Python version for calculating on point of the set
+       This is slow and for reference purposes only.
+       Use version from mandel_ext instead.
+    """
+
+    z = complex(0,0)
+
+    count = 0
+    while count < kmax and abs(z) <= 2:
+        z = z*z + c
+        count += 1
+
+    return count    
+
+
+def calculate_region(real_min, real_max, imag_min, imag_max, kmax, M, N,
+                     Mlo = 0, Mhi = None, Nlo = 0, Nhi = None):
+    """Calculate the mandelbrot set in the given region with resolution M by N
+       If Mlo, Mhi or Nlo, Nhi are specified computed only given subinterval.
+    """
+
+    from numpy import zeros
+    from mandel_ext import calculate_point  #Fast C implementation
+
+    if Mhi is None: Mhi = M
+    if Nhi is None: Nhi = N    
+
+    real_step = (real_max-real_min)/M
+    imag_step = (imag_max-imag_min)/N
+
+    A = zeros((M, N), dtype='i')   # Create M x N matrix
+
+    for i in range(Mlo, Mhi):
+        for j in range(Nlo, Nhi):
+            c = complex(real_min + i*real_step, imag_min + j*imag_step)
+            A[i,j] = calculate_point(c, kmax)
+
+    return A
+
+
+
+def calculate_region_cyclic(real_min, real_max, imag_min, imag_max, kmax,
+                            M, N, p=0, P=1, row = 1):
+    """Calculate rows p+nP, n in N of the mandelbrot set in the given region
+    with resolution M by N
+
+    This is the most efficient way of partitioning the work.
+    """
+
+
+    from numpy import zeros
+    from mandel_ext import calculate_point  #Fast C implementation
+
+    real_step = (real_max-real_min)/M
+    imag_step = (imag_max-imag_min)/N
+
+    A = zeros((M, N), dtype='i')   # Create M x N matrix
+
+    if row:
+        for i in range(M):
+            if i%P == p:
+                for j in range(N):
+                    c = complex(real_min + i*real_step, imag_min + j*imag_step)
+                    A[i,j] = calculate_point(c, kmax)
+    else:
+        for j in range(N):        
+            if j%P == p:
+                for i in range(M):
+                    c = complex(real_min + i*real_step, imag_min + j*imag_step)
+                    A[i,j] = calculate_point(c, kmax)
+    return A
+
+    
Index: trunk/anuga_core/source/pypar_dist/demos/mandelbrot_example/mandelplot.py
===================================================================
--- trunk/anuga_core/source/pypar_dist/demos/mandelbrot_example/mandelplot.py	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/demos/mandelbrot_example/mandelplot.py	(revision 8494)
@@ -0,0 +1,98 @@
+"""Plotting routine for use with the mandelbrot set
+
+   Ole Nielsen, SUT 2003
+"""
+
+
+def plot(A, kmax = None):
+    """Plot matrix A as an RGB image using the Python Imaging Library and Tkinter
+
+       A is converted to an RGB image using PIL and saved to disk
+       Then it is displayed using PhotoImage
+
+       A must be square, integer valued, two dimensional and non-negative
+
+       If kmax is omitted it will be set to the max value of A
+       
+       Ole Nielsen, SUT 2003
+    """
+    
+    #from Tkinter import Frame, Canvas, TOP, NW, PhotoImage
+    from numpy import transpose
+    from Image import new             # PIL    
+    import time
+
+    t0 = time.time()
+
+    # User definable parameters
+
+    imtype = 'ppm'      # Image format (e.g 'ppm', 'bmp', 'tiff')
+    filename ='mandel'  # Base filename
+    
+    exponent = 0.998    # Exponent for morphing colors, good with kmax = 2**15
+    rgbmax = 2**24      # Normalisation constant for RGB
+    
+
+    # Input check
+    assert len(A.shape) == 2, 'Matrix must be 2 dimensional'
+    assert A.shape[0] == A.shape[1], 'Matrix must be square'
+    msg = 'A must contain integers, I got %c' %A.dtype.char
+    assert A.dtype.char in 'iIbBhHl', msg
+    assert min(A.flat)>=0, 'A must be non-negative'
+
+    if kmax is None:
+        kmax = max(A.flat)
+
+    # Convert values from A into RGB values (0 to 255) in each band
+    N = A.shape[0]
+    A = transpose(A).astype('d') # Cast as double         
+
+    im = new("RGB", A.shape)
+
+    
+    L = []        
+    try:
+        from mandelplot_ext import normalise_and_convert
+        normalise_and_convert(A, L, kmax, rgbmax, exponent)
+        
+    except:
+        print 'WARNING: Could not import C extension from mandelplot_ext'
+                
+    
+        for i in range(A.shape[0]):
+            for j in range(A.shape[1]):    
+                
+                c = A[i,j]/kmax
+
+                if c == 1: c = 0       #Map convergent point (kmax) to black (0)
+                c = c**exponent        #Morph slightly
+                
+                c = int(c * rgbmax)    #Normalise to 256 levels per channel
+                
+                red   = c / 256 / 256
+                green = (c / 256) % 256
+                blue  = c % 256
+    
+                L.append( (red, green, blue) )
+            
+            
+    
+    # Save image to file        
+    im.putdata(L)
+    im.save(filename + '.' + imtype, imtype)
+    print 'Computed plot in %.2f seconds: ' %(time.time()-t0)
+
+    # Display image on screen
+    #answer = raw_input('Show image [Y/N][Y]?')
+    #if answer.lower() in ['n', 'no']:
+    #   import sys
+    #   sys.exit()
+
+    # Try to display using a image viewer
+    import os
+    os.system('eog %s' %(filename + '.' + imtype))
+    #os.system('xv %s' %(filename + '.' + imtype))    
+
+
+
+ 
Index: trunk/anuga_core/source/pypar_dist/demos/mandelbrot_example/mandelplot_ext.c
===================================================================
--- trunk/anuga_core/source/pypar_dist/demos/mandelbrot_example/mandelplot_ext.c	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/demos/mandelbrot_example/mandelplot_ext.c	(revision 8494)
@@ -0,0 +1,117 @@
+// Python - C extension for fast computation of the Mandelbrot plot routine
+//
+// To compile (Python2.3):
+//  gcc -c mandelplot_ext.c -I/usr/include/python2.3 -o mandelplot_ext.o -Wall -O3 
+//  gcc -shared mandelplot_ext.o  -o mandelplot_ext.so	
+//
+// kmax = 256
+// c = complex(0.5, 0.5)
+// k = calculate_point(c, kmax)
+//
+// Ole Nielsen, ANU 2004
+	
+	
+#include "Python.h"
+#include "numpy/arrayobject.h"
+
+// Computational function
+int  _normalise_and_convert(double* A, 
+			    int M, 
+			    int N,
+			    PyObject* L, 
+			    int kmax, 
+			    int rgbmax, 
+			    double exponent) {
+			     
+  /*			     
+  for i in range(A.shape[0]):
+    for j in range(A.shape[1]):    
+      
+      c = A[i,j]/kmax
+      if c == 1: c = 0       #Map convergent point (kmax) to black (0)
+      c = c**exponent        #Morph slightly
+      c = int(c * rgbmax)    #Normalise to 256 levels per channel
+ 		
+      red   = c / 256 / 256
+      green = (c / 256) % 256
+      blue  = c % 256
+      
+      L.append( (red, green, blue) ) */
+	    
+
+  
+  PyObject *tuple;
+  int i, j, iN, red, green, blue, ci;
+  double c;
+  
+  for (i=0; i<M; i++) {
+    iN = i*N;
+    
+    for (j=0; j<N; j++) {  
+    
+      c = A[iN  + j]/kmax;     //Normalise to unit interval	    	    
+      c = pow(c, exponent);        //Morph slightly
+      ci = (int) (c*rgbmax);       //Normalise to rgbmax levels per channel
+      if (ci == rgbmax) ci = 0.0;  //Map convergent point (kmax) to black (0)
+      
+
+      //Convert to RGB
+      red   = ci / 256 / 256;
+      green = (ci / 256) % 256;
+      blue  = ci % 256;			     			     
+      
+      tuple = Py_BuildValue("iii", red, green, blue);
+      PyList_Append(L, tuple);
+    }
+  }
+	
+  return 0;
+}
+
+
+// Interface to Python
+PyObject *normalise_and_convert(PyObject *self, PyObject *args) {
+  
+  //Called 
+  //normalise_and_convert(A, L, kmax, rgbmax, exponent)    
+  
+  int kmax, rgbmax, M, N;
+  double exponent;
+  PyObject *L;
+  PyArrayObject *A;
+
+  // Convert Python arguments to C  
+  if (!PyArg_ParseTuple(args, "OOiid", &A, &L, &kmax, &rgbmax, &exponent))
+    return NULL;
+    
+  M = A -> dimensions[0];
+  N = A -> dimensions[1];  
+  
+
+  // Call underlying routine
+  _normalise_and_convert((double *) A -> data, M, N, L, 
+			 kmax, rgbmax, exponent); 
+  
+  // Return None
+  return Py_BuildValue("");
+}
+
+
+// Method table for python module
+static struct PyMethodDef MethodTable[] = {
+  {"normalise_and_convert", normalise_and_convert, METH_VARARGS},  
+  {NULL, NULL}
+};
+
+
+// Module initialisation   
+void initmandelplot_ext(void){
+  Py_InitModule("mandelplot_ext", MethodTable);
+  import_array();     //Necessary for handling of numpy structures
+}
+
+
+
+
+
+
Index: trunk/anuga_core/source/pypar_dist/demos/mandelbrot_example/test_calculate_point.py
===================================================================
--- trunk/anuga_core/source/pypar_dist/demos/mandelbrot_example/test_calculate_point.py	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/demos/mandelbrot_example/test_calculate_point.py	(revision 8494)
@@ -0,0 +1,84 @@
+"""
+  Test of calculate_point within mandelbrot.py
+"""
+
+import unittest
+import Numeric
+from mandelbrot import calculate_point
+#from mandel_ext import calculate_point
+
+
+#-------------------------------------------------------------
+
+class TestCase(unittest.TestCase):
+
+    def setUp(self):
+	pass
+
+	
+    def test_basic(self):
+        """Test that c==0 always returns kmax
+	"""
+
+	c = complex(0, 0)
+	
+	for kmax in [0,1,2,3,4,10, 20, 33, 45, 100]:
+	    count = calculate_point(c, kmax)	
+            assert count == kmax    	
+
+	    
+    def test_points_clearly_outside_set(self):	
+    
+    	for kmax in [1,2,3,5,10,20,100]:
+	    for c in [complex(5, 4), complex(2,1), complex(-2,1),
+	    	      complex(2,-1), complex(-2,-1)]:
+	        count = calculate_point(c, kmax)
+	        assert count == 1
+	
+		
+    def test_arbitrary_points(self):
+    	for kmax in [10,20,100]:	    
+	    c = complex(1, 1.5)
+	    count = calculate_point(c, kmax)
+	    assert count == 2    
+	    
+	    c = complex(-1, 1.5)
+	    count = calculate_point(c, kmax)
+	    assert count == 2    	    
+	    
+	    c = complex(-1, -1.5)
+	    count = calculate_point(c, kmax)
+	    assert count == 2    	    	    
+	    
+	    c = complex(1, -1.5)
+	    count = calculate_point(c, kmax)
+	    assert count == 2    	    	    	    
+	    
+	    c = complex(1, 0)
+	    count = calculate_point(c, kmax)
+	    assert count == 3    	    	    	    	    
+	    
+	    c = complex(0.2, 1)
+	    count = calculate_point(c, kmax)
+	    assert count == 4    	    	    	    	    	    
+	    
+	    c = complex(0.2, 0.7)
+	    count = calculate_point(c, kmax)
+	    assert count == 6    	    	    	    	    	    	    
+	    
+	    c = complex(-0.2, 0.9)
+	    count = calculate_point(c, kmax)
+	    assert count == 9
+            
+
+    def test_another_known_point(self):
+        c = complex(0.5, 0.5)
+        count = calculate_point(c, 256)
+        assert count == 5        
+
+#-------------------------------------------------------------
+if __name__ == "__main__":
+
+    mysuite = unittest.makeSuite(TestCase,'test')
+    runner = unittest.TextTestRunner()
+    runner.run(mysuite)
Index: trunk/anuga_core/source/pypar_dist/demos/mandelbrot_example/test_calculate_region.py
===================================================================
--- trunk/anuga_core/source/pypar_dist/demos/mandelbrot_example/test_calculate_region.py	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/demos/mandelbrot_example/test_calculate_region.py	(revision 8494)
@@ -0,0 +1,83 @@
+"""
+  Test of calculate_region within mandelbrot.py
+"""
+
+import unittest
+import numpy
+from mandelbrot import calculate_region
+
+
+#-------------------------------------------------------------
+
+class TestCase(unittest.TestCase):
+
+    def setUp(self):
+	pass
+
+	
+    def test1(self):
+	kmax = 16
+	M = N = 5
+	
+	real_min = -2.0
+	real_max =  1.0
+	imag_min = -1.5
+	imag_max =  1.5
+        A = calculate_region(real_min, real_max, 
+		imag_min, imag_max, kmax, M, N)
+
+	assert numpy.allclose(A,  		
+	[[ 1,  1,  1,  1,  1],
+ 	[ 1,  3,  5,  5,  3],
+ 	[ 2,  4,  9,  9,  4],
+ 	[ 2,  9, 16, 16,  9],
+ 	[ 2,  3, 15, 15,  3]])
+
+
+    def test2(self):
+	kmax = 32
+	M = N = 5
+	
+	real_min = -2.0
+	real_max =  1.0
+	imag_min = -1.5
+	imag_max =  1.5
+        A = calculate_region(real_min, real_max, 
+		imag_min, imag_max, kmax, M, N)
+
+	assert numpy.allclose(A,  		
+	[[ 1,  1,  1,  1,  1],
+ 	[ 1,  3,  5,  5,  3],
+ 	[ 2,  4,  9,  9,  4],
+ 	[ 2,  9, 32, 32,  9],
+ 	[ 2,  3, 15, 15,  3]])
+	
+
+    def test3(self):
+	kmax = 32
+	M = N = 100
+	
+	real_min = -2.0
+	real_max =  1.0
+	imag_min = -1.5
+	imag_max =  1.5
+        A = calculate_region(real_min, real_max, 
+		imag_min, imag_max, kmax, M, N)
+
+	assert numpy.allclose(A[:, 50], 
+	[32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 
+	 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32,
+         32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 
+         32, 32, 32, 32, 32, 32, 32, 32,
+         32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 
+         32, 32, 32, 32, 32, 32, 32, 32,
+         32, 32, 16, 11,  9,  7,  7,  6,  5,  5,  5,  4,  4,  4,  4,  4,  
+         3, 3,  3,  3,  3,  3,  3,  3, 3,  3])
+
+	
+#-------------------------------------------------------------
+if __name__ == "__main__":
+
+    mysuite = unittest.makeSuite(TestCase,'test')
+    runner = unittest.TextTestRunner()
+    runner.run(mysuite)
Index: trunk/anuga_core/source/pypar_dist/demos/ring_example.py
===================================================================
--- trunk/anuga_core/source/pypar_dist/demos/ring_example.py	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/demos/ring_example.py	(revision 8494)
@@ -0,0 +1,27 @@
+import pypar                                       # Import module and initialise MPI 
+
+proc = pypar.size()                                # Number of processes as specified by mpirun
+myid = pypar.rank()                                # Id of of this process (myid in [0, proc-1]) 
+node = pypar.get_processor_name()                  # Host name on which current process is running
+
+print 'I am proc %d of %d on node %s' %(myid, proc, node)
+
+if myid == 0:                                      # Actions for process 0:
+  msg = 'P0'  
+  pypar.send(msg, destination=1)                   # Send message to proces 1 (right hand neighbour)
+  msg = pypar.receive(source=proc-1)               # Receive message from last process
+      
+  print 'Processor 0 received message "%s" from processor %d' %(msg, proc-1)
+
+else:                                              # Actions for all other processes:
+
+  source = myid-1                                  # Source is the process to the left
+  destination = (myid+1)%proc                      # Destination is process to the right
+                                                   # wrapped so that last processor will 
+						   # send back to proces 0  
+  
+  msg = pypar.receive(source)                      # Receive message from source 
+  msg = msg + 'P' + str(myid)                      # Update message     
+  pypar.send(msg, destination)                     # Send message to destination   
+
+pypar.finalize()                                   # Stop MPI 
Index: trunk/anuga_core/source/pypar_dist/documentation/DOC
===================================================================
--- trunk/anuga_core/source/pypar_dist/documentation/DOC	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/documentation/DOC	(revision 8494)
@@ -0,0 +1,330 @@
+------------------------------------------------
+PyPAR - Parallel Python, no-frills MPI interface
+
+Author:  Ole Nielsen (2001, 2002, 2003)
+Email:   Ole.Nielsen@anu.edu.au
+Version: See pypar.__version__
+Date:    See pypar.__date__
+------------------------------------------------
+
+CONTENTS:
+  TESTING THE INSTALLATION
+  GETTING STARTED  
+  FIRST PARALLEL PROGRAM     
+  PROGRAMMING FOR EFFICIENCY
+  PYPAR REFERENCE
+  DATA TYPES
+  STATUS OBJECT	  
+  EXTENSIONS
+  PROBLEMS
+  REFERENCES
+
+TESTING THE INSTALLATION
+
+  After having installed pypar run
+  
+    mpirun -np 2 testpypar.py
+    
+  to verify that everything works.
+  If it doesn't try to run a C/MPI program 
+  (for example ctiming.c) to see if the problem lies with
+  the local MPI installation or with pypar.
+   
+    
+GETTING STARTED
+  Assuming that the pypar C-extension compiled you should be able to
+  write 
+  >> import pypar
+  from the Python prompt. 
+  (This may not work under MPICH under Sunos - this doesn't matter, 
+  just skip this interactive bit and go to FIRST PARALLEL PROGRAM.)
+  
+  Then you can write
+  >> pypar.size()  
+  to get the number of parallel processors in play.
+  (When using pypar from the command line this number will be 1).
+  
+  Processors are numbered from 0 to pypar.size()-1.
+  Try
+  >> pypar.rank()
+  to get the processor number of the current processor.
+  (When using pypar from the command line this number will be 0).  
+  
+  Finally, try
+  >> pypar.Get_processor_name()
+  to return host name of current node 
+  
+FIRST PARALLEL PROGRAM   
+  Take a look at the file demo.py supplied with this distribution.
+  To execute it in parallel on four processors, say, run
+  
+    mpirun -np 4 demo.py
+    
+  
+  from the UNIX command line.
+  This will start four copies of the program each on its own processor. 
+    
+  The program demo.py makes a message on processor 0 and 
+  sends it around in a ring
+  - each processor adding a bit to it - until it arrives back at 
+  processor 0.
+  
+  All parallel programs must find out what processor they are running on.
+  This is accomplished by the call
+  
+    myid = pypar.rank()
+  
+  The total number of processors is obtained from
+  
+    proc = pypar.size()
+    
+  One can then have different codes for different processors by branching
+  as in
+  
+    if myid == 0
+      ...  
+  
+  To send a general Python structure A to processor p, write
+    pypar.send(A, p)
+  and to receive something from processor q, write
+    X = pypar.receive(q)  
+  
+  This will cater for any (picklable Python structure) and make parallel 
+  programs very simple and readable. While reasonably fast, it does not 
+  achieve the full bandwidth obtainable by MPI.
+
+
+PROGRAMMING FOR EFFICIENCY    
+  Really low latency communication can be achieved by sticking 
+  to Numeric arrays and specifying receive buffers whenever possible.
+
+  To send a Numeric array A to processor p, write
+    pypar.send(A, p, use_buffer=True)
+  and to receive the array from processor q, write
+    X = pypar.receive(q, buffer=X)    
+  Note that X acts as a buffer and must be pre-allocated prior to the 
+  receive statement as in Fortran and C programs using MPI.
+  
+  These forms have superseded the raw forms present in pypar
+  prior to version 1.9. The raw forms have been recast in terms of the 
+  above and have been retained for backwars compatibility.
+  See the script pytiming for an example of communication of Numeric arrays.
+      
+
+MORE PARALLEL PROGRAMMING WITH PYPAR
+
+  When you are ready to move on, have a look at the supplied 
+  demos and the script testpypar.py which all give examples
+  of parallel programming with pypar.
+  You can also look at a standard MPI reference (C or Fortran)
+  to get some ideas. The principles are the same in pypar except
+  that many calls have been simplified.
+  
+
+      
+PYPAR REFERENCE
+  Here is a list of functions provided by pypar:
+  (See section on Data types for explanation of 'vanilla').  
+  
+  Identification:
+  ---------------
+    
+  size() -- Number of processors
+  rank() -- Id of current processor
+  get_processor_name() -- Return host name of current node  
+  
+  Basic send forms:
+  --------------------
+  send(x, destination)
+    Sends data x of any type to destination with default tag. 
+        
+  send(x, destination, tag=t)
+    Sends data x of any type to destination with tag t. 
+    
+  send(x, destination, use_buffer=True)
+    Sends data x of any type to destination 
+    assuming that recipient will specify a suitable buffer.
+    
+  send(x, destination, bypass=True)
+    Send Numeric array of any type to recipient assuming 
+    that a suitable buffer has been specified and that 
+    recipient also specifies bypass=True
+        
+	
+  Basic receive forms:	
+  --------------------
+  y=receive(source)
+    receives data y of any type from source with default tag.
+      
+  y=receive(source, tag=t)
+    receives data y of any type from source with tag t.
+  
+  y,status=receive(source, return_status=True)
+    receives data y and status object from source 
+    
+  y=receive(source, buffer=x)    
+    receives data y from source and puts
+    it in x (which must be of compatible size and type).
+    It also returns a reference to x.
+    (Although it will accept all types this form is thought to be used 
+    mainly for Numeric arrays).    
+
+  Collective Communication:
+  -------------------------  
+    
+  broadcast(x, root):
+    Broadcasts x from root to all other processors.
+    All processors must issue the same bcast.
+
+  gather(x, root):
+     Gather all elements in x to buffer of 
+     size len(x) * numprocs 
+     created by this function.
+     If x is multidimensional buffer will have
+     the size of zero'th dimension multiplied by numprocs.
+     A reference to the created buffer is returned.
+
+  gather(x, root, buffer=y):
+     Gather all elements in x to specified buffer y
+     from source.
+     Buffer must have size len(x) * numprocs and 
+     shape[0] == x.shape[0]*numprocs.
+     A reference to the buffer y is returned.     
+
+  scatter(x, root):
+     Scatter all elements in x from root to all other processors
+     in a buffer created by this function.
+     A reference to the created buffer is returned.
+
+  scatter(x, root, buffer=y):
+     Scatter all elements in x from root to all other processors
+     using specified buffer y.
+     A reference to the buffer y is returned.     
+
+  reduce(x, op, root):
+     Reduce all elements in x at root
+     applying operation op elementwise and return result in 
+     buffer created by this function.  
+     A reference to the created buffer is returned.          
+
+  reduce(x, op, root, buffer=y):
+     Reduce all elements in x to specified buffer y
+     (of the same size as x)
+     at source applying operation op elementwise.
+     A reference to the buffer y is returned.               
+
+     
+  Other functions:
+  ----------------     
+		      		         
+  time() -- MPI wall time
+  barrier() -- Synchronisation point. Makes processors wait until all 
+               processors have reached this point.
+  abort() -- Terminate all processes. 
+  finalize() -- Cleanup MPI. No parallelism can take place after this point. 
+  initialized() -- True if MPI has been initialised
+
+  
+  See pypar.py for doc strings on individual functions.   
+
+
+    
+DATA TYPES
+  Pypar automatically handles different data types differently
+  There are three protocols:
+    'array': Numeric arrays of type Int ('i', 'l'), Float ('f', 'd'),
+             or Complex ('F', 'D') can be communicated
+             with the underlying mpiext.send_array and mpiext.receive_array. 
+	     This is the fastest mode.
+	     Note that even though the underlying C implementation does not
+	     support Complex as a native datatype, pypar handles them 
+	     efficiently and seemlessly by transmitting them as arrays of 
+	     floats of twice the size.  
+    'string': Text strings can be communicated with mpiext.send_string and
+              mpiext.receive_string.
+    'vanilla': All other types can be communicated using the scripts 
+               send_vanilla and receive_vanilla provided that the objects 
+	       can be serialised using
+               pickle (or cPickle). The latter mode is less efficient than the
+               first two but it can handle general structures.
+
+     Rules:
+     If keyword argument vanilla == 1, vanilla is chosen regardless of 
+     x's type.
+     Otherwise if x is a string, the string protocol is chosen
+     If x is an array, the 'array' protocol is chosen provided that x has one
+     of the admissible typecodes.
+     
+  Function that take vanilla as a keyword argument can force vanilla mode
+  on any datatype.     
+  
+STATUS OBJECT
+  A status object can be optionally returned from receive by
+  specifying return_status=True in the call.
+  The status object can subsequently be queried for information about the communication.
+  The fields are:	
+    status.source: The origin of the received message (use e.g. with pypar.any_source)
+    status.tag: The tag of the received message (use e.g. with pypar.any_tag)    
+    status.error: Error code generated by underlying C function
+    status.length: Number of elements received
+    status.size: Size (in bytes) of one element
+    status.bytes(): Number of payload bytes received (excl control info)
+
+  The status object is essential when use together with any_source or any_tag.  	
+  
+
+EXTENSIONS
+  At this stage only a subset of MPI is implemented. However,
+  most real-world MPI programs use only these simple functions.
+  If you find that you need other MPI calls, please feel free to
+  add them to the C-extension. Alternatively, drop me note and I'll
+  use that as an excuse to update pypar.
+  
+  
+PERFORMANCE
+  If you are passing simple Numeric arrays around you can reduce 
+  the communication time by using the 'buffer' keyword arguments 
+  (see REFERENCE above). These version are closer to the underlying MPI 
+  implementation in that one must provide receive buffers of the right size.
+  However, you will find that these version have lower latency and 
+  can be somewhat faster as they bypass 
+  pypar's mechanism for automatically transferring the needed buffer size.
+  Also, using simple numeric arrays will bypass pypar's pickling of general 
+  structures.
+
+  This will mainly be of interest if you are using a 'fine grained' 
+  parallelism, i.e. if you have frequent communications.	
+  
+  Try both versions and see for yourself if there is any noticable difference.
+  
+
+PROBLEMS
+  If you encounter any problems with this package please drop me a line.
+  I cannot guarantee that I can help you, but I will try.
+  
+  Ole Nielsen
+  Australian National University
+  Email: Ole.Nielsen@anu.edu.au
+   
+  
+REFERENCES   
+  To learn more about MPI in general, see WEB sites
+    http://www-unix.mcs.anl.gov/mpi/
+    http://www.redbooks.ibm.com/pubs/pdfs/redbooks/sg245380.pdf
+
+  or books  
+    "Using MPI, 2nd Edition", by Gropp, Lusk, and Skjellum, 
+    "The LAM companion to 'Using MPI...'" by Zdzislaw Meglicki
+    "Parallel Programming With MPI", by Peter S. Pacheco
+    "RS/6000 SP: Practical MPI Programming", by Yukiya Aoyama and Jun Nakano 
+     
+  To learn more about Python, see the WEB site
+    http://www.python.org
+    
+         
+
+
+
+
+
+
Index: trunk/anuga_core/source/pypar_dist/documentation/FAQ
===================================================================
--- trunk/anuga_core/source/pypar_dist/documentation/FAQ	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/documentation/FAQ	(revision 8494)
@@ -0,0 +1,46 @@
+
+Q: If I broadcast a general structure how can I specify a
+   suitable buffer for the receiving processes?
+
+A: You must create something large enough to acccommodate the 
+   structure (e.g a list conating blanks of sufficent size [' ']*N).
+   The size of N must be estimated but it is OK to be larger   
+   than the broadcasted structure.
+   
+   
+Q: If I broadcast a general structure and specify a string as buffer
+   I receve only garbage.
+   
+A: If you specify a string or a Numerical array as receive buffer
+   you must use vanilla=True in the broadcast call. Otherwise the 
+   receiving processes will think that they are receiving data with
+   the type of the specified buffer.      
+
+Q: Why do I have to specify a buffer in broadcast?
+A: Issuing the command broadcast(x,  root=0) on all processors 
+   will send the object x from processor 0 to x in every other
+   processor.
+   The same command is issued on all processors so x doubles as
+   send buffer on root and receive buffer on the receiving processors.
+
+   
+
+INSTALLATION
+--------------
+
+
+Q: I installed the pypar package exactly as instructed, everything compiled 
+   fine etc.  However, when I run python and enter `import pypar '  it says 
+   that mpiext.c hasn't been compiled, even though there is an mpiext.so 
+   sitting right in the directory.  I installed it with  --prefix=~. 
+
+A: Because you are installing into your own home directory
+   you need to update your PYTHONPATH so that python can find the installed modules.
+   It should be set to ~/python by adding something like the following to 
+   your startup script (here bash)
+
+      export PYTHONPATH=.:$HOME/python:${PYTHONPATH} 
+
+
+   This is not specific to pypar but general for all packages installed locally.
+   
Index: trunk/anuga_core/source/pypar_dist/documentation/manuals/install_notes.txt
===================================================================
--- trunk/anuga_core/source/pypar_dist/documentation/manuals/install_notes.txt	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/documentation/manuals/install_notes.txt	(revision 8494)
@@ -0,0 +1,23 @@
+Install notes for Debian Linux:
+
+# Python packages
+python2.5
+python-dev
+python-numpy
+python-numpy-dev
+python-imaging (PIL)
+
+
+# MPI
+lam4c2
+lam4-dev
+lam-runtime
+
+
+# .bashrc
+
+# If using local pypar installation
+export PYTHONPATH=$PYTHONPATH:/home/work/sandpit/pypar/source
+
+# MPI
+alias mpirun='/usr/bin/mpirun -x PYTHONPATH'
Index: trunk/anuga_core/source/pypar_dist/documentation/manuals/pypar_installation.tex
===================================================================
--- trunk/anuga_core/source/pypar_dist/documentation/manuals/pypar_installation.tex	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/documentation/manuals/pypar_installation.tex	(revision 8494)
@@ -0,0 +1,104 @@
+        
+a native MPI library (e.g.\ LAM/MPI or MPICH),
+that 
+that the MPI runtime system is up and running (e.g.\ using \texttt{lamboot}).
+If not please see the installation notes at the pypar web site.
+
+        
+%Taken from exercise 5 
+
+\subsection*{Preliminaries}
+
+\subsubsection*{The shell configuration}
+Log in using the username you created previously.
+You will need to configure a few environment variables to make 
+life simpler.
+Environment variables control the way you personally interact with the
+shell and they are defined in the file \texttt{.bashrc} which lives in
+your home directory. Edit \texttt{.bashrc} and add the following lines
+
+\begin{verbatim}
+export PATH=.:$HOME/bin:${PATH}
+export LAMRSH="ssh -x"
+\end{verbatim}
+
+The first line asks the shell to always look for files in the directory
+you are working in. The second tells the MPI runtime system to use 
+secure shell (ssh) without graphics capabilities (X11). 
+Close the shell window and open a new one for this to take effect.  
+
+\subsubsection*{Setting up the secure shell environment}
+
+Next we will make sure that you can access all machines without having to
+type in your password every time. The secure shell (ssh) has a capability
+to store your password encrypted on your disk and automatically provide
+it when needed. To accomplish this you need the following steps:
+
+\begin{enumerate} 
+  \item ssh-keygen -t rsa (Generate codes for the encryption)\\
+  \emph{When asked for a filename, just go with the suggested default.
+  When asked for a passphrase just press \textbf{Enter} twice}
+  \item cd \verb+~/.ssh+  
+  \item cp id\_rsa.pub authorized\_keys   
+\end{enumerate}
+
+Whenever you try to log into a remote host using ssh, your host will 
+negotiate with the remote host using these keys and if accepted provide
+the password automatically. To test try
+\begin{verbatim}
+  ssh ninja-n whoami
+\end{verbatim}
+where 
+\[
+  n = 1, 2, \ldots 8
+\]
+This will log you in briefly issue the command \texttt{whoami},
+display the result and log you out again.
+
+The first time you do this you will see the message
+\begin{verbatim}
+The authenticity of host 'ninja-8 (192.168.1.8)' can't be established.
+RSA key fingerprint is 06:eb:b3:e0:f1:57:35:0e:58:49:b4:48:46:fb:5c:aa.
+Are you sure you want to continue connecting (yes/no)?
+\end{verbatim}
+which means that this is the first time the encrypted connection 
+is to be established. Just type \texttt{yes}.
+You might also see a warning message about X11 forwarding which 
+you can ignore. From now on you should be able to move freely among 
+the hosts in our cluster.
+See \texttt{man ssh} for more information.
+
+\subsection*{The MPI runtime system}
+
+We are now ready to start the \emph{MPI runtime system} which comprises a
+number of servers (one on each machine) who will allow you to execute
+MPI programs in parallel.
+
+The runtime system needs a configuration file listing 
+the hosts who are participating in the cluster.
+Create a file called \texttt{.lamhosts} in your 
+home directory (\verb+~+). 
+To begin with list \emph{only} the machine you are working on, e.g.
+\begin{verbatim}
+ninja-n
+\end{verbatim} 
+where n is the number you are working on.
+Later we will add more machines.
+%namely \texttt{ninja-1} to \texttt{ninja-8}.
+
+The MPI runtime is then started by issuing the command
+\begin{verbatim}
+  lamboot -v ~/.lamhosts
+\end{verbatim}
+which reads the configuration file, logs in to all listed hosts and 
+start the MPI servers.  
+You should see a diagnostic message like
+\begin{verbatim}
+LAM 6.5.8/MPI 2 C++/ROMIO - Indiana University
+
+Executing hboot on n2 (ninja-n - 1 CPU)...
+\end{verbatim}
+and you are ready to run MPI.
+
+        
+
Index: trunk/anuga_core/source/pypar_dist/documentation/manuals/pypar_tutorial.tex
===================================================================
--- trunk/anuga_core/source/pypar_dist/documentation/manuals/pypar_tutorial.tex	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/documentation/manuals/pypar_tutorial.tex	(revision 8494)
@@ -0,0 +1,755 @@
+\documentclass[12pt]{article}
+
+\usepackage{url}
+\usepackage{times}
+\usepackage[dvips]{graphicx}
+
+%\textheight=25cm
+\textwidth=14cm
+%\topmargin=-4cm
+%\oddsidemargin=0cm
+
+
+
+\begin{document}
+
+\title{Pypar Tutorial\\
+Building a Parallel Program using Python}
+
+
+\author{Ole Nielsen \\
+Australian National University, Canberra \\
+Ole.Nielsen@anu.edu.au}
+
+\maketitle
+\section*{Introduction}
+
+This is a tutorial demonstrating essential 
+features of parallel programming and building 
+a simple Python MPI program using the MPI binding \texttt{pypar} available at
+\url{http://datamining.anu.edu.au/software/pypar}.
+
+It is assumed that \texttt{pypar} has been installed on your system.
+If not, see the installation document (\texttt{pypar\_installation.pdf}) 
+that comes with pypar
+(or get it from the web site at 
+\url{http://datamining.anu.edu.au/~ole/pypar/pypar_installation.pdf}).
+
+For a reference to all of Pypar's functionality, see the 
+reference manual (\texttt{pypar\_reference.pdf}) 
+that comes with pypar (or get it from the web site at 
+\url{http://datamining.anu.edu.au/~ole/pypar/pypar_reference.pdf}).
+
+
+\section{The Message Passing Interface --- MPI}
+MPI is the defacto standard for parallel programming of both
+distributed computer clusters of and shared memory architectures.
+MPI programming takes the form of special commands that are
+imported from the MPI library. They provide information to the program 
+about how many processes are available, which number each process has been 
+allocated and functions to communicate with other processes.
+See \url{http://www.netlib.org/utk/papers/mpi-book/mpi-book.html} for
+a comprehensive reference to MPI (based on 
+the C or Fortran programming languages).
+
+It is said that MPI is both large and small. What is meant is that the
+MPI standard has hundreds of functions in it. However, many of the
+advanced routines represent functionality that can be ignored unless
+one pursues added flexibility (data types), overlap of computation and
+communication (nonblocking send/receive), modularity (groups,
+communicators), or convenience (collective operations, topologies).
+MPI is said to be small because the majority of useful and efficient 
+parallel programs are written using only a small essential subset of MPI.
+In fact, it can be argued that parallel performance is best achieved by 
+keeping the parallelism simple. 
+
+Pypar is a Python binding to such a subset.
+
+
+\section{A simple Pypar MPI program}
+Let us start with a simple example: a Python program 
+that imports the pypar module and 
+reports the number of processors, local id and host name
+This is the Pypar equivalent of the infamous 'hello world' 
+program if you like. 
+Make a file called \texttt{ni.py} using your favourite
+editor\footnote{The author believs that your favourite editor should 
+be GNU/Emacs, but it is up to you, of course.}
+and type in the following contents:
+{\footnotesize
+\begin{verbatim}
+import pypar
+
+p = pypar.rank()
+P = pypar.size()
+node = pypar.get_processor_name()
+
+print "I am process %d of %d on node %s" %(p, P, node)
+
+pypar.finalize()
+\end{verbatim}}
+\noindent and run it as follows:
+\begin{verbatim}
+mpirun -np 4 python ni.py
+\end{verbatim}
+You should get a greeting from each of the four nodes looking 
+something like
+\begin{verbatim}
+Pypar (version 1.9) initialised MPI OK with 4 processors
+I am process 0 of 4 on node ninja-1
+I am process 3 of 4 on node ninja-4
+I am process 1 of 4 on node ninja-2
+I am process 2 of 4 on node ninja-3
+\end{verbatim}
+The order of output may be arbitrary!  If four copies of the
+program is run, print is called four times. The order in which each
+process executes the message is undetermined, based on when they each
+reach that point in their execution of the program, and how they
+travel on the network.
+\emph{Note that all the print's, though they come from different
+  processes, will send their output intact to your shell window; this
+  is generally true of output commands.
+  Input commands, like \texttt{raw_input},
+  will only work on the process with rank zero.}
+
+  
+\subsection{The MPI execution model}
+
+It is important to understand that in MPI (and therefore also in 
+Pypar), multiple identical copies of this program 
+will start simultaneously in separate processes.
+These processes may run on one machine or on multiple machines.
+This is a fundamental difference from ordinary programs, 
+where, if someone sais ``run the program'', it is
+assumed that there was only one instance of the program running.
+
+
+XXXXX
+In a nutshell, this program sets up a communication group of 4 
+processes, where each process gets its rank (\texttt{p}), prints it, 
+and exits. 
+
+.......Explain in detail like below..
+
+
+
+
+The first line,
+\texttt{\#include <stdio>}
+should be familiar to all C programmers. It includes the standard
+input/output routines like printf. The second line,
+\texttt{\#include <mpi.h>}
+includes the MPI functions. The file \texttt{mpi.h} 
+contains prototypes for all the
+MPI routines in this program; this file is located in
+\texttt{/usr/include/lam/mpi.h} in case you actually want to look at it.
+The program starts with the main... line which takes the two arguments
+argc and argv (which is normal in C): The argument \texttt{argc} is
+the number of commandline arguments given to the program (including itself)
+and \texttt{argv} is an array of strings containing them. For example
+argv[0] will contain the program name itself. We will not be using them 
+directly but MPI needs access to them through \texttt{MPI\_init}.
+Then the program declares one integer variable, \texttt{myid}. The
+first step of the program,
+\begin{verbatim} 
+      MPI_Init(&argc,&argv);
+\end{verbatim}       
+calls MPI\_Init to set up everything. 
+This should be the first command executed in all MPI programs. This
+routine takes pointers to \texttt{argc} and \texttt{argv}, 
+looks at them, pulls out the purely
+MPI-relevant things, and generally fixes them so you can use command line
+arguments as normal.
+
+\medskip
+
+Next, the program runs \texttt{MPI\_Comm\_rank}, 
+passing it the default communicator MPI\_COMM\_WORLD 
+(which is the group of \emph{all} processes started)
+and a \emph{pointer} to \texttt{myid}. Passing a pointer is the way 
+C can change the value of te argument \texttt{myid}.
+\texttt{MPI\_Comm\_rank} will set \texttt{myid} to the rank of the machine 
+on which the program is running. 
+Remember that in reality, several instances of this program
+start up in several different processes when this program is run. Each
+will receive a unique number from \texttt{MPI\_Comm\_rank}.
+Because multiple copies are running, each will execute all lines 
+in the program including the \texttt{printf} statement which prints a 
+message and the rank. 
+After doing everything else, the program calls \texttt{MPI\_Finalize},
+which generally terminates everything and shuts down MPI. This should be the
+last command executed in all MPI programs.
+
+\subsection*{Compile the MPI program}
+
+Normally, C programs are compiled with a command such as 
+\texttt{gcc hello.c -o hello.x}. However, MPI programs need access to 
+more libraries which are provided through the wrapper program 
+\texttt{mpicc}. This can be used as usual. Try to run
+\begin{verbatim} 
+  mpicc hello.c -o hello.x
+\end{verbatim}
+If you made a syntactically correct program a \emph{compiled} 
+program will appear with the name you specified after the -o option.
+If not you will see some error messages and you'll have to correct 
+those before continuing.
+
+\subsection*{Run the MPI program}
+In order to run an MPI compiled program, you must type: 
+\begin{verbatim}
+mpirun -np <number of processes> [options] <program name and arguments>
+\end{verbatim} 
+where you specify the number of processes on which you want to run 
+your parallel program, optional mpirun options, 
+and your program name and its expected arguments. 
+In this case try:
+\begin{verbatim}
+    mpirun -np 1 hello.x
+\end{verbatim}
+This will run one copy of your program and the output should like this
+\begin{verbatim} 
+Sawatdii khrap thuk thuk khon (Process 0)
+\end{verbatim}  
+
+Now try to start four processes by running \texttt{mpirun -np 4 hello.x}.
+You should see the following output:
+\begin{verbatim}
+Sawatdii khrap thuk thuk khon (Process 0)
+Sawatdii khrap thuk thuk khon (Process 3)
+Sawatdii khrap thuk thuk khon (Process 1)
+Sawatdii khrap thuk thuk khon (Process 2)
+\end{verbatim}
+Note that the order of output may be arbitrary!  If four copies of the
+program is run, printf is called four times.  The order in which each
+process executes the message is undetermined, based on when they each
+reach that point in their execution of the program, and how they
+travel on the network. Your guess is as good as mine.  
+\emph{Note that all the printf's, though they come from different
+  processes, will send their output intact to your shell window; this
+  is generally true of output commands. Input commands, like scanf,
+  will only work on the process with rank zero.}
+
+\subsubsection*{Exercise}
+The MPI command \texttt{MPI\_Comm\_size(MPI\_COMM\_WORLD, \&proc);} will
+store the total number of processes started in the integer variable
+\texttt{number\_of\_processes}. Modify the program 
+to give the following output:
+\begin{verbatim}
+Sawatdii khrap thuk thuk khon (Process 0 of 4)
+Sawatdii khrap thuk thuk khon (Process 3 of 4)
+Sawatdii khrap thuk thuk khon (Process 1 of 4)
+Sawatdii khrap thuk thuk khon (Process 2 of 4)
+\end{verbatim}
+when started with four processes.
+ 
+
+\subsection*{Running on multiple computers}  
+
+So far we have only \emph{simulated} the parallelism by running 
+multiple processes on one machine. 
+We will now add host specific information to the program and then run it 
+in parallel.
+
+Add the following declarations to your program:
+\begin{verbatim}
+int  namelen;
+char processor_name[MPI_MAX_PROCESSOR_NAME];
+\end{verbatim}
+
+Add the command
+\begin{verbatim}
+MPI_Get_processor_name(processor_name, &namelen);  
+\end{verbatim}
+This will store the hostname of the processor executing the given process 
+along with the length of the hostname. Then modify the print statement to   
+\begin{verbatim}
+  printf("Sawatdii khrap thuk thuk khon (Process %d of %d running on %s)\n", 
+         myid, number_of_processes, processor_name);
+\end{verbatim}
+Compiling and running the program you should see the folowing output
+\begin{verbatim}
+Sawatdii khrap thuk thuk khon (Process 0 of 4 running on ninja-n)
+Sawatdii khrap thuk thuk khon (Process 1 of 4 running on ninja-n)
+Sawatdii khrap thuk thuk khon (Process 2 of 4 running on ninja-n)
+Sawatdii khrap thuk thuk khon (Process 3 of 4 running on ninja-n)
+\end{verbatim}
+We are still running all out MPI processes on one computer but we are 
+now ready to run this program in parallel.
+
+Edit the file \texttt{~/.lamhosts} to contain all machines in our network
+and run \texttt{lamboot -v ~/.lamhosts} again. You should see the following 
+diagnostic message:
+\begin{verbatim} 
+LAM 6.5.8/MPI 2 C++/ROMIO - Indiana University
+
+Executing hboot on n0 (ninja-1 - 1 CPU)...
+Executing hboot on n1 (ninja-2 - 1 CPU)...
+Executing hboot on n2 (ninja-3 - 1 CPU)...
+Executing hboot on n3 (ninja-4 - 1 CPU)...
+Executing hboot on n4 (ninja-5 - 1 CPU)...
+Executing hboot on n5 (ninja-6 - 1 CPU)...
+Executing hboot on n6 (ninja-7 - 1 CPU)...
+Executing hboot on n7 (ninja-8 - 1 CPU)...
+topology done
+\end{verbatim}
+\noindent Finally try to run your program again and verify that it runs 
+on different nodes.
+What happens if you run more processes than there are physical machines?
+
+       
+\section*{Optional: synchronise the clocks}
+
+\textbf{\emph{Note: This an optional exercise for those who can't get enough!}}
+
+The clocks in our cluster are not synchronised: They show differenttimes.
+Try for example to run
+\begin{verbatim}
+ssh -x ninja-2 date; ssh -x ninja-5 date; ssh -x ninja-3 date; date
+You will see that the times listed differ significantly.
+\end{verbatim}
+
+Try and search for things like
+\texttt{synchronize time network}
+and see if you can figure out how to do it.
+
+
+\section*{MPI resources}
+
+\begin{itemize} 
+  \item \url{http://www.cs.appstate.edu/~can/classes/5530} (exellent introduction 
+  and many useful links)
+  \item http://www-unix.mcs.anl.gov/mpi/mpich (The MPICH implementation)
+  \item http://www.lam-mpi.org (The LAM-MPI implementation)
+  \item http://www.netlib.org/utk/papers/mpi-book/mpi-book.html\\
+    (The complete MPI reference)
+\end{itemize} 
+
+
+
+\section*{Exercise 6}
+
+In this exercise we will continue the introduction to MPI
+by using the local id to differentiate work and 
+by familiarising ourselves with the basic send and receive 
+primitives.
+We will then proceed to measure the fundamental characteristics of 
+the network: namely latency and bandwidth. 
+
+
+
+\subsection*{Using local id to differentiate work}        
+In Exercise 5 we had every processor write to the screen. This 
+is convenient for debugging purposes 
+and I suggest you start every parallel program 
+with a message from each processor about its rank, the total number 
+of processors and the hostname it is running on (see below).
+
+However, it is often also desirable to have only on 
+processor output to screen
+for example to report the final result of a calculation.
+
+\subsubsection*{Exercise}  
+Write a parallel program that 
+produces something like the following output when run on four processors:
+\begin{verbatim}
+  P2/4: Initialised OK on host ninja-6
+  P0/4: Initialised OK on host ninja-4
+  P0: Program terminating  
+  P1/4: Initialised OK on host ninja-5
+  P3/4: Initialised OK on host ninja-7      
+\end{verbatim}
+Arrange the program such that 
+all processors report on their initialisation but 
+only one, processor 0, reports on its termination just prior 
+to \texttt{MPI\_Finalize()}. 
+Note, as in Exercise 5, that the order of output is 
+arbitrary.
+%\emph{Hint: Use the value of myid to decide whether to print or not}.
+
+\subsection*{A simple send and receive pair}
+In this part we will use the fundamental MPI send and receive commands 
+to communicate among the processors. The commands are: 
+\begin{itemize}
+  \item \texttt{MPI\_Send}: 
+  This routine performs a basic send; this routine may block until 
+  the message
+  is received, depending on the specific implementation of MPI.
+  \begin{verbatim}  
+  int MPI_Send(void* buf, int count, MPI_Datatype datatype,
+                     int dest, int tag, MPI_Comm comm)
+  Input:
+      buf  - initial address of send buffer
+      count - number of elements in send buffer (nonnegative integer)
+      datatype - datatype of each send buffer element
+      dest - rank of destination (integer)
+      tag  - message tag (integer)
+      comm - communicator
+  \end{verbatim} 
+  \textbf{Example}: To send one integer, A say, to processor 3 with tag 13:
+  \begin{verbatim}
+    MPI_Send(&A, 1, MPI_INT, 3, 13, MPI_COMM_WORLD);  
+  \end{verbatim}
+  \item \texttt{MPI\_Recv}:
+    This routine performs a basic receive.
+  \begin{verbatim}
+  int MPI_Recv(void* buf, int count, MPI_Datatype datatype,
+                     int source, int tag, MPI_Comm comm,
+                     MPI_Status *status)
+
+  Input:
+      count - maximum number of elements in receive buffer
+             (integer)
+      datatype - datatype of each receive buffer element
+      source - rank of source (integer)
+      tag  - message tag (integer)
+      comm - communicator
+                     
+  Output:
+      buf  - initial address of receive buffer
+      status - status object, provides information about
+               message received;
+      status is a structure of type MPI_Status, the element
+      status.MPI_SOURCE is the source of the message received,
+      and the element status.MPI_TAG is the tag value.
+
+  \end{verbatim}
+  \textbf{Example}: To receive one integer, B say, from processor 7 
+  with tag 13:
+  \begin{verbatim}
+    MPI_Recv(&B, 1, MPI_INT, 7, 13, MPI_COMM_WORLD, &status);  
+  \end{verbatim}
+\end{itemize} 
+
+\noindent These calls are described in more detail in Ian Foster online book
+in \textbf{MPI Basics}: 
+\url{http://www-unix.mcs.anl.gov/dbpp/text/node96.html}.
+The chapter about \textbf{Asynchronous communication},
+\url{http://www-unix.mcs.anl.gov/dbpp/text/node98.html}, describes how 
+to query the Status Object.
+
+MPI defines the following constants for use with \texttt{MPI\_Recv}:
+\begin{itemize} 
+  \item \texttt{MPI\_ANY\_SOURCE}: Can be used to specify that a message 
+  can be received from anywhere instead of a specific source. 
+  \item \texttt{MPI\_ANY\_TAG}: Can be used to specify that a message 
+  can have any tag instead of a specific tag.
+\end{itemize}   
+
+\subsubsection*{Exercise}
+The following program segment implements a very simple
+communication pattern: Processor 0 sends a number to processor 1 and 
+both outputs a diagnostic message.
+
+Add to your previous program the declarations:
+\begin{verbatim}
+  int A, B, source, destination, tag=13;
+  MPI_Status status;
+\end{verbatim}
+and the following code segment
+\begin{verbatim} 
+  if (myid == 0) {
+    A = 42;
+    destination = 1;
+    printf("P%d: Sending value %d to MPI process %d\n", 
+           myid, A, destination);   
+    MPI_Send(&A, 1, MPI_INT, 1, 13, MPI_COMM_WORLD);
+  } else if (myid == 1) {
+    source = 0;
+    MPI_Recv(&B, 1, MPI_INT, source, 13, MPI_COMM_WORLD, &status);    
+    printf("P%d: Received value %d from MPI process %d\n", myid, B, source);
+  }
+\end{verbatim}
+make sure it can compile and run on 2 processors.
+Verify that your output looks something like  
+\begin{verbatim}
+P0/2: Initialised OK on host ninja-1
+P0: Sending value 42 to MPI process 1
+P0: Terminating
+P1/2: Initialised OK on host ninja-2
+P1: Received value 42 from MPI process 0
+\end{verbatim}
+
+
+
+\subsection*{Send messages around in a ring - Exercise}
+
+Write an MPI program which passes data
+around in a ring structure from process 0 then to process 1 etc.
+When the message reaches the last processor it should be passed back to 
+processor 0 - this forms a 'communication ring'. 
+Every process should add one to the value before passing it on 
+and write out a diagnostic message.
+
+With a starting value of 42 and running on four processors your program 
+should produce the following output:
+\begin{verbatim}
+P0/4: Initialised OK on host ninja-1
+P0: Sending value 42 to MPI process 1
+P1/4: Initialised OK on host ninja-2
+P3/4: Initialised OK on host ninja-4
+P1: Received value 42 from MPI process 0
+P1: Sending value 43 to MPI process 2
+P2/4: Initialised OK on host ninja-3
+P2: Received value 43 from MPI process 1
+P2: Sending value 44 to MPI process 3
+P3: Received value 44 from MPI process 2
+P3: Sending value 45 to MPI process 0
+P0: Received value 45 from MPI process 3
+\end{verbatim}
+\emph{Hint}: You can use the \texttt{modulus} operator to calculate 
+the destination such that the next processor after the last becomes 0:
+\begin{verbatim}
+  destination = (myid + 1) % number_of_processors;
+\end{verbatim}
+
+
+\subsection*{Timing}
+A main reason for doing parallel computing is that of faster execution.
+Therefore \emph{timing} is an integral part of this game.
+MPI provides a function \texttt{MPI\_Wtime} which counts seconds.
+To time a code segment with \texttt{MPI\_Wtime} do something like this:
+\begin{verbatim}
+  double t0;
+  
+  t0=MPI_Wtime();
+  // (Computations)
+  printf("Time = %.6f sec\n", MPI_Wtime() - t0);
+\end{verbatim}
+
+\subsubsection*{Exercise}
+Time the previous 'ring' program in such a way that only process 0
+measures the time and writes it to the screen
+
+
+\subsection*{Measure network latency and bandwidth}
+
+The time it takes to transmit a message can be approximated by the model
+\[
+  t = t_l + \alpha t_b
+\]
+where 
+\begin{itemize} 
+  \item $t_l$ is the \emph{latency}. The latency is the 
+  time it takes to start the transmission before anything 
+  gets communicated. This involves opening the network connection,
+  buffering the message and so on.
+  \item $t_b$ is the time it takes to communicate one byte once 
+  the connection is open. This is related to the \emph{bandwidth} $B$, 
+  defined as the number of bytes transmitted per second, as
+  \[
+    B = \frac{1}{t_b}
+  \]
+  \item $\alpha$ is the number of bytes communicated.
+\end{itemize}    
+
+\subsubsection*{Exercise}
+\textbf{Can you work out what the network latency is?}
+\emph{Hint:} Send a very small message (e.g.\ one integer)
+from one processor to another and and back again a number of times 
+and measure the time.    
+
+\subsubsection*{Exercise}
+\textbf{Can you work out how many megabytes the network can transmit per 
+second?} 
+Create arrays of varying size, send them from one processor 
+to another and back and measure the time.
+\emph{Hints:}
+\begin{itemize} 
+  \item To create an array of double precision floating point numbers 
+    do the following.
+    \begin{verbatim}
+      #define MAX_LEN  500000    /* Largest block */
+      double A[MAX_LEN];
+    \end{verbatim}
+  \item To populate the array with pseudo random numbers:  
+  \begin{verbatim} 
+   for (j=0; j<MAX_LEN; j++) A[j]=rand();
+   \end{verbatim}
+   \item To send only part of the array use the second argument in 
+   MPI\_Send and MPI\_Recv to control how much information is transferred.
+   \item MPI\_Send and MPI\_Recv require a \emph{pointer} to the first 
+   element of the array. Since arrays are already defined as pointers
+   simply put \texttt{A} rather than \texttt{\&A} which is what 
+   we used earlier for the 
+   simple integer variable.
+   \item One double precision number takes up 8 bytes - use that in your 
+   estimate.
+\end{itemize} 
+
+Your measurements may fluctuate based on the network traffic and the 
+load on the machines. 
+
+
+\section*{Good MPI resources}
+\begin{itemize} 
+  \item MPI Data types:\\ 
+    \url{http://www.ats.ucla.edu/at/hpc/parallel_computing/mpi-intro.htm}
+  \item Online book by Ian Foster:\\ 
+    \url{http://www-unix.mcs.anl.gov/dbpp/text/node94.html}
+\end{itemize}        
+
+
+
+
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+\section{Identifying the processors}
+
+ 
+%%%%%%%%%%%
+
+
+It is almost impossible to demonstrate parallelism with an interactive
+session at the command prompt as we did with the Python and Maple tutorials.
+Therefore we must write a \emph{program} that can be executed in parallel.
+Start an editor of your choice with a new file
+(called \texttt{paraprime.py}, say). For example:
+\begin{verbatim}
+  emacs paraprime.py &
+\end{verbatim}
+
+\noindent Write the following in the file and save it
+{\small \begin{verbatim}
+import pypar
+
+numproc = pypar.size()
+myid =    pypar.rank()
+
+print "I am proc %d of %d" %(myid, numproc)
+\end{verbatim}}
+
+Then type the following at the Unix prompt to execute the
+program normally on one processor
+{\small \begin{verbatim}
+  python paraprime.py
+\end{verbatim}}
+\noindent You should see the following output
+{\small \begin{verbatim}
+> python paraprime.py
+I am proc 0 of 1
+\end{verbatim}}
+
+
+\noindent Now let us try and run the same program on four processors:
+{\small \begin{verbatim}
+> prun -n 4 python paraprime.py
+I am proc 0 of 4
+I am proc 1 of 4
+I am proc 2 of 4
+I am proc 3 of 4
+\end{verbatim}}
+If you get a message from each of the four processors stating their
+id as above, we know that everything works. We have a parallel program!
+
+\section{Strategies for efficient parallel programming}
+A final note on efficiency.
+
+Parallel programming should lead to faster execution and/or
+ability to deal with larger problems.
+
+The ultimate goal is to be \emph{P} times faster with \emph{P} processors.
+However \emph{speedup} is usually less than \emph{P}. One must address
+three critical issues to achieve good speedup:
+\begin{itemize}
+  \item \textbf{Interprocessor communication:} The amount and the frequency
+  of messages should be kept as low as possible. Ensure that processors
+  have plenty of work to do between communications.
+  \item \textbf{Data distribution and load balancing:} If some processors
+  finish much sooner than others, the total execution time of the parallel
+  program is bounded by that of the slowest processor and we say that the
+  program is poorly load balanced.
+  Ensure that each processor get its fair share of the work load.
+  \item \textbf{Sequential parts of a program:} If half of a program, say, is
+  inherently sequential the speedup can never exceed 2 no matter how well
+  the remaining half is parallelised. This is known as Amdahls law.
+  Ensure that the all cost intensive parts get parallelised.
+\end{itemize}
+
+
+
+
+
+\section*{Exercise 8}
+
+In Exercise 6-7 you sent messages around in rings and back and forth between 
+two processors and also timed the network speed when communicating between to
+processors.
+The purpose of this exercise is to try a collective communication patters
+where on processor sends and gathers data from all the others using the 
+basic MPI\_Send and MPI\_Recv.
+
+\subsection*{Distribute and gather integers}
+
+Create a program where process 0 distributes an integer to all other processes,
+then receives an integer back from each of them.
+The other processes should receive the integer from process 0, add their 
+rank to it and pass it back to process 0.
+
+With some suitable diagnostic output your program should produce 
+something like this when run on four processors: 
+
+\begin{verbatim} 
+P0/4: Initialised OK on host ninja-1
+P0: Sending value 42 to MPI process 1
+P0: Sending value 42 to MPI process 2
+P0: Sending value 42 to MPI process 3
+P3/4: Initialised OK on host ninja-4
+P1/4: Initialised OK on host ninja-2
+P2/4: Initialised OK on host ninja-3
+P3: Received value 42 from MPI process 0
+P3: Sending value 45 to MPI process 0
+P1: Received value 42 from MPI process 0
+P1: Sending value 43 to MPI process 0
+P2: Received value 42 from MPI process 0
+P2: Sending value 44 to MPI process 0
+P0: Received value 43 from MPI process 1
+P0: Received value 44 from MPI process 2
+P0: Received value 45 from MPI process 3
+\end{verbatim}
+
+
+\subsection*{Distribute and gather arrays}
+
+In this part we will do exactly the same as above except we will use
+a double precision array as the data.
+
+\begin{enumerate}
+  \item Create a double precision array (A) of length $N=1024$ to use as buffer
+  \item On process 0 populate it with numbers $0$ to $N-1$ 
+  (using A[i] = (double) i;) 
+  \item Pass A on to all other processors (the workers).
+  \item All workers should compute the $p$-norm of the array
+  where $p$ is the rank of the worker:
+  \[
+    \left(
+      \sum_{i=0}^N A[i]^p
+    \right)^{1/p}
+  \]
+  \item Then return the result as one double precision number per worker and 
+  print them all out on processor 0.
+\end{enumerate} 
+
+\emph{Hints:}
+\begin{itemize}
+  \item To compute $x^y$ use the C-function pow(x, y);
+  \item Include the math headers in your program: \verb+#include <math.h>+
+  \item You'll also need to link in the math libraries as follows: 
+     mpicc progname.c -lm
+  %\item You might want to add many print statements reporting the communication pattern in    
+\end{itemize}   
+
+If your pogram works it should produce something like this on four processors.
+\begin{verbatim}
+P0/4: Initialised OK on host ninja-1
+P3/4: Initialised OK on host ninja-4
+P2/4: Initialised OK on host ninja-3
+P1/4: Initialised OK on host ninja-2
+P0: Received value 523776.00 from MPI process 1
+P0: Received value 18904.76 from MPI process 2
+P0: Received value 6497.76 from MPI process 3
+\end{verbatim}
+
+
+
+
+\end{document}
Index: trunk/anuga_core/source/pypar_dist/lib/python2.7/site-packages/Pypar-2.1.4-py2.7.egg-info
===================================================================
--- trunk/anuga_core/source/pypar_dist/lib/python2.7/site-packages/Pypar-2.1.4-py2.7.egg-info	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/lib/python2.7/site-packages/Pypar-2.1.4-py2.7.egg-info	(revision 8494)
@@ -0,0 +1,10 @@
+Metadata-Version: 1.0
+Name: Pypar
+Version: 2.1.4
+Summary: Pypar - Parallel Python
+Home-page: http://sourceforge.net/projects/pypar
+Author: Ole M. Nielsen
+Author-email: ole.moller.nielsen@gmail.com
+License: UNKNOWN
+Description: Pypar - Parallel Python, no-frills MPI interface
+Platform: UNKNOWN
Index: trunk/anuga_core/source/pypar_dist/lib/python2.7/site-packages/pypar/__init__.py
===================================================================
--- trunk/anuga_core/source/pypar_dist/lib/python2.7/site-packages/pypar/__init__.py	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/lib/python2.7/site-packages/pypar/__init__.py	(revision 8494)
@@ -0,0 +1,42 @@
+# =============================================================================
+# pypar.py - Parallel Python using MPI
+# Copyright (C) 2001, 2002, 2003 Ole M. Nielsen 
+#              (Center for Mathematics and its Applications ANU and APAC)
+#
+#    This program is free software; you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License as published by
+#    the Free Software Foundation; either version 2 of the License, or
+#    (at your option) any later version.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License (http://www.gnu.org/copyleft/gpl.html)
+#    for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program; if not, write to the Free Software
+#    Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307
+#
+#
+# Contact address: Ole.Moller.Nielsen@gmail.com
+#
+# Version: See pypar.__version__
+# =============================================================================
+
+"""The python module pypar.py and the C-extension mpi.c 
+implements scalable parallelism on distributed and shared 
+memory architectures using an important subset of 
+the Message Passing Interface (MPI) standard.
+"""
+
+from pypar import *
+from __metadata__ import __version__, __date__, __author__
+
+#Add path of package to PYTHONPATH to allow C-extension to be loaded
+#import sys
+#sys.path += __path__
+
+
+
+
Index: trunk/anuga_core/source/pypar_dist/lib/python2.7/site-packages/pypar/__metadata__.py
===================================================================
--- trunk/anuga_core/source/pypar_dist/lib/python2.7/site-packages/pypar/__metadata__.py	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/lib/python2.7/site-packages/pypar/__metadata__.py	(revision 8494)
@@ -0,0 +1,6 @@
+# Meta data for pypar
+__version__ = '2.1.4'
+__date__ = '22 December 2010'
+__author__ = 'Ole M. Nielsen'
+
+
Index: trunk/anuga_core/source/pypar_dist/lib/python2.7/site-packages/pypar/compile.py
===================================================================
--- trunk/anuga_core/source/pypar_dist/lib/python2.7/site-packages/pypar/compile.py	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/lib/python2.7/site-packages/pypar/compile.py	(revision 8494)
@@ -0,0 +1,206 @@
+"""compile.py - compile Python C-extension
+
+   Commandline usage: 
+     python compile.py <filename>
+
+   Usage from within Python:
+     import compile
+     compile.compile(<filename>,..)
+
+   Ole Nielsen, Oct 2001      
+"""   
+
+import numpy
+def compile(FNs=None, CC=None, LD = None, SFLAG = None, verbose = 1):
+  """compile(FNs=None, CC=None, LD = None, SFLAG = None):
+  
+     Compile FN(s) using compiler CC (e.g. mpicc), 
+     Loader LD and shared flag SFLAG.
+     If CC is absent use default compiler dependent on platform
+     if LD is absent CC is used.
+     if SFLAG is absent platform default is used
+     FNs can be either one filename or a list of filenames
+     In the latter case, the first will be used to name so file.
+  """
+  import os, string, sys, types
+  
+  # Input check
+  #
+  assert not FNs is None, 'No filename provided'
+
+  if not type(FNs) == types.ListType:
+    FNs = [FNs]
+
+  # Determine platform and compiler
+  #
+  if sys.platform == 'sunos5':  # Solaris
+    if CC:
+      compiler = CC
+    else:  
+      compiler = 'gcc'
+    if LD:
+      loader = LD
+    else:  
+      loader = compiler
+    if SFLAG:
+      sharedflag = SFLAG
+    else:  
+      sharedflag = 'G'
+      
+  elif sys.platform == 'osf1V5':  # Compaq AlphaServer
+    if CC:
+      compiler = CC
+    else:  
+      compiler = 'cc'
+    if LD:
+      loader = LD
+    else:  
+      loader = compiler
+    if SFLAG:
+      sharedflag = SFLAG
+    else:  
+      sharedflag = 'shared'    
+      
+  elif sys.platform == 'linux2':  # Linux
+    if CC:
+      compiler = CC
+    else:  
+      compiler = 'gcc'
+    if LD:
+      loader = LD
+    else:  
+      loader = compiler
+    if SFLAG:
+      sharedflag = SFLAG
+    else:  
+      sharedflag = 'shared'    
+      
+  elif sys.platform == 'darwin':  # Mac OS X:
+    if CC:
+      compiler = CC
+    else:  
+      compiler = 'cc'
+    if LD:
+      loader = LD
+    else:  
+      loader = compiler
+    if SFLAG:
+      sharedflag = SFLAG
+    else:  
+      sharedflag = 'bundle -flat_namespace -undefined suppress'
+      
+  else:
+    if verbose: print "Unrecognised platform %s - revert to default"\
+                %sys.platform
+    
+    if CC:
+      compiler = CC
+    else:  
+      compiler = 'cc'
+    if LD:
+      loader = LD
+    else:  
+      loader = 'ld'
+    if SFLAG:
+      sharedflag = SFLAG
+    else:  
+      sharedflag = 'G'
+
+            
+  # Find location of include files
+  #
+  python_include = os.path.join(os.path.join(sys.exec_prefix, 'include'),
+                                'python'+sys.version[:3])
+  
+  
+  # Check existence of Python.h
+  #
+  headerfile = python_include + '/Python.h'
+  try:
+    open(headerfile, 'r')
+  except:
+    raise """Did not find Python header file %s.
+    Make sure files for Python C-extensions are installed. 
+    In debian linux, for example, you need to install a
+    package called something like python2.1-dev""" % headerfile
+
+
+  # Get numpy include
+  numpy_include = numpy.get_include()
+  
+  
+  # Check filename(s)
+  #
+  object_files = ''
+  for FN in FNs:        
+    root, ext = os.path.splitext(FN)
+    if ext == '':
+      FN = FN + '.c'
+    elif ext.lower() != '.c':
+      msg = 'Unrecognised extension: ' + FN
+      raise Exception(msg)
+    
+    try:
+      open(FN,'r')
+    except:    
+      msg = 'Could not open: ' + FN
+      raise Exception(msg) 
+
+    if not object_files: root1 = root  # Remember first filename        
+    object_files += root + '.o '  
+  
+  
+    # Compile
+    #
+    s = '%s -c %s -I%s -I%s -o %s.o -Wall' % (compiler, FN,
+                                              python_include, numpy_include,
+                                              root)
+    if os.name == 'posix' and os.uname()[4] == 'x86_64':
+      #Extra flags for 64 bit architectures
+      s += ' -fPIC'
+      
+    
+    if verbose:
+      print s
+    else:
+      s = s + ' 2> /dev/null' # Suppress errors
+  
+    try:
+      os.system(s)
+    except:
+      raise 'Could not compile %s - please try manually' % FN  
+
+  
+  # Make shared library (*.so)
+  s = '%s -%s %s -o %s.so' % (loader, sharedflag, object_files, root1)
+
+  if os.name == 'posix' and os.uname()[4] == 'x86_64':
+      # Extra flags for 64 bit architectures using Portland compilers
+      s += ' -mcmodel=medium'
+  
+  if verbose:
+    print s
+  else:
+    s = s + ' 2> /dev/null' # Suppress warnings
+  
+  try:  
+    err=os.system(s)
+  except:
+    raise 'Could not link %s - please try manually' % root1
+    
+
+
+
+if __name__ == '__main__':
+
+  import sys
+  
+  # Get file to compile
+  #
+  if len(sys.argv) < 2:
+    print 'Usage:'
+    print '  python compile.py <filename>'
+    sys.exit()
+
+  compile(sys.argv[1], verbose=1)
+
Index: trunk/anuga_core/source/pypar_dist/lib/python2.7/site-packages/pypar/compile_pypar_locally.py
===================================================================
--- trunk/anuga_core/source/pypar_dist/lib/python2.7/site-packages/pypar/compile_pypar_locally.py	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/lib/python2.7/site-packages/pypar/compile_pypar_locally.py	(revision 8494)
@@ -0,0 +1,65 @@
+# =============================================================================
+# install.py - Installation of pypar (Parallel Python using MPI)
+# Copyright (C) 2001 Ole M. Nielsen
+#
+#    This program is free software; you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License as published by
+#    the Free Software Foundation; either version 2 of the License, or
+#    (at your option) any later version.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License (http://www.gnu.org/copyleft/gpl.html)
+#    for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program; if not, write to the Free Software
+#    Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307
+#
+#
+# Contact address: Ole.Nielsen@anu.edu.au
+#
+# Version 1.0 October 2001
+# =============================================================================
+
+"""Module compile_pypar_locally.py - Compile C extension for pypar
+"""
+
+import os, sys      
+    
+
+# Attempt to compile mpiext.so 
+#
+
+if sys.platform in ['osf1V5']:  #Compaq AlphaServer do not have mpicc (grr)
+  MPICC = 'cc -lmpi'
+else:
+  MPICC = 'mpicc'  
+
+try:
+  import compile
+  compile.compile('mpiext.c', MPICC, verbose = 1)
+except Exception, e:
+  msg = 'Could not compile C extension mpiext.c - please try manually: %s' % e
+  raise Exception(msg)
+       
+
+
+if sys.platform in ['osf1V5', 'sunos5']:  #Compaq AlphaServer or Sun
+  sys.exit()
+  # Alpha Server or Sun cannot import MPI on sequential processes
+  # OK with LAM
+
+#Check if MPI module can be initialised
+#
+# Attempt to import and initialise mpiext.so
+
+error = os.system('python -c "import mpiext" > /dev/null') 
+if error:
+  msg = 'MPI could not be initialised.'
+  raise Exception(msg)
+else:  
+  import mpiext
+  print 'MPI initialised OK'    
+
Index: trunk/anuga_core/source/pypar_dist/lib/python2.7/site-packages/pypar/network_timing.py
===================================================================
--- trunk/anuga_core/source/pypar_dist/lib/python2.7/site-packages/pypar/network_timing.py	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/lib/python2.7/site-packages/pypar/network_timing.py	(revision 8494)
@@ -0,0 +1,240 @@
+# Timing of MPI module for Python and estimation of latency and bandwidth
+# 
+# Send numerical array in a ring from processor 0 to 1 etc back to 0 
+# Perform timings and compare different sending strategies 
+#
+# OMN, OCT 2001
+
+import time, sys, pypar, numpy
+
+# The send/recv routines
+#
+import pypar
+
+#from mpiext import send_array, receive_array  #most direct methods
+#Not available in final install - use bypass form for now
+
+#--------------------------------------------------------------
+
+method = 2   # Use 
+             # 0: automatically allocated buffers
+             # 1: user-supplied buffers
+             # 2: Use bypass - let pypar use direct mpi_ext call
+             # 3 (not there): direct call to mpiext (with buffers),
+             #    only fractionally better than bypass
+                  
+vanilla = False  # Force use of vanilla format (1)
+consistency_check = False # Check correctness
+
+
+#--------------------------------------------------------------
+# linfit
+#
+def linfit(x, y):
+  """Fit a and b to the model y = ax + b. Return a,b,variance
+  """
+  
+  Sx = Sy = SSoN = SxoN = norm = varest = 0.0
+  N = len(x)
+  assert len(y) == N, "x and y must have same length"
+  
+  for i in range(N):
+    #print("x,y = %f, %f\n",x[i],y[i])
+    Sx  = Sx + x[i]
+    Sy  = Sy + y[i]
+  
+  SxoN = Sx/N
+  
+  a = 0.0 
+  for i in range(N):
+    t    = x[i] - SxoN
+    SSoN = SSoN + t*t
+    a    = a + t*y[i]
+
+
+  a = a/SSoN            # a = (N Sxy - SxSy)/(NSxx - Sx^2) */
+  b = (Sy - Sx*a)/N
+  
+  # Quality - variance estimate \sum_i r_i^2 /(m-n) 
+  for i in range(N):  
+    norm = norm + float(x[i])*x[i]
+    res = y[i] - a*x[i] - b
+    varest = varest + res*res
+
+  varest = varest/norm/(N-2)
+  return a, b, varest
+
+
+
+
+#--------------------------------------------------------------
+# Main program
+#
+MAXI  = 10         # Number of blocks 
+MAXM  = 500000     # Largest block 
+BLOCK = MAXM/MAXI  # Block size 
+
+repeats = 10
+msgid = 0
+vanilla = 0 #Select vanilla mode (slower but general)
+
+numprocs = pypar.size()
+myid = pypar.rank()
+processor_name = pypar.get_processor_name()
+
+if myid == 0:
+  # Main process - Create message, pass on, verify correctness and log timing
+  #
+  print "MAXM = %d, number of processors = %d" %(MAXM, numprocs)
+  print "Measurements are repeated %d times for reliability" %repeats
+
+if numprocs < 2:
+  print "Program needs at least two processors - aborting\n"
+  pypar.abort()
+   
+pypar.barrier() #Synchronize all before timing   
+print "I am process %d on %s" %(myid,processor_name)
+
+
+#Initialise data and timings
+#
+
+try:
+  from numpy.random import uniform, seed
+  seed(17)
+  A = uniform(0.0,100.0,MAXM)
+except:
+  print 'problem with RandomArray'
+  from numpy import ones, Float
+  A = ones(MAXM).astype('f')
+  
+elsize = A.itemsize
+#print elsize
+
+noelem  = [0]*MAXI
+bytes   = [0]*MAXI         
+avgtime = [0.0]*MAXI         
+mintime = [ 1000000.0]*MAXI      
+maxtime = [-1000000.0]*MAXI            
+
+
+
+
+if myid == 0:   
+  # Determine timer overhead 
+  cpuOH = 1.0;
+  for k in range(repeats):   # Repeat to get reliable timings 
+    t1 = pypar.time()
+    t2 = pypar.time()
+    if t2-t1 < cpuOH: cpuOH = t2-t1
+    
+  print "Timing overhead is %f seconds.\n" %cpuOH         
+
+     
+# Pass msg circularly   
+for k in range(repeats):
+  if myid == 0:
+    print "Run %d of %d" %(k+1,repeats)
+    
+  for i in range(MAXI):
+    m=BLOCK*i+1       
+   
+    noelem[i] = m
+   
+    pypar.barrier() # Synchronize 
+   
+    if myid == 0:
+      #
+      # Main process
+      #
+      t1 = pypar.time()
+
+      if method==0:
+        pypar.send(A[:m], destination=1, tag=msgid, vanilla=vanilla)
+        C = pypar.receive(numprocs-1, tag=msgid, vanilla=vanilla)
+      elif method == 1:  
+        pypar.send(A[:m], use_buffer=True, destination=1,
+                   tag=msgid, vanilla=vanilla)
+        C = pypar.receive(numprocs-1, buffer=A[:m], tag=msgid, vanilla=vanilla)
+      elif method==2:
+        pypar.send(A[:m], use_buffer=True, destination=1,
+                   tag=msgid, vanilla=vanilla, bypass=True)
+        C = pypar.receive(numprocs-1, buffer=A[:m], tag=msgid,
+                          vanilla=vanilla, bypass=True)
+      else:
+        raise 'Unknown method'
+        #send_array(A[:m], 1, msgid)    
+        #stat = receive_array(A[:m], numprocs-1, msgid)
+        #C = A[:m]
+        
+      t2 = pypar.time() - t1 - cpuOH
+      t2 = t2/numprocs
+      avgtime[i] = avgtime[i] + t2
+      if t2 < mintime[i]: mintime[i] = t2
+      if t2 > maxtime[i]: maxtime[i] = t2
+
+      # Uncomment to verify integrity of data
+      # However, this may affect accuracy of timings for some reason.
+      #
+      if consistency_check:
+        assert numpy.alltrue(C == A[:m])
+    else:
+      #
+      # Parallel process - get msg and pass it on
+      #
+
+      if method==0:
+        C = pypar.receive(myid-1, tag=msgid, vanilla=vanilla)
+        pypar.send(A[:m], destination=(myid+1)%numprocs,
+                   tag=msgid, vanilla=vanilla)
+      elif method==1:  
+        C = pypar.receive(myid-1, buffer=A[:m], tag=msgid, vanilla=vanilla)
+        pypar.send(A[:m], use_buffer=True, destination=(myid+1)%numprocs,
+                   tag=msgid, vanilla=vanilla)                        
+      elif method==2:
+        # Use pypar bypass
+        C = pypar.receive(myid-1, buffer=A[:m], tag=msgid,
+                          vanilla=vanilla, bypass=True)
+        pypar.send(A[:m], use_buffer=True, destination=(myid+1)%numprocs,
+                   tag=msgid, vanilla=vanilla, bypass=True)
+      else:
+        raise 'Unknown method'
+        # Use direct mpiext call
+        #stat = receive_array(A[:m], myid-1, msgid)                
+        #send_array(A[:m], (myid+1)%numprocs, msgid)
+
+
+# Output stats
+#
+if myid == 0:
+  print "Bytes transferred   time (micro seconds)"
+  print "                    min        avg        max "     
+  print "----------------------------------------------"
+     
+  for i in range(MAXI):
+    avgtime[i] = avgtime[i]/repeats*1.0e6 #Average micro seconds
+    mintime[i] = mintime[i]*1.0e6         #Min micro seconds       
+    maxtime[i] = maxtime[i]*1.0e6         #Min micro seconds              
+          
+    m = noelem[i]
+    bytes[i] = elsize*noelem[i]       
+      
+    print "%10d    %10d %10d %10d" %(bytes[i], mintime[i], avgtime[i], maxtime[i]) 
+
+
+  Tbw, Tlat, varest = linfit(bytes, mintime)
+  print "\nLinear regression on best timings (t = t_l + t_b * bytes):\n",
+  print "  t_b = %f\n  t_l = %f" %(Tbw,Tlat)
+  print "  Estimated relative variance = %.9f\n" %varest
+       
+  print "Estimated bandwith (1/t_b):  %.3f Mb/s" %(1.0/Tbw)   
+  print "Estimated latency:           %d micro s" %int(mintime[0]-bytes[0]*Tbw)  
+
+   
+pypar.finalize()
+
+
+
+
+  
+  
Index: trunk/anuga_core/source/pypar_dist/lib/python2.7/site-packages/pypar/pypar.py
===================================================================
--- trunk/anuga_core/source/pypar_dist/lib/python2.7/site-packages/pypar/pypar.py	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/lib/python2.7/site-packages/pypar/pypar.py	(revision 8494)
@@ -0,0 +1,878 @@
+# =============================================================================
+# pypar.py - Parallel Python using MPI
+# Copyright (C) 2001, 2002, 2003 Ole M. Nielsen
+#              (Center for Mathematics and its Applications ANU and APAC)
+#
+#    This program is free software; you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License as published by
+#    the Free Software Foundation; either version 2 of the License, or
+#    (at your option) any later version.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License (http://www.gnu.org/copyleft/gpl.html)
+#    for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program; if not, write to the Free Software
+#    Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307
+#
+#
+# Contact address: Ole.Moller.Nielsen@gmail.com
+#
+# Version: See pypar.__version__
+# =============================================================================
+
+"""Module pypar.py - Parallel Python using MPI
+
+Public functions:
+
+size() -- Number of processors
+rank() -- Id of current processor
+get_processor_name() -- Return host name of current node
+
+send() -- Blocking send (all types)
+receive() -- Blocking receive (all types)
+broadcast() -- Broadcast
+time() -- MPI wall time
+barrier() -- Synchronisation point. Makes processors wait until all processors
+             have reached this point.
+abort() -- Terminate all processes.
+finalize() -- Cleanup MPI. No parallelism can take place after this point.
+
+
+See doc strings of individual functions for detailed documentation.
+"""
+
+from numpy import zeros, reshape, product
+from __metadata__ import __version__, __date__, __author__
+
+
+# Constants
+#
+max_tag = 32767      # Max tag value (MPI_TAG_UB didn't work and returned 0)
+control_tag = 13849  # Reserved tag used to identify control information
+default_tag = 1      # Tag used as default if not specified
+
+control_sep = ':'          # Separator for fields in control info (NOT ',')
+control_data_max_size = 64 # Maximal size of string holding control data
+
+
+#---------------------------------------------------------------------------
+# Communication functions
+#--------------------------------------------------------------------------
+
+def send(x, destination, use_buffer=False, vanilla=False,
+         tag=default_tag, bypass=False):
+    """Wrapper for easy MPI send.
+       Send x to destination.
+
+       Automatically determine appropriate protocol
+       and call corresponding send function.
+       Also passes type and size information on as preceding message to
+       simplify the receive call.
+
+       The variable x can be any (picklable) type, but
+       numpy variables and text strings will most efficient.
+       Setting vanilla = 1 forces vanilla mode for any type.
+
+       If use_buffer is True, workspace x will be used for the return value.
+       In this case the corresponding receive call must specify a buffer.
+       Otherwise a new workspace will be created by receive.
+
+       If bypass is True, all admin and error checks
+       get bypassed to reduce the latency. Should only
+       be used for sending numpy arrays and should be matched
+       with a bypass in the corresponding receive command.
+
+    """
+    import types, string
+
+    if bypass:
+        send_array(x, destination, tag)
+        return
+
+    # Input check
+    errmsg = 'Destination id (%s) must be an integer.' % destination
+    assert type(destination) == types.IntType, errmsg
+
+    errmsg = 'Tag %d is reserved by pypar - please use another.' % control_tag
+    assert tag != control_tag, errmsg
+
+    # Create metadata about object to be sent
+    control_info, x = create_control_info(x, vanilla, return_object=True)
+    protocol = control_info[0]
+
+
+    # Possibly transmit control data
+    if use_buffer is False:
+        send_control_info(control_info, destination)
+
+
+    # Transmit payload data
+    if protocol == 'array':
+        send_array(x, destination, tag)
+    elif protocol in ['string', 'vanilla']:
+        send_string(x, destination, tag)
+    else:
+        raise 'Unknown protocol: %s' %protocol
+
+
+def receive(source, buffer=None, vanilla=False, tag=default_tag,
+            return_status=False, bypass=False):
+    """receive - blocking MPI receive
+
+       Receive data from source.
+
+       Optional parameters:
+         buffer: Use specified buffer for received data (faster). Default None.
+         vanilla: Specify to enforce vanilla protocol for any type.
+                  Default False
+
+         tag: Only received messages tagged as specified. Default default_tag
+         return_status: Return Status object along with result. Default False.
+
+       If no buffer is specified, receive will try to receive a
+       preceding message containing protocol, type, size and shape and
+       then create a suitable buffer.
+
+       If buffer is specified the corresponding send must specify
+       use_buffer = True.
+       The variable buffer can be any (picklable) type, but
+       numpy variables and text strings will most efficient.
+
+       Appropriate protocol will be automatically determined
+       and corresponding receive function called.
+
+
+       If bypass is True, all admin and error checks
+       get bypassed to reduce the latency. Should only
+       be used for receiving numpy arrays and should
+       be matched with a bypass in the corresponding send command.
+       Also buffer must be specified.
+    """
+
+    if bypass:
+        # errmsg = 'bypass mode must be used with specified buffer'
+        # assert buffer is not None, msg
+        stat = receive_array(buffer, source, tag)
+    else:
+
+        import types
+
+        # Input check
+        errmsg = 'Source id (%s) must be an integer.' %source
+        assert type(source) == types.IntType, errmsg
+
+        errmsg = 'Tag %d is reserved by pypar - please use another.'\
+            % control_tag
+        assert tag != control_tag, errmsg
+
+
+        # Either receive or create metadata about object to receive
+        if buffer is None:
+            control_info, source = receive_control_info(source,
+                                                        return_source=True)
+            protocol, typecode, size, shape = control_info
+        else:
+            protocol, typecode, size, shape = create_control_info(buffer,
+                                                                  vanilla)
+
+
+        # Receive payload data
+        if protocol == 'array':
+            if buffer is None:
+                buffer = zeros(size,typecode)
+                buffer = reshape(buffer, shape)
+
+            stat = receive_array(buffer, source, tag)
+
+        elif protocol == 'string':
+            if buffer is None:
+                buffer = ' '*size
+
+            stat = receive_string(buffer, source, tag)
+
+        elif protocol == 'vanilla':
+            from cPickle import dumps, loads, UnpicklingError
+            if buffer is None:
+                s = ' '*size
+            else:
+                s = dumps(buffer, protocol=2)
+                s = s + ' '*int(0.1*len(s)) #safety
+
+            stat = receive_string(s, source, tag)
+            try:
+                buffer = loads(s)   #Replace buffer with received result
+            except UnpicklingError, err:
+                raise UnpicklingError(str(err) + " - '%s'" % s)
+        else:
+            raise 'Unknown protocol: %s' %protocol
+
+    # Return received data and possibly the status object
+    if return_status:
+        return buffer, Status(stat)
+    else:
+        return buffer
+
+
+def broadcast(buffer, root, vanilla=False, bypass=False):
+    """Wrapper for MPI bcast.
+
+       Broadcast buffer from the process with rank root to all other processes.
+
+
+       Automatically determine appropriate protocol
+       and call corresponding send function.
+
+       The variable buffer can be any (picklable) type, but
+       numpy variables and text strings will most efficient.
+       Setting vanilla = 1 forces vanilla mode for any type.
+
+       If bypass is True, all admin and error checks
+       get bypassed to reduce the latency.
+
+    """
+
+    if bypass:
+        broadcast_array(buffer, root)
+        return
+
+
+    import types
+
+    # Input check
+    errmsg = 'Root id (%s) must be an integer.' %root
+    assert type(root) == types.IntType, errmsg
+
+
+    # Create metadata about object to be sent
+    protocol = create_control_info(buffer, vanilla)[0]
+
+
+    # Broadcast
+    if protocol == 'array':
+        broadcast_array(buffer, root)
+    elif protocol == 'string':
+        broadcast_string(buffer, root)
+    elif protocol == 'vanilla':
+        from cPickle import loads, dumps, UnpicklingError
+        s = dumps(buffer, protocol=2)
+        s = s + ' '*int(0.1*len(s)) #safety
+
+        broadcast_string(s, root)
+        try:
+            buffer = loads(s)
+        except UnpicklingError, err:
+            raise UnpicklingError(str(err) + " - '%s'" % s)
+    else:
+        raise 'Unknown protocol: %s' %protocol
+
+    return buffer
+
+
+def scatter(x, root, buffer=None, vanilla=False):
+    """Sends data x from process with rank root to all other processes.
+
+       Create appropriate buffer and receive data.
+       Return scattered result (same type as x)
+
+       Scatter makes only sense for arrays or strings
+    """
+
+    import types
+    from mpiext import size
+    numproc = size()         # Needed to determine buffer size
+
+    # Input check
+    errmsg = 'Root id (%s) must be an integer.' %root
+    assert type(root) == types.IntType, errmsg
+
+
+    # Create metadata about object to be sent
+    protocol, typecode, size, shape = create_control_info(x)
+
+    # Scatter
+    if protocol == 'array':
+        if buffer is None:
+            # Modify shape along axis=0 to match size
+            shape = list(shape)
+            shape[0] /= numproc
+            count = product(shape)
+
+            buffer = zeros(count, typecode)
+            buffer = reshape(buffer, shape)
+
+        scatter_array(x, buffer, root)
+    elif protocol == 'string':
+        if buffer is None:
+            buffer = ' '*(size/numproc)
+
+        scatter_string(x, buffer, root)
+    elif protocol == 'vanilla':
+        errmsg = 'Scatter is only supported for numpy arrays and strings.\n'
+        errmsg += 'If you wish to distribute a general sequence, '
+        errmsg += 'please use send and receive commands or broadcast.'
+        raise errmsg
+    else:
+        raise 'Unknown protocol: %s' %protocol
+
+    return buffer
+
+
+def gather(x, root, buffer=None, vanilla=0):
+    """Gather values from all processes to root
+
+       Create appropriate buffer and receive data.
+
+       Gather only makes sens for arrays or strings
+    """
+
+    import types
+    from mpiext import size
+    numproc = size()         #Needed to determine buffer size
+
+    # Input check
+    errmsg = 'Root id (%s) must be an integer.' %root
+    assert type(root) == types.IntType, errmsg
+
+    # Create metadata about object to be gathered
+    protocol, typecode, size, shape = create_control_info(x)
+
+    # Gather
+    if protocol == 'array':
+        if buffer is None:
+            buffer = zeros(size*numproc, typecode)
+
+            # Modify shape along axis=0 to match size
+            shape = list(shape)
+            shape[0] *= numproc
+            buffer = reshape(buffer, shape)
+
+        gather_array(x, buffer, root)
+    elif protocol == 'string':
+        if buffer is None:
+            buffer = ' '*size*numproc
+
+        gather_string(x, buffer, root)
+    elif protocol == 'vanilla':
+        errmsg = 'Gather is only supported for numpy arrays and strings.\n'
+        errmsg += 'If you wish to distribute a general sequence, '
+        errmsg += 'please use send and receive commands or broadcast.'
+        raise errmsg
+    else:
+        raise 'Unknown protocol: %s' %protocol
+
+    return buffer
+
+
+def reduce(x, op, root, buffer=None, vanilla=0, bypass=False):
+    """Reduce elements in x to buffer (of the same size as x)
+       at root applying operation op elementwise.
+
+       If bypass is True, all admin and error checks
+       get bypassed to reduce the latency.
+       The buffer must be specified explicitly in this case.
+    """
+
+    if bypass:
+        reduce_array(x, buffer, op, root)
+        return
+
+
+    import types
+    from mpiext import size
+    numproc = size()         # Needed to determine buffer size
+
+
+    # Input check
+    errmsg = 'Root id (%s) must be an integer.' % root
+    assert type(root) == types.IntType, errmsg
+
+    # Create metadata about object
+    protocol, typecode, size, shape = create_control_info(x)
+
+    # Reduce
+    if protocol == 'array':
+        if buffer is None:
+            buffer = zeros(size*numproc, typecode)
+
+            # Modify shape along axis=0 to match size
+            shape = list(shape)
+            shape[0] *= numproc
+            buffer = reshape(buffer, shape)
+
+
+        msg = 'Data array and buffer must have same type '
+        msg = 'in reduce. I got types "%s" and "%s"' % (x.dtype.char,
+                                                        buffer.dtype.char)
+        assert x.dtype.char == buffer.dtype.char, msg
+        reduce_array(x, buffer, op, root)
+
+
+    elif (protocol == 'vanilla' or protocol == 'string'):
+        raise 'Protocol: %s unsupported for reduce' % protocol
+    else:
+        raise 'Unknown protocol: %s' % protocol
+
+    return buffer
+
+#
+# Functions related to MPI_Bsend().
+#
+
+def bsend(x, destination, use_buffer=False, vanilla=False,
+         tag=default_tag, bypass=False):
+
+    if bypass is True:
+        #print "bsend_array() bypass True"
+        bsend_array(x, destination, tag)
+        return
+
+    import types, string
+
+    # Input check.
+    errmsg = 'Destination id (%s) must be an integer.' % destination
+    assert type(destination) == types.IntType, errmsg
+
+    errmsg = 'Tag %d is reserved by pypar - please use another.' % control_tag
+    assert tag != control_tag, errmsg
+
+    # Create metadata about object to be sent
+    control_info, x = create_control_info(x, vanilla, return_object=True)
+    protocol = control_info[0]
+
+    # Possibly transmit control data
+    if use_buffer is False:
+       send_control_info(control_info, destination)
+
+    # Transmit payload data
+    if protocol == 'array':
+        bsend_array(x, destination, tag)
+        #print "bsend_array() bypass False"
+    elif protocol in ['string', 'vanilla']:
+        bsend_string(x, destination, tag)
+        #print "bsend_string()"
+    else:
+        raise 'Unknown protocol: %s' %protocol
+
+def push_for_alloc(x, vanilla=False, use_buffer=False, bypass=False):
+    if bypass is True:
+        return array_push_for_alloc_and_attach(x)
+    else:
+        # Create metadata about object and retrieve protocol.
+        control_info, x = create_control_info(x, vanilla, return_object=True)
+        protocol = control_info[0]
+
+		# According to protocol, use appropriate function.
+        if protocol == 'array':
+            return array_push_for_alloc_and_attach(x)
+        elif protocol in ['string', 'vanilla']:
+            return string_push_for_alloc_and_attach(x)
+        else:
+            raise 'Unknown protocol: %s' % protocol
+
+def alloc_and_attach():
+    mpi_alloc_and_attach()
+
+def detach_and_dealloc():
+    mpi_detach_and_dealloc()
+
+def alloc(nbytes=-1):
+    return mpi_alloc(nbytes)
+
+def dealloc():
+    mpi_dealloc()
+
+def attach():
+    mpi_attach()
+
+def detach():
+    mpi_detach()
+
+
+#---------------------------------------------------------
+# AUXILIARY FUNCTIONS
+#---------------------------------------------------------
+def balance(N, P, p):
+    """Compute p'th interval when N is distributed over P bins.
+
+    This function computes boundaries of sub intervals of [0:N] such
+    that they are almost equally sized with their sizes differening
+    by no more than 1.
+
+    As such, this function  is suitable for partitioning an interval equally
+    across P processors.
+
+    Inputs:
+       N: Upper bound of full interval.
+       P: Total number of processors
+       p: Local processor id
+
+
+    Outputs:
+       Nlo: Lower bound of p'th sub-interval
+       Nhi: Upper bound of p'th sub-interval
+
+
+    Example:
+       To partition the interval [0:29] among 4 processors:
+
+       Nlo, Nhi = pypar.balance(29, 4, p)
+
+       with p in [0,1,2,3]
+
+       and the subintervals are
+
+       p          Nlo      Nhi
+       -----------------------
+       0           0        8
+       1           8       15
+       2          15       22
+       3          22       29
+
+
+
+    Note that the interval bounds following the Python convention of
+    list slicing such that the last element of Nlo:Nhi is, in fact, Nhi-1
+    """
+
+    from math import floor
+
+    L = int(floor(float(N)/P))
+    K = N - P*L
+    if p < K:
+        Nlo = p*L + p
+        Nhi = Nlo + L + 1
+    else:
+        Nlo = p*L + K
+        Nhi = Nlo + L
+
+    return Nlo, Nhi
+
+
+# Obsolete functions
+from warnings import warn
+def Wtime():
+    msg = 'pypar.Wtime will become obsolete soon. Please use pypar.time'
+    warn(msg)
+    return time()
+
+def Get_processor_name():
+    msg = 'pypar.Get_processor_name will become obsolete soon.'
+    msg += 'Please use pypar.get_processor_name'
+    warn(msg)
+    return get_processor_name()
+
+
+#---------------------------------------------------------
+# INTERNAL FUNCTIONS
+#---------------------------------------------------------
+
+class Status:
+    """ MPI Status block returned by receive if
+        specified with parameter return_status=True
+    """
+
+    def __init__(self, status_tuple):
+        self.source = status_tuple[0]  # Id of sender
+        self.tag = status_tuple[1]     # Tag of received message
+        self.error = status_tuple[2]   # MPI Error code
+        self.length = status_tuple[3]  # Number of elements transmitted
+        self.size = status_tuple[4]    # Size of one element
+
+    def __repr__(self):
+        return 'Pypar Status Object:\n  source=%d\n  tag=%d\n  error=%d\n  length=%d\n  size=%d\n' %(self.source, self.tag, self.error, self.length, self.size)
+
+    def bytes(self):
+        """Number of bytes transmitted (excl control info)
+        """
+        return self.length * self.size
+
+
+
+def create_control_info(x, vanilla=0, return_object=False):
+    """Determine which protocol to use for communication:
+       (numpy) arrays, strings, or vanilla based x's type.
+
+       There are three protocols:
+       'array':   numpy arrays of type 'i', 'l', 'f', 'd', 'F' or 'D' can be
+                  communicated  with mpiext.send_array and mpiext.receive_array.
+       'string':  Text strings can be communicated with mpiext.send_string and
+                  mpiext.receive_string.
+       'vanilla': All other types can be communicated as string representations
+                  provided that the objects
+                  can be serialised using pickle (or cPickle).
+                  The latter mode is less efficient than the
+                  first two but it can handle general structures.
+
+       Rules:
+       If keyword argument vanilla == 1, vanilla is chosen regardless of
+       x's type.
+       Otherwise if x is a string, the string protocol is chosen
+       If x is an array, the 'array' protocol is chosen provided that x has one
+       of the admissible typecodes.
+
+       The optional argument return_object asks to return object as well.
+       This is useful in case it gets modified as in the case of general structures
+       using the vanilla protocol.
+    """
+
+    import types
+
+    # Default values
+    protocol = 'vanilla'
+    typecode = ' '
+    size = 0
+    shape = ()
+
+    # Determine protocol in case
+    if not vanilla:
+        if type(x) == types.StringType:
+            protocol = 'string'
+            typecode = 'c'
+            size = len(x)
+        elif type(x).__name__ == 'ndarray': # numpy isn't imported yet
+            try:
+                import numpy
+            except:
+                print "WARNING (pypar.py): numpy module could not be imported,",
+                print "reverting to vanilla mode"
+                protocol = 'vanilla'
+            else:
+                typecode = x.dtype.char
+                if typecode in ['i', 'l', 'f', 'd', 'F', 'D']:
+                    protocol = 'array'
+                    shape = x.shape
+                    size = product(shape)
+                else:
+                    print "WARNING (pypar.py): numpy object type %s is not supported."\
+                          %(x.dtype.char)
+                    print "Only types 'i', 'l', 'f', 'd', 'F', 'D' are supported,",
+                    print "Reverting to vanilla mode."
+                    protocol = 'vanilla'
+
+    # Pickle general structures using the vanilla protocol
+    if protocol == 'vanilla':
+        from cPickle import dumps
+        x = dumps(x, protocol=2)
+        size = len(x) # Let count be length of pickled object
+
+    # Return
+    if return_object:
+        return [protocol, typecode, size, shape], x
+    else:
+        return [protocol, typecode, size, shape]
+
+
+
+#----------------------------------------------
+
+def send_control_info(control_info, destination):
+    """Send control info to destination
+    """
+    import string
+
+    # Convert to strings
+    control_info = [str(c) for c in control_info]
+
+    control_msg = string.join(control_info,control_sep)
+    if len(control_msg) > control_data_max_size:
+        errmsg = 'Length of control_info exceeds specified maximium (%d)'\
+                 %control_data_max_size
+        errmsg += ' - Please increase it (in pypar.py)'
+        raise errmsg
+
+    send_string(control_msg, destination, control_tag)
+
+
+def receive_control_info(source, return_source=False):
+    """Receive control info from source
+
+    The optional argument (due to Jim Bosch) also returns the actual source node
+    which can be used to require that the data message come from the same node.
+    """
+
+    # FIXME (Ole): Perhaps we should include actual source in the control info?
+
+    import string
+
+    msg = ' '*control_data_max_size
+
+    stat = receive_string(msg, source, control_tag)
+    # No need to create status object here - it is reserved
+    # for payload communications only
+
+    msg = msg[:stat[3]] # Trim buffer to actual received length (needed?)
+
+    control_info = msg.split(control_sep)
+
+    assert len(control_info) == 4, 'len(control_info) = %d' %len(control_info)
+    control_info[2] = eval(control_info[2]) # Convert back to int
+    control_info[3] = eval(control_info[3]) # Convert back to tuple
+
+    if return_source:
+        return control_info, int(stat[0])
+    else:
+        return control_info
+
+
+#----------------------------------------------------------------------------
+# Initialise module
+#----------------------------------------------------------------------------
+
+
+# Take care of situation where module is part of package
+import sys, os, string, os.path
+dirname = os.path.dirname(string.replace(__name__,'.',os.sep)).strip()
+
+if not dirname:
+    dirname = '.'
+
+if dirname[-1] != os.sep:
+    dirname += os.sep
+
+
+
+# Import MPI extension
+#
+# Verify existence of mpiext.so.
+
+try:
+    import mpiext
+except:
+    errmsg = 'ERROR: C extension mpiext could not be imported.\n'
+    errmsg += 'Please compile mpiext.c e.g. by running\n'
+    errmsg += '  python compile_pypar_locally.py\n'
+    errmsg += 'in the pypar directory, or by using\n'
+    errmsg += '  python setup.py install\n'
+    #raise Exception, errmsg
+    error = 1
+    print errmsg
+else:
+
+    # Determine if MPI program is allowed to run sequentially on current platform
+    # Attempting to check this automatically may case some systems to hang.
+
+    if sys.platform in ['linux2', 'sunos5', 'win32', 'darwin']:
+        # Linux (LAM,MPICH) or Sun (MPICH)
+        error = 0  #Sequential execution of MPI is allowed
+    else:
+        # Platform: Alpha 'osf1V5'
+        cmdstring = '"import mpiext, sys; mpiext.init(sys.argv); mpiext.finalize()"'
+        #s = 'cd %s; python -c %s' %(dirname, cmdstring)
+        s = 'python -c %s >/dev/null 2>/dev/null' %cmdstring
+        error = os.system(s)
+
+        # The check is performed in a separate shell.
+        # Reason: The Alpha server, LAM/Linux or the Sun cannot recover from a
+        # try:
+        #   mpiext.init(sys.argv)
+
+        # However, on LAM/Linux, this test causes system to hang.
+        # Verified (OMN 12/12/2)
+        # If lamboot is started, the system, will hang when init is called
+        # again further down in this file.
+        # If lamboot is not loaded error will be nozero as it should.
+        # I don't know how to deal with this
+        #
+        #Comparisons of two strategies using LAM
+        #
+        # Strategy 1: Assume seq execution is OK (i.e. set error = 0)
+        # Strategy 2: Try to test if mpi can be initialised (in a separate shell)
+        #
+        #
+        # Strategy 1 (currently used)
+        #                    | Lam booted  | Lam not booted
+        #-----------------------------------------------------
+        #
+        # Sequential exec    |  OK         | Not OK
+        # Parallel exec      |  OK         | Not OK
+        #
+        #
+        # Strategy 2
+        #                    | Lam booted  | Lam not booted
+        #-----------------------------------------------------
+        #
+        # Sequential exec    |  Hangs      | Not OK
+        # Parallel exec      |  Hangs      | OK
+        #
+
+
+
+# Initialise MPI
+#
+# Attempt to initialise mpiext.so
+# If this fails, define a rudimentary interface suitable for
+# sequential execution.
+
+if error:
+    print "WARNING: MPI library could not be initialised - running sequentially"
+
+    # Define rudimentary functions to keep sequential programs happy
+
+    def size(): return 1
+    def rank(): return 0
+
+    def get_processor_name():
+        import os
+        try:
+            hostname = os.environ['HOST']
+        except:
+            try:
+                hostname = os.environ['HOSTNAME']
+            except:
+                hostname = 'Unknown'
+
+        return hostname
+
+    def abort():
+        import sys
+        sys.exit()
+
+    def finalize(): pass
+
+    def barrier(): pass
+
+    def time():
+        import time
+        return time.time()
+
+else:
+    from mpiext import size, rank, barrier, time,\
+         get_processor_name,\
+         init, initialized, finalize, abort,\
+         send_string, receive_string,\
+         send_array, receive_array, broadcast_string, broadcast_array,\
+         scatter_string, scatter_array,\
+         gather_string, gather_array,\
+         reduce_array,\
+	 bsend_string, bsend_array, \
+	 mpi_alloc_and_attach, mpi_detach_and_dealloc, \
+	 mpi_alloc, mpi_dealloc, mpi_attach, mpi_detach, \
+	 string_push_for_alloc_and_attach, array_push_for_alloc_and_attach, \
+         MPI_ANY_TAG as any_tag, MPI_TAG_UB as max_tag,\
+         MPI_ANY_SOURCE as any_source,\
+         MAX, MIN, SUM, PROD, LAND, BAND,\
+         LOR, BOR, LXOR, BXOR
+
+    # Work around bug in OpenMPI (December 2009):
+    # https://bugs.launchpad.net/ubuntu/+source/petsc4py/+bug/232036
+    from ctypes import *
+    CDLL('libmpi.so', RTLD_GLOBAL)
+    # End work around
+
+    # Initialise MPI with cmd line (needed by MPICH/Linux)
+    init(sys.argv)
+
+    # Report
+    if rank() == 0:
+        print 'Pypar (version %s) initialised MPI OK with %d processors'\
+            % (__version__, size())
+
+
+
+
+
+
Index: trunk/anuga_core/source/pypar_dist/lib/python2.7/site-packages/pypar/pypar_balancer.py
===================================================================
--- trunk/anuga_core/source/pypar_dist/lib/python2.7/site-packages/pypar/pypar_balancer.py	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/lib/python2.7/site-packages/pypar/pypar_balancer.py	(revision 8494)
@@ -0,0 +1,218 @@
+# coding: UTF-8
+"""
+Simple load balancing with pypar
+(based on demo3.py from pypar demo package)
+Felix Richter <felix.richter2@uni-rostock.de>
+"""
+import sys
+import time
+
+import numpy
+import pypar
+
+PYPAR_WORKTAG = 1
+PYPAR_DIETAG = 2
+
+def mprint(txt):
+    """
+    Print message txt
+    with indentation following the node's rank
+    """
+    import pypar
+    
+    pre = " " * 8 * pypar.rank()
+    if type(txt) != type('dummy'):
+        txt = txt.__str__()
+    pat = "-%d-"
+    print pre + (pat % pypar.rank()) + txt
+
+class PyparWork(object):
+    """Abstract base class for ant work to be balanced"""
+
+    def __init__(self):
+        pass
+    
+    def uplink(self, balancer, myid, numprocs, node):
+        self.balancer = balancer
+        self.pypar_id = myid
+        self.pypar_numprocs = numprocs
+        self.pypar_node = node
+    
+    def getNumWorkItems(self):
+        pass
+    
+    def handleWorkResult(self, result, status):
+        pass
+    
+    def calcWorkResult(self, worknum):
+        pass
+    
+    def masterBeforeWork(self):
+        """Master node calls this before sending out the work"""
+        pass
+
+    def slaveBeforeWork(self):
+        """Slave nodes call this before receiving work"""
+        pass
+
+    def masterAfterWork(self):
+        """Master node calls this after receiving the last work result"""
+        pass
+
+    def slaveAfterWork(self):
+        """Slave nodes call this after sending the last work result"""
+        pass
+    
+    def msgprint(self, txt):
+        pre = " " * 8 * self.pypar_id
+        if type(txt) != type('dummy'):
+            txt = txt.__str__()
+        pat = "-%d-"
+        print pre + (pat % self.pypar_id) + txt
+        
+
+class PyparBalancer(object):
+    """The Load Balancer Class
+    Initialize it with a PyparWork-derived class instance
+    which describes the actual work to do.
+    
+    debug == True - more status messages
+    """
+    
+    def __init__(self, work, debug = False):
+        self.numprocs = pypar.size()           # Number of processes as specified by mpirun
+        self.myid = pypar.rank()               # Id of of this process (myid in [0, numproc-1]) 
+        self.node = pypar.get_processor_name() # Host name on which current process is running
+        self.debug= debug
+        self.work = work
+
+        # Added by Ole Nielsen, 15 May 2008
+        if self.numprocs < 2:
+            msg = 'PyparBalancer must run on at least 2 processes'
+            msg += ' for the Master Slave paradigm to make sense.'
+            raise Exception, msg
+
+        
+        self.work.uplink(self, self.myid, self.numprocs, self.node)
+        
+        self.numworks = self.work.getNumWorkItems()
+        print "PyparBalancer initialised on proc %d of %d on node %s" %(self.myid, self.numprocs, self.node)
+
+    def master(self):
+        numcompleted = 0
+        #--- start slaves distributing the first work slot
+        for i in range(0, min(self.numprocs-1, self.numworks)): 
+            work = i
+            slave= i+1
+            pypar.send(work, destination=slave, tag=PYPAR_WORKTAG) 
+            print '[MASTER ]: sent first work "%s" to node %d' %(work, slave)
+    
+        # dispatch the remaining work slots on dynamic load-balancing policy
+        # the quicker to do the job, the more jobs it takes
+        for work in range(self.numprocs-1, self.numworks):
+            result, status = pypar.receive(source=pypar.any_source, tag=PYPAR_WORKTAG, return_status=True) 
+            print '[MASTER ]: received result from node %d' %(status.source, )
+            #print result
+            numcompleted += 1
+            pypar.send(work, destination=status.source, tag=PYPAR_WORKTAG)
+            if self.debug: print '[MASTER ]: sent work "%s" to node %d' %(work, status.source)
+            
+            self.work.handleWorkResult(result, status)
+        
+        # all works have been dispatched out
+        print '[MASTER ]: ToDo : %d' %self.numworks
+        print '[MASTER ]: Done : %d' %numcompleted
+        
+        # I've still to take into the remaining completions   
+        while (numcompleted < self.numworks): 
+            result, status = pypar.receive(source=pypar.any_source, tag=PYPAR_WORKTAG, return_status=True) 
+            print '[MASTER ]: received (final) result from node %d' % (status.source, )
+            print result
+            numcompleted += 1
+            print '[MASTER ]: %d completed' %numcompleted
+            
+            self.work.handleWorkResult(result, status)
+            
+        print '[MASTER ]: about to terminate slaves'
+    
+        # Tell slaves to stop working
+        for i in range(1, self.numprocs): 
+            pypar.send('#', destination=i, tag=PYPAR_DIETAG) 
+            if self.debug: print '[MASTER ]: sent DIETAG to node %d' %(i,)
+            
+    
+    def slave(self):
+        if self.debug: print '[SLAVE %d]: I am processor %d of %d on node %s' % (self.myid, self.myid, self.numprocs, self.node)
+        if self.debug: print '[SLAVE %d]: Entering work loop' % (self.myid,)
+        while True:
+            result, status = pypar.receive(source=0, tag=pypar.any_tag, return_status=True) 
+            print '[SLAVE %d]: received work with tag %d from node %d'\
+                      %(self.myid, status.tag, status.source)
+           
+            if (status.tag == PYPAR_DIETAG):
+                print '[SLAVE %d]: received termination from node %d' % (self.myid, 0)
+                return
+            else:
+                worknum = result
+                if self.debug: print '[SLAVE %d]: work number is %s' % (self.myid, worknum)
+                myresult = self.work.calcWorkResult(worknum)
+                pypar.send(myresult, destination=0)
+                if self.debug: print '[SLAVE %d]: sent result to node %d' % (self.myid, 0)
+
+    def run(self):
+        if self.myid == 0:
+            self.work.masterBeforeWork()
+            self.master()
+            self.work.masterAfterWork()
+        else:
+            self.work.slaveBeforeWork()
+            self.slave()
+            self.work.slaveAfterWork()
+        
+        pypar.finalize()
+        if self.myid != 0:
+            sys.exit()
+        # und schluss.
+
+class PyparDemoWork(PyparWork):
+    """Example PyparWork implementation"""
+    def __init__(self):
+        import numpy
+        self.worklist = numpy.arange(0.0,20.0)
+        self.resultlist = numpy.zeros_like(self.worklist)
+        
+    def getNumWorkItems(self):
+        return len(self.worklist)
+    
+    def calcWorkResult(self, worknum):
+        return [worknum, self.worklist[worknum] + 1]
+
+    def handleWorkResult(self, result, status):
+        self.resultlist[result[0]] = result[1]
+        
+    def masterBeforeWork(self):
+        print self.worklist
+
+    def slaveBeforeWork(self):
+        pass
+
+    def masterAfterWork(self):
+        print self.resultlist
+
+    def slaveAfterWork(self):
+        pass
+        
+if __name__ == "__main__":
+    print "-----------------------"
+    print "::: PyParBalancer TEST "
+    print "-----------------------"
+    
+    # create instance of work class
+    pyparwork = PyparDemoWork()
+
+    # create instance of balancer class,
+    # initialize with work class
+    balancer = PyparBalancer(pyparwork, True)
+    
+    # run it
+    balancer.run()
Index: trunk/anuga_core/source/pypar_dist/lib/python2.7/site-packages/pypar/test_init.py
===================================================================
--- trunk/anuga_core/source/pypar_dist/lib/python2.7/site-packages/pypar/test_init.py	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/lib/python2.7/site-packages/pypar/test_init.py	(revision 8494)
@@ -0,0 +1,24 @@
+# Inital test of MPI module 'pypar' for Python
+# 
+# Run as 
+#   python test_init.py
+# or 
+#   mpirun -np 2 test_init.py
+#  (perhaps try number of processors more than 2)
+#
+# To verify bandwidth of your architecture please run pytiming (and ctiming) 
+#
+# OMN April 2002
+
+
+import pypar
+
+myid =    pypar.rank()
+numproc = pypar.size()
+node =    pypar.get_processor_name()
+
+print "I am processor %d of %d on node %s" %(myid, numproc, node)
+if myid == 0:
+    print 'Maximal tag is %d' %pypar.max_tag
+pypar.finalize()
+
Index: trunk/anuga_core/source/pypar_dist/lib/python2.7/site-packages/pypar/test_pypar.py
===================================================================
--- trunk/anuga_core/source/pypar_dist/lib/python2.7/site-packages/pypar/test_pypar.py	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/lib/python2.7/site-packages/pypar/test_pypar.py	(revision 8494)
@@ -0,0 +1,1152 @@
+# Test of MPI module 'pypar' for Python
+# 
+# Run as 
+#   python testpypar.py
+# or 
+#   mpirun -np 2 testpypar.py
+# (perhaps try number of processors more than 2)
+#
+# To verify bandwidth of your architecture please run pytiming (and ctiming) 
+#
+# OMN, GPC FEB 2002
+
+
+try:
+  import numpy
+except:
+  raise 'Module numpy must be present to run pypar'
+
+
+#print "Importing pypar"
+import pypar
+methods = dir(pypar)
+assert 'abort' in methods
+assert 'finalize' in methods
+assert 'get_processor_name' in methods
+assert 'time' in methods
+assert 'rank' in methods
+assert 'receive' in methods
+assert 'send' in methods
+assert 'broadcast' in methods
+assert 'size' in methods
+
+#print "Module pypar imported OK"
+#pypar.Barrier()
+
+
+myid = pypar.rank()
+numproc = pypar.size()
+node = pypar.get_processor_name()
+
+print 'I am processor %d of %d on node "%s"' %(myid, numproc, node)
+pypar.barrier()
+
+
+if numproc > 1:
+  # Test simple buffered communication (arrays, strings and general)
+  #
+  N = 17 #Number of elements
+  
+  if myid == 0:
+    # Integer arrays
+    #
+    A = numpy.array(range(N)).astype('i')
+    B = numpy.zeros(N).astype('i')
+
+    pypar.send(A, 1, use_buffer=True)
+    B = pypar.receive(numproc-1, buffer=B)
+
+    assert numpy.allclose(A, B)
+    print 'Buffered communication of numeric integer arrays OK'
+
+
+    # Long integer arrays
+    #
+    A = numpy.array(range(N)).astype('l')
+    B = numpy.zeros(N).astype('l')    
+
+    pypar.send(A, 1, use_buffer=True)
+    B = pypar.receive(numproc-1, buffer=B)
+    
+    assert numpy.allclose(A, B)
+    print 'Buffered communication of numeric long integer arrays OK'
+
+    # Real arrays
+    #
+    A = numpy.array(range(N)).astype('f')
+    B = numpy.zeros(N).astype('f')    
+
+    pypar.send(A, 1, use_buffer=True)
+    B = pypar.receive(numproc-1, buffer=B)
+    
+    assert numpy.allclose(A, B)    
+    print 'Buffered communication of numeric real arrays OK'
+
+    # Complex arrays
+    #
+    A = numpy.array(range(N)).astype('D')
+    A += 3j
+    B = numpy.zeros(N).astype('D')
+
+    pypar.send(A, 1, use_buffer=True)
+    B = pypar.receive(numproc-1, buffer=B)
+
+    assert numpy.allclose(A, B)    
+    print 'Buffered communication of numeric complex arrays OK'
+
+    # Strings (< 256 characters)
+    #
+    A = "and now to something completely different !"
+    B = " "*len(A)
+
+    pypar.send(A, 1, use_buffer=True)
+    B, status = pypar.receive(numproc-1, buffer=B, return_status=True)
+
+
+    assert A == B
+    print 'Buffered communication of strings OK'
+    
+    # A more general structure
+    #
+    A = ['ABC', (1,2,3.14), {8: 'Monty'}, numpy.array([13.45, 1.2])]
+    B = ['   ', (0,0,0.0) , {0: '     '}, numpy.zeros(2).astype('f')]    
+
+    pypar.send(A, 1, use_buffer=True)
+    B, status = pypar.receive(numproc-1, buffer=B, return_status=True)
+
+    OK = True
+    for i, a in enumerate(A):
+      b = B[i]
+
+      if type(a).__name__ == 'ndarray':
+        if not numpy.allclose(a, b):
+          OK = False
+          break
+      elif a != b:
+        OK = False
+        break
+
+    if OK is True:
+      print 'Buffered communication of general structures OK' 
+    else:
+      raise Exception    
+
+    
+  else:  
+    # Integers
+    #
+    X = numpy.zeros(N).astype('i')    
+
+    X = pypar.receive(myid-1, buffer=X)
+    pypar.send(X, (myid+1)%numproc, use_buffer=True)
+  
+    # Long integers
+    #
+    X = numpy.zeros(N).astype('l')
+    X = pypar.receive(myid-1, buffer=X)
+    pypar.send(X, (myid+1)%numproc, use_buffer=True)
+
+    # Floats
+    #
+    X = numpy.zeros(N).astype('f')
+    X = pypar.receive(myid-1, buffer=X)
+    pypar.send(X, (myid+1)%numproc, use_buffer=True)
+
+    # Complex
+    #
+    X = numpy.zeros(N).astype('D')
+    X = pypar.receive(myid-1, buffer=X)
+    pypar.send(X, (myid+1)%numproc, use_buffer=True)
+
+    # Strings
+    #
+    X = " "*256
+    X = pypar.receive(myid-1, buffer=X)
+    pypar.send(X.strip(), (myid+1)%numproc, use_buffer=True)
+
+    # General
+    #
+    X = ['   ', (0,0,0.0), {0: '     '}, numpy.zeros(2).astype('f')]
+    X = pypar.receive(myid-1, buffer=X)
+    pypar.send(X, (myid+1)%numproc, use_buffer=True)
+    
+
+
+
+
+  #Test (buffered communication of) multi dimensional arrays
+
+  M = 13  #Number of elements in dim 1
+  N = 17  #Number of elements in higher dims
+  
+  if myid == 0:
+    # 2D real arrays
+    #
+    A = numpy.array(range(M*N)).astype('f')
+    B = numpy.zeros(M*N).astype('f')
+
+    A = numpy.reshape(A, (M,N))
+    B = numpy.reshape(B, (M,N))    
+    
+    pypar.send(A, 1, use_buffer=True)
+    B = pypar.receive(numproc-1, buffer=B)
+    
+    assert numpy.allclose(A, B)    
+    print 'Buffered communication of 2D real arrays OK'
+
+    # 2D complex arrays
+    #
+    A = numpy.array(range(M*N)).astype('D')
+    B = numpy.zeros(M*N).astype('D')
+
+    A = numpy.reshape(A, (M,N))
+    B = numpy.reshape(B, (M,N))    
+
+    pypar.send(A, 1, use_buffer=True)
+    B = pypar.receive(numproc-1, buffer=B)
+    
+    assert numpy.allclose(A, B)    
+    print 'Buffered communication of 2D complex arrays OK'
+
+    # 3D real arrays
+    #
+    A = numpy.array(range(M*N*N)).astype('f')
+    B = numpy.zeros(M*N*N).astype('f')
+
+    A = numpy.reshape(A, (M,N,N))
+    B = numpy.reshape(B, (M,N,N))    
+    
+
+    pypar.send(A, 1, use_buffer=True)
+    B = pypar.receive(numproc-1, buffer=B)
+    
+    assert numpy.allclose(A, B)    
+    print 'Buffered communication of 3D real real arrays OK'
+
+    # 4D real arrays
+    #
+    A = numpy.array(range(M*N*N*M)).astype('f')
+    B = numpy.zeros(M*N*N*M).astype('f')
+
+    A = numpy.reshape(A, (M,N,N,M))
+    B = numpy.reshape(B, (M,N,N,M))    
+
+    pypar.send(A, 1, use_buffer=True)
+    B = pypar.receive(numproc-1, buffer=B)
+    
+    assert numpy.allclose(A, B)    
+    print 'Buffered communication of 4D real real arrays OK'
+
+    # 5D real arrays
+    #
+    A = numpy.array(range(M*N*2*N*M)).astype('f')
+    B = numpy.zeros(M*N*2*N*M).astype('f')
+
+    A = numpy.reshape(A, (M,N,2,N,M))
+    B = numpy.reshape(B, (M,N,2,N,M))    
+    
+    pypar.send(A, 1, use_buffer=True)
+    B = pypar.receive(numproc-1, buffer=B)
+    
+    assert numpy.allclose(A, B)    
+    print 'Buffered communication of 5D real real arrays OK'
+
+    # 5D double arrays
+    #
+    A = numpy.array(range(M*N*2*N*M)).astype('d')
+    B = numpy.zeros(M*N*2*N*M).astype('d')
+
+    A = numpy.reshape(A, (M,N,2,N,M))
+    B = numpy.reshape(B, (M,N,2,N,M))    
+    
+    pypar.send(A, 1, use_buffer=True)
+    B = pypar.receive(numproc-1, buffer=B)
+    
+    assert numpy.allclose(A, B)    
+    print 'Buffered communication of 5D double arrays OK'
+
+    # 5D complex arrays
+    #
+    A = numpy.array(range(M*N*2*N*M)).astype('D')
+    B = numpy.zeros(M*N*2*N*M).astype('D')
+
+    A = numpy.reshape(A, (M,N,2,N,M))
+    B = numpy.reshape(B, (M,N,2,N,M))    
+    
+    pypar.send(A, 1, use_buffer=True)
+    B = pypar.receive(numproc-1, buffer=B)
+    
+    assert numpy.allclose(A, B)    
+    print 'Buffered communication of 5D complex arrays OK'
+  else:  
+    # 2D real arrays
+    #
+    X = numpy.zeros(M*N).astype('f')
+    X = numpy.reshape(X, (M,N))
+    
+    X = pypar.receive(myid-1, buffer=X)
+    pypar.send(X, (myid+1)%numproc, use_buffer=True)
+  
+    # 2D complex arrays
+    #
+    X = numpy.zeros(M*N).astype('D')
+    X = numpy.reshape(X, (M,N))
+
+    X = pypar.receive(myid-1, buffer=X)
+    pypar.send(X, (myid+1)%numproc, use_buffer=True)
+  
+    # 3D real arrays
+    #
+    X = numpy.zeros(M*N*N).astype('f')
+    X = numpy.reshape(X, (M,N,N))
+    
+    X = pypar.receive(myid-1, buffer=X)
+    pypar.send(X, (myid+1)%numproc, use_buffer=True)
+
+    # 4D real arrays
+    #
+    X = numpy.zeros(M*N*N*M).astype('f')
+    X = numpy.reshape(X, (M,N,N,M))
+    
+    X = pypar.receive(myid-1, buffer=X)
+    pypar.send(X, (myid+1)%numproc, use_buffer=True)
+
+    # 5D real arrays
+    #
+    X = numpy.zeros(M*N*2*N*M).astype('f')
+    X = numpy.reshape(X, (M,N,2,N,M))
+    
+    X = pypar.receive(myid-1, buffer=X)
+    pypar.send(X, (myid+1)%numproc, use_buffer=True)
+
+    # 5D double arrays
+    #
+    X = numpy.zeros(M*N*2*N*M).astype('d')
+    X = numpy.reshape(X, (M,N,2,N,M))
+    
+    X = pypar.receive(myid-1, buffer=X)
+    pypar.send(X, (myid+1)%numproc, use_buffer=True)
+
+    # 5D complex arrays
+    #
+    X = numpy.zeros(M*N*2*N*M).astype('D')
+    X = numpy.reshape(X, (M,N,2,N,M))
+    
+    X = pypar.receive(myid-1, buffer=X)
+    pypar.send(X, (myid+1)%numproc, use_buffer=True)
+
+  # Test easy communication  - without buffers (arrays, strings and general)
+  #
+  N = 17 #Number of elements
+  
+  if myid == 0:
+    # Integer arrays
+    #
+    A = numpy.array(range(N))
+
+    pypar.send(A,1)
+    B = pypar.receive(numproc-1)
+    
+
+    assert numpy.allclose(A, B)
+    print "Simplified communication of numeric integer arrays OK"
+
+    # Long integer arrays
+    #
+    A = numpy.array(range(N)).astype('l')
+    pypar.send(A,1)
+    B=pypar.receive(numproc-1)
+    
+    assert numpy.allclose(A, B)    
+    print "Simplified communication of long integer real arrays OK"
+
+    # Real arrays
+    #
+    A = numpy.array(range(N)).astype('f')
+    pypar.send(A,1)
+    B=pypar.receive(numproc-1)
+    
+    assert numpy.allclose(A, B)    
+    print "Simplified communication of numeric real arrays OK"
+
+    # Complex arrays
+    #
+    A = numpy.array(range(N)).astype('D')
+    A += 3j
+    pypar.send(A,1)
+    B=pypar.receive(numproc-1)
+
+    assert numpy.allclose(A, B)    
+    print "Simplified communication of numeric complex arrays OK"
+
+
+    # Strings
+    #
+    A = "and now to something completely different !"
+    pypar.send(A,1)
+    B=pypar.receive(numproc-1)
+    
+    assert A == B
+    print "Simplified communication of strings OK"
+    
+    # A more general structure
+    #
+    A = ['ABC', (1,2,3.14), {8: 'Monty'}, numpy.array([13.45, 1.2])]
+    pypar.send(A,1)
+    B = pypar.receive(numproc-1)
+
+    OK = True
+    for i, a in enumerate(A):
+      b = B[i]
+
+      if type(a).__name__ == 'ndarray':
+        if not numpy.allclose(a, b):
+          OK = False
+          break
+      elif a != b:
+        OK = False
+        break
+
+    if OK is True:
+      print 'Simplified communication of general structures OK' 
+    else:
+      raise Exception    
+    
+  else:  
+    # Integers
+    #
+    X=pypar.receive(myid-1)  
+    pypar.send(X, (myid+1)%numproc)
+  
+    # Long integers
+    #
+    X=pypar.receive(myid-1)  
+    pypar.send(X, (myid+1)%numproc)    
+
+    # Floats
+    #
+    X=pypar.receive(myid-1)  
+    pypar.send(X, (myid+1)%numproc)    
+
+    # Complex
+    #
+    X=pypar.receive(myid-1)  
+    pypar.send(X, (myid+1)%numproc)    
+
+    # Strings
+    #
+    X=pypar.receive(myid-1)  
+    pypar.send(X, (myid+1)%numproc)    
+
+    # General
+    #
+    X = pypar.receive(myid-1)  
+    pypar.send(X, (myid+1)%numproc)    
+
+
+
+
+    
+  #Test (easy communication of) multi dimensional arrays
+
+  M = 13  #Number of elements in dim 1
+  N = 17  #Number of elements in higher dims
+  
+  if myid == 0:
+    # 2D real arrays
+    #
+    A = numpy.array(range(M*N)).astype('f')
+
+    A = numpy.reshape(A, (M,N))
+    
+    pypar.send(A,1)
+    B = pypar.receive(numproc-1)
+    
+    assert numpy.allclose(A, B)    
+    print "Simplified communication of 2D real arrays OK"
+
+    # 3D real arrays
+    #
+    A = numpy.array(range(M*N*N)).astype('f')
+    A = numpy.reshape(A, (M,N,N))
+    
+    pypar.send(A,1)
+    B=pypar.receive(numproc-1)
+    
+    assert numpy.allclose(A, B)    
+    print "Simplified communication of 3D real arrays OK"
+
+    # 4D real arrays
+    #
+    A = numpy.array(range(M*N*N*M)).astype('f')
+    A = numpy.reshape(A, (M,N,N,M))
+    
+    pypar.send(A,1)
+    B=pypar.receive(numproc-1)
+    
+    assert numpy.allclose(A, B)    
+    print "Simplified communication of 4D real arrays OK"
+
+    # 5D real arrays
+    #
+    A = numpy.array(range(M*N*2*N*M)).astype('f')
+    A = numpy.reshape(A, (M,N,2,N,M))
+    
+    pypar.send(A,1)
+    B=pypar.receive(numproc-1)
+    
+    assert numpy.allclose(A, B)    
+    print "Simplified communication of 5D real arrays OK"
+
+    # 5D double arrays
+    #
+    A = numpy.array(range(M*N*2*N*M)).astype('d')
+    A = numpy.reshape(A, (M,N,2,N,M))
+    
+    pypar.send(A,1)
+    B=pypar.receive(numproc-1)
+    
+    assert numpy.allclose(A, B)    
+    print "Simplified communication of 5D double arrays OK"
+
+    # 5D complex arrays
+    #
+    A = numpy.array(range(M*N*2*N*M)).astype('D')
+    A = numpy.reshape(A, (M,N,2,N,M))
+    
+    pypar.send(A,1)
+    B=pypar.receive(numproc-1)
+    
+    assert numpy.allclose(A, B)    
+    print "Simplified communication of 5D complex real arrays OK"
+
+  else:  
+    # 2D real arrays
+    #
+    
+    X = pypar.receive(myid-1)  
+    pypar.send(X, (myid+1)%numproc)
+  
+    # 3D real arrays
+    #
+    X=pypar.receive(myid-1)  
+    pypar.send(X, (myid+1)%numproc)
+
+    # 4D real arrays
+    #
+    X = pypar.receive(myid-1)  
+    pypar.send(X, (myid+1)%numproc)
+
+    # 5D real arrays
+    #
+    X=pypar.receive(myid-1)  
+    pypar.send(X, (myid+1)%numproc)
+
+    # 5D double arrays
+    #
+    X=pypar.receive(myid-1)  
+    pypar.send(X, (myid+1)%numproc)
+
+    # 5D complex arrays
+    #
+    X=pypar.receive(myid-1)  
+    pypar.send(X, (myid+1)%numproc)
+
+  # Test broadcast  - with buffers (arrays, strings and general)
+  #
+      
+  testString = ('test' + str(myid)).ljust(10)  #Buffers must have the same length on all procs!
+  pypar.broadcast(testString, 0)
+  assert testString.strip() == 'test0'
+  
+  testString = ('test' + str(myid)).ljust(10)  #Buffers must have the same length on all procs!
+  pypar.broadcast(testString, numproc-1)
+  assert testString.strip() == 'test' + str(numproc-1)
+  
+  if myid == 0:
+    print "Broadcast communication of strings OK"
+
+  ####################################################  
+  N = 17 #Number of elements
+  testArray = myid * numpy.array(range(N))
+  pypar.broadcast(testArray, 1)
+  assert numpy.allclose(testArray, 1 * testArray)
+  
+  if myid == 0:    
+    print "Broadcast communication of numeric integer array OK"
+
+
+  testArray = myid * numpy.array(range(N)).astype('f')
+  pypar.broadcast(testArray, 1)
+  assert numpy.allclose(testArray, 1 * testArray)
+  if myid == 0:
+    print "Broadcast communication of numeric real array OK"
+
+
+  M = 13
+  testArray = myid * numpy.array(range(M*N)).astype('f')
+  testArray = numpy.reshape(testArray, (M,N))  
+  pypar.broadcast(testArray, 1)
+  assert numpy.allclose(testArray, 1 * testArray)
+  if myid == 0:
+    print "Broadcast communication of 2D numeric real array OK"
+
+  testArray = myid * numpy.array(range(M*2*N)).astype('f')
+  testArray = numpy.reshape(testArray, (M,2,N))  
+  pypar.broadcast(testArray, 1)
+  assert numpy.allclose(testArray, 1 * testArray)
+  if myid == 0:
+    print "Broadcast communication of 3D numeric real array OK"
+
+  testArray = myid * numpy.array(range(M*2*N)).astype('D')
+  testArray = numpy.reshape(testArray, (M,2,N))  
+  pypar.broadcast(testArray, 1)
+  assert numpy.allclose(testArray, 1 * testArray)
+  if myid == 0:
+    print "Broadcast communication of 3D numeric complex array OK"
+
+    
+  A_x = ['ABC', myid, (1,2,3), {8: 'Monty'}, numpy.array([13.45, 1.2])]
+  A_1 = ['ABC',    1, (1,2,3), {8: 'Monty'}, numpy.array([13.45, 1.2])]
+  B = pypar.broadcast(A_x, 1)
+
+  OK = True
+  for i, a in enumerate(A_1):
+    b = B[i]
+
+    if type(a).__name__ == 'ndarray':
+      if not numpy.allclose(a, b):
+        OK = False
+        break
+    elif a != b:
+      OK = False
+      break
+
+  if OK is False:
+    raise Exception    
+    
+  if myid == 0:
+    print "Broadcast communication of general structures OK"
+  
+
+
+
+  # Test scatter  - with/without buffers (arrays, strings)
+  #
+  N = 16 # Number of elements
+
+  NP = N/numproc
+  
+  testString = 'ABCDEFGHIJKLMNOP'  #Length = 16
+  X = ' '*NP
+
+  pypar.scatter(testString, 0, buffer=X) # With buffer
+  Y = pypar.scatter(testString, 0) # With buffer automatically created
+
+  assert X==Y, 'X=%s, Y=%s' %(X,Y)
+  assert Y == testString[myid*NP:(myid+1)*NP]
+  assert X == testString[myid*NP:(myid+1)*NP]
+
+  if myid == 0:    
+    print "Scatter communication of strings OK"
+    
+
+  #Scatter Arrays
+  testArray = numpy.array(range(N)).astype('i')
+  X = numpy.zeros(NP).astype('i')
+
+  pypar.scatter(testArray, 0, buffer=X) # With buffer
+  Y = pypar.scatter(testArray, 0) # With buffer automatically created
+
+  assert numpy.allclose(X, Y)  
+  assert numpy.allclose(X, testArray[myid*NP:(myid+1)*NP])
+  assert numpy.allclose(Y, testArray[myid*NP:(myid+1)*NP])   
+
+  if myid == 0:
+    print "Scatter communication of numeric integer array OK"
+
+
+  testArray = numpy.array(range(N)).astype('f')
+  X = numpy.zeros(NP).astype('f')
+
+  pypar.scatter(testArray, 0, buffer=X) # With buffer
+  Y = pypar.scatter(testArray, 0) # With buffer automatically created
+
+  assert numpy.allclose(X, Y)  
+  assert numpy.allclose(X, testArray[myid*NP:(myid+1)*NP])
+  assert numpy.allclose(Y, testArray[myid*NP:(myid+1)*NP])   
+
+  if myid == 0:
+    print "Scatter communication of numeric real arrays OK"
+  #else:
+  #  print X, testArray, Y
+  #  assert numpy.allclose(X, Y)
+
+  testArray = numpy.array(range(N)).astype('D')
+  X = numpy.zeros(NP).astype('D')
+
+  pypar.scatter(testArray, 0, buffer=X) # With buffer
+  Y = pypar.scatter(testArray, 0) # With buffer automatically created
+
+  assert numpy.allclose(X, Y)  
+  assert numpy.allclose(X, testArray[myid*NP:(myid+1)*NP])
+  assert numpy.allclose(Y, testArray[myid*NP:(myid+1)*NP])   
+
+  if myid == 0:
+    print "Scatter communication of numeric complex array OK"
+
+
+###################################################
+#FIXME: 2D scatter doesn't work yet    
+#   M = 16
+#   N = 13
+#   MP = M/numproc
+  
+#   testArray = numpy.array(range(M*N)).astype('D')
+#   testArray = numpy.reshape(testArray, (M,N))    
+#   X = numpy.zeros(MP*N).astype('D')
+#   X = numpy.reshape(X, (MP,N))
+  
+#   raw_scatter(testArray, X, 0)
+#   Y = pypar.scatter(testArray, 0)
+  
+#   assert numpy.allclose(X, Y)  
+#   assert numpy.allclose(X, testArray[myid*MP:(myid+1)*MP,:])
+#   assert numpy.allclose(Y, testArray[myid*MP:(myid+1)*MP,:])   
+
+#   if myid == 0:
+#     print "Scatter communication of 2D numeric complex array OK"
+
+
+
+  # Test gather  - with/without buffers (arrays, strings)
+  #
+  N = 17 # Number of elements
+      
+  testString = 'AB'
+  X = '_'*(len(testString)*numproc) # Blanks caused errors when numproc >= 6 
+
+  pypar.gather(testString, 0, buffer=X) # Using buffer  
+  Y = pypar.gather(testString, 0) # Create buffer automatically
+  
+  if myid == 0:
+    assert X == 'AB' * numproc
+    assert Y == 'AB' * numproc
+    print "Gather communication of strings OK"
+  
+
+  testArray = numpy.array(range(N)).astype('i')
+  X = numpy.zeros(N*numproc).astype('i')
+
+  pypar.gather(testArray, 0, buffer=X) # Using buffer  
+  Y = pypar.gather(testArray, 0) # Create buffer automatically
+
+  if myid == 0:
+    for i in range(numproc):       
+      assert numpy.allclose(testArray, X[(i * N): ((i+1)*N)])
+
+    assert numpy.allclose(X, Y)
+    print "Gather communication of numeric integer array OK"
+    
+    
+  testArray = numpy.array(range(N)).astype('f')
+  X = numpy.zeros(N*numproc).astype('f')
+
+  pypar.gather(testArray, 0, buffer=X) # Using buffer  
+  Y = pypar.gather(testArray, 0) # Create buffer automatically
+
+  if myid == 0:
+    for i in range(numproc):       
+      assert numpy.allclose(testArray, X[(i * N): ((i+1)*N)])
+    assert numpy.allclose(X, Y)      
+    print "Gather communication of numeric real array OK"
+    
+  
+  testArray = numpy.array(range(N)).astype('D')
+  X = numpy.zeros(N*numproc).astype('D')
+
+  pypar.gather(testArray, 0, buffer=X) # Using buffer  
+  Y = pypar.gather(testArray, 0) # Create buffer automatically
+
+  if myid == 0:
+    for i in range(numproc):       
+      assert numpy.allclose(testArray, X[(i * N): ((i+1)*N)])
+    assert numpy.allclose(X, Y)      
+    print "Gather communication of numeric complex arrays OK"
+
+  M = 13  
+  testArray = numpy.array(range(M*N)).astype('D')
+  testArray = numpy.reshape(testArray, (M,N))
+  X = numpy.zeros(M*N*numproc).astype('D')
+  X = numpy.reshape(X, (M*numproc,N))
+  
+  pypar.gather(testArray, 0, buffer=X) # Using buffer  
+  Y = pypar.gather(testArray, 0) # Create buffer automatically
+
+  if myid == 0:
+    for i in range(numproc):       
+      assert numpy.allclose(testArray, X[(i * M): ((i+1)*M), :])
+    assert numpy.allclose(X, Y)      
+    print "Gather communication of 2D numeric complex arrays OK"
+  
+  ########################################################
+  # Test reduce
+  #######################################################
+  N = 17 #Number of elements
+  
+  # Create one (different) array on each processor
+  #    
+  testArray = numpy.array(range(N), 'i') * (myid+1)
+  X = numpy.zeros(N, 'i') # Buffer for results
+
+  pypar.reduce(testArray, pypar.SUM, 0, buffer=X)
+
+  if myid == 0:
+    Y = numpy.zeros(N, 'i')
+    for i in range(numproc):
+      Y = Y + numpy.array(range(N), 'i')*(i+1)    
+    assert numpy.allclose(X, Y)
+    print 'Buffered reduce using pypar.SUM OK'
+        
+  pypar.reduce(testArray, pypar.MAX, 0, buffer=X)
+  if myid == 0:
+    Y = numpy.array(range(N))*numproc
+    assert numpy.allclose(X, Y)
+    print 'Buffered reduce using pypar.MAX OK'
+
+  pypar.reduce(testArray, pypar.MIN, 0, buffer=X)
+  if myid == 0:
+    Y = numpy.array(range(N))
+    assert numpy.allclose(X, Y)
+    print 'Buffered reduce using pypar.MIN OK'
+    
+  if numproc <= 20:
+    testArray_float = testArray.astype('f')  #Make room for big results
+    X_float = X.astype('f')
+
+    pypar.reduce(testArray_float, pypar.PROD, 0, buffer=X_float)
+    if myid == 0:
+      Y = numpy.ones(N).astype('f')    
+      for i in range(numproc):
+        Y = Y*numpy.array(range(N))*(i+1)    
+      #print X_float
+      #print Y  
+      assert numpy.allclose(X_float, Y)
+      print 'Buffered reduce using pypar.PROD OK'
+  else:
+    if myid == 0:
+      print "Skipping product-reduce - try again with numproc < 20"    
+
+  pypar.reduce(testArray, pypar.LAND, 0, buffer=X)
+  if myid == 0:  
+    Y = numpy.ones(N).astype('i')    
+    for i in range(numproc):
+      Y = numpy.logical_and(Y, numpy.array(range(N)).astype('i')*(i+1))  
+    assert numpy.allclose(X, Y)
+    print 'Buffered reduce using pypar.LAND OK'
+    
+  pypar.reduce(testArray, pypar.BAND, 0, buffer=X)
+  if myid == 0:
+    Y = numpy.ones(N).astype('i')*255  #Neutral element for &   
+    for i in range(numproc):
+      Y = numpy.bitwise_and(Y, numpy.array(range(N))*(i+1))
+    assert numpy.allclose(X, Y)
+    print 'Buffered reduce using pypar.BAND OK'    
+
+  pypar.reduce(testArray, pypar.LOR, 0, buffer=X)
+  if myid == 0:  
+    Y = numpy.zeros(N).astype('i')    
+    for i in range(numproc):
+      Y = numpy.logical_or(Y, numpy.array(range(N)).astype('i')*(i+1))  
+    assert numpy.allclose(X, Y)
+    print 'Buffered reduce using pypar.LOR OK'
+  
+  pypar.reduce(testArray, pypar.BOR, 0, buffer=X)
+  if myid == 0:
+    Y = numpy.zeros(N).astype('i')   #Neutral element for |   
+    for i in range(numproc):
+      Y = numpy.bitwise_or(Y, numpy.array(range(N)).astype('i')*(i+1))
+    assert numpy.allclose(X, Y)
+    print 'Buffered reduce using pypar.BOR OK'    
+
+  pypar.reduce(testArray, pypar.LXOR, 0, buffer=X)
+  if myid == 0:  
+    Y = numpy.zeros(N).astype('i')    
+    for i in range(numproc):
+      Y = numpy.logical_xor(Y, numpy.array(range(N)).astype('i')*(i+1))  
+    assert numpy.allclose(X, Y)
+    print 'Buffered reduce using pypar.LXOR OK'
+
+  pypar.reduce(testArray, pypar.BXOR, 0, buffer=X)
+  if myid == 0:
+    Y = numpy.zeros(N).astype('i')   #Neutral element for xor ?   
+    for i in range(numproc):
+      Y = numpy.bitwise_xor(Y, numpy.array(range(N)).astype('i')*(i+1))
+    assert numpy.allclose(X, Y)
+    print 'Buffered reduce using pypar.BXOR OK'    
+
+  # FIXME: NOT YET SUPPORTED: pypar.MAXLOC, pypar.MINLOC, pypar.REPLACE
+
+
+  # Test status block (simple communication)
+  N = 17  
+  if myid == 0:
+    # Integer arrays
+    #
+    A = numpy.array(range(N)).astype('i')
+
+    pypar.send(A,1)
+    B, status = pypar.receive(numproc-1, return_status = True)
+
+    repr(status)  # Check that status can be printed
+
+    sz = A.itemsize
+    assert numpy.allclose(A, B)
+    assert len(B) == status.length, 'Reported length == %d should be %d'\
+           %(status.length, len(B))
+    assert status.size == sz, 'Reported size == %d should be %d'\
+           %(status.size, sz)
+    assert status.tag == pypar.default_tag, 'Reported tag == %d should be %d'\
+           %(status.tag, pypar.default_tag)
+    assert status.error == 0, 'Status.error = %i' % status.error
+    assert status.source == numproc-1, 'Reported source == %d should be %d'\
+           %(status.source, numproc-1)
+
+    print "Status object (numeric integer arrays) OK"
+           
+    # Real arrays
+    #
+    A = numpy.array(range(N)).astype('f')
+    pypar.send(A,1)
+    B, status = pypar.receive(numproc-1, return_status = True)    
+
+    sz = A.itemsize    
+    assert numpy.allclose(A, B)
+    assert len(B) == status.length, 'Reported length == %d should be %d'\
+           %(status.length, len(B))
+    assert status.size == sz, 'Reported size == %d should be %d'\
+           %(status.size, sz)
+    assert status.tag == pypar.default_tag, 'Reported tag == %d should be %d'\
+           %(status.tag, pypar.default_tag)
+    assert status.error == 0
+    assert status.source == numproc-1, 'Reported source == %d should be %d'\
+           %(status.source, numproc-1)
+    
+    print "Status object (numeric real arrays) OK"
+
+    # Strings
+    #
+    A = "and now to something completely different !"
+    pypar.send(A,1)
+    B, status = pypar.receive(numproc-1, return_status = True)        
+
+    sz = 1 #Characters are one byte long
+    assert A == B
+    assert len(B) == status.length, 'Reported length == %d should be %d'\
+           %(status.length, len(B))
+    assert status.size == sz, 'Reported size == %d should be %d'\
+           %(status.size, sz)
+    assert status.tag == pypar.default_tag, 'Reported tag == %d should be %d'\
+           %(status.tag, pypar.default_tag)
+    assert status.error == 0
+    assert status.source == numproc-1, 'Reported source == %d should be %d'\
+           %(status.source, numproc-1)
+    
+    print "Status object (strings) OK"
+    
+    # A more general structure
+    #
+    A = ['ABC', (1,2,3.14), {8: 'Monty'}, numpy.array([13.45, 1.2])]
+    pypar.send(A,1)
+    B, status = pypar.receive(numproc-1, return_status = True)            
+
+    OK = True
+    for i, a in enumerate(A):
+      b = B[i]
+
+      if type(a).__name__ == 'ndarray':
+        if not numpy.allclose(a, b):
+          OK = False
+          break
+      elif a != b:
+        OK = False
+        break
+
+    if OK is True:
+      print 'Status object (more general structures) OK' 
+    else:
+      raise Exception    
+    
+
+    #Length is the number of characters needed to encode the structure
+    #Can't think of a test.
+
+    sz = 1
+    assert status.size == sz, 'Reported size == %d should be %d'\
+           %(status.size, sz)
+    assert status.tag == pypar.default_tag, 'Reported tag == %d should be %d'\
+           %(status.tag, pypar.default_tag)
+    assert status.error == 0
+    assert status.source == numproc-1, 'Reported source == %d should be %d'\
+           %(status.source, numproc-1)
+
+    print "Status object (general structures) OK"
+    
+  else:  
+    # Integers
+    #
+    X=pypar.receive(myid-1)  
+    pypar.send(X, (myid+1)%numproc)
+  
+    # Floats
+    #
+    X=pypar.receive(myid-1)  
+    pypar.send(X, (myid+1)%numproc)    
+
+    # Strings
+    #
+    X=pypar.receive(myid-1)  
+    pypar.send(X, (myid+1)%numproc)    
+
+    # General
+    #
+    X = pypar.receive(myid-1)  
+    pypar.send(X, (myid+1)%numproc)    
+
+
+
+
+
+
+
+
+  # Test status block (buffered communication)
+  N = 17 # Number of elements
+  if myid == 0:
+    # Integer arrays
+    #
+    A = numpy.array(range(N)).astype('i')
+    B = numpy.zeros(N).astype('i')    
+
+    pypar.send(A, 1, use_buffer=True)
+    B, status = pypar.receive(numproc-1, buffer=B, return_status=True)
+    
+    assert numpy.allclose(A, B)
+
+    sz = A.itemsize
+    assert numpy.allclose(A, B)
+    assert len(B) == status.length, 'Reported length == %d should be %d'\
+           %(status.length, len(B))
+    assert status.size == sz, 'Reported size == %d should be %d'\
+           %(status.size, sz)
+    assert status.tag == pypar.default_tag, 'Reported tag == %d should be %d'\
+           %(status.tag, pypar.default_tag)
+    assert status.error == 0
+    assert status.source == numproc-1, 'Reported source == %d should be %d'\
+           %(status.source, numproc-1)
+
+    print "Status object (buffered numeric integer arrays) OK"
+
+
+    # Real arrays
+    #
+    A = numpy.array(range(N)).astype('f')
+    B = numpy.zeros(N).astype('f')    
+    pypar.send(A, 1, use_buffer=True)
+    B, status = pypar.receive(numproc-1, buffer=B, return_status=True)
+    
+    assert numpy.allclose(A, B)
+    sz = A.itemsize
+    assert numpy.allclose(A, B)
+    assert len(B) == status.length, 'Reported length == %d should be %d'\
+           %(status.length, len(B))
+    assert status.size == sz, 'Reported size == %d should be %d'\
+           %(status.size, sz)
+    assert status.tag == pypar.default_tag, 'Reported tag == %d should be %d'\
+           %(status.tag, pypar.default_tag)
+    assert status.error == 0
+    assert status.source == numproc-1, 'Reported source == %d should be %d'\
+           %(status.source, numproc-1)
+
+    print "Status object (buffered numeric real arrays) OK"
+
+    # Strings (< 256 characters)
+    #
+    A = "and now to something completely different !"
+    B = " "*len(A)
+
+    pypar.send(A, 1, use_buffer=True)
+    B, status = pypar.receive(numproc-1, buffer=B, return_status=True)
+
+    sz = 1 #Characters are one byte long
+    assert A == B
+    assert len(B) == status.length, 'Reported length == %d should be %d'\
+           %(status.length, len(B))
+    assert status.size == sz, 'Reported size == %d should be %d'\
+           %(status.size, sz)
+    assert status.tag == pypar.default_tag, 'Reported tag == %d should be %d'\
+           %(status.tag, pypar.default_tag)
+    assert status.error == 0
+    assert status.source == numproc-1, 'Reported source == %d should be %d'\
+           %(status.source, numproc-1)
+
+    print "Status object (buffered strings) OK"
+    
+
+    
+    # A more general structure
+    #
+    A = ['ABC', (1,2,3.14), {8: 'Monty'}, numpy.array([13.45, 1.2])]
+    B = ['   ', (0,0,0.0), {0: '     '}, numpy.zeros(2).astype('f')]    
+    pypar.send(A, 1, use_buffer=True)
+    B, status = pypar.receive(numproc-1, buffer=B, return_status=True)
+
+    #assert A == B
+    sz = 1
+    assert status.size == sz, 'Reported size == %d should be %d'\
+           %(status.size, sz)
+    assert status.tag == pypar.default_tag, 'Reported tag == %d should be %d'\
+           %(status.tag, pypar.default_tag)
+    assert status.error == 0
+    assert status.source == numproc-1, 'Reported source == %d should be %d'\
+           %(status.source, numproc-1)
+
+    
+    print "Status object (buffered general structures) OK"
+    
+   
+  else:  
+    # Integers
+    #
+    X = numpy.zeros(N).astype('i')    
+
+
+    X = pypar.receive(myid-1, buffer=X)
+    pypar.send(X, (myid+1)%numproc, use_buffer=True)
+  
+    # Floats
+    #
+    X = numpy.zeros(N).astype('f')
+
+    X = pypar.receive(myid-1, buffer=X)
+    pypar.send(X, (myid+1)%numproc, use_buffer=True)
+
+    
+    # Strings
+    #
+    X = " "*256
+    X = pypar.receive(myid-1, buffer=X)
+    pypar.send(X.strip(), (myid+1)%numproc, use_buffer=True)
+
+    # General
+    #
+    X = ['   ', (0,0,0.0), {0: '     '}, numpy.zeros(2).astype('f')]
+    X = pypar.receive(myid-1, buffer=X)
+    pypar.send(X, (myid+1)%numproc, use_buffer=True)
+
+
+pypar.finalize()
+
+
+
+
Index: trunk/anuga_core/source/pypar_dist/release_notes.txt
===================================================================
--- trunk/anuga_core/source/pypar_dist/release_notes.txt	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/release_notes.txt	(revision 8494)
@@ -0,0 +1,29 @@
+Version 95: Moved to Google Code (22 Dec 2009)
+
+Release notes from version 87 to 94 (v 2.1.4):
+
+Got setup.py to work again for MPICH2
+
+Release notes from version 77 to 87 (v 2.1.2):
+
+Got rid of raw communication forms in test suite.
+The buffered forms are now consistently shown.
+
+Lots of cleanup.
+
+
+Release notes from version 49 to 76:
+
+Fixed typecode problem that showed up in December 2009 e.g. on Ubuntu 9.10.
+Fixed OpenMPI compilation workaround (r70)
+Added compilation patch suggested by Michael McKerns
+Release structure suggested by Red Hat
+
+
+Release notes from version 36 to 49:	
+
+Implemented race-condition patch from Jim Bosch
+Functionality and demo for Bsend by C Makassikis
+Deprecated obsolete functions 
+Migrated Mandelbrot demos to numpy
+Fixed demo3.py for version 2 thanks to Felix Richter
Index: trunk/anuga_core/source/pypar_dist/source/LICENSE
===================================================================
--- trunk/anuga_core/source/pypar_dist/source/LICENSE	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/source/LICENSE	(revision 8494)
@@ -0,0 +1,674 @@
+                    GNU GENERAL PUBLIC LICENSE
+                       Version 3, 29 June 2007
+
+ Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
+ Everyone is permitted to copy and distribute verbatim copies
+ of this license document, but changing it is not allowed.
+
+                            Preamble
+
+  The GNU General Public License is a free, copyleft license for
+software and other kinds of works.
+
+  The licenses for most software and other practical works are designed
+to take away your freedom to share and change the works.  By contrast,
+the GNU General Public License is intended to guarantee your freedom to
+share and change all versions of a program--to make sure it remains free
+software for all its users.  We, the Free Software Foundation, use the
+GNU General Public License for most of our software; it applies also to
+any other work released this way by its authors.  You can apply it to
+your programs, too.
+
+  When we speak of free software, we are referring to freedom, not
+price.  Our General Public Licenses are designed to make sure that you
+have the freedom to distribute copies of free software (and charge for
+them if you wish), that you receive source code or can get it if you
+want it, that you can change the software or use pieces of it in new
+free programs, and that you know you can do these things.
+
+  To protect your rights, we need to prevent others from denying you
+these rights or asking you to surrender the rights.  Therefore, you have
+certain responsibilities if you distribute copies of the software, or if
+you modify it: responsibilities to respect the freedom of others.
+
+  For example, if you distribute copies of such a program, whether
+gratis or for a fee, you must pass on to the recipients the same
+freedoms that you received.  You must make sure that they, too, receive
+or can get the source code.  And you must show them these terms so they
+know their rights.
+
+  Developers that use the GNU GPL protect your rights with two steps:
+(1) assert copyright on the software, and (2) offer you this License
+giving you legal permission to copy, distribute and/or modify it.
+
+  For the developers' and authors' protection, the GPL clearly explains
+that there is no warranty for this free software.  For both users' and
+authors' sake, the GPL requires that modified versions be marked as
+changed, so that their problems will not be attributed erroneously to
+authors of previous versions.
+
+  Some devices are designed to deny users access to install or run
+modified versions of the software inside them, although the manufacturer
+can do so.  This is fundamentally incompatible with the aim of
+protecting users' freedom to change the software.  The systematic
+pattern of such abuse occurs in the area of products for individuals to
+use, which is precisely where it is most unacceptable.  Therefore, we
+have designed this version of the GPL to prohibit the practice for those
+products.  If such problems arise substantially in other domains, we
+stand ready to extend this provision to those domains in future versions
+of the GPL, as needed to protect the freedom of users.
+
+  Finally, every program is threatened constantly by software patents.
+States should not allow patents to restrict development and use of
+software on general-purpose computers, but in those that do, we wish to
+avoid the special danger that patents applied to a free program could
+make it effectively proprietary.  To prevent this, the GPL assures that
+patents cannot be used to render the program non-free.
+
+  The precise terms and conditions for copying, distribution and
+modification follow.
+
+                       TERMS AND CONDITIONS
+
+  0. Definitions.
+
+  "This License" refers to version 3 of the GNU General Public License.
+
+  "Copyright" also means copyright-like laws that apply to other kinds of
+works, such as semiconductor masks.
+
+  "The Program" refers to any copyrightable work licensed under this
+License.  Each licensee is addressed as "you".  "Licensees" and
+"recipients" may be individuals or organizations.
+
+  To "modify" a work means to copy from or adapt all or part of the work
+in a fashion requiring copyright permission, other than the making of an
+exact copy.  The resulting work is called a "modified version" of the
+earlier work or a work "based on" the earlier work.
+
+  A "covered work" means either the unmodified Program or a work based
+on the Program.
+
+  To "propagate" a work means to do anything with it that, without
+permission, would make you directly or secondarily liable for
+infringement under applicable copyright law, except executing it on a
+computer or modifying a private copy.  Propagation includes copying,
+distribution (with or without modification), making available to the
+public, and in some countries other activities as well.
+
+  To "convey" a work means any kind of propagation that enables other
+parties to make or receive copies.  Mere interaction with a user through
+a computer network, with no transfer of a copy, is not conveying.
+
+  An interactive user interface displays "Appropriate Legal Notices"
+to the extent that it includes a convenient and prominently visible
+feature that (1) displays an appropriate copyright notice, and (2)
+tells the user that there is no warranty for the work (except to the
+extent that warranties are provided), that licensees may convey the
+work under this License, and how to view a copy of this License.  If
+the interface presents a list of user commands or options, such as a
+menu, a prominent item in the list meets this criterion.
+
+  1. Source Code.
+
+  The "source code" for a work means the preferred form of the work
+for making modifications to it.  "Object code" means any non-source
+form of a work.
+
+  A "Standard Interface" means an interface that either is an official
+standard defined by a recognized standards body, or, in the case of
+interfaces specified for a particular programming language, one that
+is widely used among developers working in that language.
+
+  The "System Libraries" of an executable work include anything, other
+than the work as a whole, that (a) is included in the normal form of
+packaging a Major Component, but which is not part of that Major
+Component, and (b) serves only to enable use of the work with that
+Major Component, or to implement a Standard Interface for which an
+implementation is available to the public in source code form.  A
+"Major Component", in this context, means a major essential component
+(kernel, window system, and so on) of the specific operating system
+(if any) on which the executable work runs, or a compiler used to
+produce the work, or an object code interpreter used to run it.
+
+  The "Corresponding Source" for a work in object code form means all
+the source code needed to generate, install, and (for an executable
+work) run the object code and to modify the work, including scripts to
+control those activities.  However, it does not include the work's
+System Libraries, or general-purpose tools or generally available free
+programs which are used unmodified in performing those activities but
+which are not part of the work.  For example, Corresponding Source
+includes interface definition files associated with source files for
+the work, and the source code for shared libraries and dynamically
+linked subprograms that the work is specifically designed to require,
+such as by intimate data communication or control flow between those
+subprograms and other parts of the work.
+
+  The Corresponding Source need not include anything that users
+can regenerate automatically from other parts of the Corresponding
+Source.
+
+  The Corresponding Source for a work in source code form is that
+same work.
+
+  2. Basic Permissions.
+
+  All rights granted under this License are granted for the term of
+copyright on the Program, and are irrevocable provided the stated
+conditions are met.  This License explicitly affirms your unlimited
+permission to run the unmodified Program.  The output from running a
+covered work is covered by this License only if the output, given its
+content, constitutes a covered work.  This License acknowledges your
+rights of fair use or other equivalent, as provided by copyright law.
+
+  You may make, run and propagate covered works that you do not
+convey, without conditions so long as your license otherwise remains
+in force.  You may convey covered works to others for the sole purpose
+of having them make modifications exclusively for you, or provide you
+with facilities for running those works, provided that you comply with
+the terms of this License in conveying all material for which you do
+not control copyright.  Those thus making or running the covered works
+for you must do so exclusively on your behalf, under your direction
+and control, on terms that prohibit them from making any copies of
+your copyrighted material outside their relationship with you.
+
+  Conveying under any other circumstances is permitted solely under
+the conditions stated below.  Sublicensing is not allowed; section 10
+makes it unnecessary.
+
+  3. Protecting Users' Legal Rights From Anti-Circumvention Law.
+
+  No covered work shall be deemed part of an effective technological
+measure under any applicable law fulfilling obligations under article
+11 of the WIPO copyright treaty adopted on 20 December 1996, or
+similar laws prohibiting or restricting circumvention of such
+measures.
+
+  When you convey a covered work, you waive any legal power to forbid
+circumvention of technological measures to the extent such circumvention
+is effected by exercising rights under this License with respect to
+the covered work, and you disclaim any intention to limit operation or
+modification of the work as a means of enforcing, against the work's
+users, your or third parties' legal rights to forbid circumvention of
+technological measures.
+
+  4. Conveying Verbatim Copies.
+
+  You may convey verbatim copies of the Program's source code as you
+receive it, in any medium, provided that you conspicuously and
+appropriately publish on each copy an appropriate copyright notice;
+keep intact all notices stating that this License and any
+non-permissive terms added in accord with section 7 apply to the code;
+keep intact all notices of the absence of any warranty; and give all
+recipients a copy of this License along with the Program.
+
+  You may charge any price or no price for each copy that you convey,
+and you may offer support or warranty protection for a fee.
+
+  5. Conveying Modified Source Versions.
+
+  You may convey a work based on the Program, or the modifications to
+produce it from the Program, in the form of source code under the
+terms of section 4, provided that you also meet all of these conditions:
+
+    a) The work must carry prominent notices stating that you modified
+    it, and giving a relevant date.
+
+    b) The work must carry prominent notices stating that it is
+    released under this License and any conditions added under section
+    7.  This requirement modifies the requirement in section 4 to
+    "keep intact all notices".
+
+    c) You must license the entire work, as a whole, under this
+    License to anyone who comes into possession of a copy.  This
+    License will therefore apply, along with any applicable section 7
+    additional terms, to the whole of the work, and all its parts,
+    regardless of how they are packaged.  This License gives no
+    permission to license the work in any other way, but it does not
+    invalidate such permission if you have separately received it.
+
+    d) If the work has interactive user interfaces, each must display
+    Appropriate Legal Notices; however, if the Program has interactive
+    interfaces that do not display Appropriate Legal Notices, your
+    work need not make them do so.
+
+  A compilation of a covered work with other separate and independent
+works, which are not by their nature extensions of the covered work,
+and which are not combined with it such as to form a larger program,
+in or on a volume of a storage or distribution medium, is called an
+"aggregate" if the compilation and its resulting copyright are not
+used to limit the access or legal rights of the compilation's users
+beyond what the individual works permit.  Inclusion of a covered work
+in an aggregate does not cause this License to apply to the other
+parts of the aggregate.
+
+  6. Conveying Non-Source Forms.
+
+  You may convey a covered work in object code form under the terms
+of sections 4 and 5, provided that you also convey the
+machine-readable Corresponding Source under the terms of this License,
+in one of these ways:
+
+    a) Convey the object code in, or embodied in, a physical product
+    (including a physical distribution medium), accompanied by the
+    Corresponding Source fixed on a durable physical medium
+    customarily used for software interchange.
+
+    b) Convey the object code in, or embodied in, a physical product
+    (including a physical distribution medium), accompanied by a
+    written offer, valid for at least three years and valid for as
+    long as you offer spare parts or customer support for that product
+    model, to give anyone who possesses the object code either (1) a
+    copy of the Corresponding Source for all the software in the
+    product that is covered by this License, on a durable physical
+    medium customarily used for software interchange, for a price no
+    more than your reasonable cost of physically performing this
+    conveying of source, or (2) access to copy the
+    Corresponding Source from a network server at no charge.
+
+    c) Convey individual copies of the object code with a copy of the
+    written offer to provide the Corresponding Source.  This
+    alternative is allowed only occasionally and noncommercially, and
+    only if you received the object code with such an offer, in accord
+    with subsection 6b.
+
+    d) Convey the object code by offering access from a designated
+    place (gratis or for a charge), and offer equivalent access to the
+    Corresponding Source in the same way through the same place at no
+    further charge.  You need not require recipients to copy the
+    Corresponding Source along with the object code.  If the place to
+    copy the object code is a network server, the Corresponding Source
+    may be on a different server (operated by you or a third party)
+    that supports equivalent copying facilities, provided you maintain
+    clear directions next to the object code saying where to find the
+    Corresponding Source.  Regardless of what server hosts the
+    Corresponding Source, you remain obligated to ensure that it is
+    available for as long as needed to satisfy these requirements.
+
+    e) Convey the object code using peer-to-peer transmission, provided
+    you inform other peers where the object code and Corresponding
+    Source of the work are being offered to the general public at no
+    charge under subsection 6d.
+
+  A separable portion of the object code, whose source code is excluded
+from the Corresponding Source as a System Library, need not be
+included in conveying the object code work.
+
+  A "User Product" is either (1) a "consumer product", which means any
+tangible personal property which is normally used for personal, family,
+or household purposes, or (2) anything designed or sold for incorporation
+into a dwelling.  In determining whether a product is a consumer product,
+doubtful cases shall be resolved in favor of coverage.  For a particular
+product received by a particular user, "normally used" refers to a
+typical or common use of that class of product, regardless of the status
+of the particular user or of the way in which the particular user
+actually uses, or expects or is expected to use, the product.  A product
+is a consumer product regardless of whether the product has substantial
+commercial, industrial or non-consumer uses, unless such uses represent
+the only significant mode of use of the product.
+
+  "Installation Information" for a User Product means any methods,
+procedures, authorization keys, or other information required to install
+and execute modified versions of a covered work in that User Product from
+a modified version of its Corresponding Source.  The information must
+suffice to ensure that the continued functioning of the modified object
+code is in no case prevented or interfered with solely because
+modification has been made.
+
+  If you convey an object code work under this section in, or with, or
+specifically for use in, a User Product, and the conveying occurs as
+part of a transaction in which the right of possession and use of the
+User Product is transferred to the recipient in perpetuity or for a
+fixed term (regardless of how the transaction is characterized), the
+Corresponding Source conveyed under this section must be accompanied
+by the Installation Information.  But this requirement does not apply
+if neither you nor any third party retains the ability to install
+modified object code on the User Product (for example, the work has
+been installed in ROM).
+
+  The requirement to provide Installation Information does not include a
+requirement to continue to provide support service, warranty, or updates
+for a work that has been modified or installed by the recipient, or for
+the User Product in which it has been modified or installed.  Access to a
+network may be denied when the modification itself materially and
+adversely affects the operation of the network or violates the rules and
+protocols for communication across the network.
+
+  Corresponding Source conveyed, and Installation Information provided,
+in accord with this section must be in a format that is publicly
+documented (and with an implementation available to the public in
+source code form), and must require no special password or key for
+unpacking, reading or copying.
+
+  7. Additional Terms.
+
+  "Additional permissions" are terms that supplement the terms of this
+License by making exceptions from one or more of its conditions.
+Additional permissions that are applicable to the entire Program shall
+be treated as though they were included in this License, to the extent
+that they are valid under applicable law.  If additional permissions
+apply only to part of the Program, that part may be used separately
+under those permissions, but the entire Program remains governed by
+this License without regard to the additional permissions.
+
+  When you convey a copy of a covered work, you may at your option
+remove any additional permissions from that copy, or from any part of
+it.  (Additional permissions may be written to require their own
+removal in certain cases when you modify the work.)  You may place
+additional permissions on material, added by you to a covered work,
+for which you have or can give appropriate copyright permission.
+
+  Notwithstanding any other provision of this License, for material you
+add to a covered work, you may (if authorized by the copyright holders of
+that material) supplement the terms of this License with terms:
+
+    a) Disclaiming warranty or limiting liability differently from the
+    terms of sections 15 and 16 of this License; or
+
+    b) Requiring preservation of specified reasonable legal notices or
+    author attributions in that material or in the Appropriate Legal
+    Notices displayed by works containing it; or
+
+    c) Prohibiting misrepresentation of the origin of that material, or
+    requiring that modified versions of such material be marked in
+    reasonable ways as different from the original version; or
+
+    d) Limiting the use for publicity purposes of names of licensors or
+    authors of the material; or
+
+    e) Declining to grant rights under trademark law for use of some
+    trade names, trademarks, or service marks; or
+
+    f) Requiring indemnification of licensors and authors of that
+    material by anyone who conveys the material (or modified versions of
+    it) with contractual assumptions of liability to the recipient, for
+    any liability that these contractual assumptions directly impose on
+    those licensors and authors.
+
+  All other non-permissive additional terms are considered "further
+restrictions" within the meaning of section 10.  If the Program as you
+received it, or any part of it, contains a notice stating that it is
+governed by this License along with a term that is a further
+restriction, you may remove that term.  If a license document contains
+a further restriction but permits relicensing or conveying under this
+License, you may add to a covered work material governed by the terms
+of that license document, provided that the further restriction does
+not survive such relicensing or conveying.
+
+  If you add terms to a covered work in accord with this section, you
+must place, in the relevant source files, a statement of the
+additional terms that apply to those files, or a notice indicating
+where to find the applicable terms.
+
+  Additional terms, permissive or non-permissive, may be stated in the
+form of a separately written license, or stated as exceptions;
+the above requirements apply either way.
+
+  8. Termination.
+
+  You may not propagate or modify a covered work except as expressly
+provided under this License.  Any attempt otherwise to propagate or
+modify it is void, and will automatically terminate your rights under
+this License (including any patent licenses granted under the third
+paragraph of section 11).
+
+  However, if you cease all violation of this License, then your
+license from a particular copyright holder is reinstated (a)
+provisionally, unless and until the copyright holder explicitly and
+finally terminates your license, and (b) permanently, if the copyright
+holder fails to notify you of the violation by some reasonable means
+prior to 60 days after the cessation.
+
+  Moreover, your license from a particular copyright holder is
+reinstated permanently if the copyright holder notifies you of the
+violation by some reasonable means, this is the first time you have
+received notice of violation of this License (for any work) from that
+copyright holder, and you cure the violation prior to 30 days after
+your receipt of the notice.
+
+  Termination of your rights under this section does not terminate the
+licenses of parties who have received copies or rights from you under
+this License.  If your rights have been terminated and not permanently
+reinstated, you do not qualify to receive new licenses for the same
+material under section 10.
+
+  9. Acceptance Not Required for Having Copies.
+
+  You are not required to accept this License in order to receive or
+run a copy of the Program.  Ancillary propagation of a covered work
+occurring solely as a consequence of using peer-to-peer transmission
+to receive a copy likewise does not require acceptance.  However,
+nothing other than this License grants you permission to propagate or
+modify any covered work.  These actions infringe copyright if you do
+not accept this License.  Therefore, by modifying or propagating a
+covered work, you indicate your acceptance of this License to do so.
+
+  10. Automatic Licensing of Downstream Recipients.
+
+  Each time you convey a covered work, the recipient automatically
+receives a license from the original licensors, to run, modify and
+propagate that work, subject to this License.  You are not responsible
+for enforcing compliance by third parties with this License.
+
+  An "entity transaction" is a transaction transferring control of an
+organization, or substantially all assets of one, or subdividing an
+organization, or merging organizations.  If propagation of a covered
+work results from an entity transaction, each party to that
+transaction who receives a copy of the work also receives whatever
+licenses to the work the party's predecessor in interest had or could
+give under the previous paragraph, plus a right to possession of the
+Corresponding Source of the work from the predecessor in interest, if
+the predecessor has it or can get it with reasonable efforts.
+
+  You may not impose any further restrictions on the exercise of the
+rights granted or affirmed under this License.  For example, you may
+not impose a license fee, royalty, or other charge for exercise of
+rights granted under this License, and you may not initiate litigation
+(including a cross-claim or counterclaim in a lawsuit) alleging that
+any patent claim is infringed by making, using, selling, offering for
+sale, or importing the Program or any portion of it.
+
+  11. Patents.
+
+  A "contributor" is a copyright holder who authorizes use under this
+License of the Program or a work on which the Program is based.  The
+work thus licensed is called the contributor's "contributor version".
+
+  A contributor's "essential patent claims" are all patent claims
+owned or controlled by the contributor, whether already acquired or
+hereafter acquired, that would be infringed by some manner, permitted
+by this License, of making, using, or selling its contributor version,
+but do not include claims that would be infringed only as a
+consequence of further modification of the contributor version.  For
+purposes of this definition, "control" includes the right to grant
+patent sublicenses in a manner consistent with the requirements of
+this License.
+
+  Each contributor grants you a non-exclusive, worldwide, royalty-free
+patent license under the contributor's essential patent claims, to
+make, use, sell, offer for sale, import and otherwise run, modify and
+propagate the contents of its contributor version.
+
+  In the following three paragraphs, a "patent license" is any express
+agreement or commitment, however denominated, not to enforce a patent
+(such as an express permission to practice a patent or covenant not to
+sue for patent infringement).  To "grant" such a patent license to a
+party means to make such an agreement or commitment not to enforce a
+patent against the party.
+
+  If you convey a covered work, knowingly relying on a patent license,
+and the Corresponding Source of the work is not available for anyone
+to copy, free of charge and under the terms of this License, through a
+publicly available network server or other readily accessible means,
+then you must either (1) cause the Corresponding Source to be so
+available, or (2) arrange to deprive yourself of the benefit of the
+patent license for this particular work, or (3) arrange, in a manner
+consistent with the requirements of this License, to extend the patent
+license to downstream recipients.  "Knowingly relying" means you have
+actual knowledge that, but for the patent license, your conveying the
+covered work in a country, or your recipient's use of the covered work
+in a country, would infringe one or more identifiable patents in that
+country that you have reason to believe are valid.
+
+  If, pursuant to or in connection with a single transaction or
+arrangement, you convey, or propagate by procuring conveyance of, a
+covered work, and grant a patent license to some of the parties
+receiving the covered work authorizing them to use, propagate, modify
+or convey a specific copy of the covered work, then the patent license
+you grant is automatically extended to all recipients of the covered
+work and works based on it.
+
+  A patent license is "discriminatory" if it does not include within
+the scope of its coverage, prohibits the exercise of, or is
+conditioned on the non-exercise of one or more of the rights that are
+specifically granted under this License.  You may not convey a covered
+work if you are a party to an arrangement with a third party that is
+in the business of distributing software, under which you make payment
+to the third party based on the extent of your activity of conveying
+the work, and under which the third party grants, to any of the
+parties who would receive the covered work from you, a discriminatory
+patent license (a) in connection with copies of the covered work
+conveyed by you (or copies made from those copies), or (b) primarily
+for and in connection with specific products or compilations that
+contain the covered work, unless you entered into that arrangement,
+or that patent license was granted, prior to 28 March 2007.
+
+  Nothing in this License shall be construed as excluding or limiting
+any implied license or other defenses to infringement that may
+otherwise be available to you under applicable patent law.
+
+  12. No Surrender of Others' Freedom.
+
+  If conditions are imposed on you (whether by court order, agreement or
+otherwise) that contradict the conditions of this License, they do not
+excuse you from the conditions of this License.  If you cannot convey a
+covered work so as to satisfy simultaneously your obligations under this
+License and any other pertinent obligations, then as a consequence you may
+not convey it at all.  For example, if you agree to terms that obligate you
+to collect a royalty for further conveying from those to whom you convey
+the Program, the only way you could satisfy both those terms and this
+License would be to refrain entirely from conveying the Program.
+
+  13. Use with the GNU Affero General Public License.
+
+  Notwithstanding any other provision of this License, you have
+permission to link or combine any covered work with a work licensed
+under version 3 of the GNU Affero General Public License into a single
+combined work, and to convey the resulting work.  The terms of this
+License will continue to apply to the part which is the covered work,
+but the special requirements of the GNU Affero General Public License,
+section 13, concerning interaction through a network will apply to the
+combination as such.
+
+  14. Revised Versions of this License.
+
+  The Free Software Foundation may publish revised and/or new versions of
+the GNU General Public License from time to time.  Such new versions will
+be similar in spirit to the present version, but may differ in detail to
+address new problems or concerns.
+
+  Each version is given a distinguishing version number.  If the
+Program specifies that a certain numbered version of the GNU General
+Public License "or any later version" applies to it, you have the
+option of following the terms and conditions either of that numbered
+version or of any later version published by the Free Software
+Foundation.  If the Program does not specify a version number of the
+GNU General Public License, you may choose any version ever published
+by the Free Software Foundation.
+
+  If the Program specifies that a proxy can decide which future
+versions of the GNU General Public License can be used, that proxy's
+public statement of acceptance of a version permanently authorizes you
+to choose that version for the Program.
+
+  Later license versions may give you additional or different
+permissions.  However, no additional obligations are imposed on any
+author or copyright holder as a result of your choosing to follow a
+later version.
+
+  15. Disclaimer of Warranty.
+
+  THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
+APPLICABLE LAW.  EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
+HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
+OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
+THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+PURPOSE.  THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
+IS WITH YOU.  SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
+ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
+
+  16. Limitation of Liability.
+
+  IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
+WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
+THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
+GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
+USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
+DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
+PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
+EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
+SUCH DAMAGES.
+
+  17. Interpretation of Sections 15 and 16.
+
+  If the disclaimer of warranty and limitation of liability provided
+above cannot be given local legal effect according to their terms,
+reviewing courts shall apply local law that most closely approximates
+an absolute waiver of all civil liability in connection with the
+Program, unless a warranty or assumption of liability accompanies a
+copy of the Program in return for a fee.
+
+                     END OF TERMS AND CONDITIONS
+
+            How to Apply These Terms to Your New Programs
+
+  If you develop a new program, and you want it to be of the greatest
+possible use to the public, the best way to achieve this is to make it
+free software which everyone can redistribute and change under these terms.
+
+  To do so, attach the following notices to the program.  It is safest
+to attach them to the start of each source file to most effectively
+state the exclusion of warranty; and each file should have at least
+the "copyright" line and a pointer to where the full notice is found.
+
+    <one line to give the program's name and a brief idea of what it does.>
+    Copyright (C) <year>  <name of author>
+
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+Also add information on how to contact you by electronic and paper mail.
+
+  If the program does terminal interaction, make it output a short
+notice like this when it starts in an interactive mode:
+
+    <program>  Copyright (C) <year>  <name of author>
+    This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
+    This is free software, and you are welcome to redistribute it
+    under certain conditions; type `show c' for details.
+
+The hypothetical commands `show w' and `show c' should show the appropriate
+parts of the General Public License.  Of course, your program's commands
+might be different; for a GUI interface, you would use an "about box".
+
+  You should also get your employer (if you work as a programmer) or school,
+if any, to sign a "copyright disclaimer" for the program, if necessary.
+For more information on this, and how to apply and follow the GNU GPL, see
+<http://www.gnu.org/licenses/>.
+
+  The GNU General Public License does not permit incorporating your program
+into proprietary programs.  If your program is a subroutine library, you
+may consider it more useful to permit linking proprietary applications with
+the library.  If this is what you want to do, use the GNU Lesser General
+Public License instead of this License.  But first, please read
+<http://www.gnu.org/philosophy/why-not-lgpl.html>.
Index: trunk/anuga_core/source/pypar_dist/source/__init__.py
===================================================================
--- trunk/anuga_core/source/pypar_dist/source/__init__.py	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/source/__init__.py	(revision 8494)
@@ -0,0 +1,42 @@
+# =============================================================================
+# pypar.py - Parallel Python using MPI
+# Copyright (C) 2001, 2002, 2003 Ole M. Nielsen 
+#              (Center for Mathematics and its Applications ANU and APAC)
+#
+#    This program is free software; you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License as published by
+#    the Free Software Foundation; either version 2 of the License, or
+#    (at your option) any later version.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License (http://www.gnu.org/copyleft/gpl.html)
+#    for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program; if not, write to the Free Software
+#    Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307
+#
+#
+# Contact address: Ole.Moller.Nielsen@gmail.com
+#
+# Version: See pypar.__version__
+# =============================================================================
+
+"""The python module pypar.py and the C-extension mpi.c 
+implements scalable parallelism on distributed and shared 
+memory architectures using an important subset of 
+the Message Passing Interface (MPI) standard.
+"""
+
+from pypar import *
+from __metadata__ import __version__, __date__, __author__
+
+#Add path of package to PYTHONPATH to allow C-extension to be loaded
+#import sys
+#sys.path += __path__
+
+
+
+
Index: trunk/anuga_core/source/pypar_dist/source/__metadata__.py
===================================================================
--- trunk/anuga_core/source/pypar_dist/source/__metadata__.py	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/source/__metadata__.py	(revision 8494)
@@ -0,0 +1,6 @@
+# Meta data for pypar
+__version__ = '2.1.4'
+__date__ = '22 December 2010'
+__author__ = 'Ole M. Nielsen'
+
+
Index: trunk/anuga_core/source/pypar_dist/source/compile.py
===================================================================
--- trunk/anuga_core/source/pypar_dist/source/compile.py	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/source/compile.py	(revision 8494)
@@ -0,0 +1,206 @@
+"""compile.py - compile Python C-extension
+
+   Commandline usage: 
+     python compile.py <filename>
+
+   Usage from within Python:
+     import compile
+     compile.compile(<filename>,..)
+
+   Ole Nielsen, Oct 2001      
+"""   
+
+import numpy
+def compile(FNs=None, CC=None, LD = None, SFLAG = None, verbose = 1):
+  """compile(FNs=None, CC=None, LD = None, SFLAG = None):
+  
+     Compile FN(s) using compiler CC (e.g. mpicc), 
+     Loader LD and shared flag SFLAG.
+     If CC is absent use default compiler dependent on platform
+     if LD is absent CC is used.
+     if SFLAG is absent platform default is used
+     FNs can be either one filename or a list of filenames
+     In the latter case, the first will be used to name so file.
+  """
+  import os, string, sys, types
+  
+  # Input check
+  #
+  assert not FNs is None, 'No filename provided'
+
+  if not type(FNs) == types.ListType:
+    FNs = [FNs]
+
+  # Determine platform and compiler
+  #
+  if sys.platform == 'sunos5':  # Solaris
+    if CC:
+      compiler = CC
+    else:  
+      compiler = 'gcc'
+    if LD:
+      loader = LD
+    else:  
+      loader = compiler
+    if SFLAG:
+      sharedflag = SFLAG
+    else:  
+      sharedflag = 'G'
+      
+  elif sys.platform == 'osf1V5':  # Compaq AlphaServer
+    if CC:
+      compiler = CC
+    else:  
+      compiler = 'cc'
+    if LD:
+      loader = LD
+    else:  
+      loader = compiler
+    if SFLAG:
+      sharedflag = SFLAG
+    else:  
+      sharedflag = 'shared'    
+      
+  elif sys.platform == 'linux2':  # Linux
+    if CC:
+      compiler = CC
+    else:  
+      compiler = 'gcc'
+    if LD:
+      loader = LD
+    else:  
+      loader = compiler
+    if SFLAG:
+      sharedflag = SFLAG
+    else:  
+      sharedflag = 'shared'    
+      
+  elif sys.platform == 'darwin':  # Mac OS X:
+    if CC:
+      compiler = CC
+    else:  
+      compiler = 'cc'
+    if LD:
+      loader = LD
+    else:  
+      loader = compiler
+    if SFLAG:
+      sharedflag = SFLAG
+    else:  
+      sharedflag = 'bundle -flat_namespace -undefined suppress'
+      
+  else:
+    if verbose: print "Unrecognised platform %s - revert to default"\
+                %sys.platform
+    
+    if CC:
+      compiler = CC
+    else:  
+      compiler = 'cc'
+    if LD:
+      loader = LD
+    else:  
+      loader = 'ld'
+    if SFLAG:
+      sharedflag = SFLAG
+    else:  
+      sharedflag = 'G'
+
+            
+  # Find location of include files
+  #
+  python_include = os.path.join(os.path.join(sys.exec_prefix, 'include'),
+                                'python'+sys.version[:3])
+  
+  
+  # Check existence of Python.h
+  #
+  headerfile = python_include + '/Python.h'
+  try:
+    open(headerfile, 'r')
+  except:
+    raise """Did not find Python header file %s.
+    Make sure files for Python C-extensions are installed. 
+    In debian linux, for example, you need to install a
+    package called something like python2.1-dev""" % headerfile
+
+
+  # Get numpy include
+  numpy_include = numpy.get_include()
+  
+  
+  # Check filename(s)
+  #
+  object_files = ''
+  for FN in FNs:        
+    root, ext = os.path.splitext(FN)
+    if ext == '':
+      FN = FN + '.c'
+    elif ext.lower() != '.c':
+      msg = 'Unrecognised extension: ' + FN
+      raise Exception(msg)
+    
+    try:
+      open(FN,'r')
+    except:    
+      msg = 'Could not open: ' + FN
+      raise Exception(msg) 
+
+    if not object_files: root1 = root  # Remember first filename        
+    object_files += root + '.o '  
+  
+  
+    # Compile
+    #
+    s = '%s -c %s -I%s -I%s -o %s.o -Wall' % (compiler, FN,
+                                              python_include, numpy_include,
+                                              root)
+    if os.name == 'posix' and os.uname()[4] == 'x86_64':
+      #Extra flags for 64 bit architectures
+      s += ' -fPIC'
+      
+    
+    if verbose:
+      print s
+    else:
+      s = s + ' 2> /dev/null' # Suppress errors
+  
+    try:
+      os.system(s)
+    except:
+      raise 'Could not compile %s - please try manually' % FN  
+
+  
+  # Make shared library (*.so)
+  s = '%s -%s %s -o %s.so' % (loader, sharedflag, object_files, root1)
+
+  if os.name == 'posix' and os.uname()[4] == 'x86_64':
+      # Extra flags for 64 bit architectures using Portland compilers
+      s += ' -mcmodel=medium'
+  
+  if verbose:
+    print s
+  else:
+    s = s + ' 2> /dev/null' # Suppress warnings
+  
+  try:  
+    err=os.system(s)
+  except:
+    raise 'Could not link %s - please try manually' % root1
+    
+
+
+
+if __name__ == '__main__':
+
+  import sys
+  
+  # Get file to compile
+  #
+  if len(sys.argv) < 2:
+    print 'Usage:'
+    print '  python compile.py <filename>'
+    sys.exit()
+
+  compile(sys.argv[1], verbose=1)
+
Index: trunk/anuga_core/source/pypar_dist/source/compile_pypar_locally.py
===================================================================
--- trunk/anuga_core/source/pypar_dist/source/compile_pypar_locally.py	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/source/compile_pypar_locally.py	(revision 8494)
@@ -0,0 +1,65 @@
+# =============================================================================
+# install.py - Installation of pypar (Parallel Python using MPI)
+# Copyright (C) 2001 Ole M. Nielsen
+#
+#    This program is free software; you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License as published by
+#    the Free Software Foundation; either version 2 of the License, or
+#    (at your option) any later version.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License (http://www.gnu.org/copyleft/gpl.html)
+#    for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program; if not, write to the Free Software
+#    Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307
+#
+#
+# Contact address: Ole.Nielsen@anu.edu.au
+#
+# Version 1.0 October 2001
+# =============================================================================
+
+"""Module compile_pypar_locally.py - Compile C extension for pypar
+"""
+
+import os, sys      
+    
+
+# Attempt to compile mpiext.so 
+#
+
+if sys.platform in ['osf1V5']:  #Compaq AlphaServer do not have mpicc (grr)
+  MPICC = 'cc -lmpi'
+else:
+  MPICC = 'mpicc'  
+
+try:
+  import compile
+  compile.compile('mpiext.c', MPICC, verbose = 1)
+except Exception, e:
+  msg = 'Could not compile C extension mpiext.c - please try manually: %s' % e
+  raise Exception(msg)
+       
+
+
+if sys.platform in ['osf1V5', 'sunos5']:  #Compaq AlphaServer or Sun
+  sys.exit()
+  # Alpha Server or Sun cannot import MPI on sequential processes
+  # OK with LAM
+
+#Check if MPI module can be initialised
+#
+# Attempt to import and initialise mpiext.so
+
+error = os.system('python -c "import mpiext" > /dev/null') 
+if error:
+  msg = 'MPI could not be initialised.'
+  raise Exception(msg)
+else:  
+  import mpiext
+  print 'MPI initialised OK'    
+
Index: trunk/anuga_core/source/pypar_dist/source/mpi_test.c
===================================================================
--- trunk/anuga_core/source/pypar_dist/source/mpi_test.c	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/source/mpi_test.c	(revision 8494)
@@ -0,0 +1,88 @@
+/*
+  Simple MPI communication test.
+  Ole Moller Nielsen - 2011
+
+  To compile
+  mpicc mpi_test.c -lmpi -Wall
+
+  To run on one node
+  mpirun --hostfile /etc/mpihosts -host node5 -npernode 2 a.out
+
+  To run on multiple nodes
+  mpirun --hostfile /etc/mpihosts -host node5,node10 -npernode 1 a.out
+*/
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <mpi.h>
+
+
+#define M  5000     /* Data size */
+
+
+int main(int argc, char **argv) {
+   int repeats = 3, msgid = 0;
+   int myid, procs;
+   int j, k;
+
+   double A[M];
+
+   int  namelen;
+   char processor_name[MPI_MAX_PROCESSOR_NAME];
+
+   MPI_Status stat;
+
+
+   /* Initialize */
+   MPI_Init(&argc, &argv);
+   MPI_Comm_size(MPI_COMM_WORLD, &procs);
+   MPI_Comm_rank(MPI_COMM_WORLD, &myid);
+   MPI_Get_processor_name(processor_name, &namelen);
+
+   if (myid == 0) {
+     printf("Number of processes = %d\n", procs);
+     printf("Test repeated %d times for reliability\n", repeats);
+   }
+
+   if (procs < 2) {
+     printf("Program needs at least two processors - aborting\n");
+     MPI_Abort(MPI_COMM_WORLD,999);
+   }
+
+   /* Create the data */
+   for (j=0; j<M; j++) {
+      A[j]=rand();
+   }
+
+   /* Synchronize */
+   MPI_Barrier(MPI_COMM_WORLD);
+   printf("I am process %d on node %s\n", myid, processor_name);
+
+   /* Pass msg circularly a number of times*/
+   for (k=0; k<repeats; k++) {
+     if (myid == 0) {
+       printf("Run %d of %d\n", k+1, repeats);
+     }
+
+     /* Communicate*/
+     if (myid == 0) {
+	 printf("P%i: Sending to P%i\n", myid, 1);
+         MPI_Send(&A[0], M, MPI_DOUBLE, 1, msgid, MPI_COMM_WORLD);
+	 printf("P%i: Waiting to receive from P%i\n", myid, procs-1);
+         MPI_Recv(&A[0], M, MPI_DOUBLE, procs-1, msgid, MPI_COMM_WORLD, &stat);
+	 printf("P%i: Received from to P%i\n", myid, procs-1);
+       } else {
+	 printf("P%i: Waiting to receive from to P%i\n", myid, myid-1);
+         MPI_Recv(&A[0], M, MPI_DOUBLE, myid-1, msgid, MPI_COMM_WORLD, &stat);
+	 printf("P%i: Sending to to P%i\n", myid, (myid+1)%procs);
+         MPI_Send(&A[0], M, MPI_DOUBLE, (myid+1)%procs, msgid, MPI_COMM_WORLD);
+     }
+   }
+
+
+   printf("P%i: Done\n", myid);
+
+   MPI_Finalize();
+   return 0;
+}
+
Index: trunk/anuga_core/source/pypar_dist/source/mpiext.c
===================================================================
--- trunk/anuga_core/source/pypar_dist/source/mpiext.c	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/source/mpiext.c	(revision 8494)
@@ -0,0 +1,1254 @@
+/************************************************************************/
+/* PyPAR - Parallel Python using MPI                 	                */
+/* Copyright (C) 2001 - 2009 Ole Nielsen                                */
+/*                                                                      */
+/* See enclosed README file for details of installation and use.   	*/
+/*                                                                 	*/   
+/* This program is free software; you can redistribute it and/or modify */
+/* it under the terms of the GNU General Public License as published by */
+/* the Free Software Foundation; either version 2 of the License, or    */
+/* (at your option) any later version.                                  */
+/*                                                                      */     
+/* This program is distributed in the hope that it will be useful,      */
+/* but WITHOUT ANY WARRANTY; without even the implied warranty of       */
+/* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the        */
+/* GNU General Public License (http://www.gnu.org/copyleft/gpl.html)    */
+/* for more details.                                                    */
+/*                                                                      */
+/* You should have received a copy of the GNU General Public License    */
+/* along with this program; if not, write to the Free Software          */
+/*  Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307*/
+/*                                                                      */
+/*                                                                 	*/
+/* Contact address: Ole.Moller.Nielsen@gmail.com                        */
+/*                                                                 	*/
+/* version (see __version__ in pypar.py)                                */
+/* date (see __date__ in pypar.py)                                      */
+/************************************************************************/
+
+
+#include "Python.h"
+#include "mpi.h"
+#include "math.h"
+#include "numpy/arrayobject.h"
+
+
+/* Handle MPI constants export (shamelessly stolen from _cursesmodule.c)*/
+#define SetDictInt(string,ch) \
+        PyDict_SetItemString(ModDict, string, PyInt_FromLong((long) (ch)));
+
+/* Remap struct MPI_op to int (easier to handle by python)*/
+#define MAX 1
+#define MIN 2
+#define SUM 3
+#define PROD 4
+#define LAND 5
+#define BAND 6
+#define LOR 7
+#define BOR 8
+#define LXOR 9
+#define BXOR 10
+#define MAXLOC 11
+#define MINLOC 12
+/*#define REPLACE 13 // Not available on all MPI systems */
+
+
+static char errmsg[132];  /*Used to cretae exception messages*/
+
+/* MPI_Bsend() related variables. */
+static void *pt_buf;	/* Pointer to allocated buffer. */
+static int buf_size;	/* Size of buffer to allocate. */ 
+
+int length(PyArrayObject *x) {  
+  /* Compute the total length of contiguous array */
+  /* Necessary for communicating multi dimensional arrays */
+  
+  int i, length;
+  
+  /* Compute total length of contiguous data */
+  length = 1;
+  for (i=0; i<x->nd; i++) {
+    length *= x->dimensions[i];
+  }  
+    
+  return length;
+}  
+
+
+MPI_Datatype type_map(PyArrayObject *x, int *count) {  
+  /* Return the MPI Datatype corresponding to                       
+     the Python data type as follows  
+  
+     TYPE    py_type  mpi_type  bytes  symbol
+     ---------------------------------------- 
+     INT       4        6         4      'i'
+     LONG      5        8         8      'l'
+     FLOAT     6       10         4      'f'  
+     DOUBLE    12      11         8      'd'
+  
+     Also return the total number of elements in the array
+  
+     The Python datatype COMPLEX ('F') and COMPLEX_DOUBLE ('D')
+     is treated as a special case to the absence of an 
+     MPI_COMPLEX datatype:
+  
+     Complex arrays are mapped to float or double arrays with real 
+     and imaginary parts alternating and count is updated. */
+  
+  int py_type;
+  MPI_Datatype mpi_type;
+  char err_msg[64];
+
+  *count = length(x);
+  
+  py_type = PyArray_TYPE(x);
+  if (py_type == NPY_DOUBLE) 
+    mpi_type = MPI_DOUBLE;
+  else if (py_type == NPY_INT) 
+    mpi_type = MPI_INT;
+  else if (py_type == NPY_CDOUBLE) {
+    mpi_type = MPI_DOUBLE;
+    (*count) *= 2;
+  } else if (py_type == NPY_FLOAT) 
+    mpi_type = MPI_FLOAT;
+  else if (py_type == NPY_LONG)   
+    mpi_type = MPI_LONG;  
+  else if (py_type == NPY_CFLOAT) {
+    mpi_type = MPI_FLOAT;
+    (*count) *= 2;
+  } else {
+    sprintf(err_msg, 
+	    "Array must be of type int or float. I got py_type == %d", 
+	    py_type);
+    PyErr_SetString(PyExc_ValueError, err_msg);
+    return (MPI_Datatype) NULL;
+  }      
+
+  //printf("Types: py_type=%d, mpi_type=%d\n", py_type, (int) mpi_type);
+  
+  return mpi_type;
+}    
+
+
+MPI_Op op_map(int py_op) {  
+  
+  MPI_Op mpi_op;
+  
+  if (py_op == MAX) 
+    mpi_op = MPI_MAX;
+  else if (py_op == MIN)   
+    mpi_op = MPI_MIN;  
+  else if (py_op == SUM)   
+    mpi_op = MPI_SUM;  
+  else if (py_op == PROD)   
+    mpi_op = MPI_PROD;  
+  else if (py_op == LAND)   
+    mpi_op = MPI_LAND;  
+  else if (py_op == BAND)   
+    mpi_op = MPI_BAND;  
+  else if (py_op == LOR)   
+    mpi_op = MPI_LOR;  
+  else if (py_op == BOR)   
+    mpi_op = MPI_BOR;  
+  else if (py_op == LXOR)   
+    mpi_op = MPI_LXOR;  
+  else if (py_op == BXOR)   
+    mpi_op = MPI_BXOR;  
+  else if (py_op == MAXLOC)
+    mpi_op = MPI_MAXLOC;
+  else if (py_op == MINLOC)   
+    mpi_op = MPI_MINLOC;  
+  /*else if (py_op == REPLACE)*/   
+  /*  mpi_op = MPI_REPLACE; */ 
+  else {
+    PyErr_SetString(PyExc_ValueError, "Operation unknown");
+    return (MPI_Op) NULL;
+  }      
+  
+  return mpi_op;
+}  
+
+
+/*********************************************************/
+/* send_string                                           */
+/* Send string of characters                             */
+/*                                                       */
+/*********************************************************/
+static PyObject *send_string(PyObject *self, PyObject *args) {
+  char *s;
+  int destination, tag, length, error, myid;
+ 
+  /* process the parameters */
+  if (!PyArg_ParseTuple(args, "s#ii", &s, &length, &destination, &tag))
+    return NULL;
+  
+  /* call the MPI routine */
+  error = MPI_Send(s, length, MPI_CHAR, destination, tag, MPI_COMM_WORLD);
+  
+  if (error != 0) {
+    MPI_Comm_rank(MPI_COMM_WORLD, &myid);    
+    sprintf(errmsg, "Proc %d: MPI_Send failed with error code %d\n", 
+	    myid, error);
+    PyErr_SetString(PyExc_RuntimeError, errmsg);
+    return NULL;
+  }  
+      
+  Py_INCREF(Py_None);
+  return (Py_None);
+}
+
+/**********************************************************/
+/* receive_string                                         */
+/* Receive string of characters                           */
+/*                                                        */
+/**********************************************************/
+static PyObject *receive_string(PyObject *self, PyObject *args) {
+  char *s;
+  int source, tag, length, error, st_length, myid; 
+  MPI_Status status;
+
+  /* process the parameters */
+  if (!PyArg_ParseTuple(args, "s#ii", &s, &length, &source, &tag))
+    return NULL;
+    
+  /* call the MPI routine */
+  error = MPI_Recv(s, length, MPI_CHAR, source, tag, MPI_COMM_WORLD, &status);
+
+  if (error != 0) {
+    MPI_Comm_rank(MPI_COMM_WORLD, &myid);    
+    sprintf(errmsg, "Proc %d: MPI_Recv failed with error code %d\n", 
+	    myid, error);
+    PyErr_SetString(PyExc_RuntimeError, errmsg);   
+    return NULL;
+  }  
+     
+  MPI_Get_count(&status, MPI_CHAR, &st_length); 
+  /* status.st_length is not available in all MPI implementations
+  // Alternative is: MPI_Get_elements(MPI_Status *, MPI_Datatype, int *); */
+
+  /* Still include error msg in status in case exception is caught. */
+  return Py_BuildValue("(iiiii)", status.MPI_SOURCE, status.MPI_TAG,
+  status.MPI_ERROR, st_length, sizeof(char));  
+}
+
+/**********************************************************/
+/* broadcast_string                                           */
+/* Broadcast string of characters                         */
+/*                                                        */
+/**********************************************************/
+static PyObject *broadcast_string(PyObject *self, PyObject *args) {
+  char *s;
+  int source, length, error, myid; 
+
+  /* process the parameters */
+  if (!PyArg_ParseTuple(args, "s#i", &s, &length, &source))
+    return NULL;
+    
+  /* call the MPI routine */
+  error = MPI_Bcast(s, length, MPI_CHAR, source, MPI_COMM_WORLD);
+  if (error != 0) {
+    MPI_Comm_rank(MPI_COMM_WORLD, &myid);    
+    sprintf(errmsg, "Proc %d: MPI_Bcast failed with error code %d\n", 
+	    myid, error);
+    PyErr_SetString(PyExc_RuntimeError, errmsg);   
+    return NULL;
+  }  
+   
+  Py_INCREF(Py_None);
+  return (Py_None);
+}
+
+/**********************************************************/
+/* scatter_string                                         */
+/* Scatter string of characters                           */
+/*                                                        */
+/**********************************************************/
+static PyObject *scatter_string(PyObject *self, PyObject *args) {
+  char *s;
+  char *d;
+  int source, count, error, myid, numprocs; 
+
+  /* process the parameters */
+  if (!PyArg_ParseTuple(args, "s#si", &s, &count, &d, &source))
+    return NULL;
+    
+  error = MPI_Comm_size(MPI_COMM_WORLD,&numprocs);    
+  count = count/numprocs;
+  
+  /* call the MPI routine */
+  error = MPI_Scatter(s, count, MPI_CHAR, d, count, MPI_CHAR, 
+		      source, MPI_COMM_WORLD);
+
+  if (error != 0) {
+    MPI_Comm_rank(MPI_COMM_WORLD, &myid);    
+    sprintf(errmsg, "Proc %d: MPI_Scatter failed with error code %d\n", 
+	    myid, error);
+    PyErr_SetString(PyExc_RuntimeError, errmsg);   
+    return NULL;
+  }  
+   
+  Py_INCREF(Py_None);
+  return (Py_None);
+}
+
+/**********************************************************/
+/* gather_string                                         */
+/* Gather string of characters                           */
+/*                                                        */
+/**********************************************************/
+static PyObject *gather_string(PyObject *self, PyObject *args) {
+  char *s;
+  char *d;
+  int source, error, count, myid; 
+
+  /* process the parameters */
+  if (!PyArg_ParseTuple(args, "s#si", &s, &count, &d, &source))
+    return NULL;
+    
+  /* call the MPI routine */
+  error = MPI_Gather(s, count, MPI_CHAR, d, count,  MPI_CHAR, 
+		     source, MPI_COMM_WORLD);
+  
+  if (error != 0) {
+    MPI_Comm_rank(MPI_COMM_WORLD, &myid);    
+    sprintf(errmsg, "Proc %d: MPI_Gather failed with error code %d\n", 
+	    myid, error);
+    PyErr_SetString(PyExc_RuntimeError, errmsg);   
+    return NULL;
+  }  
+   
+  Py_INCREF(Py_None);
+  return (Py_None);
+}
+
+
+/**********************************************************/
+/* bsend_string                                           */
+/* Send a string of characters using MPI_Bsend().         */
+/*                                                        */
+/* Return value: PyNone.                                  */
+/**********************************************************/
+static PyObject *bsend_string(PyObject *self, PyObject *args) {
+  char *s;
+  int destination, tag, length;
+  int error, myid;
+
+
+  /* Process the parameters. */
+  if (!PyArg_ParseTuple(args, "s#ii", &s, &length, &destination, &tag))
+    return NULL;
+
+  /* Call the MPI routine. */
+  error = MPI_Bsend(s, length, MPI_CHAR, destination, tag, MPI_COMM_WORLD);
+
+
+  if (error != 0) {
+    MPI_Comm_rank(MPI_COMM_WORLD, &myid);    
+    sprintf(errmsg, "Proc %d: MPI_Bsend failed with error code %d\n", 
+	    myid, error);
+    PyErr_SetString(PyExc_RuntimeError, errmsg);
+    return NULL;
+  }  
+      
+  Py_INCREF(Py_None);
+  return (Py_None);
+}
+
+
+/**********************************************************/
+/* bsend_array                                            */
+/* Send a Numpy array using MPI_Bsend().                  */
+/* Accepted types for array: float, double, int, or long. */
+/* Return value: PyNone.                                  */
+/**********************************************************/
+static PyObject *bsend_array(PyObject *self, PyObject *args) { 
+  PyObject *input;
+  PyArrayObject *x;
+
+  int destination;
+  int tag;
+  int count;
+  MPI_Datatype mpi_type;
+
+  int error, myid;
+
+  /* Process the parameters. */
+  if (!PyArg_ParseTuple(args, "Oii", &input, &destination, &tag))
+    return NULL;
+
+  /* Make Numpy array from general sequence type (no cost if already Numpy). */
+  x = (PyArrayObject *)
+    PyArray_ContiguousFromObject(input, NPY_NOTYPE, 0, 0);
+
+  /* Input check and determination of MPI type */          
+  mpi_type = type_map(x, &count);
+  if (!mpi_type)
+    return NULL;
+
+  /* Call the MPI routine */
+  error = MPI_Bsend(x->data, count, mpi_type, destination, tag,
+          MPI_COMM_WORLD);
+
+  Py_DECREF(x); 	   
+  
+  if (error != 0) {
+    MPI_Comm_rank(MPI_COMM_WORLD, &myid);    
+    sprintf(errmsg, "Proc %d: MPI_Bsend failed with error code %d\n", 
+	    myid, error);
+    PyErr_SetString(PyExc_RuntimeError, errmsg);   
+    return NULL;
+  }  
+
+  Py_INCREF(Py_None);
+  return (Py_None);
+}
+
+
+
+
+/**********************************************************/
+/* send_array                                             */
+/* Send Numeric array of type float, double, int, or long */
+/*                                                        */
+/**********************************************************/
+static PyObject *send_array(PyObject *self, PyObject *args) {
+  PyObject *input;
+  PyArrayObject *x;
+  int destination, tag, error, count, myid;
+  MPI_Datatype mpi_type;
+  
+  /* process the parameters */
+  if (!PyArg_ParseTuple(args, "Oii", &input, &destination, &tag))
+    return NULL;
+    
+  /* Make Numpy array from general sequence type (no cost if already Numpy). */
+  x = (PyArrayObject *)
+    PyArray_ContiguousFromObject(input, NPY_NOTYPE, 0, 0);
+    
+  /* Input check and determination of MPI type */          
+  mpi_type = type_map(x, &count);
+  if (!mpi_type) return NULL;
+    
+  /* call the MPI routine */
+  error = MPI_Send(x->data, count, mpi_type, destination, tag,
+		   MPI_COMM_WORLD);
+  Py_DECREF(x); 	   
+  
+  if (error != 0) {
+    MPI_Comm_rank(MPI_COMM_WORLD, &myid);    
+    sprintf(errmsg, "Proc %d: MPI_Send failed with error code %d\n", 
+	    myid, error);
+    PyErr_SetString(PyExc_RuntimeError, errmsg);   
+    return NULL;
+  }  
+   
+  Py_INCREF(Py_None);
+  return (Py_None);
+}
+
+
+
+/*************************************************************/
+/* receive_array                                             */
+/* Receive Numeric array of type float, double, int, or long */
+/*                                                           */
+/*************************************************************/
+static PyObject *receive_array(PyObject *self, PyObject *args) {
+  PyArrayObject *x;
+  int source, tag, error, st_length, size, count, myid;
+  MPI_Datatype mpi_type;
+  MPI_Status status;
+
+    
+  if (!PyArg_ParseTuple(args, "Oii", &x, &source, &tag))
+    return NULL;    
+    
+  /* Input check and determination of MPI type */          
+  mpi_type = type_map(x, &count);
+  if (!mpi_type) return NULL;  
+      
+  /* call the MPI routine */
+  error =  MPI_Recv(x->data, count, mpi_type, source, tag,
+		    MPI_COMM_WORLD, &status);
+	 
+  /* Do not DECREF x as it must be returned to Python */
+  	 
+  if (error != 0) {
+    MPI_Comm_rank(MPI_COMM_WORLD, &myid);    
+    sprintf(errmsg, "Proc %d: MPI_Recv failed with error code %d\n", 
+	    myid, error);
+    PyErr_SetString(PyExc_RuntimeError, errmsg);   
+    return NULL;
+  }  
+   
+	 
+  MPI_Get_count(&status, mpi_type, &st_length); 
+  /* status.st_length is not available in all MPI implementations */
+  /* Alternative is: MPI_Get_elements(MPI_Status *, MPI_Datatype, int *); */
+	 
+      
+  /* FIXME: This might not be watertight on all platforms */
+  /* Need C equivalent to itemsize().*/
+  if (mpi_type == MPI_DOUBLE) {
+    size = sizeof(double);  /*8 */
+  } else if (mpi_type == MPI_LONG) {
+    size = sizeof(long); /*8? */
+  } else if (mpi_type == MPI_FLOAT) {
+    size = sizeof(float);
+  } else if (mpi_type == MPI_INT) {    
+    size = sizeof(int);  
+  } else {
+    size = 4;  
+  }
+    
+  return Py_BuildValue("(iiiii)", status.MPI_SOURCE, status.MPI_TAG,
+  status.MPI_ERROR, st_length, size);  
+}
+
+
+/*************************************************************/
+/* broadcast_array                                               */
+/* Broadcast Num.  array of type float, double, int, or long */
+/*                                                           */
+/*************************************************************/
+static PyObject *broadcast_array(PyObject *self, PyObject *args) {
+  PyArrayObject *x;
+  int source, error, count, myid;
+  MPI_Datatype mpi_type;
+
+  /* process the parameters */
+  if (!PyArg_ParseTuple(args, "Oi", &x, &source))
+    return NULL;
+
+  /* Input check and determination of MPI type */          
+  mpi_type = type_map(x, &count);
+  if (!mpi_type) return NULL;  
+      
+  /* call the MPI routine */
+  error =  MPI_Bcast(x->data, count, mpi_type, source, \
+		     MPI_COMM_WORLD);
+	 
+	 
+  if (error != 0) {
+    MPI_Comm_rank(MPI_COMM_WORLD, &myid);    
+    sprintf(errmsg, "Proc %d: MPI_Bcast failed with error code %d\n", 
+	    myid, error);
+    PyErr_SetString(PyExc_RuntimeError, errmsg);   
+    return NULL;
+  }  
+	 
+  Py_INCREF(Py_None);
+  return (Py_None);
+}
+
+/*************************************************************/
+/* scatter_array                                             */
+/* Scatter Num.    array of type float, double, int, or long */
+/*                                                           */
+/*************************************************************/
+static PyObject *scatter_array(PyObject *self, PyObject *args) {
+  PyArrayObject *x;
+  PyArrayObject *d;
+  int source, error, count, myid, numprocs;
+  MPI_Datatype mpi_type;
+
+  /* process the parameters */
+  if (!PyArg_ParseTuple(args, "OOi", &x, &d, &source))
+    return NULL;
+
+  /* Input check and determination of MPI type */          
+  mpi_type = type_map(x, &count);
+  if (!mpi_type) return NULL;  
+   
+  error = MPI_Comm_size(MPI_COMM_WORLD,&numprocs);    
+  count = count/numprocs;  
+  
+  /* call the MPI routine */
+  error = MPI_Scatter(x->data, count, mpi_type, d->data, count, 
+		      mpi_type, source,	MPI_COMM_WORLD);
+	 
+  if (error != 0) {
+    MPI_Comm_rank(MPI_COMM_WORLD, &myid);    
+    sprintf(errmsg, "Proc %d: MPI_Scatter failed with error code %d\n", 
+	    myid, error);
+    PyErr_SetString(PyExc_RuntimeError, errmsg);
+    return NULL;
+  }  
+      
+  Py_INCREF(Py_None);
+  return (Py_None);
+}
+
+
+/*************************************************************/
+/* gather_array                                              */
+/* Gather Num.     array of type float, double, int, or long */
+/*                                                           */
+/*************************************************************/
+static PyObject *gather_array(PyObject *self, PyObject *args) {
+  PyArrayObject *x;
+  PyArrayObject *d;
+  int source, error, count, myid;
+  MPI_Datatype mpi_type;
+
+  /* process the parameters */
+  if (!PyArg_ParseTuple(args, "OOi", &x, &d, &source))
+    return NULL;
+
+  /* Input check and determination of MPI type */          
+  mpi_type = type_map(x, &count);
+  if (!mpi_type) return NULL;  
+      
+  /* call the MPI routine */
+  error =  MPI_Gather(x->data, count, mpi_type, d->data, count, 
+		      mpi_type, source,	MPI_COMM_WORLD);
+
+  if (error != 0) {
+    MPI_Comm_rank(MPI_COMM_WORLD, &myid);    
+    sprintf(errmsg, "Proc %d: MPI_Gather failed with error code %d\n", 
+	    myid, error);
+    PyErr_SetString(PyExc_RuntimeError, errmsg);
+    return NULL;
+  }  
+  
+  Py_INCREF(Py_None);
+  return (Py_None);
+}
+
+
+
+/*************************************************************/
+/* reduce_array                                              */
+/* Reduce Num.     array of type float, double, int, or long */
+/*                                                           */
+/*************************************************************/
+static PyObject *reduce_array(PyObject *self, PyObject *args) {
+  PyArrayObject *x;
+  PyArrayObject *d;
+  int source, op, error, count, count1, myid;
+  MPI_Datatype mpi_type, buffer_type;
+  MPI_Op mpi_op;
+
+  /* process the parameters */
+  if (!PyArg_ParseTuple(args, "OOii", &x, &d, &op, &source)) {
+    PyErr_SetString(PyExc_RuntimeError, 
+		    "mpiext.c (reduce_array): could not parse input");
+    return NULL;
+  }
+   
+  /* Input check and determination of MPI type */          
+  mpi_type = type_map(x, &count);
+  if (!mpi_type) {
+    PyErr_SetString(PyExc_RuntimeError, 
+		    "mpiext.c (reduce_array): could not determine mpi_type");
+    return NULL;  
+  }
+
+  /* This error is caught at the pypar level - so we won't end up here
+     unless mpiext is being used independently */
+  buffer_type = type_map(d, &count1);
+  if (mpi_type != buffer_type) {
+    sprintf(errmsg, "mpiext.c (reduce_array): Input array and buffer must be of the same type.");
+    PyErr_SetString(PyExc_RuntimeError, errmsg);
+		    
+    return NULL;  
+  }
+
+  if (count != count1) {
+    PyErr_SetString(PyExc_RuntimeError, 
+		    "mpiext.c (reduce_array): Input array and buffer must have same length");
+    return NULL;  
+  }
+    
+  /* Input check and determination of MPI op */ 
+  mpi_op = op_map(op);
+  if (!mpi_op) {
+    PyErr_SetString(PyExc_RuntimeError, 
+		    "mpiext.c (reduce_array): could not determine mpi_op");
+    return NULL;  
+  }
+   
+  if (op == MAXLOC || op == MINLOC) {
+    PyErr_SetString(PyExc_RuntimeError, 
+		    "mpiext.c (reduce_array): MAXLOC and MINLOC are not implemented");
+    return NULL;  
+  }
+  else {
+    /* call the MPI routine */
+    error =  MPI_Reduce(x->data, d->data, count, mpi_type, mpi_op, source, \
+			MPI_COMM_WORLD);
+  }
+         
+  if (error != 0) {
+    MPI_Comm_rank(MPI_COMM_WORLD, &myid);    
+    sprintf(errmsg, "Proc %d: MPI_Reduce failed with error code %d\n", 
+	    myid, error);
+    PyErr_SetString(PyExc_RuntimeError, errmsg);
+    return NULL;
+  }  
+  
+  Py_INCREF(Py_None);
+  return (Py_None);
+}
+
+
+/*************************************************************/
+/* allreduce_array                                              */
+/* Allreduce Num.     array of type float, double, int, or long */
+/*                                                           */
+/*************************************************************/
+static PyObject *allreduce_array(PyObject *self, PyObject *args) {
+  PyArrayObject *x;
+  PyArrayObject *d;
+  int op, error, count, count1, myid;
+  MPI_Datatype mpi_type, buffer_type;
+  MPI_Op mpi_op;
+
+  /* process the parameters */
+  if (!PyArg_ParseTuple(args, "OOi", &x, &d, &op)) {
+    PyErr_SetString(PyExc_RuntimeError,
+		    "mpiext.c (allreduce_array): could not parse input");
+    return NULL;
+  }
+
+  /* Input check and determination of MPI type */
+  mpi_type = type_map(x, &count);
+  if (!mpi_type) {
+    PyErr_SetString(PyExc_RuntimeError,
+		    "mpiext.c (allreduce_array): could not determine mpi_type");
+    return NULL;
+  }
+
+  /* This error is caught at the pypar level - so we won't end up here
+     unless mpiext is being used independently */
+  buffer_type = type_map(d, &count1);
+  if (mpi_type != buffer_type) {
+    sprintf(errmsg, "mpiext.c (allreduce_array): Input array and buffer must be of the same type.");
+    PyErr_SetString(PyExc_RuntimeError, errmsg);
+
+    return NULL;
+  }
+
+  if (count != count1) {
+    PyErr_SetString(PyExc_RuntimeError,
+		    "mpiext.c (allreduce_array): Input array and buffer must have same length");
+    return NULL;
+  }
+
+  /* Input check and determination of MPI op */
+  mpi_op = op_map(op);
+  if (!mpi_op) {
+    PyErr_SetString(PyExc_RuntimeError,
+		    "mpiext.c (allreduce_array): could not determine mpi_op");
+    return NULL;
+  }
+
+  if (op == MAXLOC || op == MINLOC) {
+    PyErr_SetString(PyExc_RuntimeError,
+		    "mpiext.c (allreduce_array): MAXLOC and MINLOC are not implemented");
+    return NULL;
+  }
+  else {
+    /* call the MPI routine */
+    error =  MPI_Allreduce(x->data, d->data, count, mpi_type, mpi_op, \
+			MPI_COMM_WORLD);
+  }
+
+  if (error != 0) {
+    MPI_Comm_rank(MPI_COMM_WORLD, &myid);
+    sprintf(errmsg, "Proc %d: MPI_Allreduce failed with error code %d\n",
+	    myid, error);
+    PyErr_SetString(PyExc_RuntimeError, errmsg);
+    return NULL;
+  }
+
+  Py_INCREF(Py_None);
+  return (Py_None);
+}
+
+
+
+
+
+
+
+/**********************************************************/
+/* CONTRIBUTED FUNCTIONS FROM CMAKASSIKIS:                */
+/* push_for_alloc functions                               */ 
+/* mpi_alloc_and_attach                                   */
+/* mpi_detach_and_dealloc                                 */ 
+/* mpi_attach                                             */
+/* mpi_detach                                             */
+/* mpi_alloc                                              */
+/* mpi_dealloc                                            */ 
+/**********************************************************/
+
+
+/*
+ * 'push_for_alloc' functions are used to increase the current size in bytes of
+ * the buffer. No effective allocation is done at the call of these functions.
+ * They merely update 'buf_size' whose value will be used when calling the
+ * function for allocation.
+ */
+
+/*
+ * 'string_push_for_alloc_and_attach'
+ *
+ * This function is used when protocol is set to 'string' or 'vanilla'.
+ *
+ * Return value: 'buf_size'.
+ */
+
+static PyObject *string_push_for_alloc_and_attach(PyObject *self, 
+						  PyObject *args) {
+  char *s;
+  int length;
+
+  /* Process the parameters. */
+  if (!PyArg_ParseTuple(args, "s#", &s, &length))
+    return NULL;
+
+  buf_size += (length + MPI_BSEND_OVERHEAD);
+
+  return Py_BuildValue("i", buf_size);
+}
+
+/*
+ * 'array_push_for_alloc_and_attach' 
+ *
+ * This function is used when protocol is set to 'array'.
+ *
+ * Return value: 'buf_size'.
+ */
+
+static PyObject *array_push_for_alloc_and_attach(PyObject *self, 
+						 PyObject *args) {
+  PyArrayObject *array;	
+  int count;				
+  int nbytes;				
+  MPI_Datatype mpi_type;	
+
+  int error, myid;
+
+  count = nbytes = error = 0;
+  mpi_type = (MPI_Datatype) 0;
+  myid = -1;
+
+  /* Process the parameters. */
+  if (!PyArg_ParseTuple(args, "O", &array))
+    return NULL;
+
+  /* Input check and determination of MPI type */          
+  mpi_type = type_map(array, &count);
+  if (!mpi_type) 
+    return NULL;
+
+  /* Compute number of bytes. */
+  error = MPI_Type_size(mpi_type, &nbytes);
+  buf_size += (nbytes * count + MPI_BSEND_OVERHEAD);
+
+  if (error != 0) {
+    MPI_Comm_rank(MPI_COMM_WORLD, &myid);    
+    sprintf(errmsg, "Proc %d: array_push_for_alloc_and_attach: \
+	        MPI_Type_size failed with error code %d\n", myid, error);
+    PyErr_SetString(PyExc_RuntimeError, errmsg);
+    return NULL;
+  }  
+
+  return Py_BuildValue("i", buf_size);
+}
+
+
+
+
+/*
+ * 'mpi_alloc_and_attach'
+ *
+ * Allocates and attaches a buffer of size 'buf_size' (in bytes) for use with
+ * MPI_Bsend() through 'bsend_string' and 'bsend_array'.
+ *
+ * Return value: 'PyNone'.
+ *
+ */
+
+static PyObject *mpi_alloc_and_attach(PyObject *self, PyObject *args) {
+	
+  //int count = 0;
+  int error, myid;
+
+  /* Process input parameters. */
+  //if (!PyArg_ParseTuple(args, "i", &count))
+  //    return NULL;
+
+  /* Allocate MPI_Bsend()'s buffer. */
+  pt_buf = (void *) malloc (buf_size);
+
+  if (pt_buf == NULL) {
+    PyErr_SetString(PyExc_RuntimeError, 
+		    "mpi_alloc_and_attach: Not enough memory to allocate bsend buffer");
+    return NULL;
+  }
+
+  /* Attach MPI_Bsend()'s buffer. */
+  error = MPI_Buffer_attach(pt_buf, buf_size);
+
+  if (error != 0) {
+    MPI_Comm_rank(MPI_COMM_WORLD, &myid);    
+    sprintf(errmsg, "Proc %d: mpi_alloc_and_attach: MPI_Buffer_attach: \
+	                 failed with error code %d\n", myid, error);
+    PyErr_SetString(PyExc_RuntimeError, errmsg);
+    return NULL;
+  }  
+
+  Py_INCREF(Py_None);
+  return (Py_None);
+}
+
+/*
+ * 'mpi_detach_and_dealloc'
+ *
+ * Detaches and deallocates buffer used for making MPI_Bsend().
+ *
+ * Reinitializes global variables 'pt_buf' and 'buf_size'.
+ *
+ * Return value: 'PyNone'.
+ *
+ */
+static PyObject *mpi_detach_and_dealloc(PyObject *self, PyObject *args) {
+  MPI_Buffer_detach(&pt_buf, &buf_size);
+
+  free(pt_buf);
+  pt_buf = NULL;
+
+  buf_size = 0;
+
+  Py_INCREF(Py_None);
+  return (Py_None);
+}
+
+/*
+ * 'mpi_attach'
+ *
+ * Attaches a buffer in order to make a MPI_Bsend().
+ *
+ * Return value: PyNone.
+ *
+ */
+static PyObject *mpi_attach(PyObject *self, PyObject *args) {
+  int error, myid;
+
+  error = MPI_Buffer_attach(pt_buf, buf_size);
+
+  if (error != 0) {
+    MPI_Comm_rank(MPI_COMM_WORLD, &myid);    
+    sprintf(errmsg, "Proc %d: MPI_Buffer_attach: failed with error code %d\n", 
+	    myid, error);
+    PyErr_SetString(PyExc_RuntimeError, errmsg); 
+    return NULL;
+  }  
+
+  Py_INCREF(Py_None);
+  return (Py_None);
+}
+
+/*
+ * 'mpi_detach'
+ *
+ * Detach a buffer after an MPI_Bsend().
+ *
+ * Return value: 'PyNone'.
+ *
+ */
+
+static PyObject *mpi_detach(PyObject *self, PyObject *args) {
+	MPI_Buffer_detach(&pt_buf, &buf_size);
+
+	Py_INCREF(Py_None);
+	return (Py_None);
+}
+
+/*
+ * Allocates 'pt_buf'.
+ *
+ * Default: allocates a buffer of size equal to 'buf_size' (in bytes).
+ * mpi_attach() has an optional parameter which, if set, overrides the 
+ * size specified by 'buf_size'.
+ *
+ * Return value: 'buf_size'.
+ *
+ */
+static PyObject *mpi_alloc(PyObject *self, PyObject *args) {
+  int nbytes = -1;	/* Number of bytes we wish to allocate. */
+
+  /* Process the parameters. */
+  if (!PyArg_ParseTuple(args, "|i", &nbytes))
+    return NULL;
+
+  if ((nbytes < 0) && (buf_size <=0)) {
+    PyErr_SetString(PyExc_RuntimeError, 
+		    "mpi_alloc: Buffer size must be set either through push_for_alloc() or directly via alloc()'s optional parameter.");
+    return NULL;
+  }
+
+  /* Override 'buf_size' value. */
+  if (nbytes > 0) {
+    buf_size = nbytes;
+  }
+
+  pt_buf = (void *) malloc (buf_size);
+  if (pt_buf == NULL) {
+    PyErr_SetString(PyExc_RuntimeError, 
+		    "mpi_alloc: Not enough memory to allocate mpi bsend buffer");
+    return NULL;
+  }
+
+  return Py_BuildValue("i", buf_size);
+}
+
+/*
+ * Deallocate.
+ */
+
+static PyObject *mpi_dealloc(PyObject *self, PyObject *args) {
+  free(pt_buf);
+  pt_buf = NULL;
+  
+  buf_size = 0;
+
+  Py_INCREF(Py_None);
+  return (Py_None);
+}
+
+
+
+
+/*********************************************************/
+/* MPI calls such as rank, size, finalize, abort         */
+/*                                                       */
+/*********************************************************/
+static PyObject * rank(PyObject *self, PyObject *args) {
+  int error, myid;
+
+  error = MPI_Comm_rank(MPI_COMM_WORLD, &myid);
+  if (error != 0) {
+    sprintf(errmsg, "Proc ?: MPI_Comm_rank failed with error code %d\n", 
+	    error);
+    PyErr_SetString(PyExc_RuntimeError, errmsg);
+    return NULL;
+  }  
+  
+  return Py_BuildValue("i", myid);
+}
+
+static PyObject * size(PyObject *self, PyObject *args) {
+  int error, numprocs, myid; 
+  
+  error = MPI_Comm_size(MPI_COMM_WORLD, &numprocs);
+  if (error != 0) {
+    MPI_Comm_rank(MPI_COMM_WORLD, &myid);    
+    sprintf(errmsg, "Proc %d: MPI_Comm_size failed with error code %d\n", 
+	    myid, error);
+    PyErr_SetString(PyExc_RuntimeError, errmsg);
+    return NULL;
+  }  
+  
+  return Py_BuildValue("i", numprocs);
+}
+  
+static PyObject * get_processor_name(PyObject *self, PyObject *args) {  
+  char processor_name[MPI_MAX_PROCESSOR_NAME];
+  int  error, namelen, myid;
+
+  error = MPI_Get_processor_name(processor_name,&namelen);
+  if (error != 0) {
+    MPI_Comm_rank(MPI_COMM_WORLD, &myid);    
+    sprintf(errmsg, "Proc %d: Get_processor_name failed with error code %d\n", 
+	    myid, error);
+    PyErr_SetString(PyExc_RuntimeError, errmsg);
+    return NULL;
+  }  
+  
+  return Py_BuildValue("s#", processor_name, namelen);
+}   
+
+static PyObject * init(PyObject *self, PyObject *args) {  
+  PyObject *input;
+
+  int i, error, myid;
+  int argc = 0;  
+  char **argv;   
+
+  /* process input parameters */
+  if (!PyArg_ParseTuple(args, "O!", &PyList_Type, &input))
+    return NULL;
+
+  /* Reconstruct C-commandline */     
+  argc = PyList_Size(input); /* Number of commandline arguments */
+  argv = (char**) malloc((argc+1)*sizeof(char*)); 
+  
+  for (i=0; i<argc; i++)  
+    argv[i] = PyString_AsString( PyList_GetItem(input, i) );
+    
+  argv[i] = NULL; /* Lam 7.0 requires last arg to be NULL  */
+  
+  error = MPI_Init(&argc, &argv); 
+  if (error != 0) {
+    MPI_Comm_rank(MPI_COMM_WORLD, &myid);    
+    sprintf(errmsg, "Proc ?: MPI_Init failed with error code %d\n", 
+	    error);
+    PyErr_SetString(PyExc_RuntimeError, errmsg);   
+    return NULL;
+  }  
+
+  Py_INCREF(Py_None);
+  return (Py_None);
+} 
+
+
+static PyObject * initialized(PyObject *self, PyObject *args) {  
+  int error, flag, myid;
+  
+  error = MPI_Initialized(&flag);
+  
+  if (error != 0) {
+    MPI_Comm_rank(MPI_COMM_WORLD, &myid);    
+    sprintf(errmsg, "Proc %d: MPI_Initialized failed with error code %d\n", 
+	    myid, error);
+    PyErr_SetString(PyExc_RuntimeError, errmsg);
+    return NULL;
+  }  
+
+  return Py_BuildValue("i", flag);  
+} 
+
+  
+static PyObject * finalize(PyObject *self, PyObject *args) {  
+  int error, myid;
+
+  MPI_Comm_rank(MPI_COMM_WORLD, &myid);  
+  
+  error = MPI_Finalize();
+  if (error != 0) {
+    sprintf(errmsg, "Proc %d: MPI_Finalize failed with error code %d\n", 
+	    myid, error);
+    PyErr_SetString(PyExc_RuntimeError, errmsg);
+    return NULL;
+  }  
+    
+  Py_INCREF(Py_None);
+  return (Py_None);
+} 
+
+static PyObject * mpi_abort(PyObject *self, PyObject *args) {  
+  int error, code=0, myid;
+  
+  error = MPI_Abort(MPI_COMM_WORLD, code);
+  if (error != 0) {
+    MPI_Comm_rank(MPI_COMM_WORLD, &myid);    
+    sprintf(errmsg, "Proc %d: MPI_Abort failed with error code %d\n", 
+	    myid, error);
+    PyErr_SetString(PyExc_RuntimeError, errmsg);
+    return NULL;
+  }  
+  
+  Py_INCREF(Py_None);
+  return (Py_None);
+} 
+
+static PyObject * barrier(PyObject *self, PyObject *args) {  
+  int error, myid;
+  
+  error = MPI_Barrier(MPI_COMM_WORLD);
+  if (error != 0) {
+    MPI_Comm_rank(MPI_COMM_WORLD, &myid);    
+    sprintf(errmsg, "Proc %d: MPI_Barrier failed with error code %d\n", 
+	    myid, error);
+    PyErr_SetString(PyExc_RuntimeError, errmsg);
+    return NULL;
+  }  
+  
+  Py_INCREF(Py_None);
+  return (Py_None);
+}    
+
+static PyObject * Wtime(PyObject *self, PyObject *args) {     
+  double t;
+  
+  t = MPI_Wtime();
+  return Py_BuildValue("d", t);      
+}     
+ 
+ 
+/**********************************/
+/* Method table for python module */
+/**********************************/
+static struct PyMethodDef MethodTable[] = {
+  {"size", size, METH_VARARGS},  
+  {"rank", rank, METH_VARARGS},  
+  {"barrier", barrier, METH_VARARGS},          
+  {"time", Wtime, METH_VARARGS},            
+  {"get_processor_name", get_processor_name, METH_VARARGS},              
+  {"init", init, METH_VARARGS},          
+  {"initialized", initialized, METH_VARARGS},       
+  {"finalize", finalize, METH_VARARGS},        
+  {"abort", mpi_abort, METH_VARARGS},          
+  {"send_string", send_string, METH_VARARGS},
+  {"receive_string", receive_string, METH_VARARGS},      
+  {"broadcast_string", broadcast_string, METH_VARARGS},        
+  {"scatter_string", scatter_string, METH_VARARGS},        
+  {"gather_string", gather_string, METH_VARARGS},        
+  {"send_array", send_array, METH_VARARGS},
+  {"receive_array", receive_array, METH_VARARGS},    
+  {"broadcast_array", broadcast_array, METH_VARARGS},              
+  {"scatter_array", scatter_array, METH_VARARGS},              
+  {"gather_array", gather_array, METH_VARARGS},              
+  {"reduce_array", reduce_array, METH_VARARGS},
+  {"allreduce_array", allreduce_array, METH_VARARGS},
+  /* Functions providing 'MPI_Bsend' support. */
+  {"bsend_string", bsend_string, METH_VARARGS},
+  {"bsend_array", bsend_array, METH_VARARGS},
+  {"string_push_for_alloc_and_attach", string_push_for_alloc_and_attach, 
+   METH_VARARGS},
+  {"array_push_for_alloc_and_attach", array_push_for_alloc_and_attach, 
+   METH_VARARGS},
+  {"mpi_alloc_and_attach", mpi_alloc_and_attach, METH_VARARGS},
+  {"mpi_detach_and_dealloc", mpi_detach_and_dealloc, METH_VARARGS},
+  {"mpi_alloc", mpi_alloc, METH_VARARGS},
+  {"mpi_dealloc", mpi_dealloc, METH_VARARGS},
+  {"mpi_attach", mpi_attach, METH_VARARGS},
+  {"mpi_detach", mpi_detach, METH_VARARGS},
+  {NULL, NULL}
+};
+
+
+/***************************/
+/* Module initialisation   */
+/***************************/
+void initmpiext(void){
+  PyObject *m, *ModDict;
+  
+  m = Py_InitModule("mpiext", MethodTable);
+  
+  /* to handle MPI symbolic constants*/
+  ModDict = PyModule_GetDict(m); 
+  SetDictInt("MPI_ANY_TAG", MPI_ANY_TAG);
+  SetDictInt("MPI_TAG_UB", MPI_TAG_UB);  
+  SetDictInt("MPI_ANY_SOURCE", MPI_ANY_SOURCE);
+  SetDictInt("MAX", MAX);
+  SetDictInt("MIN", MIN);
+  SetDictInt("SUM", SUM);
+  SetDictInt("PROD", PROD);
+  SetDictInt("LAND", LAND);
+  SetDictInt("BAND", BAND);
+  SetDictInt("LOR", LOR);
+  SetDictInt("BOR", BOR);
+  SetDictInt("LXOR", LXOR);
+  SetDictInt("BXOR", BXOR);
+  /*SetDictInt("MAXLOC", MAXLOC);*/
+  /*SetDictInt("MINLOC", MINLOC);*/
+  /*SetDictInt("REPLACE", REPLACE);*/
+
+  /*SetDictInt("MPI_COMM_WORLD", MPI_COMM_WORLD);  */
+   
+  import_array();     /* Necessary for handling of numpy structures  */
+}
Index: trunk/anuga_core/source/pypar_dist/source/network_timing.c
===================================================================
--- trunk/anuga_core/source/pypar_dist/source/network_timing.c	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/source/network_timing.c	(revision 8494)
@@ -0,0 +1,188 @@
+/*
+  Estimate bandwidth and latency of a parallel computer using MPI.
+  Ole Moller Nielsen - 1998
+*/
+	
+#include <stdio.h>
+#include <stdlib.h>
+#include <mpi.h>
+
+
+#define MAXI  10         /* Number of blocks */
+#define MAXM  500000     /* Largest block */
+#define BLOCK MAXM/MAXI  /* Block size */
+
+
+double linfit(double* x, double* y, int N, double* a, double* b) 
+{
+  /* Given vectors y and x fit a and b to the model y = ax + b */
+  
+  double Sx=0, Sy=0, SxoN=0, SSoN=0, t=0;
+  double res, varest=0, norm=0; 
+  int i;
+  
+  for (i=0; i<N; i++)
+  {
+    /*printf("x,y = %f, %f\n",x[i],y[i]);*/
+    Sx  = Sx + x[i];
+    Sy  = Sy + y[i];
+  }
+
+  SxoN = Sx/N;
+  
+  *a = 0.0; 
+  for (i=0; i<N; i++)
+  {
+    t    = x[i] - SxoN;
+    SSoN = SSoN + t*t;
+    *a   = *a + t*y[i];
+  }
+
+  *a = (*a)/SSoN;          /* a = (N Sxy - SxSy)/(NSxx - Sx^2) */
+  *b = (Sy - Sx*(*a))/N;
+  
+  /* Quality - variance estimate \sum_i r_i^2 /(m-n) */
+  for (i=0; i<N; i++)
+  {
+    norm = norm + x[i]*x[i];
+    res = y[i] - (*a)*x[i] - (*b);
+    varest = varest + res*res;
+  } 
+  varest = varest/norm/(N-2); 
+  return(varest);
+}
+
+main(int argc, char **argv) 
+{
+   int repeats = 10, msgid = 0;
+   int myid, procs;
+   int i,j,k,m;
+
+   double t1, t2, cpuOH; 
+   double Tbw, Tlat;
+   double varest;
+      
+   int noelem[MAXI];
+   double bytes[MAXI];   
+   double mintime[MAXI];   
+   double maxtime[MAXI];      
+   double avgtime[MAXI];         
+   double A[MAXM]; 
+
+   int  namelen;
+   char processor_name[MPI_MAX_PROCESSOR_NAME];
+   
+   MPI_Status stat;
+  
+   
+   /* Initialize */
+
+   MPI_Init(&argc,&argv);
+   MPI_Comm_size(MPI_COMM_WORLD,&procs);
+   MPI_Comm_rank(MPI_COMM_WORLD,&myid);
+   MPI_Get_processor_name(processor_name,&namelen);
+   
+   if (myid==0)
+   {
+     printf("MAXM = %d, number of processors = %d\n",MAXM,procs);         
+     printf("Measurements are repeated %d times for reliability\n",repeats);
+   } 
+   
+   if (procs < 2) {
+     printf("Program needs at least two processors - aborting\n");
+     MPI_Abort(MPI_COMM_WORLD,999);
+   }
+   
+   MPI_Barrier(MPI_COMM_WORLD); /* Synchronize */   
+   printf("I am process %d on %s\n",myid,processor_name);   
+      
+   for (j=0; j<MAXM; j++) 
+   {
+      A[j]=rand();
+   }   
+   for (i=0; i<MAXI; i++) 
+   {
+      avgtime[i] =  0;         
+      mintime[i] =  1000000;      
+      maxtime[i] = -1000000;            
+   }
+   
+   /* Determine timer overhead */
+   if (myid == 0) {   
+     cpuOH = 1.0;
+     for (k=0; k<repeats; k++)   /* Repeat to get reliable timings */
+     { 
+       t1 = MPI_Wtime();
+       t2 = MPI_Wtime();
+       if (t2-t1 < cpuOH) cpuOH = t2-t1;
+     }  
+     printf("Timing overhead is %f seconds\n\n", cpuOH);              
+   }
+
+   
+        
+   /* Pass msg circularly */
+     
+   for (k=0; k<repeats; k++) {
+     if (myid == 0) {  
+       printf("Run %d of %d\n", k+1, repeats);
+     }  
+
+     for (i=0; i<MAXI; i++) {
+       /*m=BLOCK*(i+1);*/
+       m=BLOCK*i+1;       
+      
+       noelem[i] = m;
+      
+       MPI_Barrier(MPI_COMM_WORLD); /* Synchronize */
+      
+       if (myid == 0) {
+         t1=MPI_Wtime();
+         MPI_Send(&A[0],m,MPI_DOUBLE,1,msgid,MPI_COMM_WORLD);
+         MPI_Recv(&A[0],m,MPI_DOUBLE,procs-1,msgid,MPI_COMM_WORLD,&stat);
+         t2=MPI_Wtime() - t1 - cpuOH;
+	 t2 = t2/procs;
+	 avgtime[i] = avgtime[i] + t2;
+         if (t2 < mintime[i]) mintime[i] = t2;
+         if (t2 > maxtime[i]) maxtime[i] = t2;	 
+       } else {
+         MPI_Recv(&A[0],m,MPI_DOUBLE,myid-1,msgid,MPI_COMM_WORLD,&stat);
+         MPI_Send(&A[0],m,MPI_DOUBLE,(myid+1)%procs,msgid,MPI_COMM_WORLD);
+       }
+     } 
+   }
+
+   if (myid == 0) {
+     printf("Bytes transferred   time (micro seconds)\n");
+     printf("                    min        avg        max \n");     
+     printf("----------------------------------------------\n");      
+        
+     for (i=0; i<MAXI; i++) {
+       avgtime[i] = avgtime[i]/repeats*1.0e6; /*Average micro seconds*/
+       mintime[i] = mintime[i]*1.0e6;         /*Min micro seconds*/       
+       maxtime[i] = maxtime[i]*1.0e6;         /*Min micro seconds*/              
+             
+       m = noelem[i];
+       bytes[i] = (double) 8*noelem[i];       
+	 
+       /* printf("m=%d, time(min)=%lf, time(avg)=%lf, time(max)=%lf\n",
+	       m,mintime[i],avgtime[i],maxtime[i]); */
+       printf("%10d    %10d %10d %10d\n",
+	 (int) bytes[i], (int) mintime[i], (int) avgtime[i], (int)maxtime[i]); 
+     }		 
+   
+     varest=linfit(bytes, mintime, MAXI, &Tbw, &Tlat);
+     printf("\nLinear regression on best timings (t = t_l + t_b * bytes):\n");
+
+     printf("  t_b = %f\n  t_l = %f\n", Tbw, Tlat);
+     printf("  Estimated relative variance = %.9f\n\n",varest);     
+
+     printf("Estimated bandwith (1/t_b):  %.3f Mb/s\n", (1.0/Tbw));         
+     printf("Estimated latency:           %d micro s\n", 
+             (int) (mintime[0] - (float) bytes[0]* (float)Tbw));         
+     
+   }
+
+   MPI_Finalize();
+}
+
Index: trunk/anuga_core/source/pypar_dist/source/network_timing.py
===================================================================
--- trunk/anuga_core/source/pypar_dist/source/network_timing.py	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/source/network_timing.py	(revision 8494)
@@ -0,0 +1,240 @@
+# Timing of MPI module for Python and estimation of latency and bandwidth
+# 
+# Send numerical array in a ring from processor 0 to 1 etc back to 0 
+# Perform timings and compare different sending strategies 
+#
+# OMN, OCT 2001
+
+import time, sys, pypar, numpy
+
+# The send/recv routines
+#
+import pypar
+
+#from mpiext import send_array, receive_array  #most direct methods
+#Not available in final install - use bypass form for now
+
+#--------------------------------------------------------------
+
+method = 2   # Use 
+             # 0: automatically allocated buffers
+             # 1: user-supplied buffers
+             # 2: Use bypass - let pypar use direct mpi_ext call
+             # 3 (not there): direct call to mpiext (with buffers),
+             #    only fractionally better than bypass
+                  
+vanilla = False  # Force use of vanilla format (1)
+consistency_check = False # Check correctness
+
+
+#--------------------------------------------------------------
+# linfit
+#
+def linfit(x, y):
+  """Fit a and b to the model y = ax + b. Return a,b,variance
+  """
+  
+  Sx = Sy = SSoN = SxoN = norm = varest = 0.0
+  N = len(x)
+  assert len(y) == N, "x and y must have same length"
+  
+  for i in range(N):
+    #print("x,y = %f, %f\n",x[i],y[i])
+    Sx  = Sx + x[i]
+    Sy  = Sy + y[i]
+  
+  SxoN = Sx/N
+  
+  a = 0.0 
+  for i in range(N):
+    t    = x[i] - SxoN
+    SSoN = SSoN + t*t
+    a    = a + t*y[i]
+
+
+  a = a/SSoN            # a = (N Sxy - SxSy)/(NSxx - Sx^2) */
+  b = (Sy - Sx*a)/N
+  
+  # Quality - variance estimate \sum_i r_i^2 /(m-n) 
+  for i in range(N):  
+    norm = norm + float(x[i])*x[i]
+    res = y[i] - a*x[i] - b
+    varest = varest + res*res
+
+  varest = varest/norm/(N-2)
+  return a, b, varest
+
+
+
+
+#--------------------------------------------------------------
+# Main program
+#
+MAXI  = 10         # Number of blocks 
+MAXM  = 500000     # Largest block 
+BLOCK = MAXM/MAXI  # Block size 
+
+repeats = 10
+msgid = 0
+vanilla = 0 #Select vanilla mode (slower but general)
+
+numprocs = pypar.size()
+myid = pypar.rank()
+processor_name = pypar.get_processor_name()
+
+if myid == 0:
+  # Main process - Create message, pass on, verify correctness and log timing
+  #
+  print "MAXM = %d, number of processors = %d" %(MAXM, numprocs)
+  print "Measurements are repeated %d times for reliability" %repeats
+
+if numprocs < 2:
+  print "Program needs at least two processors - aborting\n"
+  pypar.abort()
+   
+pypar.barrier() #Synchronize all before timing   
+print "I am process %d on %s" %(myid,processor_name)
+
+
+#Initialise data and timings
+#
+
+try:
+  from numpy.random import uniform, seed
+  seed(17)
+  A = uniform(0.0,100.0,MAXM)
+except:
+  print 'problem with RandomArray'
+  from numpy import ones, Float
+  A = ones(MAXM).astype('f')
+  
+elsize = A.itemsize
+#print elsize
+
+noelem  = [0]*MAXI
+bytes   = [0]*MAXI         
+avgtime = [0.0]*MAXI         
+mintime = [ 1000000.0]*MAXI      
+maxtime = [-1000000.0]*MAXI            
+
+
+
+
+if myid == 0:   
+  # Determine timer overhead 
+  cpuOH = 1.0;
+  for k in range(repeats):   # Repeat to get reliable timings 
+    t1 = pypar.time()
+    t2 = pypar.time()
+    if t2-t1 < cpuOH: cpuOH = t2-t1
+    
+  print "Timing overhead is %f seconds.\n" %cpuOH         
+
+     
+# Pass msg circularly   
+for k in range(repeats):
+  if myid == 0:
+    print "Run %d of %d" %(k+1,repeats)
+    
+  for i in range(MAXI):
+    m=BLOCK*i+1       
+   
+    noelem[i] = m
+   
+    pypar.barrier() # Synchronize 
+   
+    if myid == 0:
+      #
+      # Main process
+      #
+      t1 = pypar.time()
+
+      if method==0:
+        pypar.send(A[:m], destination=1, tag=msgid, vanilla=vanilla)
+        C = pypar.receive(numprocs-1, tag=msgid, vanilla=vanilla)
+      elif method == 1:  
+        pypar.send(A[:m], use_buffer=True, destination=1,
+                   tag=msgid, vanilla=vanilla)
+        C = pypar.receive(numprocs-1, buffer=A[:m], tag=msgid, vanilla=vanilla)
+      elif method==2:
+        pypar.send(A[:m], use_buffer=True, destination=1,
+                   tag=msgid, vanilla=vanilla, bypass=True)
+        C = pypar.receive(numprocs-1, buffer=A[:m], tag=msgid,
+                          vanilla=vanilla, bypass=True)
+      else:
+        raise 'Unknown method'
+        #send_array(A[:m], 1, msgid)    
+        #stat = receive_array(A[:m], numprocs-1, msgid)
+        #C = A[:m]
+        
+      t2 = pypar.time() - t1 - cpuOH
+      t2 = t2/numprocs
+      avgtime[i] = avgtime[i] + t2
+      if t2 < mintime[i]: mintime[i] = t2
+      if t2 > maxtime[i]: maxtime[i] = t2
+
+      # Uncomment to verify integrity of data
+      # However, this may affect accuracy of timings for some reason.
+      #
+      if consistency_check:
+        assert numpy.alltrue(C == A[:m])
+    else:
+      #
+      # Parallel process - get msg and pass it on
+      #
+
+      if method==0:
+        C = pypar.receive(myid-1, tag=msgid, vanilla=vanilla)
+        pypar.send(A[:m], destination=(myid+1)%numprocs,
+                   tag=msgid, vanilla=vanilla)
+      elif method==1:  
+        C = pypar.receive(myid-1, buffer=A[:m], tag=msgid, vanilla=vanilla)
+        pypar.send(A[:m], use_buffer=True, destination=(myid+1)%numprocs,
+                   tag=msgid, vanilla=vanilla)                        
+      elif method==2:
+        # Use pypar bypass
+        C = pypar.receive(myid-1, buffer=A[:m], tag=msgid,
+                          vanilla=vanilla, bypass=True)
+        pypar.send(A[:m], use_buffer=True, destination=(myid+1)%numprocs,
+                   tag=msgid, vanilla=vanilla, bypass=True)
+      else:
+        raise 'Unknown method'
+        # Use direct mpiext call
+        #stat = receive_array(A[:m], myid-1, msgid)                
+        #send_array(A[:m], (myid+1)%numprocs, msgid)
+
+
+# Output stats
+#
+if myid == 0:
+  print "Bytes transferred   time (micro seconds)"
+  print "                    min        avg        max "     
+  print "----------------------------------------------"
+     
+  for i in range(MAXI):
+    avgtime[i] = avgtime[i]/repeats*1.0e6 #Average micro seconds
+    mintime[i] = mintime[i]*1.0e6         #Min micro seconds       
+    maxtime[i] = maxtime[i]*1.0e6         #Min micro seconds              
+          
+    m = noelem[i]
+    bytes[i] = elsize*noelem[i]       
+      
+    print "%10d    %10d %10d %10d" %(bytes[i], mintime[i], avgtime[i], maxtime[i]) 
+
+
+  Tbw, Tlat, varest = linfit(bytes, mintime)
+  print "\nLinear regression on best timings (t = t_l + t_b * bytes):\n",
+  print "  t_b = %f\n  t_l = %f" %(Tbw,Tlat)
+  print "  Estimated relative variance = %.9f\n" %varest
+       
+  print "Estimated bandwith (1/t_b):  %.3f Mb/s" %(1.0/Tbw)   
+  print "Estimated latency:           %d micro s" %int(mintime[0]-bytes[0]*Tbw)  
+
+   
+pypar.finalize()
+
+
+
+
+  
+  
Index: trunk/anuga_core/source/pypar_dist/source/pypar.py
===================================================================
--- trunk/anuga_core/source/pypar_dist/source/pypar.py	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/source/pypar.py	(revision 8494)
@@ -0,0 +1,927 @@
+# =============================================================================
+# pypar.py - Parallel Python using MPI
+# Copyright (C) 2001, 2002, 2003 Ole M. Nielsen
+#              (Center for Mathematics and its Applications ANU and APAC)
+#
+#    This program is free software; you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License as published by
+#    the Free Software Foundation; either version 2 of the License, or
+#    (at your option) any later version.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License (http://www.gnu.org/copyleft/gpl.html)
+#    for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program; if not, write to the Free Software
+#    Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307
+#
+#
+# Contact address: Ole.Moller.Nielsen@gmail.com
+#
+# Version: See pypar.__version__
+# =============================================================================
+
+"""Module pypar.py - Parallel Python using MPI
+
+Public functions:
+
+size() -- Number of processors
+rank() -- Id of current processor
+get_processor_name() -- Return host name of current node
+
+send() -- Blocking send (all types)
+receive() -- Blocking receive (all types)
+broadcast() -- Broadcast
+time() -- MPI wall time
+barrier() -- Synchronisation point. Makes processors wait until all processors
+             have reached this point.
+abort() -- Terminate all processes.
+finalize() -- Cleanup MPI. No parallelism can take place after this point.
+
+
+See doc strings of individual functions for detailed documentation.
+"""
+
+from numpy import zeros, reshape, product
+from __metadata__ import __version__, __date__, __author__
+
+
+# Constants
+#
+max_tag = 32767      # Max tag value (MPI_TAG_UB didn't work and returned 0)
+control_tag = 13849  # Reserved tag used to identify control information
+default_tag = 1      # Tag used as default if not specified
+
+control_sep = ':'          # Separator for fields in control info (NOT ',')
+control_data_max_size = 64 # Maximal size of string holding control data
+
+
+#---------------------------------------------------------------------------
+# Communication functions
+#--------------------------------------------------------------------------
+
+def send(x, destination, use_buffer=False, vanilla=False,
+         tag=default_tag, bypass=False):
+    """Wrapper for easy MPI send.
+       Send x to destination.
+
+       Automatically determine appropriate protocol
+       and call corresponding send function.
+       Also passes type and size information on as preceding message to
+       simplify the receive call.
+
+       The variable x can be any (picklable) type, but
+       numpy variables and text strings will most efficient.
+       Setting vanilla = 1 forces vanilla mode for any type.
+
+       If use_buffer is True, workspace x will be used for the return value.
+       In this case the corresponding receive call must specify a buffer.
+       Otherwise a new workspace will be created by receive.
+
+       If bypass is True, all admin and error checks
+       get bypassed to reduce the latency. Should only
+       be used for sending numpy arrays and should be matched
+       with a bypass in the corresponding receive command.
+
+    """
+    import types, string
+
+    if bypass:
+        send_array(x, destination, tag)
+        return
+
+    # Input check
+    errmsg = 'Destination id (%s) must be an integer.' % destination
+    assert type(destination) == types.IntType, errmsg
+
+    errmsg = 'Tag %d is reserved by pypar - please use another.' % control_tag
+    assert tag != control_tag, errmsg
+
+    # Create metadata about object to be sent
+    control_info, x = create_control_info(x, vanilla, return_object=True)
+    protocol = control_info[0]
+
+
+    # Possibly transmit control data
+    if use_buffer is False:
+        send_control_info(control_info, destination)
+
+
+    # Transmit payload data
+    if protocol == 'array':
+        send_array(x, destination, tag)
+    elif protocol in ['string', 'vanilla']:
+        send_string(x, destination, tag)
+    else:
+        raise 'Unknown protocol: %s' %protocol
+
+
+def receive(source, buffer=None, vanilla=False, tag=default_tag,
+            return_status=False, bypass=False):
+    """receive - blocking MPI receive
+
+       Receive data from source.
+
+       Optional parameters:
+         buffer: Use specified buffer for received data (faster). Default None.
+         vanilla: Specify to enforce vanilla protocol for any type.
+                  Default False
+
+         tag: Only received messages tagged as specified. Default default_tag
+         return_status: Return Status object along with result. Default False.
+
+       If no buffer is specified, receive will try to receive a
+       preceding message containing protocol, type, size and shape and
+       then create a suitable buffer.
+
+       If buffer is specified the corresponding send must specify
+       use_buffer = True.
+       The variable buffer can be any (picklable) type, but
+       numpy variables and text strings will most efficient.
+
+       Appropriate protocol will be automatically determined
+       and corresponding receive function called.
+
+
+       If bypass is True, all admin and error checks
+       get bypassed to reduce the latency. Should only
+       be used for receiving numpy arrays and should
+       be matched with a bypass in the corresponding send command.
+       Also buffer must be specified.
+    """
+
+    if bypass:
+        # errmsg = 'bypass mode must be used with specified buffer'
+        # assert buffer is not None, msg
+        stat = receive_array(buffer, source, tag)
+    else:
+
+        import types
+
+        # Input check
+        errmsg = 'Source id (%s) must be an integer.' %source
+        assert type(source) == types.IntType, errmsg
+
+        errmsg = 'Tag %d is reserved by pypar - please use another.'\
+            % control_tag
+        assert tag != control_tag, errmsg
+
+
+        # Either receive or create metadata about object to receive
+        if buffer is None:
+            control_info, source = receive_control_info(source,
+                                                        return_source=True)
+            protocol, typecode, size, shape = control_info
+        else:
+            protocol, typecode, size, shape = create_control_info(buffer,
+                                                                  vanilla)
+
+
+        # Receive payload data
+        if protocol == 'array':
+            if buffer is None:
+                buffer = zeros(size,typecode)
+                buffer = reshape(buffer, shape)
+
+            stat = receive_array(buffer, source, tag)
+
+        elif protocol == 'string':
+            if buffer is None:
+                buffer = ' '*size
+
+            stat = receive_string(buffer, source, tag)
+
+        elif protocol == 'vanilla':
+            from cPickle import dumps, loads, UnpicklingError
+            if buffer is None:
+                s = ' '*size
+            else:
+                s = dumps(buffer, protocol=2)
+                s = s + ' '*int(0.1*len(s)) #safety
+
+            stat = receive_string(s, source, tag)
+            try:
+                buffer = loads(s)   #Replace buffer with received result
+            except UnpicklingError, err:
+                raise UnpicklingError(str(err) + " - '%s'" % s)
+        else:
+            raise 'Unknown protocol: %s' %protocol
+
+    # Return received data and possibly the status object
+    if return_status:
+        return buffer, Status(stat)
+    else:
+        return buffer
+
+
+def broadcast(buffer, root, vanilla=False, bypass=False):
+    """Wrapper for MPI bcast.
+
+       Broadcast buffer from the process with rank root to all other processes.
+
+
+       Automatically determine appropriate protocol
+       and call corresponding send function.
+
+       The variable buffer can be any (picklable) type, but
+       numpy variables and text strings will most efficient.
+       Setting vanilla = 1 forces vanilla mode for any type.
+
+       If bypass is True, all admin and error checks
+       get bypassed to reduce the latency.
+
+    """
+
+    if bypass:
+        broadcast_array(buffer, root)
+        return
+
+
+    import types
+
+    # Input check
+    errmsg = 'Root id (%s) must be an integer.' %root
+    assert type(root) == types.IntType, errmsg
+
+
+    # Create metadata about object to be sent
+    protocol = create_control_info(buffer, vanilla)[0]
+
+
+    # Broadcast
+    if protocol == 'array':
+        broadcast_array(buffer, root)
+    elif protocol == 'string':
+        broadcast_string(buffer, root)
+    elif protocol == 'vanilla':
+        from cPickle import loads, dumps, UnpicklingError
+        s = dumps(buffer, protocol=2)
+        s = s + ' '*int(0.1*len(s)) #safety
+
+        broadcast_string(s, root)
+        try:
+            buffer = loads(s)
+        except UnpicklingError, err:
+            raise UnpicklingError(str(err) + " - '%s'" % s)
+    else:
+        raise 'Unknown protocol: %s' %protocol
+
+    return buffer
+
+
+def scatter(x, root, buffer=None, vanilla=False):
+    """Sends data x from process with rank root to all other processes.
+
+       Create appropriate buffer and receive data.
+       Return scattered result (same type as x)
+
+       Scatter makes only sense for arrays or strings
+    """
+
+    import types
+    from mpiext import size
+    numproc = size()         # Needed to determine buffer size
+
+    # Input check
+    errmsg = 'Root id (%s) must be an integer.' %root
+    assert type(root) == types.IntType, errmsg
+
+
+    # Create metadata about object to be sent
+    protocol, typecode, size, shape = create_control_info(x)
+
+    # Scatter
+    if protocol == 'array':
+        if buffer is None:
+            # Modify shape along axis=0 to match size
+            shape = list(shape)
+            shape[0] /= numproc
+            count = product(shape)
+
+            buffer = zeros(count, typecode)
+            buffer = reshape(buffer, shape)
+
+        scatter_array(x, buffer, root)
+    elif protocol == 'string':
+        if buffer is None:
+            buffer = ' '*(size/numproc)
+
+        scatter_string(x, buffer, root)
+    elif protocol == 'vanilla':
+        errmsg = 'Scatter is only supported for numpy arrays and strings.\n'
+        errmsg += 'If you wish to distribute a general sequence, '
+        errmsg += 'please use send and receive commands or broadcast.'
+        raise errmsg
+    else:
+        raise 'Unknown protocol: %s' %protocol
+
+    return buffer
+
+
+def gather(x, root, buffer=None, vanilla=0):
+    """Gather values from all processes to root
+
+       Create appropriate buffer and receive data.
+
+       Gather only makes sens for arrays or strings
+    """
+
+    import types
+    from mpiext import size
+    numproc = size()         #Needed to determine buffer size
+
+    # Input check
+    errmsg = 'Root id (%s) must be an integer.' %root
+    assert type(root) == types.IntType, errmsg
+
+    # Create metadata about object to be gathered
+    protocol, typecode, size, shape = create_control_info(x)
+
+    # Gather
+    if protocol == 'array':
+        if buffer is None:
+            buffer = zeros(size*numproc, typecode)
+
+            # Modify shape along axis=0 to match size
+            shape = list(shape)
+            shape[0] *= numproc
+            buffer = reshape(buffer, shape)
+
+        gather_array(x, buffer, root)
+    elif protocol == 'string':
+        if buffer is None:
+            buffer = ' '*size*numproc
+
+        gather_string(x, buffer, root)
+    elif protocol == 'vanilla':
+        errmsg = 'Gather is only supported for numpy arrays and strings.\n'
+        errmsg += 'If you wish to distribute a general sequence, '
+        errmsg += 'please use send and receive commands or broadcast.'
+        raise errmsg
+    else:
+        raise 'Unknown protocol: %s' %protocol
+
+    return buffer
+
+
+def reduce(x, op, root, buffer=None, vanilla=0, bypass=False):
+    """Reduce elements in x to buffer (of the same size as x)
+       at root applying operation op elementwise.
+
+       If bypass is True, all admin and error checks
+       get bypassed to reduce the latency.
+       The buffer must be specified explicitly in this case.
+    """
+
+    if bypass:
+        reduce_array(x, buffer, op, root)
+        return
+
+
+    import types
+    from mpiext import size
+    numproc = size()         # Needed to determine buffer size
+
+
+    # Input check
+    errmsg = 'Root id (%s) must be an integer.' % root
+    assert type(root) == types.IntType, errmsg
+
+    # Create metadata about object
+    protocol, typecode, size, shape = create_control_info(x)
+
+    # Reduce
+    if protocol == 'array':
+        if buffer is None:
+            buffer = zeros(size*numproc, typecode)
+
+            # Modify shape along axis=0 to match size
+            shape = list(shape)
+            shape[0] *= numproc
+            buffer = reshape(buffer, shape)
+
+
+        msg = 'Data array and buffer must have same type '
+        msg = 'in reduce. I got types "%s" and "%s"' % (x.dtype.char,
+                                                        buffer.dtype.char)
+        assert x.dtype.char == buffer.dtype.char, msg
+        reduce_array(x, buffer, op, root)
+
+
+    elif (protocol == 'vanilla' or protocol == 'string'):
+        raise 'Protocol: %s unsupported for reduce' % protocol
+    else:
+        raise 'Unknown protocol: %s' % protocol
+
+    return buffer
+
+def allreduce(x, op, buffer=None, vanilla=0, bypass=False):
+    """Allreduce elements in x to buffer (of the same size as x)
+       applying operation op elementwise.
+
+       If bypass is True, all admin and error checks
+       get bypassed to reduce the latency.
+       The buffer must be specified explicitly in this case.
+    """
+
+    if bypass:
+        allreduce_array(x, buffer, op)
+        return
+
+
+    import types
+    from mpiext import size
+    numproc = size()         # Needed to determine buffer size
+
+
+
+    # Create metadata about object
+    protocol, typecode, size, shape = create_control_info(x)
+
+    # Allreduce
+    if protocol == 'array':
+        if buffer is None:
+            buffer = zeros(size*numproc, typecode)
+
+            # Modify shape along axis=0 to match size
+            shape = list(shape)
+            shape[0] *= numproc
+            buffer = reshape(buffer, shape)
+
+
+        msg = 'Data array and buffer must have same type '
+        msg = 'in allreduce. I got types "%s" and "%s"' % (x.dtype.char,
+                                                        buffer.dtype.char)
+        assert x.dtype.char == buffer.dtype.char, msg
+        allreduce_array(x, buffer, op)
+
+
+    elif (protocol == 'vanilla' or protocol == 'string'):
+        raise 'Protocol: %s unsupported for allreduce' % protocol
+    else:
+        raise 'Unknown protocol: %s' % protocol
+
+    return buffer
+
+#
+# Functions related to MPI_Bsend().
+#
+
+def bsend(x, destination, use_buffer=False, vanilla=False,
+         tag=default_tag, bypass=False):
+
+    if bypass is True:
+        #print "bsend_array() bypass True"
+        bsend_array(x, destination, tag)
+        return
+
+    import types, string
+
+    # Input check.
+    errmsg = 'Destination id (%s) must be an integer.' % destination
+    assert type(destination) == types.IntType, errmsg
+
+    errmsg = 'Tag %d is reserved by pypar - please use another.' % control_tag
+    assert tag != control_tag, errmsg
+
+    # Create metadata about object to be sent
+    control_info, x = create_control_info(x, vanilla, return_object=True)
+    protocol = control_info[0]
+
+    # Possibly transmit control data
+    if use_buffer is False:
+       send_control_info(control_info, destination)
+
+    # Transmit payload data
+    if protocol == 'array':
+        bsend_array(x, destination, tag)
+        #print "bsend_array() bypass False"
+    elif protocol in ['string', 'vanilla']:
+        bsend_string(x, destination, tag)
+        #print "bsend_string()"
+    else:
+        raise 'Unknown protocol: %s' %protocol
+
+def push_for_alloc(x, vanilla=False, use_buffer=False, bypass=False):
+    if bypass is True:
+        return array_push_for_alloc_and_attach(x)
+    else:
+        # Create metadata about object and retrieve protocol.
+        control_info, x = create_control_info(x, vanilla, return_object=True)
+        protocol = control_info[0]
+
+		# According to protocol, use appropriate function.
+        if protocol == 'array':
+            return array_push_for_alloc_and_attach(x)
+        elif protocol in ['string', 'vanilla']:
+            return string_push_for_alloc_and_attach(x)
+        else:
+            raise 'Unknown protocol: %s' % protocol
+
+def alloc_and_attach():
+    mpi_alloc_and_attach()
+
+def detach_and_dealloc():
+    mpi_detach_and_dealloc()
+
+def alloc(nbytes=-1):
+    return mpi_alloc(nbytes)
+
+def dealloc():
+    mpi_dealloc()
+
+def attach():
+    mpi_attach()
+
+def detach():
+    mpi_detach()
+
+
+#---------------------------------------------------------
+# AUXILIARY FUNCTIONS
+#---------------------------------------------------------
+def balance(N, P, p):
+    """Compute p'th interval when N is distributed over P bins.
+
+    This function computes boundaries of sub intervals of [0:N] such
+    that they are almost equally sized with their sizes differening
+    by no more than 1.
+
+    As such, this function  is suitable for partitioning an interval equally
+    across P processors.
+
+    Inputs:
+       N: Upper bound of full interval.
+       P: Total number of processors
+       p: Local processor id
+
+
+    Outputs:
+       Nlo: Lower bound of p'th sub-interval
+       Nhi: Upper bound of p'th sub-interval
+
+
+    Example:
+       To partition the interval [0:29] among 4 processors:
+
+       Nlo, Nhi = pypar.balance(29, 4, p)
+
+       with p in [0,1,2,3]
+
+       and the subintervals are
+
+       p          Nlo      Nhi
+       -----------------------
+       0           0        8
+       1           8       15
+       2          15       22
+       3          22       29
+
+
+
+    Note that the interval bounds following the Python convention of
+    list slicing such that the last element of Nlo:Nhi is, in fact, Nhi-1
+    """
+
+    from math import floor
+
+    L = int(floor(float(N)/P))
+    K = N - P*L
+    if p < K:
+        Nlo = p*L + p
+        Nhi = Nlo + L + 1
+    else:
+        Nlo = p*L + K
+        Nhi = Nlo + L
+
+    return Nlo, Nhi
+
+
+# Obsolete functions
+from warnings import warn
+def Wtime():
+    msg = 'pypar.Wtime will become obsolete soon. Please use pypar.time'
+    warn(msg)
+    return time()
+
+def Get_processor_name():
+    msg = 'pypar.Get_processor_name will become obsolete soon.'
+    msg += 'Please use pypar.get_processor_name'
+    warn(msg)
+    return get_processor_name()
+
+
+#---------------------------------------------------------
+# INTERNAL FUNCTIONS
+#---------------------------------------------------------
+
+class Status:
+    """ MPI Status block returned by receive if
+        specified with parameter return_status=True
+    """
+
+    def __init__(self, status_tuple):
+        self.source = status_tuple[0]  # Id of sender
+        self.tag = status_tuple[1]     # Tag of received message
+        self.error = status_tuple[2]   # MPI Error code
+        self.length = status_tuple[3]  # Number of elements transmitted
+        self.size = status_tuple[4]    # Size of one element
+
+    def __repr__(self):
+        return 'Pypar Status Object:\n  source=%d\n  tag=%d\n  error=%d\n  length=%d\n  size=%d\n' %(self.source, self.tag, self.error, self.length, self.size)
+
+    def bytes(self):
+        """Number of bytes transmitted (excl control info)
+        """
+        return self.length * self.size
+
+
+
+def create_control_info(x, vanilla=0, return_object=False):
+    """Determine which protocol to use for communication:
+       (numpy) arrays, strings, or vanilla based x's type.
+
+       There are three protocols:
+       'array':   numpy arrays of type 'i', 'l', 'f', 'd', 'F' or 'D' can be
+                  communicated  with mpiext.send_array and mpiext.receive_array.
+       'string':  Text strings can be communicated with mpiext.send_string and
+                  mpiext.receive_string.
+       'vanilla': All other types can be communicated as string representations
+                  provided that the objects
+                  can be serialised using pickle (or cPickle).
+                  The latter mode is less efficient than the
+                  first two but it can handle general structures.
+
+       Rules:
+       If keyword argument vanilla == 1, vanilla is chosen regardless of
+       x's type.
+       Otherwise if x is a string, the string protocol is chosen
+       If x is an array, the 'array' protocol is chosen provided that x has one
+       of the admissible typecodes.
+
+       The optional argument return_object asks to return object as well.
+       This is useful in case it gets modified as in the case of general structures
+       using the vanilla protocol.
+    """
+
+    import types
+
+    # Default values
+    protocol = 'vanilla'
+    typecode = ' '
+    size = 0
+    shape = ()
+
+    # Determine protocol in case
+    if not vanilla:
+        if type(x) == types.StringType:
+            protocol = 'string'
+            typecode = 'c'
+            size = len(x)
+        elif type(x).__name__ == 'ndarray': # numpy isn't imported yet
+            try:
+                import numpy
+            except:
+                print "WARNING (pypar.py): numpy module could not be imported,",
+                print "reverting to vanilla mode"
+                protocol = 'vanilla'
+            else:
+                typecode = x.dtype.char
+                if typecode in ['i', 'l', 'f', 'd', 'F', 'D']:
+                    protocol = 'array'
+                    shape = x.shape
+                    size = product(shape)
+                else:
+                    print "WARNING (pypar.py): numpy object type %s is not supported."\
+                          %(x.dtype.char)
+                    print "Only types 'i', 'l', 'f', 'd', 'F', 'D' are supported,",
+                    print "Reverting to vanilla mode."
+                    protocol = 'vanilla'
+
+    # Pickle general structures using the vanilla protocol
+    if protocol == 'vanilla':
+        from cPickle import dumps
+        x = dumps(x, protocol=2)
+        size = len(x) # Let count be length of pickled object
+
+    # Return
+    if return_object:
+        return [protocol, typecode, size, shape], x
+    else:
+        return [protocol, typecode, size, shape]
+
+
+
+#----------------------------------------------
+
+def send_control_info(control_info, destination):
+    """Send control info to destination
+    """
+    import string
+
+    # Convert to strings
+    control_info = [str(c) for c in control_info]
+
+    control_msg = string.join(control_info,control_sep)
+    if len(control_msg) > control_data_max_size:
+        errmsg = 'Length of control_info exceeds specified maximium (%d)'\
+                 %control_data_max_size
+        errmsg += ' - Please increase it (in pypar.py)'
+        raise errmsg
+
+    send_string(control_msg, destination, control_tag)
+
+
+def receive_control_info(source, return_source=False):
+    """Receive control info from source
+
+    The optional argument (due to Jim Bosch) also returns the actual source node
+    which can be used to require that the data message come from the same node.
+    """
+
+    # FIXME (Ole): Perhaps we should include actual source in the control info?
+
+    import string
+
+    msg = ' '*control_data_max_size
+
+    stat = receive_string(msg, source, control_tag)
+    # No need to create status object here - it is reserved
+    # for payload communications only
+
+    msg = msg[:stat[3]] # Trim buffer to actual received length (needed?)
+
+    control_info = msg.split(control_sep)
+
+    assert len(control_info) == 4, 'len(control_info) = %d' %len(control_info)
+    control_info[2] = eval(control_info[2]) # Convert back to int
+    control_info[3] = eval(control_info[3]) # Convert back to tuple
+
+    if return_source:
+        return control_info, int(stat[0])
+    else:
+        return control_info
+
+
+#----------------------------------------------------------------------------
+# Initialise module
+#----------------------------------------------------------------------------
+
+
+# Take care of situation where module is part of package
+import sys, os, string, os.path
+dirname = os.path.dirname(string.replace(__name__,'.',os.sep)).strip()
+
+if not dirname:
+    dirname = '.'
+
+if dirname[-1] != os.sep:
+    dirname += os.sep
+
+
+
+# Import MPI extension
+#
+# Verify existence of mpiext.so.
+
+try:
+    import mpiext
+except:
+    errmsg = 'ERROR: C extension mpiext could not be imported.\n'
+    errmsg += 'Please compile mpiext.c e.g. by running\n'
+    errmsg += '  python compile_pypar_locally.py\n'
+    errmsg += 'in the pypar directory, or by using\n'
+    errmsg += '  python setup.py install\n'
+    #raise Exception, errmsg
+    error = 1
+    print errmsg
+else:
+
+    # Determine if MPI program is allowed to run sequentially on current platform
+    # Attempting to check this automatically may case some systems to hang.
+
+    if sys.platform in ['linux2', 'sunos5', 'win32', 'darwin']:
+        # Linux (LAM,MPICH) or Sun (MPICH)
+        error = 0  #Sequential execution of MPI is allowed
+    else:
+        # Platform: Alpha 'osf1V5'
+        cmdstring = '"import mpiext, sys; mpiext.init(sys.argv); mpiext.finalize()"'
+        #s = 'cd %s; python -c %s' %(dirname, cmdstring)
+        s = 'python -c %s >/dev/null 2>/dev/null' %cmdstring
+        error = os.system(s)
+
+        # The check is performed in a separate shell.
+        # Reason: The Alpha server, LAM/Linux or the Sun cannot recover from a
+        # try:
+        #   mpiext.init(sys.argv)
+
+        # However, on LAM/Linux, this test causes system to hang.
+        # Verified (OMN 12/12/2)
+        # If lamboot is started, the system, will hang when init is called
+        # again further down in this file.
+        # If lamboot is not loaded error will be nozero as it should.
+        # I don't know how to deal with this
+        #
+        #Comparisons of two strategies using LAM
+        #
+        # Strategy 1: Assume seq execution is OK (i.e. set error = 0)
+        # Strategy 2: Try to test if mpi can be initialised (in a separate shell)
+        #
+        #
+        # Strategy 1 (currently used)
+        #                    | Lam booted  | Lam not booted
+        #-----------------------------------------------------
+        #
+        # Sequential exec    |  OK         | Not OK
+        # Parallel exec      |  OK         | Not OK
+        #
+        #
+        # Strategy 2
+        #                    | Lam booted  | Lam not booted
+        #-----------------------------------------------------
+        #
+        # Sequential exec    |  Hangs      | Not OK
+        # Parallel exec      |  Hangs      | OK
+        #
+
+
+
+# Initialise MPI
+#
+# Attempt to initialise mpiext.so
+# If this fails, define a rudimentary interface suitable for
+# sequential execution.
+
+if error:
+    print "WARNING: MPI library could not be initialised - running sequentially"
+
+    # Define rudimentary functions to keep sequential programs happy
+
+    def size(): return 1
+    def rank(): return 0
+
+    def get_processor_name():
+        import os
+        try:
+            hostname = os.environ['HOST']
+        except:
+            try:
+                hostname = os.environ['HOSTNAME']
+            except:
+                hostname = 'Unknown'
+
+        return hostname
+
+    def abort():
+        import sys
+        sys.exit()
+
+    def finalize(): pass
+
+    def barrier(): pass
+
+    def time():
+        import time
+        return time.time()
+
+else:
+    from mpiext import size, rank, barrier, time,\
+         get_processor_name,\
+         init, initialized, finalize, abort,\
+         send_string, receive_string,\
+         send_array, receive_array, broadcast_string, broadcast_array,\
+         scatter_string, scatter_array,\
+         gather_string, gather_array,\
+         reduce_array,\
+         allreduce_array,\
+	 bsend_string, bsend_array, \
+	 mpi_alloc_and_attach, mpi_detach_and_dealloc, \
+	 mpi_alloc, mpi_dealloc, mpi_attach, mpi_detach, \
+	 string_push_for_alloc_and_attach, array_push_for_alloc_and_attach, \
+         MPI_ANY_TAG as any_tag, MPI_TAG_UB as max_tag,\
+         MPI_ANY_SOURCE as any_source,\
+         MAX, MIN, SUM, PROD, LAND, BAND,\
+         LOR, BOR, LXOR, BXOR
+
+    # Work around bug in OpenMPI (December 2009):
+    # https://bugs.launchpad.net/ubuntu/+source/petsc4py/+bug/232036
+    from ctypes import *
+    CDLL('libmpi.so', RTLD_GLOBAL)
+    # End work around
+
+    # Initialise MPI with cmd line (needed by MPICH/Linux)
+    init(sys.argv)
+
+    # Report
+    if rank() == 0:
+        print 'Pypar (version %s) initialised MPI OK with %d processors'\
+            % (__version__, size())
+
+
+
+
+
+
Index: trunk/anuga_core/source/pypar_dist/source/pypar_balancer.py
===================================================================
--- trunk/anuga_core/source/pypar_dist/source/pypar_balancer.py	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/source/pypar_balancer.py	(revision 8494)
@@ -0,0 +1,218 @@
+# coding: UTF-8
+"""
+Simple load balancing with pypar
+(based on demo3.py from pypar demo package)
+Felix Richter <felix.richter2@uni-rostock.de>
+"""
+import sys
+import time
+
+import numpy
+import pypar
+
+PYPAR_WORKTAG = 1
+PYPAR_DIETAG = 2
+
+def mprint(txt):
+    """
+    Print message txt
+    with indentation following the node's rank
+    """
+    import pypar
+    
+    pre = " " * 8 * pypar.rank()
+    if type(txt) != type('dummy'):
+        txt = txt.__str__()
+    pat = "-%d-"
+    print pre + (pat % pypar.rank()) + txt
+
+class PyparWork(object):
+    """Abstract base class for ant work to be balanced"""
+
+    def __init__(self):
+        pass
+    
+    def uplink(self, balancer, myid, numprocs, node):
+        self.balancer = balancer
+        self.pypar_id = myid
+        self.pypar_numprocs = numprocs
+        self.pypar_node = node
+    
+    def getNumWorkItems(self):
+        pass
+    
+    def handleWorkResult(self, result, status):
+        pass
+    
+    def calcWorkResult(self, worknum):
+        pass
+    
+    def masterBeforeWork(self):
+        """Master node calls this before sending out the work"""
+        pass
+
+    def slaveBeforeWork(self):
+        """Slave nodes call this before receiving work"""
+        pass
+
+    def masterAfterWork(self):
+        """Master node calls this after receiving the last work result"""
+        pass
+
+    def slaveAfterWork(self):
+        """Slave nodes call this after sending the last work result"""
+        pass
+    
+    def msgprint(self, txt):
+        pre = " " * 8 * self.pypar_id
+        if type(txt) != type('dummy'):
+            txt = txt.__str__()
+        pat = "-%d-"
+        print pre + (pat % self.pypar_id) + txt
+        
+
+class PyparBalancer(object):
+    """The Load Balancer Class
+    Initialize it with a PyparWork-derived class instance
+    which describes the actual work to do.
+    
+    debug == True - more status messages
+    """
+    
+    def __init__(self, work, debug = False):
+        self.numprocs = pypar.size()           # Number of processes as specified by mpirun
+        self.myid = pypar.rank()               # Id of of this process (myid in [0, numproc-1]) 
+        self.node = pypar.get_processor_name() # Host name on which current process is running
+        self.debug= debug
+        self.work = work
+
+        # Added by Ole Nielsen, 15 May 2008
+        if self.numprocs < 2:
+            msg = 'PyparBalancer must run on at least 2 processes'
+            msg += ' for the Master Slave paradigm to make sense.'
+            raise Exception, msg
+
+        
+        self.work.uplink(self, self.myid, self.numprocs, self.node)
+        
+        self.numworks = self.work.getNumWorkItems()
+        print "PyparBalancer initialised on proc %d of %d on node %s" %(self.myid, self.numprocs, self.node)
+
+    def master(self):
+        numcompleted = 0
+        #--- start slaves distributing the first work slot
+        for i in range(0, min(self.numprocs-1, self.numworks)): 
+            work = i
+            slave= i+1
+            pypar.send(work, destination=slave, tag=PYPAR_WORKTAG) 
+            print '[MASTER ]: sent first work "%s" to node %d' %(work, slave)
+    
+        # dispatch the remaining work slots on dynamic load-balancing policy
+        # the quicker to do the job, the more jobs it takes
+        for work in range(self.numprocs-1, self.numworks):
+            result, status = pypar.receive(source=pypar.any_source, tag=PYPAR_WORKTAG, return_status=True) 
+            print '[MASTER ]: received result from node %d' %(status.source, )
+            #print result
+            numcompleted += 1
+            pypar.send(work, destination=status.source, tag=PYPAR_WORKTAG)
+            if self.debug: print '[MASTER ]: sent work "%s" to node %d' %(work, status.source)
+            
+            self.work.handleWorkResult(result, status)
+        
+        # all works have been dispatched out
+        print '[MASTER ]: ToDo : %d' %self.numworks
+        print '[MASTER ]: Done : %d' %numcompleted
+        
+        # I've still to take into the remaining completions   
+        while (numcompleted < self.numworks): 
+            result, status = pypar.receive(source=pypar.any_source, tag=PYPAR_WORKTAG, return_status=True) 
+            print '[MASTER ]: received (final) result from node %d' % (status.source, )
+            print result
+            numcompleted += 1
+            print '[MASTER ]: %d completed' %numcompleted
+            
+            self.work.handleWorkResult(result, status)
+            
+        print '[MASTER ]: about to terminate slaves'
+    
+        # Tell slaves to stop working
+        for i in range(1, self.numprocs): 
+            pypar.send('#', destination=i, tag=PYPAR_DIETAG) 
+            if self.debug: print '[MASTER ]: sent DIETAG to node %d' %(i,)
+            
+    
+    def slave(self):
+        if self.debug: print '[SLAVE %d]: I am processor %d of %d on node %s' % (self.myid, self.myid, self.numprocs, self.node)
+        if self.debug: print '[SLAVE %d]: Entering work loop' % (self.myid,)
+        while True:
+            result, status = pypar.receive(source=0, tag=pypar.any_tag, return_status=True) 
+            print '[SLAVE %d]: received work with tag %d from node %d'\
+                      %(self.myid, status.tag, status.source)
+           
+            if (status.tag == PYPAR_DIETAG):
+                print '[SLAVE %d]: received termination from node %d' % (self.myid, 0)
+                return
+            else:
+                worknum = result
+                if self.debug: print '[SLAVE %d]: work number is %s' % (self.myid, worknum)
+                myresult = self.work.calcWorkResult(worknum)
+                pypar.send(myresult, destination=0)
+                if self.debug: print '[SLAVE %d]: sent result to node %d' % (self.myid, 0)
+
+    def run(self):
+        if self.myid == 0:
+            self.work.masterBeforeWork()
+            self.master()
+            self.work.masterAfterWork()
+        else:
+            self.work.slaveBeforeWork()
+            self.slave()
+            self.work.slaveAfterWork()
+        
+        pypar.finalize()
+        if self.myid != 0:
+            sys.exit()
+        # und schluss.
+
+class PyparDemoWork(PyparWork):
+    """Example PyparWork implementation"""
+    def __init__(self):
+        import numpy
+        self.worklist = numpy.arange(0.0,20.0)
+        self.resultlist = numpy.zeros_like(self.worklist)
+        
+    def getNumWorkItems(self):
+        return len(self.worklist)
+    
+    def calcWorkResult(self, worknum):
+        return [worknum, self.worklist[worknum] + 1]
+
+    def handleWorkResult(self, result, status):
+        self.resultlist[result[0]] = result[1]
+        
+    def masterBeforeWork(self):
+        print self.worklist
+
+    def slaveBeforeWork(self):
+        pass
+
+    def masterAfterWork(self):
+        print self.resultlist
+
+    def slaveAfterWork(self):
+        pass
+        
+if __name__ == "__main__":
+    print "-----------------------"
+    print "::: PyParBalancer TEST "
+    print "-----------------------"
+    
+    # create instance of work class
+    pyparwork = PyparDemoWork()
+
+    # create instance of balancer class,
+    # initialize with work class
+    balancer = PyparBalancer(pyparwork, True)
+    
+    # run it
+    balancer.run()
Index: trunk/anuga_core/source/pypar_dist/source/setup.py
===================================================================
--- trunk/anuga_core/source/pypar_dist/source/setup.py	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/source/setup.py	(revision 8494)
@@ -0,0 +1,200 @@
+# Examples of setup
+#
+# python setup.py install
+# python setup.py install --prefix=/opt/python-2.3
+# python setup.py install --home=~
+#
+# Some times you'll have to add a file called pypar.pth
+# containing the word pypar to site-packages
+#
+# See http://docs.python.org/dist/pure-pkg.html for more on distutils
+
+# FIXME: Now mpiext.c and pypar.py are assumed to be in this directory.
+# Maybe, we should put them in the default package directory, pypar.
+# The repository structure would then be
+#
+# pypar
+#     demos
+#     documentation
+#     source
+#          pypar
+
+from distutils.core import setup, Extension
+
+# FIXME (Ole): This works, but I don't know how to use it
+# Generate Python EGG if possible.
+#try:
+#   from setuptools import setup, Extension
+#except ImportError:
+#   pass
+
+import distutils.sysconfig
+import distutils.debug
+import os, sys
+import popen2 # FIXME: Replace with subprocess - http://docs.python.org/library/subprocess.html#replacing-older-functions-with-the-subprocess-module
+import string
+import tempfile
+import numpy
+import shlex
+from __metadata__ import __version__, __date__, __author__
+
+
+def setup_compiler():
+    distutils.sysconfig.get_config_vars()
+    config_vars = distutils.sysconfig._config_vars
+
+    if sys.platform == 'sunos5':
+        config_vars['LDSHARED'] = 'gcc -G'
+        config_vars['CCSHARED'] = ''
+
+
+def uniq_arr(arr):
+    """Remove repeated values from an array and return new array."""
+    ret = []
+    for i in arr:
+        if i not in ret:
+            ret.append(i)
+    return ret
+
+
+def _run_command(cmd):
+    out_file, in_file, err_file = popen2.popen3(cmd)
+    output = out_file.read() + err_file.read()
+    out_file.close()
+    in_file.close()
+    err_file.close()
+    # need this hack to get the exit status
+    print 'running ' + cmd
+    out_file = os.popen(cmd)
+    if out_file.close():
+        # close returns exit status of command.
+        return ''
+    else:
+        # no errors, out_file.close() returns None.
+        return output
+
+
+def _get_mpi_cmd():
+    """Returns the output of the command used to compile using
+    mpicc."""
+    # LAM/OPENMPI/MPICH2
+    output = _run_command('mpicc -show')
+    if output:
+        return output
+
+    # FIXME: If appears that MPICH actually needs this these hacks.
+
+    # MPICH
+    # works with MPICH version 1.2.1 (on Debian)
+    output = _run_command('mpicc -compile_info -link_info')
+    if output:
+        return output
+
+    # Old version of MPICH needs this hack.
+    tmp_base = tempfile.mktemp()
+    tmp_c = tmp_base + ".c"
+    tmp_o = tmp_base + ".o"
+    tmp_file = open(tmp_c, "w")
+    tmp_file.write('#include "mpi.h"\nint main(){return 0;}\n')
+    tmp_file.close()
+    output = _run_command("mpicc -show;"\
+                          "mpicc -echo -c %s -o %s"%(tmp_c, tmp_o))
+    os.remove(tmp_c)
+    if os.path.exists(tmp_o):
+        os.remove(tmp_o)
+    if output:
+        return output
+    else:
+        return ''
+
+
+def get_mpi_flags():
+    output = _get_mpi_cmd()
+    print output
+    if not output:
+        if sys.platform=='win32': # From Simon Frost
+            #this didn't work on my machine (Vladimir Lazunin on April 7, 2009)
+            #output = "gcc -L$MPICH_DIR\SDK.gcc\lib -lmpich -I$MPICH_DIR\SDK.gcc\include"
+
+            #"MPICH_DIR" must be set manually in environment variables
+            mpi_dir = os.getenv("MPICH_DIR")
+            if mpi_dir == None:
+            	print 'MPICH_DIR environment variable must be set'
+            	exit()
+
+            #for MPICH2
+            sdk_prefix = mpi_dir
+            lib_name = 'mpi'
+
+            #for MPICH1
+            if os.path.exists(sdk_prefix + '\\SDK'):
+                sdk_prefix += '\\SDK'
+                lib_name = 'mpich'
+            output = 'gcc -L"%(sdk_prefix)s\lib" -l"%(lib_name)s" -I"%(sdk_prefix)s\include"' % {'sdk_prefix' : sdk_prefix, 'lib_name' : lib_name}
+        else:
+            output = 'cc -L/usr/opt/mpi -lmpi -lelan'
+
+
+    # Now get the include, library dirs and the libs to link with.
+    #flags = string.split(output)
+    flags = shlex.split(output)
+    flags = uniq_arr(flags) # Remove repeated values.
+    inc_dirs = []
+    lib_dirs = []
+    libs = []
+    def_macros = []
+    undef_macros = []
+    for f in flags:
+        if f[:2] == '-I':
+            inc_dirs.append(f[2:])
+        elif f[:2] == '-L':
+            lib_dirs.append(f[2:])
+        elif f[:2] == '-l' and f[-1] != "'": # Patched by Michael McKerns July 2009
+            libs.append(f[2:])
+        elif f[:2] == '-U':
+            undef_macros.append(f[2:])
+        elif f[:2] == '-D':
+            tmp = string.split(f[2:], '=')
+            if len(tmp) == 1:
+                def_macros.append((tmp[0], None))
+            else:
+                def_macros.append(tuple(tmp))
+    return {'inc_dirs': inc_dirs, 'lib_dirs': lib_dirs, 'libs':libs,
+            'def_macros': def_macros, 'undef_macros': undef_macros}
+
+
+if __name__ == '__main__':
+    setup_compiler()
+
+    mpi_flags = get_mpi_flags()
+    mpi_flags['inc_dirs'].append(numpy.get_include())
+
+
+    # setting some extra compile flags for 64 bit architectures, utilizing
+    # distutils.sysconfig to check which compiler to use
+    if os.name == 'posix' and os.uname()[4] == 'x86_64':
+        #Extra flags for 64 bit architectures
+        extra_compile_args = ['-fPIC']
+    else:
+        extra_compile_args = None
+
+
+
+    setup(name='Pypar',
+          version=__version__,
+          description='Pypar - Parallel Python',
+          long_description='Pypar - Parallel Python, no-frills MPI interface',
+          author=__author__,
+          author_email='ole.moller.nielsen@gmail.com',
+          url='http://sourceforge.net/projects/pypar',
+          package_dir = {'pypar': ''}, # Use files in this dirctory
+          packages  = ['pypar'],
+          ext_modules = [Extension('pypar.mpiext',
+                                   ['mpiext.c'],
+                                   include_dirs=mpi_flags['inc_dirs'],
+                                   library_dirs=mpi_flags['lib_dirs'],
+                                   libraries=mpi_flags['libs'],
+                                   define_macros=mpi_flags['def_macros'],
+                                   undef_macros=mpi_flags['undef_macros'],
+                                   extra_compile_args=extra_compile_args)]
+         )
Index: trunk/anuga_core/source/pypar_dist/source/test_init.py
===================================================================
--- trunk/anuga_core/source/pypar_dist/source/test_init.py	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/source/test_init.py	(revision 8494)
@@ -0,0 +1,24 @@
+# Inital test of MPI module 'pypar' for Python
+# 
+# Run as 
+#   python test_init.py
+# or 
+#   mpirun -np 2 test_init.py
+#  (perhaps try number of processors more than 2)
+#
+# To verify bandwidth of your architecture please run pytiming (and ctiming) 
+#
+# OMN April 2002
+
+
+import pypar
+
+myid =    pypar.rank()
+numproc = pypar.size()
+node =    pypar.get_processor_name()
+
+print "I am processor %d of %d on node %s" %(myid, numproc, node)
+if myid == 0:
+    print 'Maximal tag is %d' %pypar.max_tag
+pypar.finalize()
+
Index: trunk/anuga_core/source/pypar_dist/source/test_pypar.py
===================================================================
--- trunk/anuga_core/source/pypar_dist/source/test_pypar.py	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/source/test_pypar.py	(revision 8494)
@@ -0,0 +1,1257 @@
+# Test of MPI module 'pypar' for Python
+# 
+# Run as 
+#   python testpypar.py
+# or 
+#   mpirun -np 2 testpypar.py
+# (perhaps try number of processors more than 2)
+#
+# To verify bandwidth of your architecture please run pytiming (and ctiming) 
+#
+# OMN, GPC FEB 2002
+
+
+try:
+  import numpy
+except:
+  raise 'Module numpy must be present to run pypar'
+
+
+#print "Importing pypar"
+import pypar
+methods = dir(pypar)
+assert 'abort' in methods
+assert 'finalize' in methods
+assert 'get_processor_name' in methods
+assert 'time' in methods
+assert 'rank' in methods
+assert 'receive' in methods
+assert 'send' in methods
+assert 'broadcast' in methods
+assert 'size' in methods
+
+#print "Module pypar imported OK"
+#pypar.Barrier()
+
+
+myid = pypar.rank()
+numproc = pypar.size()
+node = pypar.get_processor_name()
+
+print 'I am processor %d of %d on node "%s"' %(myid, numproc, node)
+pypar.barrier()
+
+
+if numproc > 1:
+  # Test simple buffered communication (arrays, strings and general)
+  #
+  N = 17 #Number of elements
+  
+  if myid == 0:
+    # Integer arrays
+    #
+    A = numpy.array(range(N)).astype('i')
+    B = numpy.zeros(N).astype('i')
+
+    pypar.send(A, 1, use_buffer=True)
+    B = pypar.receive(numproc-1, buffer=B)
+
+    assert numpy.allclose(A, B)
+    print 'Buffered communication of numeric integer arrays OK'
+
+
+    # Long integer arrays
+    #
+    A = numpy.array(range(N)).astype('l')
+    B = numpy.zeros(N).astype('l')    
+
+    pypar.send(A, 1, use_buffer=True)
+    B = pypar.receive(numproc-1, buffer=B)
+    
+    assert numpy.allclose(A, B)
+    print 'Buffered communication of numeric long integer arrays OK'
+
+    # Real arrays
+    #
+    A = numpy.array(range(N)).astype('f')
+    B = numpy.zeros(N).astype('f')    
+
+    pypar.send(A, 1, use_buffer=True)
+    B = pypar.receive(numproc-1, buffer=B)
+    
+    assert numpy.allclose(A, B)    
+    print 'Buffered communication of numeric real arrays OK'
+
+    # Complex arrays
+    #
+    A = numpy.array(range(N)).astype('D')
+    A += 3j
+    B = numpy.zeros(N).astype('D')
+
+    pypar.send(A, 1, use_buffer=True)
+    B = pypar.receive(numproc-1, buffer=B)
+
+    assert numpy.allclose(A, B)    
+    print 'Buffered communication of numeric complex arrays OK'
+
+    # Strings (< 256 characters)
+    #
+    A = "and now to something completely different !"
+    B = " "*len(A)
+
+    pypar.send(A, 1, use_buffer=True)
+    B, status = pypar.receive(numproc-1, buffer=B, return_status=True)
+
+
+    assert A == B
+    print 'Buffered communication of strings OK'
+    
+    # A more general structure
+    #
+    A = ['ABC', (1,2,3.14), {8: 'Monty'}, numpy.array([13.45, 1.2])]
+    B = ['   ', (0,0,0.0) , {0: '     '}, numpy.zeros(2).astype('f')]    
+
+    pypar.send(A, 1, use_buffer=True)
+    B, status = pypar.receive(numproc-1, buffer=B, return_status=True)
+
+    OK = True
+    for i, a in enumerate(A):
+      b = B[i]
+
+      if type(a).__name__ == 'ndarray':
+        if not numpy.allclose(a, b):
+          OK = False
+          break
+      elif a != b:
+        OK = False
+        break
+
+    if OK is True:
+      print 'Buffered communication of general structures OK' 
+    else:
+      raise Exception    
+
+    
+  else:  
+    # Integers
+    #
+    X = numpy.zeros(N).astype('i')    
+
+    X = pypar.receive(myid-1, buffer=X)
+    pypar.send(X, (myid+1)%numproc, use_buffer=True)
+  
+    # Long integers
+    #
+    X = numpy.zeros(N).astype('l')
+    X = pypar.receive(myid-1, buffer=X)
+    pypar.send(X, (myid+1)%numproc, use_buffer=True)
+
+    # Floats
+    #
+    X = numpy.zeros(N).astype('f')
+    X = pypar.receive(myid-1, buffer=X)
+    pypar.send(X, (myid+1)%numproc, use_buffer=True)
+
+    # Complex
+    #
+    X = numpy.zeros(N).astype('D')
+    X = pypar.receive(myid-1, buffer=X)
+    pypar.send(X, (myid+1)%numproc, use_buffer=True)
+
+    # Strings
+    #
+    X = " "*256
+    X = pypar.receive(myid-1, buffer=X)
+    pypar.send(X.strip(), (myid+1)%numproc, use_buffer=True)
+
+    # General
+    #
+    X = ['   ', (0,0,0.0), {0: '     '}, numpy.zeros(2).astype('f')]
+    X = pypar.receive(myid-1, buffer=X)
+    pypar.send(X, (myid+1)%numproc, use_buffer=True)
+    
+
+
+
+
+  #Test (buffered communication of) multi dimensional arrays
+
+  M = 13  #Number of elements in dim 1
+  N = 17  #Number of elements in higher dims
+  
+  if myid == 0:
+    # 2D real arrays
+    #
+    A = numpy.array(range(M*N)).astype('f')
+    B = numpy.zeros(M*N).astype('f')
+
+    A = numpy.reshape(A, (M,N))
+    B = numpy.reshape(B, (M,N))    
+    
+    pypar.send(A, 1, use_buffer=True)
+    B = pypar.receive(numproc-1, buffer=B)
+    
+    assert numpy.allclose(A, B)    
+    print 'Buffered communication of 2D real arrays OK'
+
+    # 2D complex arrays
+    #
+    A = numpy.array(range(M*N)).astype('D')
+    B = numpy.zeros(M*N).astype('D')
+
+    A = numpy.reshape(A, (M,N))
+    B = numpy.reshape(B, (M,N))    
+
+    pypar.send(A, 1, use_buffer=True)
+    B = pypar.receive(numproc-1, buffer=B)
+    
+    assert numpy.allclose(A, B)    
+    print 'Buffered communication of 2D complex arrays OK'
+
+    # 3D real arrays
+    #
+    A = numpy.array(range(M*N*N)).astype('f')
+    B = numpy.zeros(M*N*N).astype('f')
+
+    A = numpy.reshape(A, (M,N,N))
+    B = numpy.reshape(B, (M,N,N))    
+    
+
+    pypar.send(A, 1, use_buffer=True)
+    B = pypar.receive(numproc-1, buffer=B)
+    
+    assert numpy.allclose(A, B)    
+    print 'Buffered communication of 3D real real arrays OK'
+
+    # 4D real arrays
+    #
+    A = numpy.array(range(M*N*N*M)).astype('f')
+    B = numpy.zeros(M*N*N*M).astype('f')
+
+    A = numpy.reshape(A, (M,N,N,M))
+    B = numpy.reshape(B, (M,N,N,M))    
+
+    pypar.send(A, 1, use_buffer=True)
+    B = pypar.receive(numproc-1, buffer=B)
+    
+    assert numpy.allclose(A, B)    
+    print 'Buffered communication of 4D real real arrays OK'
+
+    # 5D real arrays
+    #
+    A = numpy.array(range(M*N*2*N*M)).astype('f')
+    B = numpy.zeros(M*N*2*N*M).astype('f')
+
+    A = numpy.reshape(A, (M,N,2,N,M))
+    B = numpy.reshape(B, (M,N,2,N,M))    
+    
+    pypar.send(A, 1, use_buffer=True)
+    B = pypar.receive(numproc-1, buffer=B)
+    
+    assert numpy.allclose(A, B)    
+    print 'Buffered communication of 5D real real arrays OK'
+
+    # 5D double arrays
+    #
+    A = numpy.array(range(M*N*2*N*M)).astype('d')
+    B = numpy.zeros(M*N*2*N*M).astype('d')
+
+    A = numpy.reshape(A, (M,N,2,N,M))
+    B = numpy.reshape(B, (M,N,2,N,M))    
+    
+    pypar.send(A, 1, use_buffer=True)
+    B = pypar.receive(numproc-1, buffer=B)
+    
+    assert numpy.allclose(A, B)    
+    print 'Buffered communication of 5D double arrays OK'
+
+    # 5D complex arrays
+    #
+    A = numpy.array(range(M*N*2*N*M)).astype('D')
+    B = numpy.zeros(M*N*2*N*M).astype('D')
+
+    A = numpy.reshape(A, (M,N,2,N,M))
+    B = numpy.reshape(B, (M,N,2,N,M))    
+    
+    pypar.send(A, 1, use_buffer=True)
+    B = pypar.receive(numproc-1, buffer=B)
+    
+    assert numpy.allclose(A, B)    
+    print 'Buffered communication of 5D complex arrays OK'
+  else:  
+    # 2D real arrays
+    #
+    X = numpy.zeros(M*N).astype('f')
+    X = numpy.reshape(X, (M,N))
+    
+    X = pypar.receive(myid-1, buffer=X)
+    pypar.send(X, (myid+1)%numproc, use_buffer=True)
+  
+    # 2D complex arrays
+    #
+    X = numpy.zeros(M*N).astype('D')
+    X = numpy.reshape(X, (M,N))
+
+    X = pypar.receive(myid-1, buffer=X)
+    pypar.send(X, (myid+1)%numproc, use_buffer=True)
+  
+    # 3D real arrays
+    #
+    X = numpy.zeros(M*N*N).astype('f')
+    X = numpy.reshape(X, (M,N,N))
+    
+    X = pypar.receive(myid-1, buffer=X)
+    pypar.send(X, (myid+1)%numproc, use_buffer=True)
+
+    # 4D real arrays
+    #
+    X = numpy.zeros(M*N*N*M).astype('f')
+    X = numpy.reshape(X, (M,N,N,M))
+    
+    X = pypar.receive(myid-1, buffer=X)
+    pypar.send(X, (myid+1)%numproc, use_buffer=True)
+
+    # 5D real arrays
+    #
+    X = numpy.zeros(M*N*2*N*M).astype('f')
+    X = numpy.reshape(X, (M,N,2,N,M))
+    
+    X = pypar.receive(myid-1, buffer=X)
+    pypar.send(X, (myid+1)%numproc, use_buffer=True)
+
+    # 5D double arrays
+    #
+    X = numpy.zeros(M*N*2*N*M).astype('d')
+    X = numpy.reshape(X, (M,N,2,N,M))
+    
+    X = pypar.receive(myid-1, buffer=X)
+    pypar.send(X, (myid+1)%numproc, use_buffer=True)
+
+    # 5D complex arrays
+    #
+    X = numpy.zeros(M*N*2*N*M).astype('D')
+    X = numpy.reshape(X, (M,N,2,N,M))
+    
+    X = pypar.receive(myid-1, buffer=X)
+    pypar.send(X, (myid+1)%numproc, use_buffer=True)
+
+  # Test easy communication  - without buffers (arrays, strings and general)
+  #
+  N = 17 #Number of elements
+  
+  if myid == 0:
+    # Integer arrays
+    #
+    A = numpy.array(range(N))
+
+    pypar.send(A,1)
+    B = pypar.receive(numproc-1)
+    
+
+    assert numpy.allclose(A, B)
+    print "Simplified communication of numeric integer arrays OK"
+
+    # Long integer arrays
+    #
+    A = numpy.array(range(N)).astype('l')
+    pypar.send(A,1)
+    B=pypar.receive(numproc-1)
+    
+    assert numpy.allclose(A, B)    
+    print "Simplified communication of long integer real arrays OK"
+
+    # Real arrays
+    #
+    A = numpy.array(range(N)).astype('f')
+    pypar.send(A,1)
+    B=pypar.receive(numproc-1)
+    
+    assert numpy.allclose(A, B)    
+    print "Simplified communication of numeric real arrays OK"
+
+    # Complex arrays
+    #
+    A = numpy.array(range(N)).astype('D')
+    A += 3j
+    pypar.send(A,1)
+    B=pypar.receive(numproc-1)
+
+    assert numpy.allclose(A, B)    
+    print "Simplified communication of numeric complex arrays OK"
+
+
+    # Strings
+    #
+    A = "and now to something completely different !"
+    pypar.send(A,1)
+    B=pypar.receive(numproc-1)
+    
+    assert A == B
+    print "Simplified communication of strings OK"
+    
+    # A more general structure
+    #
+    A = ['ABC', (1,2,3.14), {8: 'Monty'}, numpy.array([13.45, 1.2])]
+    pypar.send(A,1)
+    B = pypar.receive(numproc-1)
+
+    OK = True
+    for i, a in enumerate(A):
+      b = B[i]
+
+      if type(a).__name__ == 'ndarray':
+        if not numpy.allclose(a, b):
+          OK = False
+          break
+      elif a != b:
+        OK = False
+        break
+
+    if OK is True:
+      print 'Simplified communication of general structures OK' 
+    else:
+      raise Exception    
+    
+  else:  
+    # Integers
+    #
+    X=pypar.receive(myid-1)  
+    pypar.send(X, (myid+1)%numproc)
+  
+    # Long integers
+    #
+    X=pypar.receive(myid-1)  
+    pypar.send(X, (myid+1)%numproc)    
+
+    # Floats
+    #
+    X=pypar.receive(myid-1)  
+    pypar.send(X, (myid+1)%numproc)    
+
+    # Complex
+    #
+    X=pypar.receive(myid-1)  
+    pypar.send(X, (myid+1)%numproc)    
+
+    # Strings
+    #
+    X=pypar.receive(myid-1)  
+    pypar.send(X, (myid+1)%numproc)    
+
+    # General
+    #
+    X = pypar.receive(myid-1)  
+    pypar.send(X, (myid+1)%numproc)    
+
+
+
+
+    
+  #Test (easy communication of) multi dimensional arrays
+
+  M = 13  #Number of elements in dim 1
+  N = 17  #Number of elements in higher dims
+  
+  if myid == 0:
+    # 2D real arrays
+    #
+    A = numpy.array(range(M*N)).astype('f')
+
+    A = numpy.reshape(A, (M,N))
+    
+    pypar.send(A,1)
+    B = pypar.receive(numproc-1)
+    
+    assert numpy.allclose(A, B)    
+    print "Simplified communication of 2D real arrays OK"
+
+    # 3D real arrays
+    #
+    A = numpy.array(range(M*N*N)).astype('f')
+    A = numpy.reshape(A, (M,N,N))
+    
+    pypar.send(A,1)
+    B=pypar.receive(numproc-1)
+    
+    assert numpy.allclose(A, B)    
+    print "Simplified communication of 3D real arrays OK"
+
+    # 4D real arrays
+    #
+    A = numpy.array(range(M*N*N*M)).astype('f')
+    A = numpy.reshape(A, (M,N,N,M))
+    
+    pypar.send(A,1)
+    B=pypar.receive(numproc-1)
+    
+    assert numpy.allclose(A, B)    
+    print "Simplified communication of 4D real arrays OK"
+
+    # 5D real arrays
+    #
+    A = numpy.array(range(M*N*2*N*M)).astype('f')
+    A = numpy.reshape(A, (M,N,2,N,M))
+    
+    pypar.send(A,1)
+    B=pypar.receive(numproc-1)
+    
+    assert numpy.allclose(A, B)    
+    print "Simplified communication of 5D real arrays OK"
+
+    # 5D double arrays
+    #
+    A = numpy.array(range(M*N*2*N*M)).astype('d')
+    A = numpy.reshape(A, (M,N,2,N,M))
+    
+    pypar.send(A,1)
+    B=pypar.receive(numproc-1)
+    
+    assert numpy.allclose(A, B)    
+    print "Simplified communication of 5D double arrays OK"
+
+    # 5D complex arrays
+    #
+    A = numpy.array(range(M*N*2*N*M)).astype('D')
+    A = numpy.reshape(A, (M,N,2,N,M))
+    
+    pypar.send(A,1)
+    B=pypar.receive(numproc-1)
+    
+    assert numpy.allclose(A, B)    
+    print "Simplified communication of 5D complex real arrays OK"
+
+  else:  
+    # 2D real arrays
+    #
+    
+    X = pypar.receive(myid-1)  
+    pypar.send(X, (myid+1)%numproc)
+  
+    # 3D real arrays
+    #
+    X=pypar.receive(myid-1)  
+    pypar.send(X, (myid+1)%numproc)
+
+    # 4D real arrays
+    #
+    X = pypar.receive(myid-1)  
+    pypar.send(X, (myid+1)%numproc)
+
+    # 5D real arrays
+    #
+    X=pypar.receive(myid-1)  
+    pypar.send(X, (myid+1)%numproc)
+
+    # 5D double arrays
+    #
+    X=pypar.receive(myid-1)  
+    pypar.send(X, (myid+1)%numproc)
+
+    # 5D complex arrays
+    #
+    X=pypar.receive(myid-1)  
+    pypar.send(X, (myid+1)%numproc)
+
+  # Test broadcast  - with buffers (arrays, strings and general)
+  #
+      
+  testString = ('test' + str(myid)).ljust(10)  #Buffers must have the same length on all procs!
+  pypar.broadcast(testString, 0)
+  assert testString.strip() == 'test0'
+  
+  testString = ('test' + str(myid)).ljust(10)  #Buffers must have the same length on all procs!
+  pypar.broadcast(testString, numproc-1)
+  assert testString.strip() == 'test' + str(numproc-1)
+  
+  if myid == 0:
+    print "Broadcast communication of strings OK"
+
+  ####################################################  
+  N = 17 #Number of elements
+  testArray = myid * numpy.array(range(N))
+  pypar.broadcast(testArray, 1)
+  assert numpy.allclose(testArray, 1 * testArray)
+  
+  if myid == 0:    
+    print "Broadcast communication of numeric integer array OK"
+
+
+  testArray = myid * numpy.array(range(N)).astype('f')
+  pypar.broadcast(testArray, 1)
+  assert numpy.allclose(testArray, 1 * testArray)
+  if myid == 0:
+    print "Broadcast communication of numeric real array OK"
+
+
+  M = 13
+  testArray = myid * numpy.array(range(M*N)).astype('f')
+  testArray = numpy.reshape(testArray, (M,N))  
+  pypar.broadcast(testArray, 1)
+  assert numpy.allclose(testArray, 1 * testArray)
+  if myid == 0:
+    print "Broadcast communication of 2D numeric real array OK"
+
+  testArray = myid * numpy.array(range(M*2*N)).astype('f')
+  testArray = numpy.reshape(testArray, (M,2,N))  
+  pypar.broadcast(testArray, 1)
+  assert numpy.allclose(testArray, 1 * testArray)
+  if myid == 0:
+    print "Broadcast communication of 3D numeric real array OK"
+
+  testArray = myid * numpy.array(range(M*2*N)).astype('D')
+  testArray = numpy.reshape(testArray, (M,2,N))  
+  pypar.broadcast(testArray, 1)
+  assert numpy.allclose(testArray, 1 * testArray)
+  if myid == 0:
+    print "Broadcast communication of 3D numeric complex array OK"
+
+    
+  A_x = ['ABC', myid, (1,2,3), {8: 'Monty'}, numpy.array([13.45, 1.2])]
+  A_1 = ['ABC',    1, (1,2,3), {8: 'Monty'}, numpy.array([13.45, 1.2])]
+  B = pypar.broadcast(A_x, 1)
+
+  OK = True
+  for i, a in enumerate(A_1):
+    b = B[i]
+
+    if type(a).__name__ == 'ndarray':
+      if not numpy.allclose(a, b):
+        OK = False
+        break
+    elif a != b:
+      OK = False
+      break
+
+  if OK is False:
+    raise Exception    
+    
+  if myid == 0:
+    print "Broadcast communication of general structures OK"
+  
+
+
+
+  # Test scatter  - with/without buffers (arrays, strings)
+  #
+  N = 16 # Number of elements
+
+  NP = N/numproc
+  
+  testString = 'ABCDEFGHIJKLMNOP'  #Length = 16
+  X = ' '*NP
+
+  pypar.scatter(testString, 0, buffer=X) # With buffer
+  Y = pypar.scatter(testString, 0) # With buffer automatically created
+
+  assert X==Y, 'X=%s, Y=%s' %(X,Y)
+  assert Y == testString[myid*NP:(myid+1)*NP]
+  assert X == testString[myid*NP:(myid+1)*NP]
+
+  if myid == 0:    
+    print "Scatter communication of strings OK"
+    
+
+  #Scatter Arrays
+  testArray = numpy.array(range(N)).astype('i')
+  X = numpy.zeros(NP).astype('i')
+
+  pypar.scatter(testArray, 0, buffer=X) # With buffer
+  Y = pypar.scatter(testArray, 0) # With buffer automatically created
+
+  assert numpy.allclose(X, Y)  
+  assert numpy.allclose(X, testArray[myid*NP:(myid+1)*NP])
+  assert numpy.allclose(Y, testArray[myid*NP:(myid+1)*NP])   
+
+  if myid == 0:
+    print "Scatter communication of numeric integer array OK"
+
+
+  testArray = numpy.array(range(N)).astype('f')
+  X = numpy.zeros(NP).astype('f')
+
+  pypar.scatter(testArray, 0, buffer=X) # With buffer
+  Y = pypar.scatter(testArray, 0) # With buffer automatically created
+
+  assert numpy.allclose(X, Y)  
+  assert numpy.allclose(X, testArray[myid*NP:(myid+1)*NP])
+  assert numpy.allclose(Y, testArray[myid*NP:(myid+1)*NP])   
+
+  if myid == 0:
+    print "Scatter communication of numeric real arrays OK"
+  #else:
+  #  print X, testArray, Y
+  #  assert numpy.allclose(X, Y)
+
+  testArray = numpy.array(range(N)).astype('D')
+  X = numpy.zeros(NP).astype('D')
+
+  pypar.scatter(testArray, 0, buffer=X) # With buffer
+  Y = pypar.scatter(testArray, 0) # With buffer automatically created
+
+  assert numpy.allclose(X, Y)  
+  assert numpy.allclose(X, testArray[myid*NP:(myid+1)*NP])
+  assert numpy.allclose(Y, testArray[myid*NP:(myid+1)*NP])   
+
+  if myid == 0:
+    print "Scatter communication of numeric complex array OK"
+
+
+###################################################
+#FIXME: 2D scatter doesn't work yet    
+#   M = 16
+#   N = 13
+#   MP = M/numproc
+  
+#   testArray = numpy.array(range(M*N)).astype('D')
+#   testArray = numpy.reshape(testArray, (M,N))    
+#   X = numpy.zeros(MP*N).astype('D')
+#   X = numpy.reshape(X, (MP,N))
+  
+#   raw_scatter(testArray, X, 0)
+#   Y = pypar.scatter(testArray, 0)
+  
+#   assert numpy.allclose(X, Y)  
+#   assert numpy.allclose(X, testArray[myid*MP:(myid+1)*MP,:])
+#   assert numpy.allclose(Y, testArray[myid*MP:(myid+1)*MP,:])   
+
+#   if myid == 0:
+#     print "Scatter communication of 2D numeric complex array OK"
+
+
+
+  # Test gather  - with/without buffers (arrays, strings)
+  #
+  N = 17 # Number of elements
+      
+  testString = 'AB'
+  X = '_'*(len(testString)*numproc) # Blanks caused errors when numproc >= 6 
+
+  pypar.gather(testString, 0, buffer=X) # Using buffer  
+  Y = pypar.gather(testString, 0) # Create buffer automatically
+  
+  if myid == 0:
+    assert X == 'AB' * numproc
+    assert Y == 'AB' * numproc
+    print "Gather communication of strings OK"
+  
+
+  testArray = numpy.array(range(N)).astype('i')
+  X = numpy.zeros(N*numproc).astype('i')
+
+  pypar.gather(testArray, 0, buffer=X) # Using buffer  
+  Y = pypar.gather(testArray, 0) # Create buffer automatically
+
+  if myid == 0:
+    for i in range(numproc):       
+      assert numpy.allclose(testArray, X[(i * N): ((i+1)*N)])
+
+    assert numpy.allclose(X, Y)
+    print "Gather communication of numeric integer array OK"
+    
+    
+  testArray = numpy.array(range(N)).astype('f')
+  X = numpy.zeros(N*numproc).astype('f')
+
+  pypar.gather(testArray, 0, buffer=X) # Using buffer  
+  Y = pypar.gather(testArray, 0) # Create buffer automatically
+
+  if myid == 0:
+    for i in range(numproc):       
+      assert numpy.allclose(testArray, X[(i * N): ((i+1)*N)])
+    assert numpy.allclose(X, Y)      
+    print "Gather communication of numeric real array OK"
+    
+  
+  testArray = numpy.array(range(N)).astype('D')
+  X = numpy.zeros(N*numproc).astype('D')
+
+  pypar.gather(testArray, 0, buffer=X) # Using buffer  
+  Y = pypar.gather(testArray, 0) # Create buffer automatically
+
+  if myid == 0:
+    for i in range(numproc):       
+      assert numpy.allclose(testArray, X[(i * N): ((i+1)*N)])
+    assert numpy.allclose(X, Y)      
+    print "Gather communication of numeric complex arrays OK"
+
+  M = 13  
+  testArray = numpy.array(range(M*N)).astype('D')
+  testArray = numpy.reshape(testArray, (M,N))
+  X = numpy.zeros(M*N*numproc).astype('D')
+  X = numpy.reshape(X, (M*numproc,N))
+  
+  pypar.gather(testArray, 0, buffer=X) # Using buffer  
+  Y = pypar.gather(testArray, 0) # Create buffer automatically
+
+  if myid == 0:
+    for i in range(numproc):       
+      assert numpy.allclose(testArray, X[(i * M): ((i+1)*M), :])
+    assert numpy.allclose(X, Y)      
+    print "Gather communication of 2D numeric complex arrays OK"
+  
+  ########################################################
+  # Test reduce
+  #######################################################
+  N = 17 #Number of elements
+  
+  # Create one (different) array on each processor
+  #    
+  testArray = numpy.array(range(N), 'i') * (myid+1)
+  X = numpy.zeros(N, 'i') # Buffer for results
+
+  pypar.reduce(testArray, pypar.SUM, 0, buffer=X)
+
+  if myid == 0:
+    Y = numpy.zeros(N, 'i')
+    for i in range(numproc):
+      Y = Y + numpy.array(range(N), 'i')*(i+1)    
+    assert numpy.allclose(X, Y)
+    print 'Buffered reduce using pypar.SUM OK'
+        
+  pypar.reduce(testArray, pypar.MAX, 0, buffer=X)
+  if myid == 0:
+    Y = numpy.array(range(N))*numproc
+    assert numpy.allclose(X, Y)
+    print 'Buffered reduce using pypar.MAX OK'
+
+  pypar.reduce(testArray, pypar.MIN, 0, buffer=X)
+  if myid == 0:
+    Y = numpy.array(range(N))
+    assert numpy.allclose(X, Y)
+    print 'Buffered reduce using pypar.MIN OK'
+    
+  if numproc <= 20:
+    testArray_float = testArray.astype('f')  #Make room for big results
+    X_float = X.astype('f')
+
+    pypar.reduce(testArray_float, pypar.PROD, 0, buffer=X_float)
+    if myid == 0:
+      Y = numpy.ones(N).astype('f')    
+      for i in range(numproc):
+        Y = Y*numpy.array(range(N))*(i+1)    
+      #print X_float
+      #print Y  
+      assert numpy.allclose(X_float, Y)
+      print 'Buffered reduce using pypar.PROD OK'
+  else:
+    if myid == 0:
+      print "Skipping product-reduce - try again with numproc < 20"    
+
+  pypar.reduce(testArray, pypar.LAND, 0, buffer=X)
+  if myid == 0:  
+    Y = numpy.ones(N).astype('i')    
+    for i in range(numproc):
+      Y = numpy.logical_and(Y, numpy.array(range(N)).astype('i')*(i+1))  
+    assert numpy.allclose(X, Y)
+    print 'Buffered reduce using pypar.LAND OK'
+    
+  pypar.reduce(testArray, pypar.BAND, 0, buffer=X)
+  if myid == 0:
+    Y = numpy.ones(N).astype('i')*255  #Neutral element for &   
+    for i in range(numproc):
+      Y = numpy.bitwise_and(Y, numpy.array(range(N))*(i+1))
+    assert numpy.allclose(X, Y)
+    print 'Buffered reduce using pypar.BAND OK'    
+
+  pypar.reduce(testArray, pypar.LOR, 0, buffer=X)
+  if myid == 0:  
+    Y = numpy.zeros(N).astype('i')    
+    for i in range(numproc):
+      Y = numpy.logical_or(Y, numpy.array(range(N)).astype('i')*(i+1))  
+    assert numpy.allclose(X, Y)
+    print 'Buffered reduce using pypar.LOR OK'
+  
+  pypar.reduce(testArray, pypar.BOR, 0, buffer=X)
+  if myid == 0:
+    Y = numpy.zeros(N).astype('i')   #Neutral element for |   
+    for i in range(numproc):
+      Y = numpy.bitwise_or(Y, numpy.array(range(N)).astype('i')*(i+1))
+    assert numpy.allclose(X, Y)
+    print 'Buffered reduce using pypar.BOR OK'    
+
+  pypar.reduce(testArray, pypar.LXOR, 0, buffer=X)
+  if myid == 0:  
+    Y = numpy.zeros(N).astype('i')    
+    for i in range(numproc):
+      Y = numpy.logical_xor(Y, numpy.array(range(N)).astype('i')*(i+1))  
+    assert numpy.allclose(X, Y)
+    print 'Buffered reduce using pypar.LXOR OK'
+
+  pypar.reduce(testArray, pypar.BXOR, 0, buffer=X)
+  if myid == 0:
+    Y = numpy.zeros(N).astype('i')   #Neutral element for xor ?   
+    for i in range(numproc):
+      Y = numpy.bitwise_xor(Y, numpy.array(range(N)).astype('i')*(i+1))
+    assert numpy.allclose(X, Y)
+    print 'Buffered reduce using pypar.BXOR OK'    
+
+
+
+
+
+  # FIXME: NOT YET SUPPORTED: pypar.MAXLOC, pypar.MINLOC, pypar.REPLACE
+
+  ########################################################
+  # Test allreduce
+  #######################################################
+  N = 17 #Number of elements
+
+  # Create one (different) array on each processor
+  #
+  testArray = numpy.array(range(N), 'i') * (myid+1)
+  X = numpy.zeros(N, 'i') # Buffer for results
+
+  pypar.allreduce(testArray, pypar.SUM, buffer=X)
+
+  if myid == 0:
+    Y = numpy.zeros(N, 'i')
+    for i in range(numproc):
+      Y = Y + numpy.array(range(N), 'i')*(i+1)
+    assert numpy.allclose(X, Y)
+    print 'Buffered allreduce using pypar.SUM OK'
+
+  pypar.allreduce(testArray, pypar.MAX, buffer=X)
+  if myid == 0:
+    Y = numpy.array(range(N))*numproc
+    assert numpy.allclose(X, Y)
+    print 'Buffered allreduce using pypar.MAX OK'
+
+  pypar.allreduce(testArray, pypar.MIN, buffer=X)
+  if myid == 0:
+    Y = numpy.array(range(N))
+    assert numpy.allclose(X, Y)
+    print 'Buffered allreduce using pypar.MIN OK'
+
+  if numproc <= 20:
+    testArray_float = testArray.astype('f')  #Make room for big results
+    X_float = X.astype('f')
+
+    pypar.allreduce(testArray_float, pypar.PROD, buffer=X_float)
+    if myid == 0:
+      Y = numpy.ones(N).astype('f')
+      for i in range(numproc):
+        Y = Y*numpy.array(range(N))*(i+1)
+      #print X_float
+      #print Y
+      assert numpy.allclose(X_float, Y)
+      print 'Buffered allreduce using pypar.PROD OK'
+  else:
+    if myid == 0:
+      print "Skipping product-allreduce - try again with numproc < 20"
+
+  pypar.allreduce(testArray, pypar.LAND, buffer=X)
+  if myid == 0:
+    Y = numpy.ones(N).astype('i')
+    for i in range(numproc):
+      Y = numpy.logical_and(Y, numpy.array(range(N)).astype('i')*(i+1))
+    assert numpy.allclose(X, Y)
+    print 'Buffered allreduce using pypar.LAND OK'
+
+  pypar.allreduce(testArray, pypar.BAND, buffer=X)
+  if myid == 0:
+    Y = numpy.ones(N).astype('i')*255  #Neutral element for &
+    for i in range(numproc):
+      Y = numpy.bitwise_and(Y, numpy.array(range(N))*(i+1))
+    assert numpy.allclose(X, Y)
+    print 'Buffered allreduce using pypar.BAND OK'
+
+  pypar.allreduce(testArray, pypar.LOR, buffer=X)
+  if myid == 0:
+    Y = numpy.zeros(N).astype('i')
+    for i in range(numproc):
+      Y = numpy.logical_or(Y, numpy.array(range(N)).astype('i')*(i+1))
+    assert numpy.allclose(X, Y)
+    print 'Buffered allreduce using pypar.LOR OK'
+
+  pypar.allreduce(testArray, pypar.BOR, buffer=X)
+  if myid == 0:
+    Y = numpy.zeros(N).astype('i')   #Neutral element for |
+    for i in range(numproc):
+      Y = numpy.bitwise_or(Y, numpy.array(range(N)).astype('i')*(i+1))
+    assert numpy.allclose(X, Y)
+    print 'Buffered allreduce using pypar.BOR OK'
+
+  pypar.allreduce(testArray, pypar.LXOR, buffer=X)
+  if myid == 0:
+    Y = numpy.zeros(N).astype('i')
+    for i in range(numproc):
+      Y = numpy.logical_xor(Y, numpy.array(range(N)).astype('i')*(i+1))
+    assert numpy.allclose(X, Y)
+    print 'Buffered allreduce using pypar.LXOR OK'
+
+  pypar.allreduce(testArray, pypar.BXOR, buffer=X)
+  if myid == 0:
+    Y = numpy.zeros(N).astype('i')   #Neutral element for xor ?
+    for i in range(numproc):
+      Y = numpy.bitwise_xor(Y, numpy.array(range(N)).astype('i')*(i+1))
+    assert numpy.allclose(X, Y)
+    print 'Buffered allreduce using pypar.BXOR OK'
+
+
+
+
+
+
+
+  # Test status block (simple communication)
+  N = 17  
+  if myid == 0:
+    # Integer arrays
+    #
+    A = numpy.array(range(N)).astype('i')
+
+    pypar.send(A,1)
+    B, status = pypar.receive(numproc-1, return_status = True)
+
+    repr(status)  # Check that status can be printed
+
+    sz = A.itemsize
+    assert numpy.allclose(A, B)
+    assert len(B) == status.length, 'Reported length == %d should be %d'\
+           %(status.length, len(B))
+    assert status.size == sz, 'Reported size == %d should be %d'\
+           %(status.size, sz)
+    assert status.tag == pypar.default_tag, 'Reported tag == %d should be %d'\
+           %(status.tag, pypar.default_tag)
+    assert status.error == 0, 'Status.error = %i' % status.error
+    assert status.source == numproc-1, 'Reported source == %d should be %d'\
+           %(status.source, numproc-1)
+
+    print "Status object (numeric integer arrays) OK"
+           
+    # Real arrays
+    #
+    A = numpy.array(range(N)).astype('f')
+    pypar.send(A,1)
+    B, status = pypar.receive(numproc-1, return_status = True)    
+
+    sz = A.itemsize    
+    assert numpy.allclose(A, B)
+    assert len(B) == status.length, 'Reported length == %d should be %d'\
+           %(status.length, len(B))
+    assert status.size == sz, 'Reported size == %d should be %d'\
+           %(status.size, sz)
+    assert status.tag == pypar.default_tag, 'Reported tag == %d should be %d'\
+           %(status.tag, pypar.default_tag)
+    assert status.error == 0
+    assert status.source == numproc-1, 'Reported source == %d should be %d'\
+           %(status.source, numproc-1)
+    
+    print "Status object (numeric real arrays) OK"
+
+    # Strings
+    #
+    A = "and now to something completely different !"
+    pypar.send(A,1)
+    B, status = pypar.receive(numproc-1, return_status = True)        
+
+    sz = 1 #Characters are one byte long
+    assert A == B
+    assert len(B) == status.length, 'Reported length == %d should be %d'\
+           %(status.length, len(B))
+    assert status.size == sz, 'Reported size == %d should be %d'\
+           %(status.size, sz)
+    assert status.tag == pypar.default_tag, 'Reported tag == %d should be %d'\
+           %(status.tag, pypar.default_tag)
+    assert status.error == 0
+    assert status.source == numproc-1, 'Reported source == %d should be %d'\
+           %(status.source, numproc-1)
+    
+    print "Status object (strings) OK"
+    
+    # A more general structure
+    #
+    A = ['ABC', (1,2,3.14), {8: 'Monty'}, numpy.array([13.45, 1.2])]
+    pypar.send(A,1)
+    B, status = pypar.receive(numproc-1, return_status = True)            
+
+    OK = True
+    for i, a in enumerate(A):
+      b = B[i]
+
+      if type(a).__name__ == 'ndarray':
+        if not numpy.allclose(a, b):
+          OK = False
+          break
+      elif a != b:
+        OK = False
+        break
+
+    if OK is True:
+      print 'Status object (more general structures) OK' 
+    else:
+      raise Exception    
+    
+
+    #Length is the number of characters needed to encode the structure
+    #Can't think of a test.
+
+    sz = 1
+    assert status.size == sz, 'Reported size == %d should be %d'\
+           %(status.size, sz)
+    assert status.tag == pypar.default_tag, 'Reported tag == %d should be %d'\
+           %(status.tag, pypar.default_tag)
+    assert status.error == 0
+    assert status.source == numproc-1, 'Reported source == %d should be %d'\
+           %(status.source, numproc-1)
+
+    print "Status object (general structures) OK"
+    
+  else:  
+    # Integers
+    #
+    X=pypar.receive(myid-1)  
+    pypar.send(X, (myid+1)%numproc)
+  
+    # Floats
+    #
+    X=pypar.receive(myid-1)  
+    pypar.send(X, (myid+1)%numproc)    
+
+    # Strings
+    #
+    X=pypar.receive(myid-1)  
+    pypar.send(X, (myid+1)%numproc)    
+
+    # General
+    #
+    X = pypar.receive(myid-1)  
+    pypar.send(X, (myid+1)%numproc)    
+
+
+
+
+
+
+
+
+  # Test status block (buffered communication)
+  N = 17 # Number of elements
+  if myid == 0:
+    # Integer arrays
+    #
+    A = numpy.array(range(N)).astype('i')
+    B = numpy.zeros(N).astype('i')    
+
+    pypar.send(A, 1, use_buffer=True)
+    B, status = pypar.receive(numproc-1, buffer=B, return_status=True)
+    
+    assert numpy.allclose(A, B)
+
+    sz = A.itemsize
+    assert numpy.allclose(A, B)
+    assert len(B) == status.length, 'Reported length == %d should be %d'\
+           %(status.length, len(B))
+    assert status.size == sz, 'Reported size == %d should be %d'\
+           %(status.size, sz)
+    assert status.tag == pypar.default_tag, 'Reported tag == %d should be %d'\
+           %(status.tag, pypar.default_tag)
+    assert status.error == 0
+    assert status.source == numproc-1, 'Reported source == %d should be %d'\
+           %(status.source, numproc-1)
+
+    print "Status object (buffered numeric integer arrays) OK"
+
+
+    # Real arrays
+    #
+    A = numpy.array(range(N)).astype('f')
+    B = numpy.zeros(N).astype('f')    
+    pypar.send(A, 1, use_buffer=True)
+    B, status = pypar.receive(numproc-1, buffer=B, return_status=True)
+    
+    assert numpy.allclose(A, B)
+    sz = A.itemsize
+    assert numpy.allclose(A, B)
+    assert len(B) == status.length, 'Reported length == %d should be %d'\
+           %(status.length, len(B))
+    assert status.size == sz, 'Reported size == %d should be %d'\
+           %(status.size, sz)
+    assert status.tag == pypar.default_tag, 'Reported tag == %d should be %d'\
+           %(status.tag, pypar.default_tag)
+    assert status.error == 0
+    assert status.source == numproc-1, 'Reported source == %d should be %d'\
+           %(status.source, numproc-1)
+
+    print "Status object (buffered numeric real arrays) OK"
+
+    # Strings (< 256 characters)
+    #
+    A = "and now to something completely different !"
+    B = " "*len(A)
+
+    pypar.send(A, 1, use_buffer=True)
+    B, status = pypar.receive(numproc-1, buffer=B, return_status=True)
+
+    sz = 1 #Characters are one byte long
+    assert A == B
+    assert len(B) == status.length, 'Reported length == %d should be %d'\
+           %(status.length, len(B))
+    assert status.size == sz, 'Reported size == %d should be %d'\
+           %(status.size, sz)
+    assert status.tag == pypar.default_tag, 'Reported tag == %d should be %d'\
+           %(status.tag, pypar.default_tag)
+    assert status.error == 0
+    assert status.source == numproc-1, 'Reported source == %d should be %d'\
+           %(status.source, numproc-1)
+
+    print "Status object (buffered strings) OK"
+    
+
+    
+    # A more general structure
+    #
+    A = ['ABC', (1,2,3.14), {8: 'Monty'}, numpy.array([13.45, 1.2])]
+    B = ['   ', (0,0,0.0), {0: '     '}, numpy.zeros(2).astype('f')]    
+    pypar.send(A, 1, use_buffer=True)
+    B, status = pypar.receive(numproc-1, buffer=B, return_status=True)
+
+    #assert A == B
+    sz = 1
+    assert status.size == sz, 'Reported size == %d should be %d'\
+           %(status.size, sz)
+    assert status.tag == pypar.default_tag, 'Reported tag == %d should be %d'\
+           %(status.tag, pypar.default_tag)
+    assert status.error == 0
+    assert status.source == numproc-1, 'Reported source == %d should be %d'\
+           %(status.source, numproc-1)
+
+    
+    print "Status object (buffered general structures) OK"
+    
+   
+  else:  
+    # Integers
+    #
+    X = numpy.zeros(N).astype('i')    
+
+
+    X = pypar.receive(myid-1, buffer=X)
+    pypar.send(X, (myid+1)%numproc, use_buffer=True)
+  
+    # Floats
+    #
+    X = numpy.zeros(N).astype('f')
+
+    X = pypar.receive(myid-1, buffer=X)
+    pypar.send(X, (myid+1)%numproc, use_buffer=True)
+
+    
+    # Strings
+    #
+    X = " "*256
+    X = pypar.receive(myid-1, buffer=X)
+    pypar.send(X.strip(), (myid+1)%numproc, use_buffer=True)
+
+    # General
+    #
+    X = ['   ', (0,0,0.0), {0: '     '}, numpy.zeros(2).astype('f')]
+    X = pypar.receive(myid-1, buffer=X)
+    pypar.send(X, (myid+1)%numproc, use_buffer=True)
+
+
+pypar.finalize()
+
+
+
+
Index: trunk/anuga_core/source/pypar_dist/source/test_reduce.c
===================================================================
--- trunk/anuga_core/source/pypar_dist/source/test_reduce.c	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/source/test_reduce.c	(revision 8494)
@@ -0,0 +1,57 @@
+/*
+  This is to test of Reduce with all choices of root process 
+  independently of pypar.
+
+  Adapted from http://mpi.deino.net/mpi_functions/MPI_Reduce.html
+
+  Ole Moller Nielsen - 2009
+*/
+	
+
+
+
+#include "mpi.h"
+#include <stdio.h>
+#include <stdlib.h>
+ 
+
+int main( int argc, char *argv[] )
+{
+    int errs = 0;
+    int rank, size, root;
+    int *sendbuf, *recvbuf, i;
+    int minsize = 2, count;
+    MPI_Comm comm;
+ 
+    MPI_Init( &argc, &argv );
+ 
+    comm = MPI_COMM_WORLD;
+    /* Determine the sender and receiver */
+    MPI_Comm_rank( comm, &rank );
+    MPI_Comm_size( comm, &size );
+ 
+    for (count = 1; count < 130000; count = count * 2) {
+        sendbuf = (int *)malloc( count * sizeof(int) );
+        recvbuf = (int *)malloc( count * sizeof(int) );
+        for (root = 0; root < size; root ++) {
+            for (i=0; i<count; i++) sendbuf[i] = i;
+            for (i=0; i<count; i++) recvbuf[i] = -1;
+            MPI_Reduce( sendbuf, recvbuf, count, MPI_INT, MPI_SUM, root, comm );
+            if (rank == root) {
+                for (i=0; i<count; i++) {
+                    if (recvbuf[i] != i * size) {
+                        errs++;
+                    }
+                }
+            }
+        }
+        free( sendbuf );
+        free( recvbuf );
+    }
+ 
+    MPI_Finalize();
+
+    
+    printf("Errors encountered: %i\n", errs);
+    return errs;
+}
Index: trunk/anuga_core/source/pypar_dist/supporting_material/archive/old_revision_control_logs/_inundation_pypar_dist_ (log) - AnuGA - Trac.htm
===================================================================
--- trunk/anuga_core/source/pypar_dist/supporting_material/archive/old_revision_control_logs/_inundation_pypar_dist_ (log) - AnuGA - Trac.htm	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/supporting_material/archive/old_revision_control_logs/_inundation_pypar_dist_ (log) - AnuGA - Trac.htm	(revision 8494)
@@ -0,0 +1,413 @@
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"><head><title>/inundation/pypar_dist/ (log) - AnuGA - Trac</title>
+
+
+ 
+  
+  <link rel="search" href="https://datamining.anu.edu.au/anuga/search">
+  <link rel="help" href="https://datamining.anu.edu.au/anuga/wiki/TracGuide">
+  <link rel="alternate" href="https://datamining.anu.edu.au/anuga/log/inundation/pypar_dist/?format=rss" title="RSS Feed" type="application/rss+xml">
+  <link rel="up" href="https://datamining.anu.edu.au/anuga/browser/inundation/pypar_dist/" title="Parent directory">
+  <link rel="start" href="https://datamining.anu.edu.au/anuga/wiki">
+  <link rel="shortcut icon" href="https://datamining.anu.edu.au/trac/trac.ico" type="image/x-icon">
+  <link rel="icon" href="https://datamining.anu.edu.au/trac/trac.ico" type="image/x-icon">
+  <style type="text/css">
+   @import url(/trac/css/trac.css);
+   @import url(/trac/css/browser.css);
+   
+
+  </style>
+  <script src="_inundation_pypar_dist_%20%28log%29%20-%20AnuGA%20-%20Trac_files/trac.js" type="text/javascript"></script></head><body>
+
+
+<div id="banner">
+
+<div id="header">
+  <a id="logo" href="http://trac.edgewall.com/"><img src="_inundation_pypar_dist_%20%28log%29%20-%20AnuGA%20-%20Trac_files/trac_banner.png" alt="Trac" height="73" width="236"></a>
+  <hr>
+</div>
+
+<form id="search" action="/anuga/search" method="get">
+ <div>
+  <label for="proj-search">Search:</label>
+  <input id="proj-search" name="q" size="10" value="" type="text">
+  <input value="Search" type="submit">
+  <input name="wiki" value="on" type="hidden">
+  <input name="changeset" value="on" type="hidden">
+  <input name="ticket" value="on" type="hidden">
+ </div>
+</form>
+
+<div id="metanav" class="nav">
+ <h2>Navigation</h2>
+ <ul>
+  <li class="first">
+    logged in as ole </li>
+    <li><a href="https://datamining.anu.edu.au/anuga/logout">Logout</a>
+  </li>
+  <li><a href="https://datamining.anu.edu.au/anuga/settings">Settings</a></li>
+  <li><a accesskey="6" href="https://datamining.anu.edu.au/anuga/wiki/TracGuide">Help/Guide</a></li>
+  <li style="display: none;"><a accesskey="5" href="http://projects.edgewall.com/trac/wiki/TracFaq">FAQ</a></li>
+  <li style="display: none;"><a accesskey="0" href="https://datamining.anu.edu.au/anuga/wiki/TracAccessibility">Accessibility</a></li>
+  <li class="last"><a accesskey="9" href="https://datamining.anu.edu.au/anuga/about_trac">About Trac</a></li>
+ </ul>
+</div>
+
+</div>
+
+
+
+
+
+<div id="mainnav" class="nav">
+ <ul><li><a href="https://datamining.anu.edu.au/anuga/wiki" accesskey="1">Wiki</a></li><li><a href="https://datamining.anu.edu.au/anuga/timeline" accesskey="2">Timeline</a></li><li><a href="https://datamining.anu.edu.au/anuga/roadmap" accesskey="3">Roadmap</a></li><li><a href="https://datamining.anu.edu.au/anuga/browser/" class="active">Browse Source</a></li><li><a href="https://datamining.anu.edu.au/anuga/report">View Tickets</a></li><li><a href="https://datamining.anu.edu.au/anuga/newticket" accesskey="7">New Ticket</a></li><li><a href="https://datamining.anu.edu.au/anuga/search" accesskey="4">Search</a></li></ul>
+</div>
+
+<div id="main">
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+<div id="ctxtnav" class="nav">
+ <ul>
+  <li class="last"><a href="https://datamining.anu.edu.au/anuga/browser/inundation/pypar_dist/?rev=3444">View Latest Revision</a></li>
+ </ul>
+</div>
+
+<div id="content" class="log">
+ 
+ <h1><a class="first" title="Go to root directory" href="https://datamining.anu.edu.au/anuga/browser/">root</a><span class="sep">/</span><a title="Go to directory" href="https://datamining.anu.edu.au/anuga/browser/inundation/">inundation</a><span class="sep">/</span><a title="Go to directory" href="https://datamining.anu.edu.au/anuga/browser/inundation/pypar_dist/">pypar_dist</a></h1>
+
+
+ <div id="jumprev">
+  <form action="" method="get">
+   <div>
+    <label for="rev">View revision:</label>
+    <input id="rev" name="rev" value="3444" size="4" type="text">
+   </div>
+  </form>
+ </div>
+
+ <table id="chglist" class="listing">
+  <thead>
+   <tr>
+    <th class="data">Date</th>
+    <th class="rev">Rev</th>
+    <th class="chgset">Chgset</th>
+    <th class="author">Author</th>
+    <th class="summary">Log Message</th>
+   </tr>
+  </thead>
+  <tbody>
+   
+    <tr class="odd">
+     <td class="date">08/02/06 16:23:28</td>
+     <td class="rev">
+      <a href="https://datamining.anu.edu.au/anuga/browser/inundation/pypar_dist/?rev=3444">3444</a>
+     </td>
+     <td class="chgset">
+      <a class="block-link" href="https://datamining.anu.edu.au/anuga/changeset/3444">3444</a>
+     </td>
+     <td class="author">ole</td>
+     <td class="summary">Comment about compilation of pypar ...</td>
+    </tr>
+   
+    <tr class="even">
+     <td class="date">08/02/06 16:21:24</td>
+     <td class="rev">
+      <a href="https://datamining.anu.edu.au/anuga/browser/inundation/pypar_dist/?rev=3443">3443</a>
+     </td>
+     <td class="chgset">
+      <a class="block-link" href="https://datamining.anu.edu.au/anuga/changeset/3443">3443</a>
+     </td>
+     <td class="author">ole</td>
+     <td class="summary">Attempt to add compiler flags for AMD opteron 64 ...</td>
+    </tr>
+   
+    <tr class="odd">
+     <td class="date">07/31/06 17:19:43</td>
+     <td class="rev">
+      <a href="https://datamining.anu.edu.au/anuga/browser/inundation/pypar_dist/?rev=3433">3433</a>
+     </td>
+     <td class="chgset">
+      <a class="block-link" href="https://datamining.anu.edu.au/anuga/changeset/3433">3433</a>
+     </td>
+     <td class="author">ole</td>
+     <td class="summary">Removed path again - seems to be unnecessary ...</td>
+    </tr>
+   
+    <tr class="even">
+     <td class="date">07/31/06 17:11:36</td>
+     <td class="rev">
+      <a href="https://datamining.anu.edu.au/anuga/browser/inundation/pypar_dist/?rev=3432">3432</a>
+     </td>
+     <td class="chgset">
+      <a class="block-link" href="https://datamining.anu.edu.au/anuga/changeset/3432">3432</a>
+     </td>
+     <td class="author">ole</td>
+     <td class="summary">Added path for c-extension back in ...</td>
+    </tr>
+   
+    <tr class="odd">
+     <td class="date">07/25/06 13:04:49</td>
+     <td class="rev">
+      <a href="https://datamining.anu.edu.au/anuga/browser/inundation/pypar_dist/?rev=3415">3415</a>
+     </td>
+     <td class="chgset">
+      <a class="block-link" href="https://datamining.anu.edu.au/anuga/changeset/3415">3415</a>
+     </td>
+     <td class="author">jack</td>
+     <td class="summary">Deleted several .pyc files because they can be automatically generated.</td>
+    </tr>
+   
+    <tr class="even">
+     <td class="date">06/06/06 12:25:00</td>
+     <td class="rev">
+      <a href="https://datamining.anu.edu.au/anuga/browser/inundation/pypar_dist/?rev=3096">3096</a>
+     </td>
+     <td class="chgset">
+      <a class="block-link" href="https://datamining.anu.edu.au/anuga/changeset/3096">3096</a>
+     </td>
+     <td class="author">ole</td>
+     <td class="summary">Allowed parallel code to access pypar directly in its directory pypar_dist ...</td>
+    </tr>
+   
+    <tr class="odd">
+     <td class="date">05/23/06 16:44:11</td>
+     <td class="rev">
+      <a href="https://datamining.anu.edu.au/anuga/browser/inundation/pypar_dist/?rev=2949">2949</a>
+     </td>
+     <td class="chgset">
+      <a class="block-link" href="https://datamining.anu.edu.au/anuga/changeset/2949">2949</a>
+     </td>
+     <td class="author">ole</td>
+     <td class="summary">Updated compile and install for Opteron 64 using the Portland group ...</td>
+    </tr>
+   
+    <tr class="even">
+     <td class="date">04/27/06 15:44:41</td>
+     <td class="rev">
+      <a href="https://datamining.anu.edu.au/anuga/browser/inundation/pypar_dist/?rev=2778">2778</a>
+     </td>
+     <td class="chgset">
+      <a class="block-link" href="https://datamining.anu.edu.au/anuga/changeset/2778">2778</a>
+     </td>
+     <td class="author">steve</td>
+     <td class="summary">On our 64 bit machine, ran into problem in pmesh/mesh.py where seg[<a class="missing" href="https://datamining.anu.edu.au/anuga/changeset/0">0</a>], ...</td>
+    </tr>
+   
+    <tr class="odd">
+     <td class="date">11/22/05 13:38:37</td>
+     <td class="rev">
+      <a href="https://datamining.anu.edu.au/anuga/browser/inundation/pypar_dist/?rev=2049">2049</a>
+     </td>
+     <td class="chgset">
+      <a class="block-link" href="https://datamining.anu.edu.au/anuga/changeset/2049">2049</a>
+     </td>
+     <td class="author">steve</td>
+     <td class="summary">Conflict on my PC with pypar in inundation and installed in site-packages</td>
+    </tr>
+   
+    <tr class="even">
+     <td class="date">09/28/05 18:00:46</td>
+     <td class="rev">
+      <a href="https://datamining.anu.edu.au/anuga/browser/inundation/pypar/?rev=1855">1855</a>
+     </td>
+     <td class="chgset">
+      <a class="block-link" href="https://datamining.anu.edu.au/anuga/changeset/1855">1855</a>
+     </td>
+     <td class="author">ole</td>
+     <td class="summary">Better interface to meshing for karratha + some pypar work ...</td>
+    </tr>
+   
+    <tr class="odd">
+     <td class="date">09/23/05 09:47:30</td>
+     <td class="rev">
+      <a href="https://datamining.anu.edu.au/anuga/browser/inundation/pypar/?rev=1852">1852</a>
+     </td>
+     <td class="chgset">
+      <a class="block-link" href="https://datamining.anu.edu.au/anuga/changeset/1852">1852</a>
+     </td>
+     <td class="author">ole</td>
+     <td class="summary">Added prerequisites for mandelbrot_example</td>
+    </tr>
+   
+    <tr class="even">
+     <td class="date">09/23/05 09:39:21</td>
+     <td class="rev">
+      <a href="https://datamining.anu.edu.au/anuga/browser/inundation/pypar/?rev=1851">1851</a>
+     </td>
+     <td class="chgset">
+      <a class="block-link" href="https://datamining.anu.edu.au/anuga/changeset/1851">1851</a>
+     </td>
+     <td class="author">ole</td>
+     <td class="summary">Removed strict dependency on <a class="missing" href="https://datamining.anu.edu.au/anuga/wiki/RandomArray">RandomArray?</a> ...</td>
+    </tr>
+   
+    <tr class="odd">
+     <td class="date">08/24/05 23:36:27</td>
+     <td class="rev">
+      <a href="https://datamining.anu.edu.au/anuga/browser/inundation/pypar/?rev=1756">1756</a>
+     </td>
+     <td class="chgset">
+      <a class="block-link" href="https://datamining.anu.edu.au/anuga/changeset/1756">1756</a>
+     </td>
+     <td class="author">ole</td>
+     <td class="summary">Moved remotely</td>
+    </tr>
+   
+    <tr class="even">
+     <td class="date">08/01/05 18:31:10</td>
+     <td class="rev">
+      <a href="https://datamining.anu.edu.au/anuga/browser/pypar/?rev=129">129</a>
+     </td>
+     <td class="chgset">
+      <a class="block-link" href="https://datamining.anu.edu.au/anuga/changeset/129">129</a>
+     </td>
+     <td class="author">ole</td>
+     <td class="summary">More docu</td>
+    </tr>
+   
+    <tr class="odd">
+     <td class="date">07/11/05 23:24:44</td>
+     <td class="rev">
+      <a href="https://datamining.anu.edu.au/anuga/browser/pypar/?rev=127">127</a>
+     </td>
+     <td class="chgset">
+      <a class="block-link" href="https://datamining.anu.edu.au/anuga/changeset/127">127</a>
+     </td>
+     <td class="author">steve</td>
+     <td class="summary"></td>
+    </tr>
+   
+    <tr class="even">
+     <td class="date">07/11/05 18:07:06</td>
+     <td class="rev">
+      <a href="https://datamining.anu.edu.au/anuga/browser/pypar/?rev=126">126</a>
+     </td>
+     <td class="chgset">
+      <a class="block-link" href="https://datamining.anu.edu.au/anuga/changeset/126">126</a>
+     </td>
+     <td class="author">ole</td>
+     <td class="summary">Added bypass to reduce and broadcast ...</td>
+    </tr>
+   
+    <tr class="odd">
+     <td class="date">07/11/05 15:23:29</td>
+     <td class="rev">
+      <a href="https://datamining.anu.edu.au/anuga/browser/pypar/?rev=124">124</a>
+     </td>
+     <td class="chgset">
+      <a class="block-link" href="https://datamining.anu.edu.au/anuga/changeset/124">124</a>
+     </td>
+     <td class="author">ole</td>
+     <td class="summary">Doc files and ring_example ...</td>
+    </tr>
+   
+    <tr class="even">
+     <td class="date">07/11/05 15:20:03</td>
+     <td class="rev">
+      <a href="https://datamining.anu.edu.au/anuga/browser/pypar/?rev=123">123</a>
+     </td>
+     <td class="chgset">
+      <a class="block-link" href="https://datamining.anu.edu.au/anuga/changeset/123">123</a>
+     </td>
+     <td class="author">ole</td>
+     <td class="summary">Files for 64 bit machine + latest Cvs version ...</td>
+    </tr>
+   
+    <tr class="odd">
+     <td class="date">05/19/05 17:19:16</td>
+     <td class="rev">
+      <a href="https://datamining.anu.edu.au/anuga/browser/pypar/?rev=87">87</a>
+     </td>
+     <td class="chgset">
+      <a class="block-link" href="https://datamining.anu.edu.au/anuga/changeset/87">87</a>
+     </td>
+     <td class="author">ole</td>
+     <td class="summary">Added old CVS log</td>
+    </tr>
+   
+    <tr class="even">
+     <td class="date">05/19/05 17:15:12</td>
+     <td class="rev">
+      <a href="https://datamining.anu.edu.au/anuga/browser/pypar/?rev=86">86</a>
+     </td>
+     <td class="chgset">
+      <a class="block-link" href="https://datamining.anu.edu.au/anuga/changeset/86">86</a>
+     </td>
+     <td class="author">ole</td>
+     <td class="summary"></td>
+    </tr>
+   
+    <tr class="odd">
+     <td class="date">05/19/05 17:14:23</td>
+     <td class="rev">
+      <a href="https://datamining.anu.edu.au/anuga/browser/pypar/?rev=85">85</a>
+     </td>
+     <td class="chgset">
+      <a class="block-link" href="https://datamining.anu.edu.au/anuga/changeset/85">85</a>
+     </td>
+     <td class="author">ole</td>
+     <td class="summary">Added pypar files</td>
+    </tr>
+   
+    <tr class="even">
+     <td class="date">05/19/05 17:03:55</td>
+     <td class="rev">
+      <a href="https://datamining.anu.edu.au/anuga/browser/pypar/?rev=84">84</a>
+     </td>
+     <td class="chgset">
+      <a class="block-link" href="https://datamining.anu.edu.au/anuga/changeset/84">84</a>
+     </td>
+     <td class="author">ole</td>
+     <td class="summary">Created folder for pypar (parallel python) to be moved from CVS</td>
+    </tr>
+   
+  </tbody>
+ </table>
+
+</div>
+<script type="text/javascript">searchHighlight()</script>
+
+
+<div id="altlinks">
+ <h3>Download in other formats:</h3>
+ <ul>
+  <li class="first last">
+   <a href="https://datamining.anu.edu.au/anuga/log/inundation/pypar_dist/?format=rss" class="rss">RSS Feed</a>
+  </li>
+ </ul>
+</div>
+
+
+</div>
+
+<div id="footer">
+ <hr>
+ <a id="tracpowered" href="http://trac.edgewall.com/"><img src="_inundation_pypar_dist_%20%28log%29%20-%20AnuGA%20-%20Trac_files/trac_logo_mini.png" alt="Trac Powered" height="30" width="107"></a>
+ <p class="left">
+  Powered by <a href="https://datamining.anu.edu.au/anuga/about_trac"><strong>Trac 0.8.1</strong></a><br>
+  By <a href="http://www.edgewall.com/">Edgewall Software</a>.
+ </p>
+ <p class="right">
+  Visit the Trac open source project at<br><a href="http://trac.edgewall.com/">http://trac.edgewall.com/</a>
+ </p>
+</div>
+
+
+
+ 
+</body></html>
Index: trunk/anuga_core/source/pypar_dist/supporting_material/archive/old_revision_control_logs/_inundation_pypar_dist_ (log) - AnuGA - Trac_files/trac.js
===================================================================
--- trunk/anuga_core/source/pypar_dist/supporting_material/archive/old_revision_control_logs/_inundation_pypar_dist_ (log) - AnuGA - Trac_files/trac.js	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/supporting_material/archive/old_revision_control_logs/_inundation_pypar_dist_ (log) - AnuGA - Trac_files/trac.js	(revision 8494)
@@ -0,0 +1,159 @@
+// Used for dynamically updating the height of a textarea
+function resizeTextArea(id, rows) {
+  var textarea = document.getElementById(id);
+  if (!textarea || (typeof(textarea.rows) == "undefined")) return;
+  textarea.rows = rows;
+}
+
+// A better way than for example hardcoding foo.onload
+function addEvent(element, type, func){
+  if (element.addEventListener) {
+    element.addEventListener(type, func, false);
+    return true;
+  } else if (element.attachEvent) {
+    return element.attachEvent("on" + type, func);
+  }
+  return false;
+}
+
+// Adapted from http://www.kryogenix.org/code/browser/searchhi/
+function searchHighlight() {
+  if (!document.createElement) return;
+
+  var div = document.getElementById("searchable");
+  if (!div) return;
+
+  function getSearchWords(url) {
+    if (url.indexOf('?') == -1) return [];
+    var queryString = url.substr(url.indexOf('?') + 1);
+    var params = queryString.split('&');
+    for (var p in params) {
+      var param = params[p].split('=');
+      if (param.length < 2) continue;
+      if (param[0] == 'q' || param[0] == 'p') { // q= for Google, p= for Yahoo
+        return unescape(param[1].replace(/\+/g, ' ')).split(/\s+/);
+      }
+    }
+    return [];
+  }
+
+  function highlightWord(node, word, searchwordindex) {
+    // If this node is a text node and contains the search word, highlight it by
+    // surrounding it with a span element
+    if (node.nodeType == 3) { // Node.TEXT_NODE
+      var pos = node.nodeValue.toLowerCase().indexOf(word.toLowerCase());
+      if (pos >= 0 && !/^searchword\d$/.test(node.parentNode.className)) {
+        var span = document.createElement("span");
+        span.className = "searchword" + (searchwordindex % 5);
+        span.appendChild(document.createTextNode(
+            node.nodeValue.substr(pos, word.length)));
+        var newNode = node.splitText(pos);
+        newNode.nodeValue = newNode.nodeValue.substr(word.length);
+        node.parentNode.insertBefore(span, newNode);
+        return true;
+      }
+    } else if (!node.nodeName.match(/button|select|textarea/i)) {
+      // Recurse into child nodes
+      for (var i = 0; i < node.childNodes.length; i++) {
+        if (highlightWord(node.childNodes[i], word, searchwordindex)) i++;
+      }
+    }
+    return false;
+  }
+
+  var words = getSearchWords(document.URL);
+  if (!words.length) words = getSearchWords(document.referrer);
+  if (words.length) {
+    for (var w in words) {
+      if (words[w].length) highlightWord(div, words[w], w);
+    }
+  }
+}
+
+function enableControl(id, enabled) {
+  if (typeof(enabled) == "undefined") enabled = true;
+  var control = document.getElementById(id);
+  if (!control) return;
+  control.disabled = !enabled;
+  var labels = document.getElementsByTagName("label");
+  for (var i = 0; i < labels.length; i++) {
+    if (labels[i].htmlFor == id) {
+      labels[i].className = enabled ? "enabled" : "disabled";
+    }
+  }
+}
+
+function addWikiFormattingToolbar(textarea) {
+  if ((typeof(document["selection"]) == "undefined")
+   && (typeof(textarea["setSelectionRange"]) == "undefined")) {
+    return;
+  }
+  
+  var toolbar = document.createElement("div");
+  toolbar.className = "wikitoolbar";
+
+  function addButton(id, title, fn) {
+    var a = document.createElement("a");
+    a.href = "#";
+    a.id = id;
+    a.title = title;
+    a.onclick = function() { try { fn() } catch (e) { } return false };
+    a.tabIndex = 400;
+    toolbar.appendChild(a);
+  }
+
+  function encloseSelection(prefix, suffix) {
+    textarea.focus();
+    var start, end, sel, scrollPos, subst;
+    if (typeof(document["selection"]) != "undefined") {
+      sel = document.selection.createRange().text;
+    } else if (typeof(textarea["setSelectionRange"]) != "undefined") {
+      start = textarea.selectionStart;
+      end = textarea.selectionEnd;
+      scrollPos = textarea.scrollTop;
+      sel = textarea.value.substring(start, end);
+    }
+    if (sel.match(/ $/)) { // exclude ending space char, if any
+      sel = sel.substring(0, sel.length - 1);
+      suffix = suffix + " ";
+    }
+    subst = prefix + sel + suffix;
+    if (typeof(document["selection"]) != "undefined") {
+      var range = document.selection.createRange().text = subst;
+      textarea.caretPos -= suffix.length;
+    } else if (typeof(textarea["setSelectionRange"]) != "undefined") {
+      textarea.value = textarea.value.substring(0, start) + subst +
+                       textarea.value.substring(end);
+      if (sel) {
+        textarea.setSelectionRange(start + subst.length, start + subst.length);
+      } else {
+        textarea.setSelectionRange(start + prefix.length, start + prefix.length);
+      }
+      textarea.scrollTop = scrollPos;
+    }
+  }
+
+  addButton("strong", "Bold text: '''Example'''", function() {
+    encloseSelection("'''", "'''");
+  });
+  addButton("em", "Italic text: ''Example''", function() {
+    encloseSelection("''", "''");
+  });
+  addButton("heading", "Heading: == Example ==", function() {
+    encloseSelection("\n== ", " ==\n", "Heading");
+  });
+  addButton("link", "Link: [http://www.example.com/ Example]", function() {
+    encloseSelection("[", "]");
+  });
+  addButton("code", "Code block: {{{ example }}}", function() {
+    encloseSelection("\n{{{\n", "\n}}}\n");
+  });
+  addButton("hr", "Horizontal rule: ----", function() {
+    encloseSelection("\n----\n", "");
+  });
+
+  textarea.parentNode.insertBefore(toolbar, textarea);
+  var br = document.createElement("br");
+  br.style.clear = "left";
+  textarea.parentNode.insertBefore(br, textarea);
+}
Index: trunk/anuga_core/source/pypar_dist/supporting_material/archive/old_revision_control_logs/original_cvs_log.txt
===================================================================
--- trunk/anuga_core/source/pypar_dist/supporting_material/archive/old_revision_control_logs/original_cvs_log.txt	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/supporting_material/archive/old_revision_control_logs/original_cvs_log.txt	(revision 8494)
@@ -0,0 +1,2157 @@
+
+RCS file: /home/cvs/pypar/DOC,v
+Working file: DOC
+head: 1.26
+branch:
+locks: strict
+access list:
+symbolic names:
+	release-1-9-2: 1.26
+	release-1-9-1: 1.25
+	release-1-9: 1.23
+	release-1-8-2: 1.22
+	release-1-8-1: 1.21
+	release-1-8: 1.19
+	release-1-7: 1.17
+	release-1-6-5: 1.17
+	release-1-6-4: 1.17
+	release-1-6-3-1: 1.15
+	release-1-6-3: 1.15
+	SG-release-1-1: 1.13
+	SG-release-1-0: 1.13
+	release-0-1: 1.13
+	release-1-6-2: 1.13
+	release-1-6-1: 1.13
+	release-1-6-0: 1.13
+	release-1-5-4: 1.12
+	release-1-5-3: 1.11
+	release-1-5-2: 1.11
+	release-1-5-1: 1.11
+	release-1-5: 1.11
+	release-1-4-1: 1.10
+	release-1-4-0: 1.10
+	release-1-4: 1.9
+	release-1-3: 1.9
+	release-1-2-4: 1.9
+	release-1-2-3: 1.9
+	release-1-2-2: 1.7
+	revision-1-2-1: 1.7
+	release-1-2: 1.7
+	release-1-1: 1.7
+	release-1-0-2: 1.6
+	release-1-0-1: 1.5
+	release-1-0: 1.4
+keyword substitution: kv
+total revisions: 26;	selected revisions: 26
+description:
+----------------------------
+revision 1.26
+date: 2004-12-29 08:55:52 +0000;  author: ole;  state: Exp;  lines: +2 -2
+Update DOC
+----------------------------
+revision 1.25
+date: 2004-01-07 11:39:32 +0000;  author: ole;  state: Exp;  lines: +1 -1
+Nothing
+----------------------------
+revision 1.24
+date: 2004-01-07 11:36:17 +0000;  author: ole;  state: Exp;  lines: +19 -12
+Changed naming conventions to be more consistent
+----------------------------
+revision 1.23
+date: 2003-12-03 10:13:58 +0000;  author: ole;  state: Exp;  lines: +82 -56
+Updated doc for obsoletion of raw forms
+----------------------------
+revision 1.22
+date: 2003-11-16 10:04:48 +0000;  author: ole;  state: Exp;  lines: +16 -10
+Fixed up gather, scatter and reduce to automatically calculate buffer size. Also cleaned up control_info. Updated web site and release script
+----------------------------
+revision 1.21
+date: 2003-11-13 11:36:04 +0000;  author: ole;  state: Exp;  lines: +1 -2
+Added support for the complex datatype
+----------------------------
+revision 1.20
+date: 2003-11-12 07:44:17 +0000;  author: ole;  state: Exp;  lines: +1 -1
+*** empty log message ***
+----------------------------
+revision 1.19
+date: 2003-11-11 11:05:12 +0000;  author: ole;  state: Exp;  lines: +1 -1
+Dates and cleaned up old comment
+----------------------------
+revision 1.18
+date: 2003-11-11 10:57:37 +0000;  author: ole;  state: Exp;  lines: +26 -6
+Status object optionally returned from receive. Reserved control_tag, trimmed string buffers to actual length. Minor clean up
+----------------------------
+revision 1.17
+date: 2003-02-04 01:54:18 +0000;  author: ole;  state: Exp;  lines: +30 -23
+Changed argument names in gather, reduce, ... and updated doc
+----------------------------
+revision 1.16
+date: 2003-01-15 02:17:12 +0000;  author: ole;  state: Exp;  lines: +17 -3
+Added paragraph about performance and raw versions in Documentation.
+----------------------------
+revision 1.15
+date: 2002-12-18 03:10:35 +0000;  author: ole;  state: Exp;  lines: +0 -5
+Fixed reintroduced problem about hanging under LAM linux.
+It is not possible to call init in a separate shell and then later within
+the python script.
+----------------------------
+revision 1.14
+date: 2002-12-17 23:47:37 +0000;  author: ole;  state: Exp;  lines: +6 -1
+Nothing
+----------------------------
+revision 1.13
+date: 2002-10-18 04:24:03 +0000;  author: ole;  state: Exp;  lines: +1 -1
+*** empty log message ***
+----------------------------
+revision 1.12
+date: 2002-10-18 04:20:42 +0000;  author: ole;  state: Exp;  lines: +1 -1
+doc
+----------------------------
+revision 1.11
+date: 2002-04-30 04:57:21 +0000;  author: ole;  state: Exp;  lines: +45 -9
+Doc
+----------------------------
+revision 1.10
+date: 2002-03-04 02:41:06 +0000;  author: ole;  state: Exp;  lines: +1 -1
+x
+----------------------------
+revision 1.9
+date: 2002-02-17 22:40:10 +0000;  author: ole;  state: Exp;  lines: +1 -1
+x
+----------------------------
+revision 1.8
+date: 2002-02-17 04:58:37 +0000;  author: ole;  state: Exp;  lines: +5 -1
+minor
+----------------------------
+revision 1.7
+date: 2002-02-14 07:20:40 +0000;  author: ole;  state: Exp;  lines: +3 -0
+Added bcast - thanks to Gian Paolo Ciceri
+----------------------------
+revision 1.6
+date: 2002-02-10 23:05:31 +0000;  author: ole;  state: Exp;  lines: +2 -2
+Version 1.0.2
+Bugfix from Gian Paulo Ciceri preventing pypar from running under Python 2.2
+----------------------------
+revision 1.5
+date: 2002-02-08 03:58:58 +0000;  author: ole;  state: Exp;  lines: +3 -1
+Got it to run on a Sun Enterprise server under Mpich.
+Only small changes.
+----------------------------
+revision 1.4
+date: 2002-02-07 01:51:07 +0000;  author: ole;  state: Exp;  lines: +2 -2
+*** empty log message ***
+----------------------------
+revision 1.3
+date: 2002-02-07 01:40:47 +0000;  author: ole;  state: Exp;  lines: +31 -1
+*** empty log message ***
+----------------------------
+revision 1.2
+date: 2002-02-07 01:33:16 +0000;  author: ole;  state: Exp;  lines: +157 -2
+*** empty log message ***
+----------------------------
+revision 1.1
+date: 2002-02-07 00:49:27 +0000;  author: ole;  state: Exp;
+Documemtation file
+=============================================================================
+
+RCS file: /home/cvs/pypar/FAQ,v
+Working file: FAQ
+head: 1.3
+branch:
+locks: strict
+access list:
+symbolic names:
+	release-1-9-2: 1.3
+keyword substitution: kv
+total revisions: 3;	selected revisions: 3
+description:
+----------------------------
+revision 1.3
+date: 2004-02-08 17:09:47 +0000;  author: ole;  state: Exp;  lines: +8 -0
+no message
+----------------------------
+revision 1.2
+date: 2004-02-08 16:56:16 +0000;  author: ole;  state: Exp;  lines: +18 -0
+no message
+----------------------------
+revision 1.1
+date: 2004-02-08 16:53:49 +0000;  author: ole;  state: Exp;
+added FAQ
+=============================================================================
+
+RCS file: /home/cvs/pypar/Makefile,v
+Working file: Makefile
+head: 1.2
+branch:
+locks: strict
+access list:
+symbolic names:
+	release-1-9-2: 1.2
+	release-1-9-1: 1.2
+	release-1-9: 1.2
+	release-1-8-2: 1.2
+	release-1-8-1: 1.2
+	release-1-8: 1.2
+	release-1-7: 1.2
+	release-1-6-5: 1.2
+	release-1-6-4: 1.2
+	release-1-6-3-1: 1.2
+	release-1-6-3: 1.2
+	SG-release-1-1: 1.2
+	SG-release-1-0: 1.2
+	release-0-1: 1.2
+	release-1-6-2: 1.2
+	release-1-6-1: 1.2
+	release-1-6-0: 1.2
+	release-1-5-4: 1.2
+	release-1-5-3: 1.2
+	release-1-5-2: 1.2
+	release-1-5-1: 1.2
+	release-1-5: 1.2
+	release-1-4-1: 1.1.1.1
+	release-1-4-0: 1.1.1.1
+	release-1-4: 1.1.1.1
+	release-1-3: 1.1.1.1
+	release-1-2-4: 1.1.1.1
+	release-1-2-3: 1.1.1.1
+	release-1-2-2: 1.1.1.1
+	revision-1-2-1: 1.1.1.1
+	release-1-2: 1.1.1.1
+	release-1-1: 1.1.1.1
+	release-1-0-2: 1.1.1.1
+	release-1-0-1: 1.1.1.1
+	release-1-0: 1.1.1.1
+	start: 1.1.1.1
+	ole: 1.1.1
+keyword substitution: kv
+total revisions: 3;	selected revisions: 3
+description:
+----------------------------
+revision 1.2
+date: 2002-04-30 05:02:08 +0000;  author: ole;  state: Exp;  lines: +2 -2
+Rev 1.5
+----------------------------
+revision 1.1
+date: 2002-01-22 03:28:17 +0000;  author: ole;  state: Exp;
+branches:  1.1.1;
+Initial revision
+----------------------------
+revision 1.1.1.1
+date: 2002-01-22 03:28:17 +0000;  author: ole;  state: Exp;  lines: +0 -0
+Initial CVS import
+=============================================================================
+
+RCS file: /home/cvs/pypar/README,v
+Working file: README
+head: 1.37
+branch:
+locks: strict
+access list:
+symbolic names:
+	release-1-9-2: 1.37
+	release-1-9-1: 1.37
+	release-1-9: 1.34
+	release-1-8-2: 1.31
+	release-1-8-1: 1.29
+	release-1-8: 1.28
+	release-1-7: 1.26
+	release-1-6-5: 1.24
+	release-1-6-4: 1.23
+	release-1-6-3-1: 1.21
+	release-1-6-3: 1.20
+	SG-release-1-1: 1.20
+	SG-release-1-0: 1.19
+	release-0-1: 1.19
+	release-1-6-2: 1.19
+	release-1-6-1: 1.17
+	release-1-6-0: 1.17
+	release-1-5-4: 1.16
+	release-1-5-3: 1.15
+	release-1-5-2: 1.15
+	release-1-5-1: 1.15
+	release-1-5: 1.13
+	release-1-4-1: 1.12
+	release-1-4-0: 1.12
+	release-1-4: 1.10
+	release-1-3: 1.10
+	release-1-2-4: 1.9
+	release-1-2-3: 1.9
+	release-1-2-2: 1.7
+	revision-1-2-1: 1.7
+	release-1-2: 1.7
+	release-1-1: 1.7
+	release-1-0-2: 1.6
+	release-1-0-1: 1.5
+	release-1-0: 1.5
+	start: 1.1.1.1
+	ole: 1.1.1
+keyword substitution: kv
+total revisions: 38;	selected revisions: 38
+description:
+----------------------------
+revision 1.37
+date: 2004-01-07 11:42:07 +0000;  author: ole;  state: Exp;  lines: +2 -2
+Nothing
+----------------------------
+revision 1.36
+date: 2004-01-07 11:39:32 +0000;  author: ole;  state: Exp;  lines: +3 -1
+Nothing
+----------------------------
+revision 1.35
+date: 2004-01-07 11:36:17 +0000;  author: ole;  state: Exp;  lines: +15 -3
+Changed naming conventions to be more consistent
+----------------------------
+revision 1.34
+date: 2003-12-07 10:49:56 +0000;  author: ole;  state: Exp;  lines: +4 -3
+*** empty log message ***
+----------------------------
+revision 1.33
+date: 2003-12-03 10:13:58 +0000;  author: ole;  state: Exp;  lines: +2 -6
+Updated doc for obsoletion of raw forms
+----------------------------
+revision 1.32
+date: 2003-11-19 04:18:15 +0000;  author: ole;  state: Exp;  lines: +4 -0
+Finalised obsoletion of raw forms
+----------------------------
+revision 1.31
+date: 2003-11-16 12:01:27 +0000;  author: ole;  state: Exp;  lines: +0 -7
+Nothing
+----------------------------
+revision 1.30
+date: 2003-11-16 10:04:48 +0000;  author: ole;  state: Exp;  lines: +13 -4
+Fixed up gather, scatter and reduce to automatically calculate buffer size. Also cleaned up control_info. Updated web site and release script
+----------------------------
+revision 1.29
+date: 2003-11-13 11:36:04 +0000;  author: ole;  state: Exp;  lines: +18 -6
+Added support for the complex datatype
+----------------------------
+revision 1.28
+date: 2003-11-11 11:05:12 +0000;  author: ole;  state: Exp;  lines: +6 -5
+Dates and cleaned up old comment
+----------------------------
+revision 1.27
+date: 2003-11-11 10:57:37 +0000;  author: ole;  state: Exp;  lines: +6 -1
+Status object optionally returned from receive. Reserved control_tag, trimmed string buffers to actual length. Minor clean up
+----------------------------
+revision 1.26
+date: 2003-11-07 03:29:04 +0000;  author: ole;  state: Exp;  lines: +45 -23
+Added support for arrays of arbitray dimension
+----------------------------
+revision 1.25
+date: 2003-10-09 02:03:53 +0000;  author: ole;  state: Exp;  lines: +5 -0
+darwin fix, error msg fix
+----------------------------
+revision 1.24
+date: 2003-08-30 10:17:02 +0000;  author: ole;  state: Exp;  lines: +29 -12
+Work towards version 1.6.5 -
+Fixed Null argument needed in commandline by LAM 7.0
+----------------------------
+revision 1.23
+date: 2003-01-10 04:32:21 +0000;  author: ole;  state: Exp;  lines: +2 -1
+Added __version__ etc
+----------------------------
+revision 1.22
+date: 2003-01-09 23:27:42 +0000;  author: ole;  state: Exp;  lines: +3 -2
+Nothing
+----------------------------
+revision 1.21
+date: 2003-01-09 23:24:51 +0000;  author: ole;  state: Exp;  lines: +5 -2
+x
+----------------------------
+revision 1.20
+date: 2002-11-12 03:44:31 +0000;  author: ole;  state: Exp;  lines: +3 -3
+Added status.bytes() to get number of bytes transmitted
+----------------------------
+revision 1.19
+date: 2002-10-29 03:40:51 +0000;  author: ole;  state: Exp;  lines: +4 -1
+*** empty log message ***
+----------------------------
+revision 1.18
+date: 2002-10-29 03:38:12 +0000;  author: ole;  state: Exp;  lines: +26 -11
+Added Windows installation from Simon Frost
+----------------------------
+revision 1.17
+date: 2002-10-18 04:24:03 +0000;  author: ole;  state: Exp;  lines: +2 -4
+*** empty log message ***
+----------------------------
+revision 1.16
+date: 2002-10-18 04:20:42 +0000;  author: ole;  state: Exp;  lines: +39 -16
+doc
+----------------------------
+revision 1.15
+date: 2002-05-01 08:50:54 +0000;  author: ole;  state: Exp;  lines: +1 -1
+x
+----------------------------
+revision 1.14
+date: 2002-05-01 08:43:07 +0000;  author: ole;  state: Exp;  lines: +17 -9
+Updates of feature list
+----------------------------
+revision 1.13
+date: 2002-04-30 04:57:21 +0000;  author: ole;  state: Exp;  lines: +11 -4
+Doc
+----------------------------
+revision 1.12
+date: 2002-03-04 02:44:36 +0000;  author: ole;  state: Exp;  lines: +1 -1
+x
+----------------------------
+revision 1.11
+date: 2002-03-04 02:41:06 +0000;  author: ole;  state: Exp;  lines: +3 -1
+x
+----------------------------
+revision 1.10
+date: 2002-02-21 04:24:40 +0000;  author: ole;  state: Exp;  lines: +6 -4
+ver 1.3
+----------------------------
+revision 1.9
+date: 2002-02-17 22:40:10 +0000;  author: ole;  state: Exp;  lines: +20 -2
+x
+----------------------------
+revision 1.8
+date: 2002-02-17 04:58:37 +0000;  author: ole;  state: Exp;  lines: +4 -3
+minor
+----------------------------
+revision 1.7
+date: 2002-02-14 07:20:40 +0000;  author: ole;  state: Exp;  lines: +15 -4
+Added bcast - thanks to Gian Paolo Ciceri
+----------------------------
+revision 1.6
+date: 2002-02-10 23:05:31 +0000;  author: ole;  state: Exp;  lines: +32 -23
+Version 1.0.2
+Bugfix from Gian Paulo Ciceri preventing pypar from running under Python 2.2
+----------------------------
+revision 1.5
+date: 2002-02-07 01:58:31 +0000;  author: ole;  state: Exp;  lines: +3 -1
+*** empty log message ***
+----------------------------
+revision 1.4
+date: 2002-02-07 01:49:42 +0000;  author: ole;  state: Exp;  lines: +21 -0
+*** empty log message ***
+----------------------------
+revision 1.3
+date: 2002-02-07 01:33:16 +0000;  author: ole;  state: Exp;  lines: +27 -37
+*** empty log message ***
+----------------------------
+revision 1.2
+date: 2002-02-07 00:49:05 +0000;  author: ole;  state: Exp;  lines: +16 -10
+x
+----------------------------
+revision 1.1
+date: 2002-01-22 03:28:17 +0000;  author: ole;  state: Exp;
+branches:  1.1.1;
+Initial revision
+----------------------------
+revision 1.1.1.1
+date: 2002-01-22 03:28:17 +0000;  author: ole;  state: Exp;  lines: +0 -0
+Initial CVS import
+=============================================================================
+
+RCS file: /home/cvs/pypar/__init__.py,v
+Working file: __init__.py
+head: 1.7
+branch:
+locks: strict
+access list:
+symbolic names:
+	release-1-9-2: 1.7
+	release-1-9-1: 1.7
+	release-1-9: 1.7
+	release-1-8-2: 1.7
+	release-1-8-1: 1.7
+	release-1-8: 1.7
+	release-1-7: 1.7
+	release-1-6-5: 1.7
+	release-1-6-4: 1.7
+	release-1-6-3-1: 1.6
+	release-1-6-3: 1.5
+	SG-release-1-1: 1.4
+	SG-release-1-0: 1.4
+	release-0-1: 1.4
+	release-1-6-2: 1.4
+	release-1-6-1: 1.3
+	release-1-6-0: 1.3
+	release-1-5-4: 1.3
+	release-1-5-3: 1.3
+	release-1-5-2: 1.3
+	release-1-5-1: 1.3
+	release-1-5: 1.3
+	release-1-4-1: 1.3
+	release-1-4-0: 1.3
+	release-1-4: 1.3
+	release-1-3: 1.3
+	release-1-2-4: 1.3
+	release-1-2-3: 1.3
+	release-1-2-2: 1.3
+	revision-1-2-1: 1.3
+	release-1-2: 1.3
+	release-1-1: 1.3
+	release-1-0-2: 1.3
+	release-1-0-1: 1.3
+	release-1-0: 1.3
+keyword substitution: kv
+total revisions: 7;	selected revisions: 7
+description:
+----------------------------
+revision 1.7
+date: 2003-01-10 04:32:21 +0000;  author: ole;  state: Exp;  lines: +3 -0
+Added __version__ etc
+----------------------------
+revision 1.6
+date: 2003-01-09 23:24:51 +0000;  author: ole;  state: Exp;  lines: +5 -2
+x
+----------------------------
+revision 1.5
+date: 2003-01-07 22:54:30 +0000;  author: ole;  state: Exp;  lines: +2 -2
+Import all funtions in __init__
+----------------------------
+revision 1.4
+date: 2002-10-22 07:12:38 +0000;  author: ole;  state: Exp;  lines: +4 -1
+Fixed use as a package (I think)
+----------------------------
+revision 1.3
+date: 2002-02-06 07:00:22 +0000;  author: ole;  state: Exp;  lines: +2 -0
+*** empty log message ***
+----------------------------
+revision 1.2
+date: 2002-01-23 05:54:13 +0000;  author: ole;  state: Exp;  lines: +1 -0
+Modified codes to work with Python packages
+----------------------------
+revision 1.1
+date: 2002-01-23 03:54:44 +0000;  author: ole;  state: Exp;
+Added to work as a Python module
+=============================================================================
+
+RCS file: /home/cvs/pypar/compile.py,v
+Working file: compile.py
+head: 1.6
+branch:
+locks: strict
+access list:
+symbolic names:
+	release-1-9-2: 1.6
+	release-1-9-1: 1.6
+	release-1-9: 1.6
+	release-1-8-2: 1.6
+	release-1-8-1: 1.6
+	release-1-8: 1.6
+	release-1-7: 1.6
+	release-1-6-5: 1.6
+	release-1-6-4: 1.6
+	release-1-6-3-1: 1.6
+	release-1-6-3: 1.6
+	SG-release-1-1: 1.6
+	SG-release-1-0: 1.6
+	release-0-1: 1.6
+	release-1-6-2: 1.6
+	release-1-6-1: 1.6
+	release-1-6-0: 1.6
+	release-1-5-4: 1.6
+	release-1-5-3: 1.6
+	release-1-5-2: 1.5
+	release-1-5-1: 1.5
+	release-1-5: 1.5
+	release-1-4-1: 1.5
+	release-1-4-0: 1.4
+	release-1-4: 1.4
+	release-1-3: 1.4
+	release-1-2-4: 1.4
+	release-1-2-3: 1.4
+	release-1-2-2: 1.4
+	revision-1-2-1: 1.4
+	release-1-2: 1.4
+	release-1-1: 1.4
+	release-1-0-2: 1.4
+	release-1-0-1: 1.4
+	release-1-0: 1.3
+	start: 1.1.1.1
+	ole: 1.1.1
+keyword substitution: kv
+total revisions: 7;	selected revisions: 7
+description:
+----------------------------
+revision 1.6
+date: 2002-09-17 06:33:33 +0000;  author: ole;  state: Exp;  lines: +1 -1
+Auto compilation in packages
+----------------------------
+revision 1.5
+date: 2002-03-19 01:06:04 +0000;  author: ole;  state: Exp;  lines: +14 -0
+Added support for 'darwin' MAC OS X, BSD Unix for Macintosh
+(Thanks to Gavin Huttley)
+----------------------------
+revision 1.4
+date: 2002-02-08 03:58:58 +0000;  author: ole;  state: Exp;  lines: +10 -5
+Got it to run on a Sun Enterprise server under Mpich.
+Only small changes.
+----------------------------
+revision 1.3
+date: 2002-01-23 03:58:13 +0000;  author: ole;  state: Exp;  lines: +42 -28
+New multiple version of compile.py from pytools
+----------------------------
+revision 1.2
+date: 2002-01-23 00:42:25 +0000;  author: ole;  state: Exp;  lines: +21 -2
+*** empty log message ***
+----------------------------
+revision 1.1
+date: 2002-01-22 03:28:17 +0000;  author: ole;  state: Exp;
+branches:  1.1.1;
+Initial revision
+----------------------------
+revision 1.1.1.1
+date: 2002-01-22 03:28:17 +0000;  author: ole;  state: Exp;  lines: +0 -0
+Initial CVS import
+=============================================================================
+
+RCS file: /home/cvs/pypar/ctiming.c,v
+Working file: ctiming.c
+head: 1.4
+branch:
+locks: strict
+access list:
+symbolic names:
+	release-1-9-2: 1.4
+	release-1-9-1: 1.4
+	release-1-9: 1.4
+	release-1-8-2: 1.4
+	release-1-8-1: 1.4
+	release-1-8: 1.4
+	release-1-7: 1.4
+	release-1-6-5: 1.4
+	release-1-6-4: 1.4
+	release-1-6-3-1: 1.4
+	release-1-6-3: 1.4
+	SG-release-1-1: 1.4
+	SG-release-1-0: 1.4
+	release-0-1: 1.4
+	release-1-6-2: 1.4
+	release-1-6-1: 1.4
+	release-1-6-0: 1.4
+	release-1-5-4: 1.4
+	release-1-5-3: 1.4
+	release-1-5-2: 1.4
+	release-1-5-1: 1.4
+	release-1-5: 1.4
+	release-1-4-1: 1.4
+	release-1-4-0: 1.4
+	release-1-4: 1.4
+	release-1-3: 1.4
+	release-1-2-4: 1.4
+	release-1-2-3: 1.4
+	release-1-2-2: 1.4
+	revision-1-2-1: 1.4
+	release-1-2: 1.4
+	release-1-1: 1.4
+	release-1-0-2: 1.4
+	release-1-0-1: 1.4
+	release-1-0: 1.4
+	start: 1.1.1.1
+	ole: 1.1.1
+keyword substitution: kv
+total revisions: 5;	selected revisions: 5
+description:
+----------------------------
+revision 1.4
+date: 2002-02-06 23:54:55 +0000;  author: ole;  state: Exp;  lines: +4 -0
+Streamlined interface
+----------------------------
+revision 1.3
+date: 2002-02-06 05:40:28 +0000;  author: ole;  state: Exp;  lines: +3 -2
+x
+----------------------------
+revision 1.2
+date: 2002-02-06 05:28:51 +0000;  author: ole;  state: Exp;  lines: +10 -9
+Made output from pytimimng and ctiming more userfriendly
+----------------------------
+revision 1.1
+date: 2002-01-22 03:28:17 +0000;  author: ole;  state: Exp;
+branches:  1.1.1;
+Initial revision
+----------------------------
+revision 1.1.1.1
+date: 2002-01-22 03:28:17 +0000;  author: ole;  state: Exp;  lines: +0 -0
+Initial CVS import
+=============================================================================
+
+RCS file: /home/cvs/pypar/demo.py,v
+Working file: demo.py
+head: 1.5
+branch:
+locks: strict
+access list:
+symbolic names:
+	release-1-9-2: 1.5
+	release-1-9-1: 1.5
+	release-1-9: 1.4
+	release-1-8-2: 1.4
+	release-1-8-1: 1.4
+	release-1-8: 1.4
+	release-1-7: 1.3
+	release-1-6-5: 1.3
+	release-1-6-4: 1.3
+	release-1-6-3-1: 1.3
+	release-1-6-3: 1.3
+	SG-release-1-1: 1.3
+	SG-release-1-0: 1.1
+	release-0-1: 1.1
+	release-1-6-2: 1.1
+	release-1-6-1: 1.1
+	release-1-6-0: 1.1
+	release-1-5-4: 1.1
+	release-1-5-3: 1.1
+	release-1-5-2: 1.1
+	release-1-5-1: 1.1
+	release-1-5: 1.1
+	release-1-4-1: 1.1
+	release-1-4-0: 1.1
+	release-1-4: 1.1
+	release-1-3: 1.1
+	release-1-2-4: 1.1
+	release-1-2-3: 1.1
+	release-1-2-2: 1.1
+	revision-1-2-1: 1.1
+	release-1-2: 1.1
+	release-1-1: 1.1
+	release-1-0-2: 1.1
+	release-1-0-1: 1.1
+	release-1-0: 1.1
+keyword substitution: kv
+total revisions: 5;	selected revisions: 5
+description:
+----------------------------
+revision 1.5
+date: 2004-01-07 11:36:17 +0000;  author: ole;  state: Exp;  lines: +2 -2
+Changed naming conventions to be more consistent
+----------------------------
+revision 1.4
+date: 2003-11-11 10:57:37 +0000;  author: ole;  state: Exp;  lines: +5 -4
+Status object optionally returned from receive. Reserved control_tag, trimmed string buffers to actual length. Minor clean up
+----------------------------
+revision 1.3
+date: 2002-11-12 04:20:40 +0000;  author: ole;  state: Exp;  lines: +2 -2
+Nothing
+----------------------------
+revision 1.2
+date: 2002-11-12 03:44:31 +0000;  author: ole;  state: Exp;  lines: +6 -1
+Added status.bytes() to get number of bytes transmitted
+----------------------------
+revision 1.1
+date: 2002-02-07 00:49:05 +0000;  author: ole;  state: Exp;
+x
+=============================================================================
+
+RCS file: /home/cvs/pypar/Attic/demo1.py,v
+Working file: demo1.py
+head: 1.3
+branch:
+locks: strict
+access list:
+symbolic names:
+	start: 1.1.1.1
+	ole: 1.1.1
+keyword substitution: kv
+total revisions: 4;	selected revisions: 4
+description:
+----------------------------
+revision 1.3
+date: 2002-02-07 00:49:05 +0000;  author: ole;  state: dead;  lines: +0 -0
+x
+----------------------------
+revision 1.2
+date: 2002-02-06 04:55:04 +0000;  author: ole;  state: Exp;  lines: +2 -2
+Fixed differences between Alphaserver and LAM/Linux
+----------------------------
+revision 1.1
+date: 2002-01-22 03:28:17 +0000;  author: ole;  state: Exp;
+branches:  1.1.1;
+Initial revision
+----------------------------
+revision 1.1.1.1
+date: 2002-01-22 03:28:17 +0000;  author: ole;  state: Exp;  lines: +0 -0
+Initial CVS import
+=============================================================================
+
+RCS file: /home/cvs/pypar/demo2.py,v
+Working file: demo2.py
+head: 1.12
+branch:
+locks: strict
+access list:
+symbolic names:
+	release-1-9-2: 1.12
+	release-1-9-1: 1.12
+	release-1-9: 1.11
+	release-1-8-2: 1.11
+	release-1-8-1: 1.11
+	release-1-8: 1.11
+	release-1-7: 1.11
+	release-1-6-5: 1.11
+	release-1-6-4: 1.11
+	release-1-6-3-1: 1.11
+	release-1-6-3: 1.11
+	SG-release-1-1: 1.11
+	SG-release-1-0: 1.11
+	release-1-1: 1.11
+	release-0-1: 1.11
+	release-1-6-2: 1.11
+	release-1-6-1: 1.11
+	release-1-6-0: 1.11
+	release-1-5-4: 1.11
+	release-1-5-3: 1.11
+	release-1-5-2: 1.11
+	release-1-5-1: 1.11
+	release-1-5: 1.11
+	release-1-4-1: 1.8
+	release-1-4-0: 1.4
+	release-1-4: 1.4
+	release-1-3: 1.4
+	release-1-2-4: 1.4
+	release-1-2-3: 1.4
+	release-1-2-2: 1.3
+	revision-1-2-1: 1.3
+	start: 1.1.1.1
+	ole: 1.1.1
+keyword substitution: kv
+total revisions: 13;	selected revisions: 13
+description:
+----------------------------
+revision 1.12
+date: 2004-01-07 11:36:17 +0000;  author: ole;  state: Exp;  lines: +2 -2
+Changed naming conventions to be more consistent
+----------------------------
+revision 1.11
+date: 2002-04-08 05:23:48 +0000;  author: ole;  state: Exp;  lines: +1 -1
+x
+----------------------------
+revision 1.10
+date: 2002-04-08 05:22:32 +0000;  author: ole;  state: Exp;  lines: +3 -2
+x
+----------------------------
+revision 1.9
+date: 2002-04-08 05:19:36 +0000;  author: ole;  state: Exp;  lines: +10 -4
+Added more reporting and pypar.Finalize()
+----------------------------
+revision 1.8
+date: 2002-03-22 05:40:05 +0000;  author: ole;  state: Exp;  lines: +4 -2
+Finished
+----------------------------
+revision 1.7
+date: 2002-03-22 02:36:08 +0000;  author: ole;  state: Exp;  lines: +21 -13
+Finalised demo2
+----------------------------
+revision 1.6
+date: 2002-03-22 00:52:51 +0000;  author: ole;  state: Exp;  lines: +9 -1
+x
+----------------------------
+revision 1.5
+date: 2002-03-22 00:50:11 +0000;  author: ole;  state: Exp;  lines: +71 -114
+Added simple demo for work distribution (demo2)
+Ole demo2 moved to dem3
+----------------------------
+revision 1.4
+date: 2002-02-17 04:58:37 +0000;  author: ole;  state: Exp;  lines: +42 -26
+minor
+----------------------------
+revision 1.3
+date: 2002-02-16 00:21:24 +0000;  author: ole;  state: Exp;  lines: +107 -55
+Status block
+----------------------------
+revision 1.2
+date: 2002-02-07 00:49:05 +0000;  author: ole;  state: dead;  lines: +0 -0
+x
+----------------------------
+revision 1.1
+date: 2002-01-22 03:28:17 +0000;  author: ole;  state: Exp;
+branches:  1.1.1;
+Initial revision
+----------------------------
+revision 1.1.1.1
+date: 2002-01-22 03:28:17 +0000;  author: ole;  state: Exp;  lines: +0 -0
+Initial CVS import
+=============================================================================
+
+RCS file: /home/cvs/pypar/demo3.py,v
+Working file: demo3.py
+head: 1.3
+branch:
+locks: strict
+access list:
+symbolic names:
+	release-1-9-2: 1.3
+	release-1-9-1: 1.3
+	release-1-9: 1.3
+	release-1-8-2: 1.3
+	release-1-8-1: 1.3
+	release-1-8: 1.3
+	release-1-7: 1.2
+	release-1-6-5: 1.2
+	release-1-6-4: 1.2
+	release-1-6-3-1: 1.2
+	release-1-6-3: 1.2
+	SG-release-1-1: 1.2
+	SG-release-1-0: 1.1
+	release-1-1: 1.1
+	release-0-1: 1.1
+	release-1-6-2: 1.1
+	release-1-6-1: 1.1
+	release-1-6-0: 1.1
+	release-1-5-4: 1.1
+	release-1-5-3: 1.1
+	release-1-5-2: 1.1
+	release-1-5-1: 1.1
+	release-1-5: 1.1
+	release-1-4-1: 1.1
+keyword substitution: kv
+total revisions: 3;	selected revisions: 3
+description:
+----------------------------
+revision 1.3
+date: 2003-11-11 10:57:37 +0000;  author: ole;  state: Exp;  lines: +9 -9
+Status object optionally returned from receive. Reserved control_tag, trimmed string buffers to actual length. Minor clean up
+----------------------------
+revision 1.2
+date: 2002-11-12 03:44:31 +0000;  author: ole;  state: Exp;  lines: +2 -2
+Added status.bytes() to get number of bytes transmitted
+----------------------------
+revision 1.1
+date: 2002-03-22 00:50:11 +0000;  author: ole;  state: Exp;
+Added simple demo for work distribution (demo2)
+Ole demo2 moved to dem3
+=============================================================================
+
+RCS file: /home/cvs/pypar/demo4.py,v
+Working file: demo4.py
+head: 1.3
+branch:
+locks: strict
+access list:
+symbolic names:
+	release-1-9-2: 1.3
+	release-1-9-1: 1.3
+	release-1-9: 1.3
+	release-1-8-2: 1.3
+	release-1-8-1: 1.3
+	release-1-8: 1.3
+	release-1-7: 1.2
+	release-1-6-5: 1.2
+	release-1-6-4: 1.2
+	release-1-6-3-1: 1.2
+	release-1-6-3: 1.2
+	SG-release-1-1: 1.2
+	SG-release-1-0: 1.2
+keyword substitution: kv
+total revisions: 3;	selected revisions: 3
+description:
+----------------------------
+revision 1.3
+date: 2003-11-11 10:57:37 +0000;  author: ole;  state: Exp;  lines: +4 -4
+Status object optionally returned from receive. Reserved control_tag, trimmed string buffers to actual length. Minor clean up
+----------------------------
+revision 1.2
+date: 2002-11-12 04:20:40 +0000;  author: ole;  state: Exp;  lines: +2 -2
+Nothing
+----------------------------
+revision 1.1
+date: 2002-11-12 03:44:31 +0000;  author: ole;  state: Exp;
+Added status.bytes() to get number of bytes transmitted
+=============================================================================
+
+RCS file: /home/cvs/pypar/distribute_files,v
+Working file: distribute_files
+head: 1.1
+branch:
+locks: strict
+access list:
+symbolic names:
+	release-1-9-2: 1.1
+	release-1-9-1: 1.1
+	release-1-9: 1.1
+	release-1-8-2: 1.1
+	release-1-8-1: 1.1
+	release-1-8: 1.1
+	release-1-7: 1.1
+	release-1-6-5: 1.1
+	release-1-6-4: 1.1
+keyword substitution: kv
+total revisions: 1;	selected revisions: 1
+description:
+----------------------------
+revision 1.1
+date: 2003-02-11 04:19:40 +0000;  author: ole;  state: Exp;
+Addede script for distributing files and publishing on the web.
+=============================================================================
+
+RCS file: /home/cvs/pypar/install.py,v
+Working file: install.py
+head: 1.8
+branch:
+locks: strict
+access list:
+symbolic names:
+	release-1-9-2: 1.8
+	release-1-9-1: 1.8
+	release-1-9: 1.8
+	release-1-8-2: 1.8
+	release-1-8-1: 1.8
+	release-1-8: 1.8
+	release-1-7: 1.8
+	release-1-6-5: 1.8
+	release-1-6-4: 1.8
+	release-1-6-3-1: 1.8
+	release-1-6-3: 1.8
+	SG-release-1-1: 1.7
+	SG-release-1-0: 1.7
+	release-0-1: 1.7
+	release-1-6-2: 1.7
+	release-1-6-1: 1.7
+	release-1-6-0: 1.7
+	release-1-5-4: 1.7
+	release-1-5-3: 1.7
+	release-1-5-2: 1.7
+	release-1-5-1: 1.6
+	release-1-5: 1.6
+	release-1-4-1: 1.5
+	release-1-4-0: 1.5
+	release-1-4: 1.5
+	release-1-3: 1.5
+	release-1-2-4: 1.5
+	release-1-2-3: 1.5
+	release-1-2-2: 1.5
+	revision-1-2-1: 1.5
+	release-1-2: 1.5
+	release-1-1: 1.5
+	release-1-0-2: 1.5
+	release-1-0-1: 1.5
+	release-1-0: 1.4
+	start: 1.1.1.1
+	ole: 1.1.1
+keyword substitution: kv
+total revisions: 9;	selected revisions: 9
+description:
+----------------------------
+revision 1.8
+date: 2002-12-18 03:10:35 +0000;  author: ole;  state: Exp;  lines: +1 -1
+Fixed reintroduced problem about hanging under LAM linux.
+It is not possible to call init in a separate shell and then later within
+the python script.
+----------------------------
+revision 1.7
+date: 2002-06-18 07:13:56 +0000;  author: ole;  state: Exp;  lines: +7 -1
+Fixed new problem on Alphaserver. It no longer supports mpicc, so
+cc -lmpi must be used instead (grrr)
+----------------------------
+revision 1.6
+date: 2002-04-30 04:33:47 +0000;  author: ole;  state: Exp;  lines: +6 -6
+Changed extension module from mpi.c to mpiext.c
+(import mpi caused Alphaserver to crash if not present).
+----------------------------
+revision 1.5
+date: 2002-02-08 03:58:58 +0000;  author: ole;  state: Exp;  lines: +2 -2
+Got it to run on a Sun Enterprise server under Mpich.
+Only small changes.
+----------------------------
+revision 1.4
+date: 2002-01-23 06:48:20 +0000;  author: ole;  state: Exp;  lines: +13 -26
+*** empty log message ***
+----------------------------
+revision 1.3
+date: 2002-01-23 04:29:28 +0000;  author: ole;  state: Exp;  lines: +1 -5
+
+Brought back sequential stub
+----------------------------
+revision 1.2
+date: 2002-01-23 00:29:57 +0000;  author: ole;  state: Exp;  lines: +3 -34
+Simplified
+----------------------------
+revision 1.1
+date: 2002-01-22 03:28:17 +0000;  author: ole;  state: Exp;
+branches:  1.1.1;
+Initial revision
+----------------------------
+revision 1.1.1.1
+date: 2002-01-22 03:28:17 +0000;  author: ole;  state: Exp;  lines: +0 -0
+Initial CVS import
+=============================================================================
+
+RCS file: /home/cvs/pypar/Attic/mpi.c,v
+Working file: mpi.c
+head: 1.22
+branch:
+locks: strict
+access list:
+symbolic names:
+	release-1-4-1: 1.20
+	release-1-4-0: 1.20
+	release-1-4: 1.19
+	release-1-3: 1.18
+	release-1-2-4: 1.15
+	release-1-2-3: 1.14
+	release-1-2-2: 1.13
+	revision-1-2-1: 1.13
+	release-1-2: 1.11
+	release-1-1: 1.10
+	release-1-0-2: 1.9
+	release-1-0-1: 1.8
+	release-1-0: 1.7
+	start: 1.1.1.1
+	ole: 1.1.1
+keyword substitution: kv
+total revisions: 23;	selected revisions: 23
+description:
+----------------------------
+revision 1.22
+date: 2002-04-30 04:15:58 +0000;  author: ole;  state: dead;  lines: +0 -0
+*** empty log message ***
+----------------------------
+revision 1.21
+date: 2002-04-30 03:56:49 +0000;  author: ole;  state: Exp;  lines: +36 -11
+Modified init so that python command line gets passed onto MPI_init.
+This is necessary for MPICH/Linux.
+----------------------------
+revision 1.20
+date: 2002-03-04 02:41:06 +0000;  author: ole;  state: Exp;  lines: +1 -1
+x
+----------------------------
+revision 1.19
+date: 2002-02-27 00:57:12 +0000;  author: ole;  state: Exp;  lines: +4 -0
+Fixed test and ran with 22 procs on SUN
+----------------------------
+revision 1.18
+date: 2002-02-21 04:24:40 +0000;  author: ole;  state: Exp;  lines: +1 -1
+ver 1.3
+----------------------------
+revision 1.17
+date: 2002-02-21 03:53:09 +0000;  author: ole;  state: Exp;  lines: +4 -4
+Removed REPLACE
+----------------------------
+revision 1.16
+date: 2002-02-21 03:48:22 +0000;  author: ole;  state: Exp;  lines: +162 -0
+added gather and reduce
+----------------------------
+revision 1.15
+date: 2002-02-19 07:45:44 +0000;  author: ole;  state: Exp;  lines: +4 -2
+Fixed crash on Sun Enterprise (alloc **argv)
+----------------------------
+revision 1.14
+date: 2002-02-17 22:40:10 +0000;  author: ole;  state: Exp;  lines: +1 -12
+x
+----------------------------
+revision 1.13
+date: 2002-02-16 00:40:55 +0000;  author: ole;  state: Exp;  lines: +13 -4
+Fixed st_length
+----------------------------
+revision 1.12
+date: 2002-02-16 00:21:24 +0000;  author: ole;  state: Exp;  lines: +21 -3
+Status block
+----------------------------
+revision 1.11
+date: 2002-02-15 22:46:55 +0000;  author: ole;  state: Exp;  lines: +51 -0
+Scatter
+----------------------------
+revision 1.10
+date: 2002-02-14 07:20:40 +0000;  author: ole;  state: Exp;  lines: +108 -99
+Added bcast - thanks to Gian Paolo Ciceri
+----------------------------
+revision 1.9
+date: 2002-02-10 23:05:31 +0000;  author: ole;  state: Exp;  lines: +1 -1
+Version 1.0.2
+Bugfix from Gian Paulo Ciceri preventing pypar from running under Python 2.2
+----------------------------
+revision 1.8
+date: 2002-02-08 03:58:58 +0000;  author: ole;  state: Exp;  lines: +1 -1
+Got it to run on a Sun Enterprise server under Mpich.
+Only small changes.
+----------------------------
+revision 1.7
+date: 2002-02-07 01:49:42 +0000;  author: ole;  state: Exp;  lines: +26 -8
+*** empty log message ***
+----------------------------
+revision 1.6
+date: 2002-02-06 04:55:04 +0000;  author: ole;  state: Exp;  lines: +1 -0
+Fixed differences between Alphaserver and LAM/Linux
+----------------------------
+revision 1.5
+date: 2002-02-06 04:18:07 +0000;  author: ole;  state: Exp;  lines: +1 -1
+x
+----------------------------
+revision 1.4
+date: 2002-02-06 04:14:50 +0000;  author: ole;  state: Exp;  lines: +38 -36
+Moved type_map to beginning of file to avaoid irritating warning about
+implicit declaration.
+----------------------------
+revision 1.3
+date: 2002-02-06 04:10:57 +0000;  author: ole;  state: Exp;  lines: +5 -5
+*** empty log message ***
+----------------------------
+revision 1.2
+date: 2002-02-06 03:23:51 +0000;  author: ole;  state: Exp;  lines: +5 -5
+Test program to figure out MPI types
+----------------------------
+revision 1.1
+date: 2002-01-22 03:28:17 +0000;  author: ole;  state: Exp;
+branches:  1.1.1;
+Initial revision
+----------------------------
+revision 1.1.1.1
+date: 2002-01-22 03:28:17 +0000;  author: ole;  state: Exp;  lines: +0 -0
+Initial CVS import
+=============================================================================
+
+RCS file: /home/cvs/pypar/mpiext.c,v
+Working file: mpiext.c
+head: 1.18
+branch:
+locks: strict
+access list:
+symbolic names:
+	release-1-9-2: 1.18
+	release-1-9-1: 1.18
+	release-1-9: 1.17
+	release-1-8-2: 1.15
+	release-1-8-1: 1.14
+	release-1-8: 1.13
+	release-1-7: 1.11
+	release-1-6-5: 1.10
+	release-1-6-4: 1.9
+	release-1-6-3-1: 1.7
+	release-1-6-3: 1.6
+	SG-release-1-1: 1.5
+	SG-release-1-0: 1.4
+	release-1-1: 1.4
+	release-0-1: 1.4
+	release-1-6-2: 1.4
+	release-1-6-1: 1.4
+	release-1-6-0: 1.4
+	release-1-5-4: 1.4
+	release-1-5-3: 1.4
+	release-1-5-2: 1.3
+	release-1-5-1: 1.3
+	release-1-5: 1.3
+keyword substitution: kv
+total revisions: 18;	selected revisions: 18
+description:
+----------------------------
+revision 1.18
+date: 2004-01-07 11:36:17 +0000;  author: ole;  state: Exp;  lines: +27 -21
+Changed naming conventions to be more consistent
+----------------------------
+revision 1.17
+date: 2003-11-20 10:08:12 +0000;  author: ole;  state: Exp;  lines: +10 -4
+Introduced bypass argument for efficiency
+----------------------------
+revision 1.16
+date: 2003-11-17 12:03:18 +0000;  author: ole;  state: Exp;  lines: +200 -65
+Work towards ver 2.0: Exceptions moved to c-extension, revamping of control_info, lots of clean-up
+----------------------------
+revision 1.15
+date: 2003-11-16 10:04:48 +0000;  author: ole;  state: Exp;  lines: +23 -22
+Fixed up gather, scatter and reduce to automatically calculate buffer size. Also cleaned up control_info. Updated web site and release script
+----------------------------
+revision 1.14
+date: 2003-11-13 11:36:04 +0000;  author: ole;  state: Exp;  lines: +71 -44
+Added support for the complex datatype
+----------------------------
+revision 1.13
+date: 2003-11-11 11:05:12 +0000;  author: ole;  state: Exp;  lines: +0 -5
+Dates and cleaned up old comment
+----------------------------
+revision 1.12
+date: 2003-11-11 10:57:37 +0000;  author: ole;  state: Exp;  lines: +21 -11
+Status object optionally returned from receive. Reserved control_tag, trimmed string buffers to actual length. Minor clean up
+----------------------------
+revision 1.11
+date: 2003-11-07 03:29:04 +0000;  author: ole;  state: Exp;  lines: +25 -8
+Added support for arrays of arbitray dimension
+----------------------------
+revision 1.10
+date: 2003-08-30 10:17:02 +0000;  author: ole;  state: Exp;  lines: +3 -1
+Work towards version 1.6.5 -
+Fixed Null argument needed in commandline by LAM 7.0
+----------------------------
+revision 1.9
+date: 2003-01-10 04:32:21 +0000;  author: ole;  state: Exp;  lines: +1 -1
+Added __version__ etc
+----------------------------
+revision 1.8
+date: 2003-01-09 23:27:42 +0000;  author: ole;  state: Exp;  lines: +1 -1
+Nothing
+----------------------------
+revision 1.7
+date: 2003-01-09 23:24:51 +0000;  author: ole;  state: Exp;  lines: +3 -3
+x
+----------------------------
+revision 1.6
+date: 2002-12-20 02:06:49 +0000;  author: ole;  state: Exp;  lines: +11 -0
+Studied problems with auto detetction of lam runtime
+----------------------------
+revision 1.5
+date: 2002-11-12 03:44:31 +0000;  author: ole;  state: Exp;  lines: +11 -5
+Added status.bytes() to get number of bytes transmitted
+----------------------------
+revision 1.4
+date: 2002-10-18 03:24:05 +0000;  author: ole;  state: Exp;  lines: +5 -80
+
+Removed auto compile from pypar
+Removed obsolte stuff from extension
+----------------------------
+revision 1.3
+date: 2002-04-30 05:02:08 +0000;  author: ole;  state: Exp;  lines: +1 -1
+Rev 1.5
+----------------------------
+revision 1.2
+date: 2002-04-30 04:33:47 +0000;  author: ole;  state: Exp;  lines: +2 -2
+Changed extension module from mpi.c to mpiext.c
+(import mpi caused Alphaserver to crash if not present).
+----------------------------
+revision 1.1
+date: 2002-04-30 04:15:58 +0000;  author: ole;  state: Exp;
+*** empty log message ***
+=============================================================================
+
+RCS file: /home/cvs/pypar/Attic/mpitypetest.c,v
+Working file: mpitypetest.c
+head: 1.3
+branch:
+locks: strict
+access list:
+symbolic names:
+keyword substitution: kv
+total revisions: 3;	selected revisions: 3
+description:
+----------------------------
+revision 1.3
+date: 2002-02-06 06:58:58 +0000;  author: ole;  state: dead;  lines: +0 -0
+*** empty log message ***
+----------------------------
+revision 1.2
+date: 2002-02-06 04:14:50 +0000;  author: ole;  state: Exp;  lines: +1 -1
+Moved type_map to beginning of file to avaoid irritating warning about
+implicit declaration.
+----------------------------
+revision 1.1
+date: 2002-02-06 03:23:51 +0000;  author: ole;  state: Exp;
+Test program to figure out MPI types
+=============================================================================
+
+RCS file: /home/cvs/pypar/pypar.py,v
+Working file: pypar.py
+head: 1.69
+branch:
+locks: strict
+access list:
+symbolic names:
+	release-1-9-2: 1.69
+	release-1-9-1: 1.67
+	release-1-9: 1.65
+	release-1-8-2: 1.61
+	release-1-8-1: 1.60
+	release-1-8: 1.59
+	release-1-7: 1.58
+	release-1-6-5: 1.56
+	release-1-6-4: 1.55
+	release-1-6-3-1: 1.51
+	release-1-6-3: 1.50
+	SG-release-1-1: 1.46
+	SG-release-1-0: 1.43
+	release-0-1: 1.43
+	release-1-6-2: 1.43
+	release-1-6-1: 1.38
+	release-1-6-0: 1.37
+	release-1-5-4: 1.37
+	release-1-5-3: 1.37
+	release-1-5-2: 1.33
+	release-1-5-1: 1.32
+	release-1-5: 1.32
+	release-1-4-1: 1.24
+	release-1-4-0: 1.24
+	release-1-4: 1.23
+	release-1-3: 1.22
+	release-1-2-4: 1.19
+	release-1-2-3: 1.18
+	release-1-2-2: 1.17
+	revision-1-2-1: 1.17
+	release-1-2: 1.16
+	release-1-1: 1.15
+	release-1-0-2: 1.14
+	release-1-0-1: 1.12
+	release-1-0: 1.11
+	start: 1.1.1.1
+	ole: 1.1.1
+keyword substitution: kv
+total revisions: 70;	selected revisions: 70
+description:
+----------------------------
+revision 1.69
+date: 2004-12-27 03:29:24 +0000;  author: ole;  state: Exp;  lines: +40 -0
+Doc for function balance
+----------------------------
+revision 1.68
+date: 2004-11-10 07:09:46 +0000;  author: ole;  state: Exp;  lines: +1 -1
+Fixed spelling error in init for alpha server
+----------------------------
+revision 1.67
+date: 2004-01-07 11:42:07 +0000;  author: ole;  state: Exp;  lines: +2 -2
+Nothing
+----------------------------
+revision 1.66
+date: 2004-01-07 11:36:17 +0000;  author: ole;  state: Exp;  lines: +105 -51
+Changed naming conventions to be more consistent
+----------------------------
+revision 1.65
+date: 2003-12-03 10:13:58 +0000;  author: ole;  state: Exp;  lines: +16 -23
+Updated doc for obsoletion of raw forms
+----------------------------
+revision 1.64
+date: 2003-11-20 10:08:12 +0000;  author: ole;  state: Exp;  lines: +65 -43
+Introduced bypass argument for efficiency
+----------------------------
+revision 1.63
+date: 2003-11-19 04:18:15 +0000;  author: ole;  state: Exp;  lines: +1 -1
+Finalised obsoletion of raw forms
+----------------------------
+revision 1.62
+date: 2003-11-17 12:03:18 +0000;  author: ole;  state: Exp;  lines: +490 -566
+Work towards ver 2.0: Exceptions moved to c-extension, revamping of control_info, lots of clean-up
+----------------------------
+revision 1.61
+date: 2003-11-16 10:04:48 +0000;  author: ole;  state: Exp;  lines: +41 -25
+Fixed up gather, scatter and reduce to automatically calculate buffer size. Also cleaned up control_info. Updated web site and release script
+----------------------------
+revision 1.60
+date: 2003-11-13 11:36:04 +0000;  author: ole;  state: Exp;  lines: +13 -10
+Added support for the complex datatype
+----------------------------
+revision 1.59
+date: 2003-11-11 10:57:37 +0000;  author: ole;  state: Exp;  lines: +79 -57
+Status object optionally returned from receive. Reserved control_tag, trimmed string buffers to actual length. Minor clean up
+----------------------------
+revision 1.58
+date: 2003-11-07 03:29:04 +0000;  author: ole;  state: Exp;  lines: +36 -22
+Added support for arrays of arbitray dimension
+----------------------------
+revision 1.57
+date: 2003-10-09 02:03:53 +0000;  author: ole;  state: Exp;  lines: +4 -4
+darwin fix, error msg fix
+----------------------------
+revision 1.56
+date: 2003-08-30 10:17:02 +0000;  author: ole;  state: Exp;  lines: +2 -2
+Work towards version 1.6.5 -
+Fixed Null argument needed in commandline by LAM 7.0
+----------------------------
+revision 1.55
+date: 2003-02-04 01:54:18 +0000;  author: ole;  state: Exp;  lines: +46 -41
+Changed argument names in gather, reduce, ... and updated doc
+----------------------------
+revision 1.54
+date: 2003-01-13 03:04:49 +0000;  author: ole;  state: Exp;  lines: +3 -2
+Added Ben's comments
+----------------------------
+revision 1.53
+date: 2003-01-10 04:32:21 +0000;  author: ole;  state: Exp;  lines: +7 -4
+Added __version__ etc
+----------------------------
+revision 1.52
+date: 2003-01-09 23:27:42 +0000;  author: ole;  state: Exp;  lines: +1 -1
+Nothing
+----------------------------
+revision 1.51
+date: 2003-01-09 23:24:51 +0000;  author: ole;  state: Exp;  lines: +1 -1
+x
+----------------------------
+revision 1.50
+date: 2002-12-20 02:06:49 +0000;  author: ole;  state: Exp;  lines: +30 -10
+Studied problems with auto detetction of lam runtime
+----------------------------
+revision 1.49
+date: 2002-12-18 04:46:38 +0000;  author: ole;  state: Exp;  lines: +1 -0
+*** empty log message ***
+----------------------------
+revision 1.48
+date: 2002-12-18 03:10:35 +0000;  author: ole;  state: Exp;  lines: +20 -12
+Fixed reintroduced problem about hanging under LAM linux.
+It is not possible to call init in a separate shell and then later within
+the python script.
+----------------------------
+revision 1.47
+date: 2002-12-13 04:26:32 +0000;  author: ole;  state: Exp;  lines: +6 -2
+Fast algorithm for evalf (not so fast yet)
+----------------------------
+revision 1.46
+date: 2002-11-12 04:33:06 +0000;  author: ole;  state: Exp;  lines: +1 -3
+Nothing
+----------------------------
+revision 1.45
+date: 2002-11-12 03:44:31 +0000;  author: ole;  state: Exp;  lines: +12 -6
+Added status.bytes() to get number of bytes transmitted
+----------------------------
+revision 1.44
+date: 2002-11-12 02:29:39 +0000;  author: ole;  state: Exp;  lines: +8 -3
+dirname = '.'
+----------------------------
+revision 1.43
+date: 2002-10-29 03:40:51 +0000;  author: ole;  state: Exp;  lines: +1 -1
+*** empty log message ***
+----------------------------
+revision 1.42
+date: 2002-10-29 03:38:12 +0000;  author: ole;  state: Exp;  lines: +5 -4
+Added Windows installation from Simon Frost
+----------------------------
+revision 1.41
+date: 2002-10-23 07:24:43 +0000;  author: ole;  state: Exp;  lines: +11 -3
+*** empty log message ***
+----------------------------
+revision 1.40
+date: 2002-10-23 06:50:37 +0000;  author: ole;  state: Exp;  lines: +6 -2
+*** empty log message ***
+----------------------------
+revision 1.39
+date: 2002-10-22 07:12:38 +0000;  author: ole;  state: Exp;  lines: +6 -12
+Fixed use as a package (I think)
+----------------------------
+revision 1.38
+date: 2002-10-18 04:43:04 +0000;  author: ole;  state: Exp;  lines: +4 -9
+*** empty log message ***
+----------------------------
+revision 1.37
+date: 2002-10-18 03:24:05 +0000;  author: ole;  state: Exp;  lines: +8 -18
+
+Removed auto compile from pypar
+Removed obsolte stuff from extension
+----------------------------
+revision 1.36
+date: 2002-09-17 06:46:55 +0000;  author: howard;  state: Exp;  lines: +2 -1
+*** empty log message ***
+----------------------------
+revision 1.35
+date: 2002-09-17 06:33:33 +0000;  author: ole;  state: Exp;  lines: +8 -9
+Auto compilation in packages
+----------------------------
+revision 1.34
+date: 2002-07-23 04:43:37 +0000;  author: ole;  state: Exp;  lines: +13 -1
+Entered code for handling situation where pypar is a package
+(Commented out)
+----------------------------
+revision 1.33
+date: 2002-06-18 07:13:56 +0000;  author: ole;  state: Exp;  lines: +10 -5
+Fixed new problem on Alphaserver. It no longer supports mpicc, so
+cc -lmpi must be used instead (grrr)
+----------------------------
+revision 1.32
+date: 2002-04-30 05:19:38 +0000;  author: ole;  state: Exp;  lines: +46 -46
+Fixed problem where init MPI crashes on Alpha Server
+----------------------------
+revision 1.31
+date: 2002-04-30 05:02:08 +0000;  author: ole;  state: Exp;  lines: +1 -1
+Rev 1.5
+----------------------------
+revision 1.30
+date: 2002-04-30 04:57:21 +0000;  author: ole;  state: Exp;  lines: +4 -4
+Doc
+----------------------------
+revision 1.29
+date: 2002-04-30 04:33:47 +0000;  author: ole;  state: Exp;  lines: +29 -36
+Changed extension module from mpi.c to mpiext.c
+(import mpi caused Alphaserver to crash if not present).
+----------------------------
+revision 1.28
+date: 2002-04-30 04:15:58 +0000;  author: ole;  state: Exp;  lines: +8 -35
+*** empty log message ***
+----------------------------
+revision 1.27
+date: 2002-04-30 03:56:49 +0000;  author: ole;  state: Exp;  lines: +7 -3
+Modified init so that python command line gets passed onto MPI_init.
+This is necessary for MPICH/Linux.
+----------------------------
+revision 1.26
+date: 2002-04-10 04:16:54 +0000;  author: ole;  state: Exp;  lines: +3 -1
+Added afiliation
+----------------------------
+revision 1.25
+date: 2002-04-04 06:48:16 +0000;  author: ole;  state: Exp;  lines: +3 -1
+Enabled pypar to find mpi.so in whatever dir pypar.py lives in
+----------------------------
+revision 1.24
+date: 2002-03-04 02:44:36 +0000;  author: ole;  state: Exp;  lines: +1 -1
+x
+----------------------------
+revision 1.23
+date: 2002-02-27 00:57:12 +0000;  author: ole;  state: Exp;  lines: +6 -3
+Fixed test and ran with 22 procs on SUN
+----------------------------
+revision 1.22
+date: 2002-02-21 04:24:40 +0000;  author: ole;  state: Exp;  lines: +1 -1
+ver 1.3
+----------------------------
+revision 1.21
+date: 2002-02-21 03:53:09 +0000;  author: ole;  state: Exp;  lines: +1 -1
+Removed REPLACE
+----------------------------
+revision 1.20
+date: 2002-02-21 03:48:22 +0000;  author: ole;  state: Exp;  lines: +106 -1
+added gather and reduce
+----------------------------
+revision 1.19
+date: 2002-02-19 07:45:44 +0000;  author: ole;  state: Exp;  lines: +9 -17
+Fixed crash on Sun Enterprise (alloc **argv)
+----------------------------
+revision 1.18
+date: 2002-02-17 22:40:10 +0000;  author: ole;  state: Exp;  lines: +1 -12
+x
+----------------------------
+revision 1.17
+date: 2002-02-16 00:21:24 +0000;  author: ole;  state: Exp;  lines: +57 -14
+Status block
+----------------------------
+revision 1.16
+date: 2002-02-15 22:46:55 +0000;  author: ole;  state: Exp;  lines: +59 -1
+Scatter
+----------------------------
+revision 1.15
+date: 2002-02-14 07:20:40 +0000;  author: ole;  state: Exp;  lines: +42 -7
+Added bcast - thanks to Gian Paolo Ciceri
+----------------------------
+revision 1.14
+date: 2002-02-10 23:05:31 +0000;  author: ole;  state: Exp;  lines: +3 -2
+Version 1.0.2
+Bugfix from Gian Paulo Ciceri preventing pypar from running under Python 2.2
+----------------------------
+revision 1.13
+date: 2002-02-08 04:10:17 +0000;  author: ole;  state: Exp;  lines: +2 -2
+release 1-0-1
+----------------------------
+revision 1.12
+date: 2002-02-08 03:58:58 +0000;  author: ole;  state: Exp;  lines: +24 -5
+Got it to run on a Sun Enterprise server under Mpich.
+Only small changes.
+----------------------------
+revision 1.11
+date: 2002-02-07 01:51:07 +0000;  author: ole;  state: Exp;  lines: +7 -5
+*** empty log message ***
+----------------------------
+revision 1.10
+date: 2002-02-07 01:33:16 +0000;  author: ole;  state: Exp;  lines: +4 -3
+*** empty log message ***
+----------------------------
+revision 1.9
+date: 2002-02-06 06:49:31 +0000;  author: ole;  state: Exp;  lines: +4 -2
+Added test of pypar
+----------------------------
+revision 1.8
+date: 2002-02-06 04:55:04 +0000;  author: ole;  state: Exp;  lines: +37 -17
+Fixed differences between Alphaserver and LAM/Linux
+----------------------------
+revision 1.7
+date: 2002-01-24 05:31:29 +0000;  author: ole;  state: Exp;  lines: +2 -1
+Got subdir-structure to work on Alpha Server
+----------------------------
+revision 1.6
+date: 2002-01-23 23:41:01 +0000;  author: ole;  state: Exp;  lines: +13 -6
+*** empty log message ***
+----------------------------
+revision 1.5
+date: 2002-01-23 06:49:20 +0000;  author: ole;  state: Exp;  lines: +1 -1
+*** empty log message ***
+----------------------------
+revision 1.4
+date: 2002-01-23 06:48:20 +0000;  author: ole;  state: Exp;  lines: +15 -16
+*** empty log message ***
+----------------------------
+revision 1.3
+date: 2002-01-23 05:54:13 +0000;  author: ole;  state: Exp;  lines: +21 -13
+Modified codes to work with Python packages
+----------------------------
+revision 1.2
+date: 2002-01-23 04:29:28 +0000;  author: ole;  state: Exp;  lines: +47 -8
+
+Brought back sequential stub
+----------------------------
+revision 1.1
+date: 2002-01-22 03:28:17 +0000;  author: ole;  state: Exp;
+branches:  1.1.1;
+Initial revision
+----------------------------
+revision 1.1.1.1
+date: 2002-01-22 03:28:17 +0000;  author: ole;  state: Exp;  lines: +0 -0
+Initial CVS import
+=============================================================================
+
+RCS file: /home/cvs/pypar/Attic/pypartest.py,v
+Working file: pypartest.py
+head: 1.2
+branch:
+locks: strict
+access list:
+symbolic names:
+keyword substitution: kv
+total revisions: 2;	selected revisions: 2
+description:
+----------------------------
+revision 1.2
+date: 2002-02-06 06:53:26 +0000;  author: ole;  state: dead;  lines: +0 -0
+x
+----------------------------
+revision 1.1
+date: 2002-02-06 06:49:31 +0000;  author: ole;  state: Exp;
+Added test of pypar
+=============================================================================
+
+RCS file: /home/cvs/pypar/pytiming,v
+Working file: pytiming
+head: 1.8
+branch:
+locks: strict
+access list:
+symbolic names:
+	release-1-9-2: 1.8
+	release-1-9-1: 1.8
+	release-1-9: 1.8
+	release-1-8-2: 1.5
+	release-1-8-1: 1.5
+	release-1-8: 1.5
+	release-1-7: 1.4
+	release-1-6-5: 1.4
+	release-1-6-4: 1.4
+	release-1-6-3-1: 1.4
+	release-1-6-3: 1.4
+	SG-release-1-1: 1.4
+	SG-release-1-0: 1.4
+	release-0-1: 1.4
+	release-1-6-2: 1.4
+	release-1-6-1: 1.4
+	release-1-6-0: 1.4
+	release-1-5-4: 1.4
+	release-1-5-3: 1.4
+	release-1-5-2: 1.4
+	release-1-5-1: 1.4
+	release-1-5: 1.4
+	release-1-4-1: 1.4
+	release-1-4-0: 1.4
+	release-1-4: 1.4
+	release-1-3: 1.4
+	release-1-2-4: 1.4
+	release-1-2-3: 1.4
+	release-1-2-2: 1.4
+	revision-1-2-1: 1.4
+	release-1-2: 1.4
+	release-1-1: 1.4
+	release-1-0-2: 1.4
+	release-1-0-1: 1.4
+	release-1-0: 1.4
+	start: 1.1.1.1
+	ole: 1.1.1
+keyword substitution: kv
+total revisions: 9;	selected revisions: 9
+description:
+----------------------------
+revision 1.8
+date: 2003-12-03 10:20:48 +0000;  author: ole;  state: Exp;  lines: +10 -7
+disabled mpiext direct functions
+----------------------------
+revision 1.7
+date: 2003-11-20 10:08:12 +0000;  author: ole;  state: Exp;  lines: +39 -11
+Introduced bypass argument for efficiency
+----------------------------
+revision 1.6
+date: 2003-11-17 12:03:18 +0000;  author: ole;  state: Exp;  lines: +17 -19
+Work towards ver 2.0: Exceptions moved to c-extension, revamping of control_info, lots of clean-up
+----------------------------
+revision 1.5
+date: 2003-11-11 10:57:37 +0000;  author: ole;  state: Exp;  lines: +2 -2
+Status object optionally returned from receive. Reserved control_tag, trimmed string buffers to actual length. Minor clean up
+----------------------------
+revision 1.4
+date: 2002-02-06 23:54:55 +0000;  author: ole;  state: Exp;  lines: +5 -4
+Streamlined interface
+----------------------------
+revision 1.3
+date: 2002-02-06 05:40:28 +0000;  author: ole;  state: Exp;  lines: +0 -1
+x
+----------------------------
+revision 1.2
+date: 2002-02-06 05:28:51 +0000;  author: ole;  state: Exp;  lines: +8 -7
+Made output from pytimimng and ctiming more userfriendly
+----------------------------
+revision 1.1
+date: 2002-01-22 03:28:17 +0000;  author: ole;  state: Exp;
+branches:  1.1.1;
+Initial revision
+----------------------------
+revision 1.1.1.1
+date: 2002-01-22 03:28:17 +0000;  author: ole;  state: Exp;  lines: +0 -0
+Initial CVS import
+=============================================================================
+
+RCS file: /home/cvs/pypar/release,v
+Working file: release
+head: 1.6
+branch:
+locks: strict
+access list:
+symbolic names:
+	release-1-9-2: 1.6
+	release-1-9-1: 1.5
+	release-1-9: 1.4
+	release-1-8-2: 1.4
+	release-1-8-1: 1.2
+	release-1-8: 1.2
+	release-1-7: 1.2
+	release-1-6-5: 1.2
+	release-1-6-4: 1.2
+keyword substitution: kv
+total revisions: 6;	selected revisions: 6
+description:
+----------------------------
+revision 1.6
+date: 2004-09-19 01:04:12 +0000;  author: ole;  state: Exp;  lines: +9 -4
+*** empty log message ***
+----------------------------
+revision 1.5
+date: 2004-01-07 11:51:06 +0000;  author: ole;  state: Exp;  lines: +8 -0
+Nothing
+----------------------------
+revision 1.4
+date: 2003-11-16 10:24:58 +0000;  author: ole;  state: Exp;  lines: +10 -2
+*** empty log message ***
+----------------------------
+revision 1.3
+date: 2003-11-16 10:04:48 +0000;  author: ole;  state: Exp;  lines: +19 -0
+Fixed up gather, scatter and reduce to automatically calculate buffer size. Also cleaned up control_info. Updated web site and release script
+----------------------------
+revision 1.2
+date: 2003-02-11 04:25:39 +0000;  author: ole;  state: Exp;  lines: +9 -12
+Changed to use /tmp
+----------------------------
+revision 1.1
+date: 2003-02-11 04:19:40 +0000;  author: ole;  state: Exp;
+Addede script for distributing files and publishing on the web.
+=============================================================================
+
+RCS file: /home/cvs/pypar/ring_example.py,v
+Working file: ring_example.py
+head: 1.1
+branch:
+locks: strict
+access list:
+symbolic names:
+	release-1-9-2: 1.1
+	release-1-9-1: 1.1
+keyword substitution: kv
+total revisions: 1;	selected revisions: 1
+description:
+----------------------------
+revision 1.1
+date: 2004-01-07 11:51:06 +0000;  author: ole;  state: Exp;
+Nothing
+=============================================================================
+
+RCS file: /home/cvs/pypar/runpytiming,v
+Working file: runpytiming
+head: 1.2
+branch:
+locks: strict
+access list:
+symbolic names:
+	release-1-9-2: 1.2
+	release-1-9-1: 1.2
+	release-1-9: 1.2
+	release-1-8-2: 1.2
+	release-1-8-1: 1.2
+	release-1-8: 1.2
+	release-1-7: 1.2
+	release-1-6-5: 1.2
+	release-1-6-4: 1.2
+	release-1-6-3-1: 1.2
+	release-1-6-3: 1.2
+	SG-release-1-1: 1.2
+	SG-release-1-0: 1.2
+	release-0-1: 1.2
+	release-1-6-2: 1.2
+	release-1-6-1: 1.2
+	release-1-6-0: 1.2
+	release-1-5-4: 1.2
+	release-1-5-3: 1.2
+	release-1-5-2: 1.2
+	release-1-5-1: 1.2
+	release-1-5: 1.2
+	release-1-4-1: 1.2
+	release-1-4-0: 1.2
+	release-1-4: 1.2
+	release-1-3: 1.2
+	release-1-2-4: 1.2
+	release-1-2-3: 1.2
+	release-1-2-2: 1.2
+	revision-1-2-1: 1.2
+	release-1-2: 1.2
+	release-1-1: 1.2
+	release-1-0-2: 1.2
+	release-1-0-1: 1.2
+	release-1-0: 1.2
+	start: 1.1.1.1
+	ole: 1.1.1
+keyword substitution: kv
+total revisions: 3;	selected revisions: 3
+description:
+----------------------------
+revision 1.2
+date: 2002-02-07 01:52:05 +0000;  author: ole;  state: Exp;  lines: +1 -3
+*** empty log message ***
+----------------------------
+revision 1.1
+date: 2002-01-22 03:28:17 +0000;  author: ole;  state: Exp;
+branches:  1.1.1;
+Initial revision
+----------------------------
+revision 1.1.1.1
+date: 2002-01-22 03:28:17 +0000;  author: ole;  state: Exp;  lines: +0 -0
+Initial CVS import
+=============================================================================
+
+RCS file: /home/cvs/pypar/setup.py,v
+Working file: setup.py
+head: 1.3
+branch:
+locks: strict
+access list:
+symbolic names:
+	release-1-9-2: 1.3
+	release-1-9-1: 1.3
+	release-1-9: 1.3
+	release-1-8-2: 1.3
+	release-1-8-1: 1.3
+	release-1-8: 1.3
+	release-1-7: 1.3
+	release-1-6-5: 1.3
+	release-1-6-4: 1.3
+	release-1-6-3-1: 1.2
+	release-1-6-3: 1.2
+	SG-release-1-1: 1.2
+	SG-release-1-0: 1.2
+	release-1-1: 1.2
+	release-0-1: 1.2
+	release-1-6-2: 1.2
+	release-1-6-1: 1.1
+	release-1-6-0: 1.1
+	release-1-5-4: 1.1
+	release-1-5-3: 1.1
+keyword substitution: kv
+total revisions: 3;	selected revisions: 3
+description:
+----------------------------
+revision 1.3
+date: 2003-01-13 03:04:49 +0000;  author: ole;  state: Exp;  lines: +12 -1
+Added Ben's comments
+----------------------------
+revision 1.2
+date: 2002-10-29 03:38:12 +0000;  author: ole;  state: Exp;  lines: +5 -1
+Added Windows installation from Simon Frost
+----------------------------
+revision 1.1
+date: 2002-10-18 03:46:29 +0000;  author: ole;  state: Exp;
+Distutils setup from Prabhu Ramachandran
+=============================================================================
+
+RCS file: /home/cvs/pypar/test_init.py,v
+Working file: test_init.py
+head: 1.2
+branch:
+locks: strict
+access list:
+symbolic names:
+	release-1-9-2: 1.2
+	release-1-9-1: 1.2
+	release-1-9: 1.2
+	release-1-8-2: 1.2
+	release-1-8-1: 1.2
+	release-1-8: 1.2
+	release-1-7: 1.1
+	release-1-6-5: 1.1
+	release-1-6-4: 1.1
+	release-1-6-3-1: 1.1
+	release-1-6-3: 1.1
+	SG-release-1-1: 1.1
+	SG-release-1-0: 1.1
+	release-1-1: 1.1
+	release-0-1: 1.1
+	release-1-6-2: 1.1
+	release-1-6-1: 1.1
+	release-1-6-0: 1.1
+	release-1-5-4: 1.1
+	release-1-5-3: 1.1
+	release-1-5-2: 1.1
+	release-1-5-1: 1.1
+	release-1-5: 1.1
+keyword substitution: kv
+total revisions: 2;	selected revisions: 2
+description:
+----------------------------
+revision 1.2
+date: 2003-11-11 10:57:37 +0000;  author: ole;  state: Exp;  lines: +2 -1
+Status object optionally returned from receive. Reserved control_tag, trimmed string buffers to actual length. Minor clean up
+----------------------------
+revision 1.1
+date: 2002-04-30 04:57:21 +0000;  author: ole;  state: Exp;
+Doc
+=============================================================================
+
+RCS file: /home/cvs/pypar/testpypar.py,v
+Working file: testpypar.py
+head: 1.21
+branch:
+locks: strict
+access list:
+symbolic names:
+	release-1-9-2: 1.21
+	release-1-9-1: 1.21
+	release-1-9: 1.20
+	release-1-8-2: 1.20
+	release-1-8-1: 1.18
+	release-1-8: 1.17
+	release-1-7: 1.16
+	release-1-6-5: 1.15
+	release-1-6-4: 1.15
+	release-1-6-3-1: 1.14
+	release-1-6-3: 1.14
+	SG-release-1-1: 1.14
+	SG-release-1-0: 1.14
+	release-0-1: 1.14
+	release-1-6-2: 1.14
+	release-1-6-1: 1.14
+	release-1-6-0: 1.14
+	release-1-5-4: 1.14
+	release-1-5-3: 1.14
+	release-1-5-2: 1.14
+	release-1-5-1: 1.14
+	release-1-5: 1.14
+	release-1-4-1: 1.13
+	release-1-4-0: 1.13
+	release-1-4: 1.13
+	release-1-3: 1.11
+	release-1-2-4: 1.6
+	release-1-2-3: 1.6
+	release-1-2-2: 1.6
+	revision-1-2-1: 1.6
+	release-1-2: 1.6
+	release-1-1: 1.4
+	release-1-0-2: 1.2
+	release-1-0-1: 1.2
+	release-1-0: 1.2
+keyword substitution: kv
+total revisions: 21;	selected revisions: 21
+description:
+----------------------------
+revision 1.21
+date: 2004-01-07 11:36:17 +0000;  author: ole;  state: Exp;  lines: +62 -41
+Changed naming conventions to be more consistent
+----------------------------
+revision 1.20
+date: 2003-11-16 10:38:44 +0000;  author: ole;  state: Exp;  lines: +7 -0
+Added test for complex multidim bcast
+----------------------------
+revision 1.19
+date: 2003-11-16 10:04:48 +0000;  author: ole;  state: Exp;  lines: +91 -50
+Fixed up gather, scatter and reduce to automatically calculate buffer size. Also cleaned up control_info. Updated web site and release script
+----------------------------
+revision 1.18
+date: 2003-11-13 11:36:04 +0000;  author: ole;  state: Exp;  lines: +161 -3
+Added support for the complex datatype
+----------------------------
+revision 1.17
+date: 2003-11-11 10:57:37 +0000;  author: ole;  state: Exp;  lines: +246 -4
+Status object optionally returned from receive. Reserved control_tag, trimmed string buffers to actual length. Minor clean up
+----------------------------
+revision 1.16
+date: 2003-11-07 03:29:04 +0000;  author: ole;  state: Exp;  lines: +192 -3
+Added support for arrays of arbitray dimension
+----------------------------
+revision 1.15
+date: 2003-02-04 01:54:18 +0000;  author: ole;  state: Exp;  lines: +4 -0
+Changed argument names in gather, reduce, ... and updated doc
+----------------------------
+revision 1.14
+date: 2002-04-30 03:56:49 +0000;  author: ole;  state: Exp;  lines: +3 -3
+Modified init so that python command line gets passed onto MPI_init.
+This is necessary for MPICH/Linux.
+----------------------------
+revision 1.13
+date: 2002-02-27 03:13:56 +0000;  author: ole;  state: Exp;  lines: +15 -12
+Fixed up test and ran on 32 procs
+----------------------------
+revision 1.12
+date: 2002-02-27 00:57:12 +0000;  author: ole;  state: Exp;  lines: +23 -15
+Fixed test and ran with 22 procs on SUN
+----------------------------
+revision 1.11
+date: 2002-02-21 04:17:33 +0000;  author: ole;  state: Exp;  lines: +2 -1
+x
+----------------------------
+revision 1.10
+date: 2002-02-21 04:15:43 +0000;  author: ole;  state: Exp;  lines: +2 -1
+x
+----------------------------
+revision 1.9
+date: 2002-02-21 04:08:38 +0000;  author: ole;  state: Exp;  lines: +2 -1
+x
+----------------------------
+revision 1.8
+date: 2002-02-21 03:53:09 +0000;  author: ole;  state: Exp;  lines: +1 -1
+Removed REPLACE
+----------------------------
+revision 1.7
+date: 2002-02-21 03:48:22 +0000;  author: ole;  state: Exp;  lines: +152 -1
+added gather and reduce
+----------------------------
+revision 1.6
+date: 2002-02-15 22:46:55 +0000;  author: ole;  state: Exp;  lines: +44 -0
+Scatter
+----------------------------
+revision 1.5
+date: 2002-02-15 04:06:34 +0000;  author: ole;  state: Exp;  lines: +5 -4
+Fixed comparison of numeric arrays
+----------------------------
+revision 1.4
+date: 2002-02-14 07:49:54 +0000;  author: ole;  state: Exp;  lines: +8 -0
+*** empty log message ***
+----------------------------
+revision 1.3
+date: 2002-02-14 07:20:40 +0000;  author: ole;  state: Exp;  lines: +42 -2
+Added bcast - thanks to Gian Paolo Ciceri
+----------------------------
+revision 1.2
+date: 2002-02-06 23:54:20 +0000;  author: ole;  state: Exp;  lines: +2 -0
+x
+----------------------------
+revision 1.1
+date: 2002-02-06 06:52:42 +0000;  author: ole;  state: Exp;
+Test of pypar
+=============================================================================
+
+RCS file: /home/cvs/pypar/documentation/pypar_installation.tex,v
+Working file: documentation/pypar_installation.tex
+head: 1.1
+branch:
+locks: strict
+access list:
+symbolic names:
+	release-1-9-2: 1.1
+keyword substitution: kv
+total revisions: 1;	selected revisions: 1
+description:
+----------------------------
+revision 1.1
+date: 2004-02-08 16:58:56 +0000;  author: ole;  state: Exp;
+Added beginnings of documentation
+=============================================================================
+
+RCS file: /home/cvs/pypar/documentation/pypar_tutorial.tex,v
+Working file: documentation/pypar_tutorial.tex
+head: 1.1
+branch:
+locks: strict
+access list:
+symbolic names:
+	release-1-9-2: 1.1
+keyword substitution: kv
+total revisions: 1;	selected revisions: 1
+description:
+----------------------------
+revision 1.1
+date: 2004-02-08 16:58:56 +0000;  author: ole;  state: Exp;
+Added beginnings of documentation
+=============================================================================
Index: trunk/anuga_core/source/pypar_dist/supporting_material/contrib/0203h.htm
===================================================================
--- trunk/anuga_core/source/pypar_dist/supporting_material/contrib/0203h.htm	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/supporting_material/contrib/0203h.htm	(revision 8494)
@@ -0,0 +1,312 @@
+<BASE HREF="http://www.unixreview.com/documents/s=2425/uni1016550801055/">
+<html>
+<head>
+<TITLE>Regular Expressions: Erlang Is Worth a Look</TITLE>
+<!-- hide from older browsers, should show em anyway so they will upgrade-->
+<LINK REL=stylesheet HREF="/unxrev.css" TYPE="text/css">
+<META http-equiv="expires" content="-1">
+<META http-equiv= "pragma" CONTENT="no-cache">
+<META name="author" content="Randy Reames">
+<META name="ROBOTS" content="ALL">
+<META name="description" content="News, reviews, features, and commentary for Unix and Linux professionals.">
+<META name="KEYWORDS" content="UNIX, open source, security, unix, storage, BSD, Linux, Red Hat, Debian, Suse, solaris, AIX, HP-UX, database, linux training, unix training, certification, backup, system administration, intrusion detection, reviews, XML, network, Perl, Shell, TCP/IP, secure shell">
+</HEAD>
+<BODY bgColor=#FFFFFF leftMargin=0  rightMargin=0 topMargin=4 marginheight="4">
+<A name="top"></A> 
+<TABLE border=0 cellpadding=3 cellspacing=0 width="98%">
+  <TR> 
+    <TD width="125" align="center" valign="middle"><A href="/"> </A><IMG src="/cmp_logo.gif" width="69" height="59"> 
+    </TD>
+    <TD align=left valign="top" width="160"> 
+      <P> <A href="/"><IMG alt="Welcome to UnixReview.com" border=0 
+                  src="/logosm.gif" width="151" height="89"></A> 
+    </TD>
+    <TD align="right" valign="middle"> 
+            <A target="_top" href="http://newads.cmpnet.com/event.ng/Type=click&ProfileID=6941&RunID=41981&AdID=31293&GroupID=1&FamilyID=2784&TagValues=178.2994&Redirect=https://www.sdmediagroup.com/circ/subscribe.cgi%3Fuser_keycode%3D2cck"><IMG src="http://img.cmpnet.com/ads/graphics/cs/ar/cuj_sub_468.gif" border=1 height=60 width=468 alt=""></A> </TD>
+</TABLE>
+    
+<TABLE border=0 cellPadding=0 cellSpacing=0 width="98%">
+  <TR> 
+    <TD vAlign=top width=125 align="center"> 
+      <TABLE bgcolor=#000000 border=0 cellpadding=0 cellspacing=0 
+            width="100%">
+        <TR> 
+          <TD> 
+            <TABLE border=0 cellpadding=3 cellspacing=1 width="100%">
+              <TR bgcolor="#CCCCCC" align="center" valign="top"> 
+                <TD class="bluetop">Main Menu</TD>
+              </TR>
+              <TR align="left" valign="top" bgcolor="#FFFFFF"> 
+                <TD class="bluebottom" nowrap>   <A href="/" class="sidenav">Home</A><BR>
+                    <A href="/articles/" class="sidenav">Archives</A><BR>
+                    <A href="/reviews/" class="sidenav">Reviews<BR>
+                  </A>   <A href="/articles/books/" class="sidenav">Books</A><BR>
+                    <A href="/glink/" class="sidenav">Geek Links</A><BR>
+                    <A href="/people/" class="sidenav">Contact Us</A> 
+                </TD>
+              </TR>
+            </TABLE>
+          </TD>
+        </TR>
+      </TABLE>
+      <BR>
+      <TABLE bgcolor=#000000 border=0 cellpadding=0 cellspacing=0 
+            width="100%">
+        <TR> 
+          <TD align="left" valign="top"> 
+            <TABLE border=0 cellpadding=3 cellspacing=1 width="100%">
+              <TR bgcolor="#CCCCCC" align="center" valign="top"> 
+                <TD class="bluetop">Sections</TD>
+              </TR>
+              <TR bgcolor="#FFFFFF"> 
+                <TD class="bluebottom" nowrap align="left" valign="top">
+				  <A href="/columns/laird/" class="sidenav">Regular <BR>
+  Expressions</A><BR> 
+				  <A href="/tool/" class="sidenav">Tool of the Month</A><BR>
+				  <A href="/opensource/" class="sidenav">Open 
+                  Source</A><BR>
+                    <A href="/certify/" class="sidenav">Certification</A><BR>
+                    <A href="/columns/schaefer/" class="sidenav">Shell 
+                  Corner</A><BR>
+				 
+                    <A href="/columns/preston/index.htm" class="sidenav">lost+found</A>
+
+</TD>
+              </TR>
+            </TABLE>
+          </TD>
+        </TR>
+      </TABLE>
+            <P> <A target="_top" href="http://newads.cmpnet.com/event.ng/Type=click&ProfileID=6948&RunID=41296&AdID=30834&GroupID=1&FamilyID=1&TagValues=2994.3003&Redirect=http://www.usenix.org/events/usenix02/"><IMG src="http://img.cmpnet.com/ads/graphics/cs/ar/usenix_2002_125.gif" border=1 height=125 width=125 alt="Click Here!"></A> 
+      <P> 
+      <FORM ACTION="/search.htm" METHOD="GET">
+        <INPUT type="text" name="search" size=12>
+        <BR>
+        <INPUT type="image" src="/search2.gif" alt="Search" border="0" value="Search" name="submit">
+      </FORM>
+      <P>
+      <TABLE bgcolor=#000000 border=0 cellpadding=0 cellspacing=0 
+            width="100%">
+        <tbody> 
+        <TR> 
+          <TD> 
+            <TABLE border=0 cellpadding=3 cellspacing=1 width="100%">
+              <tbody> 
+              <TR align="center" valign="top"> 
+                <TD bgcolor=#cccccc class="bluetop">Newsletter</TD>
+              </TR>
+              <TR align="center"> 
+                <TD bgcolor=#ffffff class="bluebottom"><A href="/subscribe/index.htm"><IMG src="/newsletter.gif" width="120" height="69" border="0" alt="Get the Newsletter"><BR>
+                  Get the Newsletter</A> </TD>
+              </TR>
+              </tbody> 
+            </TABLE>
+          </TD>
+        </TR>
+        </tbody> 
+      </TABLE>
+      <BR>
+    </TD>
+    <TD> </TD>
+    <TD vAlign=top align="left"> 
+            
+
+<A href="/print/documentID=24422">Print-Friendly Version</A><BR><BR> 
+
+<TABLE cellSpacing=0 cellPadding=3 width="100%" border=0>
+  <TBODY> 
+  <TR vAlign=center bgColor=#dedbce>
+        <TD class=headline align=left>Regular Expressions: Erlang Is Worth a Look</TD>
+    <TD class=headdate align=right>March 2002</TD>
+      </TR>
+  <TR vAlign=top align=left> 
+        <TD colSpan=2> 
+      <P> by Cameron Laird and Kathryn Soraiz 
+      <P>
+Let's write highly-reliable, high-performance distributed applications.
+While we're at it, let's choose a development language with a "track
+record" of high productivity.
+<P>
+We'll use the Erlang programming language.
+<P>
+<H2>Erlang for Real</H2>
+      <P> The name "Erlang" honors a Danish mathematician by that name, 
+        whose work in stochastic theories flourished at the beginning of the twentieth 
+        century. Although Erlang's name-recognition is low among U.S. programmers 
+        ("Erlang? Isn't that one of the new Winter Olypmics events?"), its accomplishments 
+        are plenty serious. First implemented in 1986 by an engineer at Ericsson 
+        Enterprise, Erlang is the language in which many production telecommunications 
+        and other mission-critical applications are written. Its use on general-purpose 
+        "personal computers" and servers is also growing. 
+      <P> One of the reasons for the interest in Erlang is that the software world 
+        is finally acknowledging the importance of distributed computation. This 
+        has been a good season for distributed computing -- <I>Scientific American</I> 
+        has popularized the topic; the new Usenet newsgroup comp.distributed has 
+        hosted a handful of worthwhile discussions since its launch in mid-February; 
+        and the Global Grid Forum (<A href="http://www.gridforum.org/" target="_blank">http://www.gridforum.org/</A>) 
+        has become one of the hottest events in computing. Ideas about large-scale 
+        operating systems (OSes) are moving out of research laboratories; the 
+        .NET and Web Services marketing machines are starting to propagate technical 
+        content; and peer-to-peer and CORBA partisans are fighting back with significant 
+        engineering advances. This is territory that Erlang experts have been 
+        exploring for more than a decade. 
+      <P>Erlang was originally designed as a "language for programming <I>distributed 
+        fault-tolerant soft real-time non-stop applications</I>." That formulation 
+        correctly suggests there's more to Erlang than just its capacity for distributed 
+        calculations. Erlang is a functional programming language, with deep roots 
+        in logic and functional semantics that contribute to program correctness. 
+        Like such wildly different languages as Ada and Eiffel, Erlang's correctness 
+        characteristics make it easier for programmers to compose applications 
+        that behave predictably. Erlang generalizes correctness to "fault-tolerance" 
+        with its powerful "exception" mechanisms. Ericsson telephone switches 
+        need very high uptimes -- 99.999% and up. Erlang's fault-tolerance, related 
+        to the exceptions in Java and Python, helps achieve this level of reliability. 
+      <P>
+<H2>See for Yourself</H2>
+<P>
+What's it like to work in a language shaped for fault-tolerance,
+high productivity, and distribution?  As with most of the subjects
+we choose for "Regular Expressions," the best answer is likely to
+be, "See for yourself."  An open-source version of Erlang has been
+available since 1998, and extensive resources are available on the
+Web.  
+      <P> Source or binary downloads for the recent versions of the standard Erlang 
+        distribution are just over 10 megabytes in size. Although this is toward 
+        the top of the range for the languages "Regular Expressions" most often 
+        covers, it's significantly smaller than typical Java installations. In 
+        our experience, Unix generations of Erlang from source always go smoothly; 
+        whatever errors the latest releases have, they don't show up during installation 
+        or simple tests. 
+      <P> Once that's done, Erlang's "Hello, World" preliminaries go quickly enough. 
+        Erlang's home site even has a "Getting Started" page at <A href="http://www.erlang.org/starting.html" target="_blank">http://www.erlang.org/starting.html</A>. 
+        Like other typical modern interpreters, the Erlang language processor, 
+        <CODE>erl</CODE>, supports both interactive and "batch" operation. You 
+        can bring it up interactively as a convenient calculator: 
+      <PRE>     #> erl
+     Erlang (BEAM) emulator version 5.1 [source]
+
+     Eshell V5.1  (abort with ^G)   
+     1> pwd().
+     /usr/local/src/otp_src_R8B-0
+     ok   
+     2> ls().
+     AUTHORS           EPLICENCE         Makefile          Makefile.in
+     README            bin               bootstrap         config.cache
+     config.log        config.status     configure         configure.in
+     erts              lib               make
+     ok
+     3> 123 * 456 - 999.
+     55089 </PRE> 
+   
+Notice that a period terminates every complete line.
+
+      <P>
+<H2>Safe Power</H2>
+<P>
+The Erlang downloads are so large because the
+standard distribution is intended to "include batteries."
+Every Erlang installation builds in functions 
+that mediate ASN encoding, manage CORBA transactions,
+perform cryptographic transformations, implement such
+standard networking protocols as FTP, HTTP, SNMP, and SSL,
+and even connect to Java classes.
+      <P> This powerful library combines with Erlang's succinct expressivity to 
+        make remarkable results possible in just a few lines. It's hard to show 
+        Erlang at its best by simply looking at common programming tasks; it doesn't 
+        do the things that Perl or C do any better than those languages. Erlang 
+        comes into its own on applications that must run continuously (think of 
+        air-traffic control), cooperatively (across many machines), and without 
+        surprises. Erlang is largely immune to the buffer overflows, memory leaks, 
+        off-by-one, and other mundane errors that affect many applications. 
+      <P>
+A favorite example of Erlang's brevity is this generic
+client-server:
+   <PRE>     -module(cs).
+     -export([start/3, rpc/2, loop/2]).
+
+     start(Name, Data, Fun) ->
+         register(Name,
+             spawn(fun() ->
+                 loop(Data, Fun)
+             end)).
+
+     rpc(Name, Q) ->         
+         Tag = ref(),
+         Name ! -query, self(), Tag, Q),
+         receive
+             (Tag, Reply) -> Reply
+         end.
+
+     loop(Data, Fun) ->
+         receive
+             (query, Pid, Tag, Q) ->
+                 (Reply, Data1) = Fun(Q, Data),
+                 Pid ! (Tag, Reply),
+                 loop(Data1, Fun)
+     end. 
+   </PRE>
+   <P>
+Invocation of <CODE>start()</CODE> launches a complete
+network endpoint. The code above handles communications
+and concurrency, while all that's left for an application
+developer is to bind <CODE>Fun</CODE> to a definition
+that executes the application-specific operation.
+<P>
+If your programs really, REALLY have to work right every
+time, consider using Erlang.
+<P>
+<H2>Administrative Wrap-Up</H2>
+      <P> We're collecting notes and hyperlinks on Erlang at <A href="http://starbase.neosoft.com/~claird/comp.lang.functional/Erlang.html" target="_blank">http://starbase.neosoft.com/~claird/comp.lang.functional/Erlang.html</A>. 
+        One ambition for this page is to introduce a functional language for the 
+        "working programmer" in a way that makes the academic abstractions found 
+        in this specialty more appealing. As always, e-mail to us @regularexpressions.com 
+        is welcome. Incidentally, if you want to try out parallel programming 
+        in a more mainstream language than Erlang, consider PyPAR (<A href="http://datamining.anu.edu.au/pypar" target="_blank">http://datamining.anu.edu.au/pypar</A>). 
+        This well-behaved Python extension was just released in its 1.0 version 
+        last month. 
+      <P>
+Next time, we'll look at a completely
+different approach to construction of correct programs:
+use of static syntax analyzers.  Until then, good luck
+in solving your problems at a high level.
+    </TD>
+      </TR>
+  </TBODY>
+</TABLE>
+
+ </TD>
+    <TD vAlign=top align="left">
+      <TABLE cellSpacing=1 cellPadding=3 width="130" 
+        border=0>
+        <TBODY> 
+        <TR vAlign=bottom align=right bgColor=#ffffff><TD>            <A target="_top" href="http://newads.cmpnet.com/event.ng/Type=click&ProfileID=9939&RunID=37627&AdID=25450&GroupID=1&FamilyID=2749&TagValues=1910.2994&Redirect=http://www.sdmagazine.com"><IMG src="http://img.cmpnet.com/ads/graphics/cs/ar/clear_1x1.gif" border="0" height=1 width=1 alt=""></A></TD> 
+          
+        </TR>
+        </TBODY> 
+      </TABLE>
+    </TD>
+  <TR> 
+    <TD vAlign=top width=150 align="center"> </TD>
+    <TD> </TD>
+    <TD vAlign=top width="100%" colspan="2"> 
+      <HR width="170" align="left" noshade size="1">
+      <A href="/" class="headline">Home</A> | <A href="#top" class="headline">Top</A> 
+      <BR>
+      <P> 
+                <A target="_top" href="http://newads.cmpnet.com/event.ng/Type=click&ProfileID=6936&RunID=41295&AdID=30835&GroupID=1&FamilyID=1&TagValues=179.2994&Redirect=http://www.usenix.org/events/usenix02/"><IMG src="http://img.cmpnet.com/ads/graphics/cs/ar/usenix_2002_468.gif" border=1 height=60 width=468 alt="Click Here!"></A> 
+        <BR>
+        <SPAN class="privacy">Copyright © 2002 <I>UnixReview.com</I>, UnixReview.com's 
+        <A href="http://www.cmp.com/delivery/privacy.html">Privacy Policy<BR>
+        </A></SPAN><SPAN class="privacy">Comments about the Web site: <A href="mailto:scoady@cmp.com?subject=UnixReview">scoady@cmp.com 
+        </A><BR>
+        SDMG Web Sites: <A href="http://www.cuj.com/" target="_new">C/C++ Users 
+        Journal</A>, <A href="http://www.ddj.com/" target="_new">Dr. Dobb's Journal</A>, 
+        <A href="http://msdn.microsoft.com/msdnmag/" target="_new">MSDN Magazine</A>, 
+        <A href="http://www.sysadminmag.com/" target="_new">Sys Admin</A>, <A href="http://www.sdexpo.com/" target="_new">SD 
+        Expo</A>, <A href="http://www.sdmagazine.com/" target="_new">SD Magazine</A>, 
+        <A href="http://www.unixreview.com/">UnixReview.com</A>, <A href="http://www.wd-mag.com/" target="_new">Windows 
+        Developer</A>, <A href="http://www.tpj.com" target="_blank">TPJ</A>, 
+        <A href="http://www.byte.com/" target="_blank"> BYTE.com</A></SPAN></P>
+    </TD>
+</TABLE><DIV align="right"><FONT size=-2>www3</FONT></DIV>
+  </BODY></HTML>
Index: trunk/anuga_core/source/pypar_dist/supporting_material/contrib/Dreed.reply
===================================================================
--- trunk/anuga_core/source/pypar_dist/supporting_material/contrib/Dreed.reply	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/supporting_material/contrib/Dreed.reply	(revision 8494)
@@ -0,0 +1,233 @@
+
+Hi Ole,
+
+Yes, that appears to work - no more segmentation faults:
+
+mpirun -np 2 python demo.py
+MPI initialised OK with 2 processors
+I am proc 0 of 2 on node fourier
+I am proc 1 of 2 on node hopper
+Processor 0 sending message "MSGP0" to processor 1
+Processor 1 received message "MSGP0" from processor 0
+Size of msg was 5 bytes
+Processor 1 sending msg "MSGP0->P1" to 0
+Processor 0 received message "MSGP0->P1" from processor 1
+Size of msg was 9 bytes
+
+Thanks!
+
+Dave
+
+
+
+> Date: Sat, 30 Aug 2003 19:47:39 +1000
+> Reply-To: Ole.Nielsen@anu.edu.au
+> Content-Type: text/plain; charset=us-ascii
+> Content-Disposition: inline
+> User-Agent: Mutt/1.5.4i
+> From: ole@cuttlefish.anu.edu.au (Ole Moller Nielsen)
+> Content-Length: 9270
+> 
+> 
+> Hello Dave
+> 
+> Regarding your problem I may have the solution (thanks to input from Doug Orr).
+> It seems that newer versions of LAM require the last commandline argument to be NULL
+> so try to modify mpiext.c as follows around line 472 and let me know if it works for you.
+> The symbol + means add the line and - means remove the line.
+> 
+> Cheers
+> Ole
+> 
+> ----PATCH for mpiext.c
+> 
+> /* Reconstruct C-commandline */
+> /*                           */
+> argc = PyList_Size(input); //Number of commandline arguments
+> - argv = (char**) malloc((argc)*sizeof(char*));
+> + argv = (char**) malloc((argc+1)*sizeof(char*));
+>      
+> for (i=0; i<argc; i++)
+>   argv[i] = PyString_AsString( PyList_GetItem(input, i) );
+> 	   
+> + argv[i] = NULL;
+> 
+> -------------------------------	   
+> 
+> 
+> 
+> 		
+> On Tue, Aug 05, 2003 at 11:14:20AM -0400, Dave Reed wrote:
+> > 
+> > Hi,
+> > 
+> > I'm trying to install the lam-7.0 version of MPI and pypar on a
+> > Solaris 9 system for one of my colleagues. I've got lam-7.0 working
+> > fine (successfully ran one of its test cases). I then installed pypar
+> > 1.6.4 using:
+> > 
+> > python setup.py build
+> > python setup.py install
+> > 
+> > and that worked without any errors; however, I can't seem to get any
+> > of the examples to work.
+> > 
+> > Details of the system are:
+> > relevant details of the system are:
+> > Solaris 9, gcc 3.2.3, python 2.3, Numeric-23.0
+> > 
+> > When I try:
+> > 
+> > mpirun -np 2 python demo.py 
+> > -----------------------------------------------------------------------------
+> > It seems that [at least] one of the processes that was started with
+> > mpirun did not invoke MPI_INIT before quitting (it is possible that
+> > more than one process did not invoke MPI_INIT -- mpirun was only
+> > notified of the first one, which was on node n0).
+> > 
+> > mpirun can *only* be used with MPI programs (i.e., programs that
+> > invoke MPI_INIT and MPI_FINALIZE).  You can use the "lamexec" program
+> > to run non-MPI programs over the lambooted nodes.
+> > -----------------------------------------------------------------------------
+> > 
+> > mpirun C python demo.py
+> > 
+> > produces the same results.
+> > 
+> > mpirun C hello
+> > 
+> > runs the hello demo that comes with lam-7.0 fine.
+> > 
+> > when I execute:
+> > python
+> > import pypar
+> > 
+> > I get a segmentation fault. Here's the backtrace
+> > 
+> > #0  0xfef33400 in strlen () from /usr/lib/libc.so.1
+> > #1  0xfece5c40 in sfh_argv_dup ()
+> >    from /usr/local/lib/python2.3/site-packages/pypar/mpiext.so
+> > #2  0xfece67b0 in ao_parse ()
+> >    from /usr/local/lib/python2.3/site-packages/pypar/mpiext.so
+> > #3  0xfecb7be0 in lam_linit ()
+> >    from /usr/local/lib/python2.3/site-packages/pypar/mpiext.so
+> > #4  0xfecb82fc in lam_mpi_init ()
+> >    from /usr/local/lib/python2.3/site-packages/pypar/mpiext.so
+> > #5  0xfecb5a70 in MPI_Init ()
+> >    from /usr/local/lib/python2.3/site-packages/pypar/mpiext.so
+> > #6  0xfecb4a08 in init (self=0x1, args=0xffbfc7e0) at lib/pypar/mpiext.c:480
+> > #7  0xff25b340 in PyCFunction_Call (func=0xc5620, arg=0xc6650, kw=0x0)
+> >     at Objects/methodobject.c:93
+> > #8  0xff29a39c in call_function (pp_stack=0xffbfc968, oparg=1)
+> >     at Python/ceval.c:3439
+> > #9  0xff297694 in eval_frame (f=0x9a8f0) at Python/ceval.c:2116
+> > #10 0xff2987a8 in PyEval_EvalCodeEx (co=0xbf9a0, globals=0x1, locals=0x0, 
+> >     args=0x0, argcount=0, kws=0x0, kwcount=0, defs=0x0, defcount=0, 
+> >     closure=0x0) at Python/ceval.c:2663
+> > #11 0xff294c34 in PyEval_EvalCode (co=0xbf9a0, globals=0x0, locals=0xbc5d0)
+> >     at Python/ceval.c:537
+> > #12 0xff2bd2fc in PyImport_ExecCodeModuleEx (name=0xffbfd558 "pypar.pypar", 
+> >     co=0xbf9a0, 
+> >     pathname=0xffbfcb70 "/usr/local/lib/python2.3/site-packages/pypar/pypar.pyc") at Python/import.c:621
+> > #13 0xff2c016c in load_source_module (name=0xffbfd558 "pypar.pypar", 
+> >     pathname=0xffbfcb70 "/usr/local/lib/python2.3/site-packages/pypar/pypar.pyc", fp=0x22e8) at Python/import.c:894
+> > #14 0xff2bdf64 in load_module (name=0xffbfd558 "pypar.pypar", fp=0xfefc02b4, 
+> >     buf=0xffbfd068 "/usr/local/lib/python2.3/site-packages/pypar/pypar.py", 
+> >     type=1, loader=0x0) at Python/import.c:1699
+> > #15 0xff2bf104 in import_submodule (mod=0xbb1b0, subname=0xffbfd55e "pypar", 
+> >     fullname=0xffbfd558 "pypar.pypar") at Python/import.c:2290
+> > #16 0xff2bebd4 in load_next (mod=0xbb1b0, altmod=0xff318028, 
+> >     p_name=0xffbfd55e, buf=0xffbfd558 "pypar.pypar", p_buflen=0xffbfd554)
+> >     at Python/import.c:2111
+> > #17 0xff2c0890 in import_module_ex (name=0x0, globals=0xbb1b0, locals=0x48d20, 
+> >     fromlist=0x45590) at Python/import.c:1957
+> > #18 0xff2be7e8 in PyImport_ImportModuleEx (name=0xbb1f4 "pypar", 
+> >     globals=0x48d20, locals=0x48d20, fromlist=0x45590) at Python/import.c:1998
+> > #19 0xff28cda0 in builtin___import__ (self=0x0, args=0xbb1f4)
+> >     at Python/bltinmodule.c:45
+> > #20 0xff25b340 in PyCFunction_Call (func=0x29e68, arg=0xb3f60, kw=0x0)
+> >     at Objects/methodobject.c:93
+> > #21 0xff22e090 in PyObject_Call (func=0x29e68, arg=0xb3f60, kw=0x0)
+> >     at Objects/abstract.c:1755
+> > #22 0xff299f84 in PyEval_CallObjectWithKeywords (func=0x29e68, arg=0xb3f60, 
+> >     kw=0x0) at Python/ceval.c:3346
+> > #23 0xff29740c in eval_frame (f=0xf71d0) at Python/ceval.c:1996
+> > #24 0xff2987a8 in PyEval_EvalCodeEx (co=0xacee0, globals=0x1, locals=0x0, 
+> >     args=0x0, argcount=0, kws=0x0, kwcount=0, defs=0x0, defcount=0, 
+> >     closure=0x0) at Python/ceval.c:2663
+> > #25 0xff294c34 in PyEval_EvalCode (co=0xacee0, globals=0x0, locals=0x48d20)
+> >     at Python/ceval.c:537
+> > #26 0xff2bd2fc in PyImport_ExecCodeModuleEx (name=0xffbfed28 "pypar", 
+> >     co=0xacee0, 
+> >     pathname=0xffbfde48 "/usr/local/lib/python2.3/site-packages/pypar/__init__.pyc") at Python/import.c:621
+> > #27 0xff2c016c in load_source_module (name=0xffbfed28 "pypar", 
+> >     pathname=0xffbfde48 "/usr/local/lib/python2.3/site-packages/pypar/__init__.pyc", fp=0x22e8) at Python/import.c:894
+> > #28 0xff2bdf64 in load_module (name=0xffbfed28 "pypar", fp=0xfefc02a4, 
+> >     buf=0xffbfe340 "/usr/local/lib/python2.3/site-packages/pypar/__init__.py", 
+> >     type=1, loader=0x0) at Python/import.c:1699
+> > #29 0xff2c0624 in load_package (name=0xffbfed28 "pypar", pathname=0xb2338 "")
+> >     at Python/import.c:951
+> > #30 0xff2be11c in load_module (name=0xffbfed28 "pypar", fp=0x0, 
+> >     buf=0xffbfe838 "/usr/local/lib/python2.3/site-packages/pypar", type=5, 
+> >     loader=0x0) at Python/import.c:1722
+> > #31 0xff2bf104 in import_submodule (mod=0xff318028, 
+> >     subname=0xffbfed28 "pypar", fullname=0xffbfed28 "pypar")
+> >     at Python/import.c:2290
+> > #32 0xff2bebd4 in load_next (mod=0xff318028, altmod=0xff318028, 
+> >     p_name=0xffbfed28, buf=0xffbfed28 "pypar", p_buflen=0xffbfed24)
+> >     at Python/import.c:2111
+> > #33 0xff2c0890 in import_module_ex (name=0x0, globals=0xff318028, 
+> >     locals=0x3a9c0, fromlist=0xff318028) at Python/import.c:1957
+> > #34 0xff2be7e8 in PyImport_ImportModuleEx (name=0xbb1f4 "pypar", 
+> >     globals=0x3a9c0, locals=0x3a9c0, fromlist=0xff318028)
+> >     at Python/import.c:1998
+> > #35 0xff28cda0 in builtin___import__ (self=0x0, args=0xbb1f4)
+> >     at Python/bltinmodule.c:45
+> > #36 0xff25b340 in PyCFunction_Call (func=0x29e68, arg=0x39900, kw=0x0)
+> >     at Objects/methodobject.c:93
+> > #37 0xff22e090 in PyObject_Call (func=0x29e68, arg=0x39900, kw=0x0)
+> >     at Objects/abstract.c:1755
+> > #38 0xff299f84 in PyEval_CallObjectWithKeywords (func=0x29e68, arg=0x39900, 
+> >     kw=0x0) at Python/ceval.c:3346
+> > #39 0xff29740c in eval_frame (f=0xedb78) at Python/ceval.c:1996
+> > #40 0xff2987a8 in PyEval_EvalCodeEx (co=0xb0ce0, globals=0x0, locals=0x0, 
+> >     args=0x0, argcount=0, kws=0x0, kwcount=0, defs=0x0, defcount=0, 
+> >     closure=0x0) at Python/ceval.c:2663
+> > #41 0xff294c34 in PyEval_EvalCode (co=0xb0ce0, globals=0x0, locals=0x3a9c0)
+> >     at Python/ceval.c:537
+> > #42 0xff2caf3c in run_node (n=0x2a140, filename=0x3a9c0 "", globals=0x3a9c0, 
+> >     locals=0x3a9c0, flags=0x3a9c0) at Python/pythonrun.c:1205
+> > #43 0xff2ca000 in PyRun_InteractiveOneFlags (fp=0xffffffff, 
+> >     filename=0xff2fed08 "<stdin>", flags=0xffbff79c) at Python/pythonrun.c:697
+> > #44 0xff2c9d60 in PyRun_InteractiveLoopFlags (fp=0xfefc0274, 
+> >     filename=0xff2fed08 "<stdin>", flags=0xffbff79c) at Python/pythonrun.c:630
+> > #45 0xff2c9cac in PyRun_AnyFileExFlags (fp=0xfefc0274, 
+> >     filename=0xff2fed08 "<stdin>", closeit=0, flags=0xffbff79c)
+> >     at Python/pythonrun.c:593
+> > #46 0xff2d2a08 in Py_Main (argc=5120, argv=0x16bc) at Modules/main.c:415
+> > 
+> > 
+> > Any suggestions or ideas for getting this working?
+> > 
+> > Thanks,
+> > Dave
+> > 
+> > -- 
+> > David Reed, Ph.D.                   e-mail: dreed@capital.edu
+> > Math/Computer Science Dept.         http://capital2.capital.edu/faculty/dreed
+> > Capital University                  phone:  614-236-6133
+> > 2199 E. Main St.                    fax:    614-236-6518
+> > Columbus, OH 43209
+> 
+> -- 
+> -------------------------------------------------------------------  
+> Ole Nielsen                     | Email: Ole.Nielsen@anu.edu.au
+> -------------------------------------------------------------------  
+> Mathematical Sciences Institute | Phone: +61 2 6125 3873 (Direct)
+> Australian National University  | Fax:   +61 2 6125 5549
+> Canberra ACT 0200               | 
+> Australia                       | 
+> -------------------------------------------------------------------  
+> ANU CRICOS # 00120C
+> -------------------------------------------------------------------  
+> 
Index: trunk/anuga_core/source/pypar_dist/supporting_material/contrib/Makefile
===================================================================
--- trunk/anuga_core/source/pypar_dist/supporting_material/contrib/Makefile	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/supporting_material/contrib/Makefile	(revision 8494)
@@ -0,0 +1,25 @@
+#####################################
+# Compilation of C extension mpi.c  #
+# for use with pypar                #
+#	                            #
+# Ole Nielsen, Jan 2002             #
+##################################### 
+
+CC          =   mpicc #MPI C compiler
+
+
+### Dependencies ###
+#
+default: mpi.so
+
+mpi.so: mpi.c Makefile install.py
+	python install.py
+
+ctiming: ctiming.c Makefile
+	$(CC) ctiming.c -o ctiming
+
+clean:
+	rm -f *.o so_locations *~ runpytiming.o* runpytiming.e*
+
+veryclean: clean
+	rm -f *.so *.pyc ctiming
Index: trunk/anuga_core/source/pypar_dist/supporting_material/contrib/ciceri
===================================================================
--- trunk/anuga_core/source/pypar_dist/supporting_material/contrib/ciceri	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/supporting_material/contrib/ciceri	(revision 8494)
@@ -0,0 +1,56 @@
+Hello,
+first of all thanks a lot for your work with pypar.
+
+Now my remark:
+trying out testpypar.py with python 2.2, raw_receive(string)
+doesn't work: the vanilla method is called instead of.
+
+Looking your sources, in pypar.py:220 (about)
+
+if type(x).__name__ == 'string':
+
+doesn't work with (my) python 2.2 (GNU/Linux RH7.2 on i686),
+where type(aString) == 'str'.
+
+So I've done this simple workaround:
+
+if type(x).__name__[0:3] == 'str':
+
+(this should work for both python2.1 and python2.2)
+
+HTH
+Thanks again for your effort.
+/gp
+
+
+P.S.: perhaps I've mis-compiled my python2.2, but since
+I've accepted all the defaults, I suppose that someone other
+will find this very same issue.
+
+P.S.: in my installation, pypar works with the latest LAM 6.5.6
+
+-- 
+Discussion: How do you feel about Open Source firms making
+millions through public offerings?
+
+"I wish these companies were making the same millions without
+distributing any non-free, user-subjugating software." --
+                                              Richard Stallman
+
+"We're so forward that sometimes we reach ourself." --
+                                              g.p.
+
+"It is amazing how much you can achieve when you don't have
+to do the real work yourself." --
+                                              Joe Celko
+
+   Gian Paolo Ciceri        Via B.Diotti 45 - 20153 Milano MI ITALY
+       CTO @ Louise         mobile :   ++39 347 4106213
+                                   :   ++39 348 3658272
+                            eMail  :   gp.ciceri@acm.org,
+                                       gp.ciceri@computer.org
+                                   :   gp.ciceri@louise.it
+                            webSite:   http://www.louise.it
+                            ICQ #  :   94620118
+
+
Index: trunk/anuga_core/source/pypar_dist/supporting_material/contrib/ciceri2
===================================================================
--- trunk/anuga_core/source/pypar_dist/supporting_material/contrib/ciceri2	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/supporting_material/contrib/ciceri2	(revision 8494)
@@ -0,0 +1,68 @@
+Hello,
+I'm not sure you'll like all the things I've
+done with your pypar, so any remark you've to
+to, it's welcome.
+
+My problem: I'm trying to do a little master-slave
+parallel program with pypar, something like the
+standard LAM one-step tutorial.
+To do so, I've found that the actual pypar implementation
+lacks of two things (please, please correct me if I'm wrong):
+
+1. MPI constants in the pypar module (like MPI_ANY_SOURCE, MPI_ANY_TAG)
+2. The capability to read the MPI_Status structure after
+an MPI_Recv.
+
+for 1. one can always stick the constant -1 inside the code
+but for 2. I've found that the actual implementation doesn't export
+MPI_Status to python.
+
+
+So I've just modified mpi.c (and pypar.py, testpypar.py) to
+1. export some MPI constants (trivial)
+2. export MPI_Status structure members along with the error code, in a
+"i(iiii)" (touple of an integer and a four-integer other touple) (ugly)
+3. modify raw_receive methods accordingly (trivial again,
+but pypar API slighty changes).
+
+
+Even worse, I've modified an old mpi.c version (before bcat),
+so please take into account this.
+
+I've attached all the stuff (along with the ms.py demo program).
+If you want to spend some time reviewing it, I'll be thankful.
+
+Best Regards
+/gp
+
+-- 
+"customers, snakes."
+                                                 g.p.
+
+"Non è il caldo ma
+sei tu che alzi la temperatura,
+non i fiori ma
+sei tu che profumi l'atmosfera,
+alzati, girati,
+muoviti, risiediti,
+non è il caldo ma
+sei tu che sei bella vera."
+                                                 max
+
+"anyone can create a cryptography product that he himself cannot break."
+                                                 Bruce Schneier
+
+"If you don't know the things, study them."
+                                                 g.p.
+
+"But writing non-free software is another matter.  Non-free software is
+a plan for subjugating and dividing the people who use it--an
+antisocial practice.  Our goal is to bring that practice to a halt."
+                                                 Richard Stallman
+
+   Gian Paolo Ciceri        via B.Diotti 45 - 20153 Milano MI ITALY
+       CTO @ Louise         mobile :   ++39 348 3658272
+                            eMail  :   gp.ciceri@acm.org, 
+gp.ciceri@louise.it
+                            webSite:   http://www.louise.it
+                            ICQ    :   94620118
Index: trunk/anuga_core/source/pypar_dist/supporting_material/contrib/ciceri3
===================================================================
--- trunk/anuga_core/source/pypar_dist/supporting_material/contrib/ciceri3	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/supporting_material/contrib/ciceri3	(revision 8494)
@@ -0,0 +1,33 @@
+Hello,
+I've just finished with gather and do some work for reduce.
+
+I'd like to have remarks from you about:
+1. the MPI_Op decoding hack.
+2. How to implement reduce with the operation MAXLOC and MINLOC
+
+Have fun
+/gp
+
+-- 
+Discussion: How do you feel about Open Source firms making
+millions through public offerings?
+
+"I wish these companies were making the same millions without
+distributing any non-free, user-subjugating software." --
+                                              Richard Stallman
+
+"We're so forward that sometimes we reach ourself." --
+                                              g.p.
+
+"It is amazing how much you can achieve when you don't have
+to do the real work yourself." --
+                                              Joe Celko
+
+   Gian Paolo Ciceri        Via B.Diotti 45 - 20153 Milano MI ITALY
+       CTO @ Louise         mobile :   ++39 347 4106213
+                                   :   ++39 348 3658272
+                            eMail  :   gp.ciceri@acm.org,
+                                       gp.ciceri@computer.org
+                                   :   gp.ciceri@louise.it
+                            webSite:   http://www.louise.it
+                            ICQ #  :   94620118
Index: trunk/anuga_core/source/pypar_dist/supporting_material/contrib/compile.py
===================================================================
--- trunk/anuga_core/source/pypar_dist/supporting_material/contrib/compile.py	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/supporting_material/contrib/compile.py	(revision 8494)
@@ -0,0 +1,175 @@
+"""compile.py - compile Python C-extension
+
+   Commandline usage: 
+     python compile.py <filename>
+
+   Usage from within Python:
+     import compile
+     compile.compile(<filename>,..)
+
+   Ole Nielsen, Oct 2001      
+"""   
+
+def compile(FNs=None, CC=None, LD = None, SFLAG = None, verbose = 1):
+  """compile(FNs=None, CC=None, LD = None, SFLAG = None):
+  
+     Compile FN(s) using compiler CC (e.g. mpicc), 
+     Loader LD and shared flag SFLAG.
+     If CC is absent use default compiler dependent on platform
+     if LD is absent CC is used.
+     if SFLAG is absent platform default is used
+     FNs can be either one filename or a list of filenames
+     In the latter case, the first will be used to name so file.
+  """
+  import os, string, sys, types
+  
+  # Input check
+  #
+  assert not FNs is None, "No filename provided"
+
+  if not type(FNs) == types.ListType:
+    FNs = [FNs]
+
+  # Determine platform and compiler
+  #
+  if sys.platform == 'sunos5':  #Solaris
+    if CC:
+      compiler = CC
+    else:  
+      compiler = 'gcc'
+    if LD:
+      loader = LD
+    else:  
+      loader = compiler
+    if SFLAG:
+      sharedflag = SFLAG
+    else:  
+      sharedflag = 'G'
+      
+  elif sys.platform == 'osf1V5':  #Compaq AlphaServer
+    if CC:
+      compiler = CC
+    else:  
+      compiler = 'cc'
+    if LD:
+      loader = LD
+    else:  
+      loader = compiler
+    if SFLAG:
+      sharedflag = SFLAG
+    else:  
+      sharedflag = 'shared'    
+      
+  elif sys.platform == 'linux2':  #Linux
+    if CC:
+      compiler = CC
+    else:  
+      compiler = 'gcc'
+    if LD:
+      loader = LD
+    else:  
+      loader = compiler
+    if SFLAG:
+      sharedflag = SFLAG
+    else:  
+      sharedflag = 'shared'    
+      
+  else:
+    if verbose: print "Unrecognised platform %s - revert to default"\
+                %sys.platform
+    
+    if CC:
+      compiler = CC
+    else:  
+      compiler = 'cc'
+    if LD:
+      loader = LD
+    else:  
+      loader = 'ld'
+    if SFLAG:
+      sharedflag = SFLAG
+    else:  
+      sharedflag = 'G'
+
+            
+  # Find location of include files
+  #
+  include = os.path.join(os.path.join(sys.exec_prefix, 'include'),
+                     'python'+sys.version[:3])
+  
+  
+  # Check existence of Python.h
+  #
+  headerfile = include + '/Python.h'
+  try:
+    open(headerfile, 'r')
+  except:
+    raise """Did not find Python header file %s.
+    Make sure files for Python C-extensions are installed. 
+    In debian linux, for example, you need to install a
+    package called something like python2.1-dev""" %headerfile
+  
+  
+  # Check filename(s)
+  #
+  object_files = ''
+  for FN in FNs:        
+    root, ext = os.path.splitext(FN)
+    if ext == '':
+      FN = FN + '.c'
+    elif ext.lower() != '.c':
+      raise Exception, "Unrecognised extension: " + FN
+    
+    try:
+      open(FN,'r')
+    except:    
+      raise Exception, "Could not open: " + FN
+
+    if not object_files: root1 = root  #Remember first filename        
+    object_files += root + '.o '  
+  
+  
+    # Compile
+    #
+    
+    s = "%s -c %s -I%s" %(compiler, FN, include)
+    if verbose:
+      print s
+    else:
+      s = s + ' 2> /dev/null' #Suppress errors
+  
+    try:
+      os.system(s)
+    except:
+      raise 'Could not compile %s - please try manually' %FN  
+
+  
+  # Make shared library (*.so)
+  
+  s = "%s -%s %s -o %s.so" %(loader, sharedflag, object_files, root1)
+  if verbose:
+    print s
+  else:
+    s = s + ' 2> /dev/null' #Suppress warnings
+  
+  try:  
+    err=os.system(s)
+  except:
+    raise 'Could not link %s - please try manually' %root1
+    
+
+
+
+if __name__ == '__main__':
+
+  import sys
+  
+  # Get file to compile
+  #
+  if len(sys.argv) < 2:
+    print "Usage:"
+    print "  python compile.py <filename>"
+    sys.exit()
+
+  compile(sys.argv[1], verbose=1)
+
Index: trunk/anuga_core/source/pypar_dist/supporting_material/contrib/copy2bergen
===================================================================
--- trunk/anuga_core/source/pypar_dist/supporting_material/contrib/copy2bergen	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/supporting_material/contrib/copy2bergen	(revision 8494)
@@ -0,0 +1,13 @@
+#!/usr/bin/env python
+#
+# Copy all files in this dir to bergen
+
+import sys, os
+
+dir = os.getcwd()
+user = os.environ['USERNAME']
+host = 'bergen'
+
+s = 'scp * %s@%s:%s' %(user,host,dir)
+print s
+os.system(s)
Index: trunk/anuga_core/source/pypar_dist/supporting_material/contrib/demo2.py
===================================================================
--- trunk/anuga_core/source/pypar_dist/supporting_material/contrib/demo2.py	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/supporting_material/contrib/demo2.py	(revision 8494)
@@ -0,0 +1,105 @@
+#!/usr/bin/env python
+# Test of MPI module 'pypar' for Python
+# Demonstrates a 'master-slave' parallel program
+# 
+# Run as 
+#   python demo2.py
+# or 
+#   mpirun -np 4 demo2.py
+#
+# GPC, FEB 2002
+
+try:
+  import Numeric
+except:
+  raise 'Module Numeric must be present to run pypar'
+  
+try:
+  import pypar
+except:
+  raise 'Module pypar must be present to run parallel'
+
+print "Modules Numeric, pypar imported OK"
+
+WORKTAG = 1
+DIETAG =  2
+
+
+def master():
+    numCompleted = 0
+    
+    print "[MASTER]: I am processor %d of %d on node %s" %(MPI_myid, MPI_numproc, MPI_node)
+    
+    # start slaves distributing the first work slot
+    for i in range(1, MPI_numproc): 
+        work = workList[i]
+        pypar.raw_send(work, i, WORKTAG) 
+        print "[MASTER]: sent work '%s' to node '%d'" %(work, i)
+
+    # dispach the remaining work slots on dynamic load-balancing policy
+    # the quicker to do the job, the more jobs it takes
+    for work in workList[MPI_numproc:]:
+        result = '  '
+        err = pypar.raw_receive(result, pypar.MPI_ANY_SOURCE, pypar.MPI_ANY_TAG) 
+        print "[MASTER]: received result '%s' from node '%d'" %(result, err[1][0])
+        numCompleted += 1
+        pypar.raw_send(work, err[1][0], WORKTAG)
+        print "[MASTER]: sent work '%s' to node '%d'" %(work, err[1][0])
+    
+    # all works have been dispatched out
+    print  "[MASTER]: toDo : %d" %numWorks
+    print  "[MASTER]: done : %d" %numCompleted
+    
+    # I've still to take into the remaining completions   
+    while(numCompleted < numWorks): 
+        result = '  '
+        err = pypar.raw_receive(result, pypar.MPI_ANY_SOURCE, pypar.MPI_ANY_TAG) 
+        print "[MASTER]: received (final) result '%s' from node '%d'" %(result, err[1][0])
+        numCompleted += 1
+        print "[MASTER]: %d completed" %numCompleted
+        
+    print "[MASTER]: about to terminate slaves"
+
+    # say slaves to stop working
+    for i in range(1, MPI_numproc): 
+        pypar.raw_send('#', i, DIETAG) 
+        print "[MASTER]: sent (final) work '%s' to node '%d'" %(0, i)
+        
+    return
+    
+def slave():
+
+    print "[SLAVE %d]: I am processor %d of %d on node %s" %(MPI_myid, MPI_myid, MPI_numproc, MPI_node)
+
+    while 1:
+        result = ' '
+        err = pypar.raw_receive(result, pypar.MPI_ANY_SOURCE, pypar.MPI_ANY_TAG) 
+        print "[SLAVE %d]: received work '%s' with tag '%d' from node '%d'" %(MPI_myid, result, err[1][1], err[1][0])
+       
+        if (err[1][1] == DIETAG):
+            print "[SLAVE %d]: received termination from node '%d'" %(MPI_myid, 0)
+            return
+        else:
+            result = 'X'+result
+            pypar.raw_send(result, 0)
+            print "[SLAVE %d]: sent result '%s' to node '%d'" %(MPI_myid, result, 0)
+            
+       
+
+if __name__ == '__main__':
+    MPI_myid =    pypar.rank()
+    MPI_numproc = pypar.size()
+    MPI_node =    pypar.Get_processor_name()
+
+    workList = ('_dummy_', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j')
+    _workList = ('_dummy_', 'a', 'b', 'c', 'd', 'e', 'f')
+    numWorks = len(workList) - 1
+
+    if MPI_myid == 0:
+        master()
+    else:
+        slave()
+
+    pypar.Finalize()
+    print "MPI environment finalized."
+        	
Index: trunk/anuga_core/source/pypar_dist/supporting_material/contrib/fra_DOUG.husk
===================================================================
--- trunk/anuga_core/source/pypar_dist/supporting_material/contrib/fra_DOUG.husk	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/supporting_material/contrib/fra_DOUG.husk	(revision 8494)
@@ -0,0 +1,96 @@
+From doug@arbor.net  Fri Aug 29 05:22:22 2003
+Return-Path: <doug@arbor.net>
+X-Original-To: ole@localhost
+Delivered-To: ole@localhost.anu.edu.au
+Received: from localhost (localhost [127.0.0.1])
+	by cuttlefish.anu.edu.au (Postfix) with ESMTP id 8D5C63E96
+	for <ole@localhost>; Fri, 29 Aug 2003 05:22:22 +1000 (EST)
+Received: from wintermute.anu.edu.au [150.203.33.75]
+	by localhost with POP3 (fetchmail-6.2.1)
+	for ole@localhost (single-drop); Fri, 29 Aug 2003 05:22:22 +1000 (EST)
+Received: from anu.edu.au (anumail4.anu.edu.au [150.203.2.44])
+	by maths.anu.edu.au (8.11.6/8.11.6) with ESMTP id h7SJM6k23100
+	for <ole@maths.anu.edu.au>; Fri, 29 Aug 2003 05:22:07 +1000 (EST)
+Received: from state.arbornetworks.com (state.arbornetworks.com [204.181.64.20])
+	by anu.edu.au (8.12.9/8.12.9) with ESMTP id h7SJM4mQ004241
+	for <Ole.Nielsen@anu.edu.au>; Fri, 29 Aug 2003 05:22:05 +1000 (EST)
+Received: from arbor-orr.arbor.net (unknown [10.0.1.106])
+	by state.arbornetworks.com (Arbor Networks mail relay) with ESMTP id EF33A63481
+	for <Ole.Nielsen@anu.edu.au>; Thu, 28 Aug 2003 15:22:01 -0400 (EDT)
+Message-Id: <5.1.0.14.2.20030828104019.03efe4d0@state.arbornetworks.com>
+X-Sender: doug@arbor.net
+X-Mailer: QUALCOMM Windows Eudora Version 5.1
+Date: Thu, 28 Aug 2003 15:22:00 -0400
+To: Ole.Nielsen@anu.edu.au
+From: Doug Orr <doug@arbor.net>
+Subject: Re: pypar
+In-Reply-To: <20030828115708.GB3384@anu.edu.au>
+References: <5.1.0.14.2.20030827235617.04380488@state.arbornetworks.com>
+ <5.1.0.14.2.20030827235617.04380488@state.arbornetworks.com>
+Mime-Version: 1.0
+Content-Type: text/plain; charset="us-ascii"; format=flowed
+X-Sender-Domain: arbor.net
+X-Spam-Score: (-2)
+X-Spam-Tests: EMAIL_ATTRIBUTION,IN_REP_TO,MSG_ID_ADDED_BY_MTA_3,QUOTED_EMAIL_TEXT,REFERENCES,REPLY_WITH_QUOTES
+X-Scanned-By: MIMEDefang 2.36
+X-UIDL: 79381bbce14513976771cae5228e1cf3
+X-Spam-Status: No, hits=-25.8 required=5.0
+	tests=EMAIL_ATTRIBUTION,IN_REP_TO,MSG_ID_ADDED_BY_MTA_3,
+	      QUOTED_EMAIL_TEXT,REFERENCES,REPLY_WITH_QUOTES
+	autolearn=ham version=2.53
+X-Spam-Level: 
+X-Spam-Checker-Version: SpamAssassin 2.53 (1.174.2.15-2003-03-30-exp)
+
+
+Hey Ole,
+
+One more suggestion -- it's obvious in retrospect, but it would be useful 
+to note in your install docs that you need to have Numeric, pypar, and 
+python 2.3 (for the current version of Numeric, anyway) installed on all 
+hosts.  It's what I *thought*, but trying without that set up produced no 
+valuable diagnostic indicating why I was losing.  I would have saved a 
+little time if I had gone ahead and done all of the installs as my first 
+step...
+
+Not a big deal, but it's always good to hear from the people currently 
+enduring pain, since they always for get what the pain was after it's over :)
+
+         -Doug
+
+At 09:57 PM 8/28/2003 +1000, Ole Moller Nielsen wrote:
+>Hi Doug
+>
+>Thank you very much for your mail
+>
+>May I ask what platform you are using: Solaris, Linux, Windows?
+>
+>Pypar was tested on all the platforms I had access to when I wrote it.
+>These days I use it exclusively with Linux.
+>
+>Cheers
+>Ole
+>
+>
+>On Wed, Aug 27, 2003 at 11:59:29PM -0400, Doug Orr wrote:
+> > hi ole,
+> >
+> > I'm trying pypar with lam7.0 and it was crashing in MPI_Init.  it looks
+> > like they're assuming the last argument in the argv list is NULL.
+> >
+> > I assume someone has probably already pointed this out, but just in case...
+> >
+> >       -doug
+> >
+>
+>--
+>-------------------------------------------------------------------
+>Ole Nielsen                     | Email: Ole.Nielsen@anu.edu.au
+>-------------------------------------------------------------------
+>Mathematical Sciences Institute | Phone: +61 2 6125 3873 (Direct)
+>Australian National University  | Fax:   +61 2 6125 5549
+>Canberra ACT 0200               |
+>Australia                       |
+>-------------------------------------------------------------------
+>ANU CRICOS # 00120C
+>-------------------------------------------------------------------
+
Index: trunk/anuga_core/source/pypar_dist/supporting_material/contrib/fra_DOUG.husk2
===================================================================
--- trunk/anuga_core/source/pypar_dist/supporting_material/contrib/fra_DOUG.husk2	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/supporting_material/contrib/fra_DOUG.husk2	(revision 8494)
@@ -0,0 +1,89 @@
+From doug@arbor.net  Thu Aug 28 22:46:22 2003
+Return-Path: <doug@arbor.net>
+X-Original-To: ole@localhost
+Delivered-To: ole@localhost.anu.edu.au
+Received: from localhost (localhost [127.0.0.1])
+	by cuttlefish.anu.edu.au (Postfix) with ESMTP id 376FE3E96
+	for <ole@localhost>; Thu, 28 Aug 2003 22:46:22 +1000 (EST)
+Received: from wintermute.anu.edu.au [150.203.33.75]
+	by localhost with POP3 (fetchmail-6.2.1)
+	for ole@localhost (single-drop); Thu, 28 Aug 2003 22:46:22 +1000 (EST)
+Received: from anu.edu.au (anumail3.anu.edu.au [150.203.2.43])
+	by maths.anu.edu.au (8.11.6/8.11.6) with ESMTP id h7SCjok20775
+	for <ole@maths.anu.edu.au>; Thu, 28 Aug 2003 22:45:50 +1000 (EST)
+Received: from state.arbornetworks.com (state.arbornetworks.com [204.181.64.20])
+	by anu.edu.au (8.12.9/8.12.9) with ESMTP id h7SCjnah001230
+	for <Ole.Nielsen@anu.edu.au>; Thu, 28 Aug 2003 22:45:50 +1000 (EST)
+Received: from arbor-orr.arbor.net (plmomi-l10-816.dsl.tds.net [134.215.219.54])
+	by state.arbornetworks.com (Arbor Networks mail relay) with ESMTP id 8FC8E6355C
+	for <Ole.Nielsen@anu.edu.au>; Thu, 28 Aug 2003 08:45:47 -0400 (EDT)
+Message-Id: <5.1.0.14.2.20030828084434.043a5f68@state.arbornetworks.com>
+X-Sender: doug@arbor.net
+X-Mailer: QUALCOMM Windows Eudora Version 5.1
+Date: Thu, 28 Aug 2003 08:45:44 -0400
+To: Ole.Nielsen@anu.edu.au
+From: Doug Orr <doug@arbor.net>
+Subject: Re: pypar
+In-Reply-To: <20030828115708.GB3384@anu.edu.au>
+References: <5.1.0.14.2.20030827235617.04380488@state.arbornetworks.com>
+ <5.1.0.14.2.20030827235617.04380488@state.arbornetworks.com>
+Mime-Version: 1.0
+Content-Type: text/plain; charset="us-ascii"; format=flowed
+X-Sender-Domain: arbor.net
+X-Spam-Score: (-2)
+X-Spam-Tests: EMAIL_ATTRIBUTION,IN_REP_TO,MSG_ID_ADDED_BY_MTA_3,QUOTED_EMAIL_TEXT,REFERENCES,REPLY_WITH_QUOTES
+X-Scanned-By: MIMEDefang 2.36
+X-UIDL: 477221179dfe4b86b3d53a28e3ab6647
+X-Spam-Status: No, hits=-25.8 required=5.0
+	tests=EMAIL_ATTRIBUTION,IN_REP_TO,MSG_ID_ADDED_BY_MTA_3,
+	      QUOTED_EMAIL_TEXT,REFERENCES,REPLY_WITH_QUOTES
+	autolearn=ham version=2.53
+X-Spam-Level: 
+X-Spam-Checker-Version: SpamAssassin 2.53 (1.174.2.15-2003-03-30-exp)
+
+At 09:57 PM 8/28/2003 +1000, Ole Moller Nielsen wrote:
+>Hi Doug
+>
+>Thank you very much for your mail
+>
+>May I ask what platform you are using: Solaris, Linux, Windows?
+>
+>Pypar was tested on all the platforms I had access to when I wrote it.
+>These days I use it exclusively with Linux.
+
+yes of course, ole.  the package looks very nice so far.  thanks for 
+writing it!
+
+linux... redhat 7.1 or 2 with the lam mpi package.  it's trivial, but if 
+you want a diff for the fix, I can give it to you...
+
+         -doug
+
+
+>Cheers
+>Ole
+>
+>
+>On Wed, Aug 27, 2003 at 11:59:29PM -0400, Doug Orr wrote:
+> > hi ole,
+> >
+> > I'm trying pypar with lam7.0 and it was crashing in MPI_Init.  it looks
+> > like they're assuming the last argument in the argv list is NULL.
+> >
+> > I assume someone has probably already pointed this out, but just in case...
+> >
+> >       -doug
+> >
+>
+>--
+>-------------------------------------------------------------------
+>Ole Nielsen                     | Email: Ole.Nielsen@anu.edu.au
+>-------------------------------------------------------------------
+>Mathematical Sciences Institute | Phone: +61 2 6125 3873 (Direct)
+>Australian National University  | Fax:   +61 2 6125 5549
+>Canberra ACT 0200               |
+>Australia                       |
+>-------------------------------------------------------------------
+>ANU CRICOS # 00120C
+>-------------------------------------------------------------------
+
Index: trunk/anuga_core/source/pypar_dist/supporting_material/contrib/fra_DOUG.husk3
===================================================================
--- trunk/anuga_core/source/pypar_dist/supporting_material/contrib/fra_DOUG.husk3	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/supporting_material/contrib/fra_DOUG.husk3	(revision 8494)
@@ -0,0 +1,108 @@
+Hi Ole,
+
+Sorry I don't have the original handy and I'm running out of the house, but 
+here's the affected line... shouldn't be hard to find.  It's around line 
+472 in mpiext.c.  I'll let you know how things go as I start to use it more!
+
+   /* Reconstruct C-commandline */
+   /*                           */
+   argc = PyList_Size(input); //Number of commandline arguments
+- argv = (char**) malloc((argc)*sizeof(char*));
++ argv = (char**) malloc((argc+1)*sizeof(char*));
+
+   for (i=0; i<argc; i++)
+     argv[i] = PyString_AsString( PyList_GetItem(input, i) );
+
++ argv[i] = NULL;
+
+
+
+
+At 06:53 PM 8/29/2003 +1000, Ole Moller Nielsen wrote:
+>Hi Doug
+>
+>Thank you so much for your input.
+>I'll put it into the todo list.
+>I will be teaching a high performamnce course next semester in Thailand
+>and updating pypar is on the agenda.
+>
+>I gather that youi got it to work by fixing pypar to honor
+>the 'NULL' assumption you mentioned, right?
+>And, yes, I'd be very happy to see the diff or even the fixed version
+>if you don't mind.
+>
+>(I'll add you to the growing list of contributors)
+>
+>
+>Cheers
+>Ole
+>
+>On Thu, Aug 28, 2003 at 03:22:00PM -0400, Doug Orr wrote:
+> >
+> > Hey Ole,
+> >
+> > One more suggestion -- it's obvious in retrospect, but it would be useful
+> > to note in your install docs that you need to have Numeric, pypar, and
+> > python 2.3 (for the current version of Numeric, anyway) installed on all
+> > hosts.  It's what I *thought*, but trying without that set up produced no
+> > valuable diagnostic indicating why I was losing.  I would have saved a
+> > little time if I had gone ahead and done all of the installs as my first
+> > step...
+> >
+> > Not a big deal, but it's always good to hear from the people currently
+> > enduring pain, since they always for get what the pain was after it's over
+> > :)
+> >
+> >         -Doug
+> >
+> > At 09:57 PM 8/28/2003 +1000, Ole Moller Nielsen wrote:
+> > >Hi Doug
+> > >
+> > >Thank you very much for your mail
+> > >
+> > >May I ask what platform you are using: Solaris, Linux, Windows?
+> > >
+> > >Pypar was tested on all the platforms I had access to when I wrote it.
+> > >These days I use it exclusively with Linux.
+> > >
+> > >Cheers
+> > >Ole
+> > >
+> > >
+> > >On Wed, Aug 27, 2003 at 11:59:29PM -0400, Doug Orr wrote:
+> > >> hi ole,
+> > >>
+> > >> I'm trying pypar with lam7.0 and it was crashing in MPI_Init.  it looks
+> > >> like they're assuming the last argument in the argv list is NULL.
+> > >>
+> > >> I assume someone has probably already pointed this out, but just in
+> > >case...
+> > >>
+> > >>       -doug
+> > >>
+> > >
+> > >--
+> > >-------------------------------------------------------------------
+> > >Ole Nielsen                     | Email: Ole.Nielsen@anu.edu.au
+> > >-------------------------------------------------------------------
+> > >Mathematical Sciences Institute | Phone: +61 2 6125 3873 (Direct)
+> > >Australian National University  | Fax:   +61 2 6125 5549
+> > >Canberra ACT 0200               |
+> > >Australia                       |
+> > >-------------------------------------------------------------------
+> > >ANU CRICOS # 00120C
+> > >-------------------------------------------------------------------
+> >
+>
+>--
+>-------------------------------------------------------------------
+>Ole Nielsen                     | Email: Ole.Nielsen@anu.edu.au
+>-------------------------------------------------------------------
+>Mathematical Sciences Institute | Phone: +61 2 6125 3873 (Direct)
+>Australian National University  | Fax:   +61 2 6125 5549
+>Canberra ACT 0200               |
+>Australia                       |
+>-------------------------------------------------------------------
+>ANU CRICOS # 00120C
+>-------------------------------------------------------------------
+
Index: trunk/anuga_core/source/pypar_dist/supporting_material/contrib/frost_pypar.py
===================================================================
--- trunk/anuga_core/source/pypar_dist/supporting_material/contrib/frost_pypar.py	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/supporting_material/contrib/frost_pypar.py	(revision 8494)
@@ -0,0 +1,639 @@
+# =============================================================================
+# pypar.py - Parallel Python using MPI
+# Copyright (C) 2001, 2002 Ole M. Nielsen 
+#              (Center for Mathematics and its Applications ANU and APAC)
+#              and Gian Paolo Ciceri (Milano, Italy)          
+#
+#    This program is free software; you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License as published by
+#    the Free Software Foundation; either version 2 of the License, or
+#    (at your option) any later version.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License (http://www.gnu.org/copyleft/gpl.html)
+#    for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program; if not, write to the Free Software
+#    Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307
+#
+#
+# Contact addresses: Ole.Nielsen@anu.edu.au, gp.ciceri@acm.org         
+#
+# version 1.6, 18 October 2002                                      
+# =============================================================================
+
+"""Module pypar.py - Parallel Python using MPI
+
+Public functions:
+
+size() -- Number of processors
+rank() -- Id of current processor
+Get_processor_name() -- Return host name of current node
+
+send() -- Blocking send (all types)
+receive() -- Blocking receive (all types)
+raw_send() -- Blocking send (Numeric arrays and strings)
+raw_receive() -- Blocking receive (Numeric arrays and strings)
+bcast() -- Broadcast
+Wtime() -- MPI wall time
+Barrier() -- Synchronisation point. Makes processors wait until all processors
+             have reached this point.
+Abort() -- Terminate all processes. 
+Finalize() -- Cleanup MPI. No parallelism can take place after this point. 
+
+
+See doc strings of individual functions for detailed documentation.
+"""
+
+import os, sys
+
+# -----------------------------------------------------------------------------
+# Options directory with default values - to be set by user
+#
+
+options = { 
+  'vanilla_bufsize': None,   # Buffersize (chars) used for vanilla sends
+  'verbose': 0
+}
+
+# Constants
+#
+control_tag = 32            # Tag used to identify control information
+control_data_max_size = 256 # Maximal size of string holding control data
+
+
+#------------------------------------------------------------------------
+# MPI Status block (can be queried by user after each receive
+#------------------------------------------------------------------------
+
+class Status:
+  def __init__(self):
+    pass
+  
+  def set_values(self, status_tuple):
+    self.source = status_tuple[0] 
+    self.tag = status_tuple[1] 
+    self.error = status_tuple[2] 
+    self.length = status_tuple[3]
+    
+status = Status() #Initialise status object      
+
+#---------------------------------------------------------------------------
+# Communication functions
+#--------------------------------------------------------------------------
+
+
+def raw_send(x, destination, tag=0, vanilla=0):
+  """Wrapper for raw MPI send.
+     Send x to destination with tag.
+     
+     Automatically determine appropriate protocol
+     and call corresponding send function.
+     
+     The variable x can be any (picklable) type, but
+     Numeric variables and text strings will most efficient.
+     Setting vanilla = 1 forces vanilla mode for any type.
+
+  """
+
+  protocol = get_control_info(x, vanilla)[0]
+  if protocol == 'array':
+    send_array(x, destination, tag)  
+  elif protocol == 'string':
+    send_string(x, destination, tag)            
+  else:  
+    send_vanilla(x, destination, tag)
+
+      
+def raw_receive(x, source, tag=0, vanilla=0):
+  """Wrapper for raw MPI receive.
+     Receive something of same size as x from source with tag.
+     
+     Automatically determine appropriate protocol
+     and call corresponding receive function.
+     
+     The variable x can be any (picklable) type, but
+     Numeric variables and text strings will most efficient.
+     Setting vanilla = 1 forces vanilla mode for any type.
+  """
+
+  
+  protocol = get_control_info(x, vanilla)[0]
+  if protocol == 'array':
+    err, stat = receive_array(x, source, tag) 
+    if not err:
+      status.set_values(stat)
+    else:
+      raise 'receive_array failed with error code %d' %err  
+  elif protocol == 'string':
+    err, stat = receive_string(x, source, tag)
+    if not err:
+      status.set_values(stat)
+    else:
+      raise 'receive_string failed with error code %d' %err  
+  else:  
+    x = receive_vanilla(x, source, tag)
+
+  return x
+
+
+def send(x, destination, tag=0, vanilla=0):
+  """Wrapper for easy MPI send.
+     Send x to destination with tag.
+     
+     Automatically determine appropriate protocol
+     and call corresponding send function.
+     Also passes type and size information on as preceding message to
+     simplify the receive call.
+     
+     The variable x can be any (picklable) type, but
+     Numeric variables and text strings will most efficient.
+     Setting vanilla = 1 forces vanilla mode for any type.
+
+  """
+  import string
+
+  control_info = get_control_info(x, vanilla)
+  protocol = control_info[0]
+  
+  if protocol == 'array':
+    send_control_info(control_info, destination)
+    
+    send_array(x, destination, tag)    
+  elif protocol == 'string':
+    send_control_info(control_info, destination)    
+    
+    send_string(x, destination, tag)          
+  elif protocol == 'vanilla':
+    from cPickle import dumps     
+    s = dumps(x, 1)
+    control_info[2] = str(len(s))
+
+    send_control_info(control_info, destination)   
+    
+    send_string(s, destination, tag)
+  else:
+    raise "Unknown values for protocol: %s" %protocol    
+
+      
+def receive(source, tag=0):
+  """Wrapper for easy MPI receive.
+     Receive data from source with tag.
+     
+     Assumes preceding message containing protocol, type, size.
+     Create appropriate buffer and receive data.
+  """
+
+  control_info = receive_control_info(source)
+  
+  protocol = control_info[0]
+  typecode = control_info[1]
+  size =     control_info[2]  
+  
+  if protocol == 'array':
+    import Numeric
+    x = Numeric.zeros(size,typecode)
+    err, stat = receive_array(x, source, tag)    
+    if not err:
+      status.set_values(stat)
+    else:
+      raise 'receive_array failed with error code %d' %err  
+  elif protocol == 'string':
+    x = ' '*size
+    err, stat = receive_string(x, source, tag)       
+    if not err:
+      status.set_values(stat)
+    else:
+      raise 'receive_string failed with error code %d' %err  
+  elif protocol == 'vanilla':
+    from cPickle import loads 
+    s = ' '*size    
+    err, stat = receive_string(s, source, tag)
+    if not err:
+      status.set_values(stat)
+    else:
+      raise 'receive_string failed with error code %d' %err  
+    
+    x = loads(s)
+  else:
+    raise "Unknown values for protocol: %s" %protocol
+      
+  return x  
+
+def bcast(x, source, vanilla=0):
+  """Wrapper for MPI bcast.
+     Broadcast x from source.
+     
+     Automatically determine appropriate protocol
+     and call corresponding send function.
+     
+     The variable x can be any (picklable) type, but
+     Numeric variables and text strings will most efficient.
+     Setting vanilla = 1 forces vanilla mode for any type.
+
+  """
+
+  protocol = get_control_info(x, vanilla)[0]
+  if protocol == 'array':
+    bcast_array(x, source)    
+  elif protocol == 'string':
+    bcast_string(x, source)          
+  elif protocol == 'vanilla':
+    from cPickle import loads, dumps 
+    s = dumps(x, 1)
+    s = s + ' '*int(0.1*len(s)) #safety 
+    bcast_string(s, source)
+    x = loads(s)
+  else:
+    raise "Unknown values for protocol: %s" %protocol  
+    
+  return x         
+
+def raw_scatter(s, nums, d, source, vanilla=0):
+  """Wrapper for MPI scatter.
+     Scatter the first nums elements in s to processor d
+     (of the same nums size) from source.
+     
+     Automatically determine appropriate protocol
+     and call corresponding send function.
+     
+     The variable s can be any (picklable) type, but
+     Numeric variables and text strings will most efficient.
+     Setting vanilla = 1 forces vanilla mode for any type.
+
+  """
+
+  protocol = get_control_info(s, vanilla)[0]
+  if protocol == 'array':
+    scatter_array(s, nums, d, source)    
+  elif protocol == 'string':
+    scatter_string(s, nums, d, source)          
+  elif protocol == 'vanilla':
+    raise "Protocol: %s unsupported for scatter" %protocol
+  else:
+    raise "Unknown values for protocol: %s" %protocol  
+    
+  return d         
+
+
+def scatter(s, nums, source, vanilla=0):
+  """Wrapper for easy MPI Scatter receive.
+     Receive data from source with tag.
+     
+     Create appropriate buffer and receive data.
+  """
+
+  control_info = get_control_info(s)
+  
+  protocol = control_info[0]
+  typecode = control_info[1]
+  size =  nums   
+  
+  if protocol == 'array':
+    import Numeric
+    x = Numeric.zeros(size,typecode)
+    scatter_array(s, size, x, source)    
+  elif protocol == 'string':
+    x = ' '*size
+    scatter_string(s, size, x, source)          
+  elif protocol == 'vanilla':
+    raise "Protocol: %s unsupported for scatter" %protocol
+  else:
+    raise "Unknown values for protocol: %s" %protocol
+      
+  return x  
+
+def raw_gather(s, nums, d, source, vanilla=0):
+  """Wrapper for MPI gather.
+     Gather first nums elements in s to d (of the same size) from source.
+     
+     Automatically determine appropriate protocol
+     and call corresponding send function.
+     
+     The variable s can be any (picklable) type, but
+     Numeric variables and text strings will most efficient.
+     Setting vanilla = 1 forces vanilla mode for any type.
+
+  """
+
+  protocol = get_control_info(s, vanilla)[0]
+  if protocol == 'array':
+    gather_array(s, nums, d, source)    
+  elif protocol == 'string':
+    gather_string(s, nums, d, source)          
+  elif protocol == 'vanilla':
+    raise "Protocol: %s unsupported for gather" %protocol
+  else:
+    raise "Unknown values for protocol: %s" %protocol  
+    
+  return d         
+
+
+def gather(s, nums, source, vanilla=0):
+  """Wrapper for easy MPI Gather receive.
+     Receive data from source with tag.
+     
+     Create appropriate buffer and receive data.
+  """
+
+  control_info = get_control_info(s)
+  
+  protocol = control_info[0]
+  typecode = control_info[1]
+  s_size =  nums   
+  
+  if protocol == 'array':
+    import Numeric
+    x = Numeric.zeros(s_size * size(),typecode)
+    gather_array(s, s_size, x, source)    
+  elif protocol == 'string':
+    x = ' '*s_size*size()
+    gather_string(s, s_size, x, source)          
+  elif protocol == 'vanilla':
+    raise "Protocol: %s unsupported for gather" %protocol
+  else:
+    raise "Unknown values for protocol: %s" %protocol
+      
+  return x  
+
+
+
+def raw_reduce(s, d, nums, op, source, vanilla=0):
+  """Wrapper for MPI_Reduce.
+     Reduce nums elements in s to d (of the same size) at source
+     applying operation op.
+     
+     Automatically determine appropriate protocol
+     and call corresponding send function.
+    
+  """
+
+  protocol = get_control_info(s, vanilla)[0]
+  if protocol == 'array':
+    if not s.typecode() == d.typecode():
+      raise "Input array and buffer must have the same typecode"
+    reduce_array(s, d, nums, op, source)          
+  elif (protocol == 'vanilla' or protocol == 'string'):
+    raise "Protocol: %s unsupported for reduce" %protocol
+  else:
+    raise "Unknown values for protocol: %s" %protocol  
+    
+  return d         
+
+
+def reduce(s, nums, op, source, vanilla=0):
+  """Wrapper for easy MPI Gather receive.
+     Receive data from source with tag.
+     
+     Create appropriate buffer and receive data.
+  """
+
+  control_info = get_control_info(s)
+  
+  protocol = control_info[0]
+  typecode = control_info[1]
+  s_size =  nums   
+  
+  if protocol == 'array':
+    import Numeric
+    x = Numeric.zeros(s_size * size(),typecode)
+    reduce_array(s, x, s_size, op, source)    
+  elif (protocol == 'vanilla' or protocol == 'string'):
+    raise "Protocol: %s unsupported for reduce" %protocol
+  else:
+    raise "Unknown values for protocol: %s" %protocol
+      
+  return x  
+
+
+      
+
+#---------------------------------------------------------
+# AUXILIARY FUNCTIONS
+#---------------------------------------------------------
+def get_control_info(x, vanilla=0):
+  """Determine which protocol to use for communication:
+     (Numeric) arrays, strings, or vanilla based x's type.
+
+     There are three protocols:
+     'array':   Numeric arrays of type 'i', 'l', 'f', or 'd' can be communicated
+                with mpiext.send_array and mpiext.receive_array.
+     'string':  Text strings can be communicated with mpiext.send_string and
+                mpiext.receive_string.
+     'vanilla': All other types can be communicated using the scripts send_vanilla and
+                receive_vanilla provided that the objects can be serialised using
+                pickle (or cPickle). The latter mode is less efficient than the
+                first two but it can handle complex structures.
+
+     Rules:
+     If keyword argument vanilla == 1, vanilla is chosen regardless of 
+     x's type.
+     Otherwise if x is a string, the string protocol is chosen
+     If x is an array, the 'array' protocol is chosen provided that x has one
+     of the admissible typecodes.
+  """
+
+  protocol = 'vanilla'
+  typecode = ' '
+  size = '0'
+  
+  if not vanilla:
+    #if type(x).__name__ == 'string':  # OK in Python 2.1 but not 2.2
+    if type(x).__name__[0:3] == 'str': # Fixed by Gian Paolo Ciceri 10/2/2   
+      protocol = 'string'
+      typecode = 'c'
+      size = len(x)
+    elif type(x).__name__ == 'array':
+      try:
+        import Numeric
+
+        typecode = x.typecode() 
+        if typecode in ['i', 'l', 'f', 'd']:
+          protocol = 'array'
+          size = len(x)
+        else:    
+          print "WARNING (pypar.py): Numeric object type %s is not supported."\
+                %(x.typecode())
+          print "Only types 'i', 'l', 'f', 'd' are supported,",
+          print "Reverting to vanilla mode."
+          protocol = 'vanilla'
+  
+      except:
+        print "WARNING (pypar.py): Numeric module could not be imported,",
+        print "reverting to vanilla mode"
+        protocol = 'vanilla'
+
+
+  control_info = [protocol, typecode, str(size)]
+
+  return control_info
+
+
+
+#----------------------------------------------
+
+def send_control_info(control_info, destination):
+  """Send control info to destination
+  """
+  import string
+  
+  msg = string.join(control_info,',')
+  send_string(msg, destination, control_tag)
+
+  
+def receive_control_info(source):
+  """Receive control info from source
+  """
+  import string
+  
+  msg = ' '*control_data_max_size
+  err, stat = receive_string(msg, source, control_tag)
+  if err:
+    raise Exception
+  #NB: Do not set status block here  
+
+  control_info = msg.split(',')
+  assert len(control_info) == 3
+  control_info[2] = int(control_info[2])
+
+  return control_info
+
+
+
+
+#
+# Used only by raw communication
+#
+def send_vanilla(x, destination, tag=0):
+  from cPickle import dumps
+  from mpiext import send_string as send
+  
+  s=dumps(x, 1)
+  send(s, destination, tag)     
+  return len(s)
+
+
+def receive_vanilla(x, source, tag=0):
+  from cPickle import loads, dumps
+  from mpiext import receive_string as receive
+
+
+  #Create buffer of the right size
+  #(assuming that x is similar to sent x).
+
+  if options['vanilla_bufsize']:
+    s = ' '*options['vanilla_bufsize']
+  else:  
+    s = dumps(x, 1)
+    s = s + ' '*int(0.1*len(s)) #safety
+                  
+  receive(s, source, tag)
+
+  return loads(s)
+   
+
+
+#----------------------------------------------------------------------------
+# Initialise module
+#----------------------------------------------------------------------------
+
+# Take care of situation where module is part of package
+import os, string, os.path
+dirname = os.path.dirname(string.replace(__name__,'.',os.sep))
+if dirname and dirname[-1] != os.sep:
+  dirname += os.sep   
+
+extension = dirname+'mpiext'
+
+# Import MPI extension
+#
+# Verify existence of mpiext.so.
+
+try:
+  import mpiext
+except:
+  errmsg = 'ERROR: compiled C extension %s.c not accessible:\n' %extension
+  errmsg += 'Please compile it e.g. by using python setup.py install\n'
+  raise Exception, errmsg
+  
+
+# Determine if MPI program is allowed to run sequentially on current platform
+# Attempting to check this automatically may case some systems to hang.
+#
+if sys.platform in ['linux2', 'sunos5', 'win32']:  #Linux (LAM,MPICH) or Sun (MPICH) or Win32 (MPICH)
+  error = 0  #Sequential execution of MPI is allowed
+else:  
+  cmdstring = '"import mpiext; import sys; mpiext.init(sys.argv)"'
+  error = os.system('python -c %s >/dev/null 2>/dev/null' %cmdstring) 
+  # The check is performed in a separate shell.
+  # Reason: The Alpha server or the Sun cannot recover from a 
+  # try: 
+  #   mpiext.init(sys.argv)
+
+  # On LAM/Linux, this test causes system to hang.  
+
+
+# Initialise MPI
+#
+# Attempt to initialise mpiext.so
+# If this fails, define a rudimentary interface suitable for
+# sequential execution.
+
+if error:
+  print "WARNING: MPI library could not be initialised - running sequentially"
+
+  # Define rudimentary functions to keep sequential programs happy
+
+  def size(): return 1
+  def rank(): return 0
+
+  def Get_processor_name():
+    import os
+    try:
+      hostname = os.environ['HOST']
+    except:
+      try:  
+        hostname = os.environ['HOSTNAME']  
+      except:
+        hostname = 'Unknown'  
+
+    return hostname
+      
+
+  def Abort():
+    import sys
+    sys.exit()
+
+  def Finalize(): pass
+  
+  def Barrier(): pass  
+
+  def Wtime():
+    import time
+    return time.time()
+
+else:
+
+  from mpiext import size, rank, Barrier, Wtime, Get_processor_name,\
+                  init, Finalize, Abort, send_string, receive_string,\
+                  send_array, receive_array, bcast_string, bcast_array,\
+                  scatter_string, scatter_array,\
+                  gather_string, gather_array,\
+                  reduce_array,\
+                  MPI_ANY_TAG as any_tag, MPI_ANY_SOURCE as any_source,\
+                  MAX, MIN, SUM, PROD, LAND, BAND,\
+                  LOR, BOR, LXOR, BXOR
+
+  init(sys.argv) #Initialise MPI with cmd line (needed by MPICH/Linux)  
+
+  if rank() == 0:     
+    print "MPI initialised OK with %d processors" %size()
+
+
+
+
+
+
Index: trunk/anuga_core/source/pypar_dist/supporting_material/contrib/frost_setup.py
===================================================================
--- trunk/anuga_core/source/pypar_dist/supporting_material/contrib/frost_setup.py	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/supporting_material/contrib/frost_setup.py	(revision 8494)
@@ -0,0 +1,131 @@
+#!/usr/bin/env python
+
+from distutils.core import setup, Extension
+import distutils.sysconfig
+import os, sys
+import popen2
+import string
+import tempfile
+
+def setup_compiler():
+    distutils.sysconfig.get_config_vars()
+    config_vars = distutils.sysconfig._config_vars
+    if sys.platform == 'sunos5':
+        config_vars['LDSHARED'] = "gcc -G"
+        config_vars['CCSHARED'] = ""
+
+
+def uniq_arr(arr):
+    """Remove repeated values from an array and return new array."""
+    ret = []
+    for i in arr:
+        if i not in ret:
+            ret.append(i)
+    return ret
+
+
+def _run_command(cmd):
+    out_file, in_file, err_file = popen2.popen3(cmd)
+    output = out_file.read() + err_file.read()
+    out_file.close()
+    in_file.close()
+    err_file.close()
+    # need this hack to get the exit status
+    out_file = os.popen(cmd)
+    if out_file.close():
+        # close returns exit status of command.
+        return ""
+    else:
+        # no errors, out_file.close() returns None.
+        return output
+
+
+def _get_mpi_cmd():
+    """Returns the output of the command used to compile using
+    mpicc."""    
+    # LAM
+    output = _run_command("mpicc -showme")
+    if output:
+        return output
+
+    # MPICH
+    # works with MPICH version 1.2.1 (on Debian)
+    output = _run_command("mpicc -compile_info -link_info")
+    if output:
+        return output
+
+    # old version of MPICH needs this hack.
+    tmp_base = tempfile.mktemp()
+    tmp_c = tmp_base + ".c"
+    tmp_o = tmp_base + ".o"
+    tmp_file = open(tmp_c, "w")
+    tmp_file.write('#include "mpi.h"\nint main(){return 0;}\n')
+    tmp_file.close()
+    output = _run_command("mpicc -show;"\
+                          "mpicc -echo -c %s -o %s"%(tmp_c, tmp_o))
+    os.remove(tmp_c)
+    if os.path.exists(tmp_o):
+        os.remove(tmp_o)
+    if output:
+        return output
+    else:
+        return ""
+
+
+def get_mpi_flags():
+    output = _get_mpi_cmd()
+    if not output:
+        if sys.platform=='win32':
+            output = "gcc -L$MPICH_DIR\SDK.gcc\lib -lmpich -I$MPICH_DIR\SDK.gcc\include"
+        else:
+            output = "cc -lmpi"
+        
+    # now get the include, library dirs and the libs to link with.
+    flags = string.split(output)
+    flags = uniq_arr(flags) # remove repeated values.
+    inc_dirs = []
+    lib_dirs = []
+    libs = []
+    def_macros = []
+    undef_macros = []
+    for f in flags:
+        if f[:2] == '-I':
+            inc_dirs.append(f[2:])
+        elif f[:2] == '-L':
+            lib_dirs.append(f[2:])
+        elif f[:2] == '-l':
+            libs.append(f[2:])
+        elif f[:2] == '-U':
+            undef_macros.append(f[2:])
+        elif f[:2] == '-D':
+            tmp = string.split(f[2:], '=')
+            if len(tmp) == 1:
+                def_macros.append((tmp[0], None))
+            else:
+                def_macros.append(tuple(tmp))
+    return {'inc_dirs': inc_dirs, 'lib_dirs': lib_dirs, 'libs':libs,
+            'def_macros': def_macros, 'undef_macros': undef_macros}
+        
+
+if __name__ == "__main__":
+    setup_compiler()
+    
+    mpi_flags = get_mpi_flags()
+
+    setup(name="Pypar",
+          version="1.5",
+          description="Pypar - Parallel Python",
+          long_description="Pypar - Parallel Python, no-frills MPI interface",
+          author="Ole Nielsen",
+          author_email="Ole.Nielsen@anu.edu.au",
+          url="http://datamining.anu.edu.au/pypar",
+          package_dir = {'': 'lib'},
+          packages  = ['pypar'],
+          ext_modules = [Extension('pypar.mpiext',
+                                   ['lib/pypar/mpiext.c'],
+                                   include_dirs=mpi_flags['inc_dirs'],
+                                   library_dirs=mpi_flags['lib_dirs'],
+                                   libraries=mpi_flags['libs'],
+                                   define_macros=mpi_flags['def_macros'],
+                                   undef_macros=mpi_flags['undef_macros'])]
+          )
Index: trunk/anuga_core/source/pypar_dist/supporting_material/contrib/install.py
===================================================================
--- trunk/anuga_core/source/pypar_dist/supporting_material/contrib/install.py	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/supporting_material/contrib/install.py	(revision 8494)
@@ -0,0 +1,56 @@
+# =============================================================================
+# install.py - Installation of pypar (Parallel Python using MPI)
+# Copyright (C) 2001 Ole M. Nielsen
+#
+#    This program is free software; you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License as published by
+#    the Free Software Foundation; either version 2 of the License, or
+#    (at your option) any later version.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License (http://www.gnu.org/copyleft/gpl.html)
+#    for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program; if not, write to the Free Software
+#    Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307
+#
+#
+# Contact address: Ole.Nielsen@anu.edu.au
+#
+# Version 1.0 October 2001
+# =============================================================================
+
+"""Module install.py - Compile C extension for pypar
+"""
+
+import os, sys      
+    
+
+# Attempt to compile mpi.so 
+#
+try:
+  import compile
+  compile.compile('mpi.c', 'mpicc', verbose = 1)
+except:
+  raise "Could compile C extension mpi.c - please try manually"       
+
+
+if sys.platform in ['osf1V5', 'sunos5']:  #Compaq AlphaServer or Sun
+  sys.exit()
+  # Alpha Server or Sun cannot import MPI on sequential processes
+  # OK with LAM
+
+#Check if MPI module can be initialised
+#
+# Attempt to import and initialise mpi.so
+
+error = os.system('python -c "import mpi" > /dev/null') 
+if error:
+  raise "MPI could not be initialised."
+else:  
+  import mpi
+  print "MPI initialised OK"    
+
Index: trunk/anuga_core/source/pypar_dist/supporting_material/contrib/mpi.c
===================================================================
--- trunk/anuga_core/source/pypar_dist/supporting_material/contrib/mpi.c	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/supporting_material/contrib/mpi.c	(revision 8494)
@@ -0,0 +1,635 @@
+/************************************************************************/
+/* PyPAR - Parallel Python using MPI                 	                */
+/* Copyright (C) 2001, 2002 Ole M. Nielsen, Gian Paolo Ciceri           */
+/*                                                                      */
+/* See enclosed README file for details of installation and use.   	*/
+/*                                                                 	*/   
+/* This program is free software; you can redistribute it and/or modify */
+/* it under the terms of the GNU General Public License as published by */
+/* the Free Software Foundation; either version 2 of the License, or    */
+/* (at your option) any later version.                                  */
+/*                                                                      */     
+/* This program is distributed in the hope that it will be useful,      */
+/* but WITHOUT ANY WARRANTY; without even the implied warranty of       */
+/* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the        */
+/* GNU General Public License (http://www.gnu.org/copyleft/gpl.html)    */
+/* for more details.                                                    */
+/*                                                                      */
+/* You should have received a copy of the GNU General Public License    */
+/* along with this program; if not, write to the Free Software          */
+/*  Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307*/
+/*                                                                      */
+/*                                                                 	*/
+/* Contact addresses: Ole.Nielsen@anu.edu.au, gp.ciceri@acm.org         */
+/*                                                                 	*/
+/* version 1.2.1, 16 February 2002                                      */
+/*   Status block, MPI_ANY_TAG, MPI_ANY_SOURCE exported                 */
+/* Version 1.2, 15 February 2002                                       	*/   
+/*   Scatter added by Gian Paolo Ciceri                                 */
+/* Version 1.1, 14 February 2002                                       	*/   
+/*   Bcast added by Gian Paolo Ciceri                                   */
+/* Version 1.0.2, 10 February 2002                                      */
+/*   Modified by Gian Paulo Ciceri to allow pypar run under Python 2.2  */
+/* Version 1.0.1, 8 February 2002                                       */
+/*   Modified to install on SUN enterprise systems under Mpich          */
+/* Version 1.0, 7 February 2002                                         */
+/*   First public release for Python 2.1 (OMN)                          */
+/************************************************************************/
+
+
+#include "Python.h"
+#include "mpi.h"
+#include "Numeric/arrayobject.h"
+
+// to handle MPI constants export (shamelessly stolen from _cursesmodule.c)
+#define SetDictInt(string,ch) \
+        PyDict_SetItemString(ModDict, string, PyInt_FromLong((long) (ch)));
+
+// kludge to remap struct MPI_op to int (easier to handle by python)
+#define mpi_MAX 1
+#define mpi_MIN 2
+#define mpi_SUM 3
+#define mpi_PROD 4
+#define mpi_LAND 5
+#define mpi_BAND 6
+#define mpi_LOR 7
+#define mpi_BOR 8
+#define mpi_LXOR 9
+#define mpi_BXOR 10
+#define mpi_MAXLOC 11
+#define mpi_MINLOC 12
+#define mpi_REPLACE 13
+
+
+
+MPI_Datatype type_map(PyArrayObject *x) {  
+
+  //
+  // TYPE    py_type  mpi_type  bytes  symbol
+  // ---------------------------------------- 
+  // INT       4        6         4      'i'
+  // LONG      5        8         8      'l'
+  // FLOAT     6       10         4      'f'  
+  // DOUBLE    7       11         8      'd'
+  
+  
+  int py_type;
+  MPI_Datatype mpi_type;
+  
+  if (x -> nd != 1) {
+    PyErr_SetString(PyExc_ValueError, "Array must be 1 dimensional");
+    return (MPI_Datatype) 0;
+  }      
+      
+  py_type = x -> descr -> type_num;     
+  if (py_type == PyArray_DOUBLE) 
+    mpi_type = MPI_DOUBLE;
+  else if (py_type == PyArray_LONG)   
+    mpi_type = MPI_LONG;  
+  else if (py_type == PyArray_FLOAT) 
+    mpi_type = MPI_FLOAT;
+  else if (py_type == PyArray_INT) 
+    mpi_type = MPI_INT;
+  else {
+    PyErr_SetString(PyExc_ValueError, "Array must be of type int or float");
+    return 0;
+  }      
+
+  //printf("Types: %d %d.\n", py_type, mpi_type);
+  
+  return mpi_type;
+}    
+
+MPI_Op op_map(int py_op) {  
+  
+  MPI_Op mpi_op;
+  
+  if (py_op == mpi_MAX) 
+    mpi_op = MPI_MAX;
+  else if (py_op == mpi_MIN)   
+    mpi_op = MPI_MIN;  
+  else if (py_op == mpi_SUM)   
+    mpi_op = MPI_SUM;  
+  else if (py_op == mpi_PROD)   
+    mpi_op = MPI_PROD;  
+  else if (py_op == mpi_LAND)   
+    mpi_op = MPI_LAND;  
+  else if (py_op == mpi_BAND)   
+    mpi_op = MPI_BAND;  
+  else if (py_op == mpi_LOR)   
+    mpi_op = MPI_LOR;  
+  else if (py_op == mpi_BOR)   
+    mpi_op = MPI_BOR;  
+  else if (py_op == mpi_LXOR)   
+    mpi_op = MPI_LXOR;  
+  else if (py_op == mpi_BXOR)   
+    mpi_op = MPI_BXOR;  
+  else if (py_op == mpi_MAXLOC)
+    mpi_op = MPI_MAXLOC;
+  else if (py_op == mpi_MINLOC)   
+    mpi_op = MPI_MINLOC;  
+  else if (py_op == mpi_REPLACE)   
+    mpi_op = MPI_REPLACE;  
+  else {
+    PyErr_SetString(PyExc_ValueError, "Operation unknown");
+    return 0;
+  }      
+
+  //printf("Op: %d.\n", py_op);
+  
+  return mpi_op;
+}  
+
+/*********************************************************/
+/* send_string                                           */
+/* Send string of characters                             */
+/*                                                       */
+/*********************************************************/
+static PyObject *send_string(PyObject *self, PyObject *args) {
+  char *s;
+  int destination, tag, length, err;
+ 
+  /* process the parameters */
+  if (!PyArg_ParseTuple(args, "s#ii", &s, &length, &destination, &tag))
+    return NULL;
+  
+  /* call the MPI routine */
+  err = MPI_Send(s, length, MPI_CHAR, destination, tag, MPI_COMM_WORLD);
+
+  return Py_BuildValue("i", err);
+}
+
+/**********************************************************/
+/* receive_string                                         */
+/* Receive string of characters                           */
+/*                                                        */
+/**********************************************************/
+static PyObject *receive_string(PyObject *self, PyObject *args) {
+  char *s;
+  int source, tag, length, err, st_length; 
+  MPI_Status status;
+
+  /* process the parameters */
+  if (!PyArg_ParseTuple(args, "s#ii", &s, &length, &source, &tag))
+    return NULL;
+    
+  /* call the MPI routine */
+  err = MPI_Recv(s, length, MPI_CHAR, source, tag, MPI_COMM_WORLD, &status);
+   
+  MPI_Get_count(&status, MPI_CHAR, &st_length); 
+  // status.st_length is not available in all MPI implementations
+  //Alternative is: MPI_Get_elements(MPI_Status *, MPI_Datatype, int *);
+
+  return Py_BuildValue("i(iiii)", err, status.MPI_SOURCE, status.MPI_TAG,
+  status.MPI_ERROR, st_length);  
+}
+
+/**********************************************************/
+/* bcast_string                                           */
+/* Broadcast string of characters                         */
+/*                                                        */
+/**********************************************************/
+static PyObject *bcast_string(PyObject *self, PyObject *args) {
+  char *s;
+  int source, length, err; 
+
+  /* process the parameters */
+  if (!PyArg_ParseTuple(args, "s#i", &s, &length, &source))
+    return NULL;
+    
+  /* call the MPI routine */
+  err = MPI_Bcast(s, length, MPI_CHAR, source, MPI_COMM_WORLD);
+   
+  return Py_BuildValue("i", err);  
+}
+
+/**********************************************************/
+/* scatter_string                                         */
+/* Scatter string of characters                           */
+/*                                                        */
+/**********************************************************/
+static PyObject *scatter_string(PyObject *self, PyObject *args) {
+  char *s;
+  char *d;
+  int source, length, err; 
+
+  /* process the parameters */
+  if (!PyArg_ParseTuple(args, "sisi", &s, &length, &d, &source))
+    return NULL;
+    
+  /* call the MPI routine */
+  err = MPI_Scatter(s, length, MPI_CHAR, d, length,  MPI_CHAR, source, MPI_COMM_WORLD);
+   
+  return Py_BuildValue("i", err);  
+}
+
+/**********************************************************/
+/* gather_string                                         */
+/* Gather string of characters                           */
+/*                                                        */
+/**********************************************************/
+static PyObject *gather_string(PyObject *self, PyObject *args) {
+  char *s;
+  char *d;
+  int source, length, err; 
+
+  /* process the parameters */
+  if (!PyArg_ParseTuple(args, "sisi", &s, &length, &d, &source))
+    return NULL;
+    
+  /* call the MPI routine */
+  err = MPI_Gather(s, length, MPI_CHAR, d, length,  MPI_CHAR, source, MPI_COMM_WORLD);
+   
+  return Py_BuildValue("i", err);  
+}
+
+
+/**********************************************************/
+/* send_array                                             */
+/* Send Numeric array of type float, double, int, or long */
+/*                                                        */
+/**********************************************************/
+static PyObject *send_array(PyObject *self, PyObject *args) {
+  PyObject *input;
+  PyArrayObject *x;
+  int destination, tag, err;
+  MPI_Datatype mpi_type;
+  
+  /* process the parameters */
+  if (!PyArg_ParseTuple(args, "Oii", &input, &destination, &tag))
+    return NULL;
+    
+  /* Make Numeric array from general sequence type (no cost if already Numeric)*/    
+  x = (PyArrayObject *)
+    PyArray_ContiguousFromObject(input, PyArray_NOTYPE, 0, 0);
+    
+  /* Input check and determination of MPI type */          
+  mpi_type = type_map(x);
+  if (!mpi_type) return NULL;
+    
+  /* call the MPI routine */
+  err = MPI_Send(x->data, x->dimensions[0], mpi_type, destination, tag,\
+           MPI_COMM_WORLD);
+	   
+  Py_DECREF(x); 	   
+	   
+  return Py_BuildValue("i", err);
+}
+
+/*************************************************************/
+/* receive_array                                             */
+/* Receive Numeric array of type float, double, int, or long */
+/*                                                           */
+/*************************************************************/
+static PyObject *receive_array(PyObject *self, PyObject *args) {
+  PyArrayObject *x;
+  int source, tag, err, st_length;
+  MPI_Datatype mpi_type;
+  MPI_Status status;
+
+  /* process the parameters */
+  if (!PyArg_ParseTuple(args, "Oii", &x, &source, &tag))
+    return NULL;
+
+  /* Input check and determination of MPI type */          
+  mpi_type = type_map(x);
+  if (!mpi_type) return NULL;  
+      
+  /* call the MPI routine */
+  err =  MPI_Recv(x->data, x->dimensions[0], mpi_type, source, tag, \
+         MPI_COMM_WORLD, &status);
+	 
+  MPI_Get_count(&status, mpi_type, &st_length); 
+  // status.st_length is not available in all MPI implementations
+  //Alternative is: MPI_Get_elements(MPI_Status *, MPI_Datatype, int *);
+	 
+      
+  return Py_BuildValue("i(iiii)", err, status.MPI_SOURCE, status.MPI_TAG,
+  status.MPI_ERROR, st_length);  
+}
+
+
+/*************************************************************/
+/* bcast_array                                               */
+/* Broadcast Num.  array of type float, double, int, or long */
+/*                                                           */
+/*************************************************************/
+static PyObject *bcast_array(PyObject *self, PyObject *args) {
+  PyArrayObject *x;
+  int source, err;
+  MPI_Datatype mpi_type;
+  MPI_Status status;
+
+  /* process the parameters */
+  if (!PyArg_ParseTuple(args, "Oi", &x, &source))
+    return NULL;
+
+  /* Input check and determination of MPI type */          
+  mpi_type = type_map(x);
+  if (!mpi_type) return NULL;  
+      
+  /* call the MPI routine */
+  err =  MPI_Bcast(x->data, x->dimensions[0], mpi_type, source, \
+         MPI_COMM_WORLD);
+      
+  return Py_BuildValue("i", err);
+}
+
+/*************************************************************/
+/* scatter_array                                             */
+/* Scatter Num.    array of type float, double, int, or long */
+/*                                                           */
+/*************************************************************/
+static PyObject *scatter_array(PyObject *self, PyObject *args) {
+  PyArrayObject *x;
+  PyArrayObject *d;
+  int length, source, err;
+  MPI_Datatype mpi_type;
+  MPI_Status status;
+
+  /* process the parameters */
+  if (!PyArg_ParseTuple(args, "OiOi", &x, &length, &d, &source))
+    return NULL;
+
+  /* Input check and determination of MPI type */          
+  mpi_type = type_map(x);
+  if (!mpi_type) return NULL;  
+      
+  /* call the MPI routine */
+  err =  MPI_Scatter(x->data, length, mpi_type, d->data, length, mpi_type, source, \
+         MPI_COMM_WORLD);
+      
+  return Py_BuildValue("i", err);
+}
+
+
+/*************************************************************/
+/* gather_array                                              */
+/* Gather Num.     array of type float, double, int, or long */
+/*                                                           */
+/*************************************************************/
+static PyObject *gather_array(PyObject *self, PyObject *args) {
+  PyArrayObject *x;
+  PyArrayObject *d;
+  int length, source, err;
+  MPI_Datatype mpi_type;
+  MPI_Status status;
+
+  /* process the parameters */
+  if (!PyArg_ParseTuple(args, "OiOi", &x, &length, &d, &source))
+    return NULL;
+
+  /* Input check and determination of MPI type */          
+  mpi_type = type_map(x);
+  if (!mpi_type) return NULL;  
+      
+  /* call the MPI routine */
+  err =  MPI_Gather(x->data, length, mpi_type, d->data, length, mpi_type, source, \
+         MPI_COMM_WORLD);
+      
+  return Py_BuildValue("i", err);
+}
+
+
+/*************************************************************/
+/* reduce_array                                              */
+/* Reduce Num.     array of type float, double, int, or long */
+/*                                                           */
+/*************************************************************/
+static PyObject *reduce_array(PyObject *self, PyObject *args) {
+  PyArrayObject *x;
+  PyArrayObject *d;
+  int length, source, op, err;
+  MPI_Datatype mpi_type;
+  MPI_Status status;
+  MPI_Op mpi_op;
+
+  /* process the parameters */
+  if (!PyArg_ParseTuple(args, "OOiii", &x, &d, &length, &op, &source))
+    return NULL;
+   
+  /* Input check and determination of MPI type */          
+  mpi_type = type_map(x);
+  if (!mpi_type) return NULL;  
+  
+  /* Input check and determination of MPI op */ 
+  //printf("op: %d\n", op);         
+  mpi_op = op_map(op);
+  if (!mpi_op) return NULL;  
+   
+         
+  if (op == mpi_MAXLOC || op == mpi_MINLOC) {
+    //not implemented
+    return Py_BuildValue("i", -666);
+  }
+  else {
+  /* call the MPI routine */
+  err =  MPI_Reduce(x->data, d->data, length, mpi_type, mpi_op, source, \
+         MPI_COMM_WORLD);
+  }
+         
+      
+  return Py_BuildValue("i", err);
+}
+
+
+
+/*********************************************************/
+/* MPI calls rank, size, finalize, abort                 */
+/*                                                       */
+/*********************************************************/
+
+static PyObject * rank(PyObject *self, PyObject *args) {
+  int myid;
+
+  MPI_Comm_rank(MPI_COMM_WORLD,&myid);
+  return Py_BuildValue("i", myid);
+}
+
+static PyObject * size(PyObject *self, PyObject *args) {
+  int numprocs; 
+
+  MPI_Comm_size(MPI_COMM_WORLD,&numprocs);
+  return Py_BuildValue("i", numprocs);
+}
+  
+static PyObject * Get_processor_name(PyObject *self, PyObject *args) {  
+  char processor_name[MPI_MAX_PROCESSOR_NAME];
+  int  namelen;
+
+  MPI_Get_processor_name(processor_name,&namelen);
+  return Py_BuildValue("s#", processor_name, namelen);
+}   
+  
+static PyObject * Finalize(PyObject *self, PyObject *args) {  
+  int error;
+
+  error = MPI_Finalize();
+  return Py_BuildValue("i", error);  
+} 
+
+static PyObject * Abort(PyObject *self, PyObject *args) {  
+  int error, code=0;
+  
+  error = MPI_Abort(MPI_COMM_WORLD, code);
+  return Py_BuildValue("i", error);  
+} 
+
+static PyObject * Barrier(PyObject *self, PyObject *args) {  
+  int error;
+  
+  error = MPI_Barrier(MPI_COMM_WORLD);
+  return Py_BuildValue("i", error);    
+}    
+
+static PyObject * Wtime(PyObject *self, PyObject *args) {     
+  double t;
+  
+  t = MPI_Wtime();
+  return Py_BuildValue("d", t);      
+}     
+ 
+/**********************************/
+/* Method table for python module */
+/**********************************/
+
+static struct PyMethodDef MethodTable[] = {
+  {"size", size, METH_VARARGS},  
+  {"rank", rank, METH_VARARGS},  
+  {"Barrier", Barrier, METH_VARARGS},          
+  {"Wtime", Wtime, METH_VARARGS},            
+  {"Get_processor_name", Get_processor_name, METH_VARARGS},              
+  {"Finalize", Finalize, METH_VARARGS},        
+  {"Abort", Abort, METH_VARARGS},          
+  {"send_string", send_string, METH_VARARGS},
+  {"receive_string", receive_string, METH_VARARGS},      
+  {"bcast_string", bcast_string, METH_VARARGS},        
+  {"scatter_string", scatter_string, METH_VARARGS},        
+  {"gather_string", gather_string, METH_VARARGS},        
+  {"send_array", send_array, METH_VARARGS},
+  {"receive_array", receive_array, METH_VARARGS},    
+  {"bcast_array", bcast_array, METH_VARARGS},              
+  {"scatter_array", scatter_array, METH_VARARGS},              
+  {"gather_array", gather_array, METH_VARARGS},              
+  {"reduce_array", reduce_array, METH_VARARGS},              
+  {NULL, NULL}
+};
+
+
+/***************************/
+/* Initialisation Function */
+/***************************/
+
+void initmpi(){
+  int error, argc = 0; //Dummy
+  char **argv;         //Dummy
+  PyObject *m, *ModDict;
+
+  //printf("Initialising MPI\n");
+  error = MPI_Init(&argc, &argv); 
+  //printf("MPI Initialised\n");  
+  
+  //FIXME: Make a sensible errorcheck here
+  
+  m = Py_InitModule("mpi", MethodTable);
+  
+  // to handle MPI symbolic constants
+  ModDict = PyModule_GetDict(m); 
+  SetDictInt("MPI_ANY_TAG", MPI_ANY_TAG);
+  SetDictInt("MPI_ANY_SOURCE", MPI_ANY_SOURCE);
+  SetDictInt("mpi_MAX", mpi_MAX);
+  SetDictInt("mpi_MIN", mpi_MIN);
+  SetDictInt("mpi_SUM", mpi_SUM);
+  SetDictInt("mpi_PROD", mpi_PROD);
+  SetDictInt("mpi_LAND", mpi_LAND);
+  SetDictInt("mpi_BAND", mpi_BAND);
+  SetDictInt("mpi_LOR", mpi_LOR);
+  SetDictInt("mpi_BOR", mpi_BOR);
+  SetDictInt("mpi_LXOR", mpi_LXOR);
+  SetDictInt("mpi_BXOR", mpi_BXOR);
+  SetDictInt("mpi_MAXLOC", mpi_MAXLOC);
+  SetDictInt("mpi_MINLOC", mpi_MINLOC);
+  SetDictInt("mpi_REPLACE", mpi_REPLACE);
+
+  //SetDictInt("MPI_COMM_WORLD", MPI_COMM_WORLD);  
+   
+  import_array();     //Necessary for handling of NumPY structures  
+}
+
+ 
+ 
+ 
+ 
+/*********************************************************************/
+/* OBSOLETE STUFF  					             */
+/*********************************************************************/
+
+/******************************************/
+/* keep for doc of Numeric arrays etc     */ 
+/******************************************/
+
+void print_real_array(PyArrayObject *x) {  
+  int i;
+  for (i=0; i<x->dimensions[0]; i++) {
+    printf("%f ", *(double*) (x->data + i*x->strides[0]));
+  }
+}
+
+void print_int_array(PyArrayObject *x) {  
+  int i;
+  for (i=0; i<x->dimensions[0]; i++) {
+    printf("%d ", *(int*) (x->data + i*x->strides[0]));
+  }
+}
+
+
+/*********************************************************/
+/* send_real_array                                       */
+/* Send Numeric array of double floating point numbers   */
+/*                                                       */
+/*********************************************************/
+static PyObject *send_real_array(PyObject *self, PyObject *args) {
+  PyObject *input;
+  PyArrayObject *x;
+  int destination, tag, err;
+  
+  /* process the parameters */
+  if (!PyArg_ParseTuple(args, "Oii", &input, &destination, &tag))
+    return NULL;
+
+  /* Make Numeric array from general sequence type (no cost if already Numeric)*/    
+  x = (PyArrayObject *)
+    PyArray_ContiguousFromObject(input, PyArray_DOUBLE, 0, 0);
+
+  /* call the MPI routine */
+  err = MPI_Send(x->data, x->dimensions[0], MPI_DOUBLE, destination, tag,\
+           MPI_COMM_WORLD);
+
+  Py_DECREF(x); 	   	   	   
+  return Py_BuildValue("i", err);
+}
+
+/**********************************************************/
+/* receive_real_array                                     */
+/* Receive Numeric array of double floating point numbers */
+/*                                                        */
+/**********************************************************/
+static PyObject *receive_real_array(PyObject *self, PyObject *args) {
+  PyArrayObject *x;
+  int source, tag, err;
+  MPI_Status status;
+
+  /* process the parameters */
+  if (!PyArg_ParseTuple(args, "Oii", &x, &source, &tag))
+    return NULL;
+  
+  /* call the MPI routine */
+  err =  MPI_Recv(x->data, x->dimensions[0], MPI_DOUBLE, source, tag, \
+         MPI_COMM_WORLD, &status);
+
+
+  return Py_BuildValue("i", err);
+}
+ 
+
+
Index: trunk/anuga_core/source/pypar_dist/supporting_material/contrib/ms.py
===================================================================
--- trunk/anuga_core/source/pypar_dist/supporting_material/contrib/ms.py	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/supporting_material/contrib/ms.py	(revision 8494)
@@ -0,0 +1,105 @@
+#!/usr/bin/env python
+# Test of MPI module 'pypar' for Python
+# Demonstrates a 'master-slave' parallel program
+# 
+# Run as 
+#   python demo2.py
+# or 
+#   mpirun -np 4 demo2.py
+#
+# GPC, FEB 2002
+
+try:
+  import Numeric
+except:
+  raise 'Module Numeric must be present to run pypar'
+  
+try:
+  import pypar
+except:
+  raise 'Module pypar must be present to run parallel'
+
+print "Modules Numeric, pypar imported OK"
+
+WORKTAG = 1
+DIETAG =  2
+
+
+def master():
+    numCompleted = 0
+    
+    print "[MASTER]: I am processor %d of %d on node %s" %(MPI_myid, MPI_numproc, MPI_node)
+    
+    # start slaves distributing the first work slot
+    for i in range(1, MPI_numproc): 
+        work = workList[i]
+        pypar.raw_send(work, i, WORKTAG) 
+        print "[MASTER]: sent work '%s' to node '%d'" %(work, i)
+
+    # dispach the remaining work slots on dynamic load-balancing policy
+    # the quicker to do the job, the more jobs it takes
+    for work in workList[MPI_numproc:]:
+        result = '  '
+        err = pypar.raw_receive(result, pypar.MPI_ANY_SOURCE, pypar.MPI_ANY_TAG) 
+        print "[MASTER]: received result '%s' from node '%d'" %(result, err[1][0])
+        numCompleted += 1
+        pypar.raw_send(work, err[1][0], WORKTAG)
+        print "[MASTER]: sent work '%s' to node '%d'" %(work, err[1][0])
+    
+    # all works have been dispatched out
+    print  "[MASTER]: toDo : %d" %numWorks
+    print  "[MASTER]: done : %d" %numCompleted
+    
+    # I've still to take into the remaining completions   
+    while(numCompleted < numWorks): 
+        result = '  '
+        err = pypar.raw_receive(result, pypar.MPI_ANY_SOURCE, pypar.MPI_ANY_TAG) 
+        print "[MASTER]: received (final) result '%s' from node '%d'" %(result, err[1][0])
+        numCompleted += 1
+        print "[MASTER]: %d completed" %numCompleted
+        
+    print "[MASTER]: about to terminate slaves"
+
+    # say slaves to stop working
+    for i in range(1, MPI_numproc): 
+        pypar.raw_send('#', i, DIETAG) 
+        print "[MASTER]: sent (final) work '%s' to node '%d'" %(0, i)
+        
+    return
+    
+def slave():
+
+    print "[SLAVE %d]: I am processor %d of %d on node %s" %(MPI_myid, MPI_myid, MPI_numproc, MPI_node)
+
+    while 1:
+        result = ' '
+        err = pypar.raw_receive(result, pypar.MPI_ANY_SOURCE, pypar.MPI_ANY_TAG) 
+        print "[SLAVE %d]: received work '%s' with tag '%d' from node '%d'" %(MPI_myid, result, err[1][1], err[1][0])
+       
+        if (err[1][1] == DIETAG):
+            print "[SLAVE %d]: received termination from node '%d'" %(MPI_myid, 0)
+            return
+        else:
+            result = 'X'+result
+            pypar.raw_send(result, 0)
+            print "[SLAVE %d]: sent result '%s' to node '%d'" %(MPI_myid, result, 0)
+            
+       
+
+if __name__ == '__main__':
+    MPI_myid =    pypar.rank()
+    MPI_numproc = pypar.size()
+    MPI_node =    pypar.Get_processor_name()
+
+    workList = ('_dummy_', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j')
+    _workList = ('_dummy_', 'a', 'b', 'c', 'd', 'e', 'f')
+    numWorks = len(workList) - 1
+
+    if MPI_myid == 0:
+        master()
+    else:
+        slave()
+
+    pypar.Finalize()
+    print "MPI environment finalized."
+        	
Index: trunk/anuga_core/source/pypar_dist/supporting_material/contrib/pypar.py
===================================================================
--- trunk/anuga_core/source/pypar_dist/supporting_material/contrib/pypar.py	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/supporting_material/contrib/pypar.py	(revision 8494)
@@ -0,0 +1,678 @@
+# =============================================================================
+# pypar.py - Parallel Python using MPI
+# Copyright (C) 2001, 2002 Ole M. Nielsen, Gian Paolo Ciceri          
+#
+#    This program is free software; you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License as published by
+#    the Free Software Foundation; either version 2 of the License, or
+#    (at your option) any later version.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License (http://www.gnu.org/copyleft/gpl.html)
+#    for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program; if not, write to the Free Software
+#    Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307
+#
+#
+# Contact addresses: Ole.Nielsen@anu.edu.au, gp.ciceri@acm.org         
+#
+# version 1.2.1, 16 February 2002                                      
+#   Status block, MPI_ANY_TAG, MPI_ANY_SOURCE exported                 
+# Version 1.2, 15 February 2002                                       	   
+#   Scatter added by Gian Paolo Ciceri                                   
+# Version 1.1, 14 February 2002                                       	   
+#   Bcast added by Gian Paolo Ciceri                                   
+# Version 1.0.2, 10 February 2002                                          
+#   Modified by Gian Paulo Ciceri to allow pypar run under Python 2.2  
+# Version 1.0.1, 8 February 2002                                       
+#   Modified to install on SUN enterprise systems under Mpich          
+# Version 1.0, 7 February 2002                                         
+#   First public release for Python 2.1 (OMN)                          
+# =============================================================================
+
+"""Module pypar.py - Parallel Python using MPI
+
+Public functions:
+
+size() -- Number of processors
+rank() -- Id of current processor
+Get_processor_name() -- Return host name of current node
+
+send() -- Blocking send (all types)
+receive() -- Blocking receive (all types)
+raw_send() -- Blocking send (Numeric arrays and strings)
+raw_receive() -- Blocking receive (Numeric arrays and strings)
+bcast() -- Broadcast
+Wtime() -- MPI wall time
+Barrier() -- Synchronisation point. Makes processors wait until all processors
+             have reached this point.
+Abort() -- Terminate all processes. 
+Finalize() -- Cleanup MPI. No parallelism can take place after this point. 
+
+
+See doc strings of individual functions for detailed documentation.
+"""
+
+import os, sys
+
+# -----------------------------------------------------------------------------
+# Options directory with default values - to be set by user
+#
+
+options = { 
+  'vanilla_bufsize': None   # Buffersize (chars) used for vanilla sends
+}
+
+# Constants
+#
+control_tag = 32            # Tag used to identify control information
+control_data_max_size = 256 # Maximal size of string holding control data
+
+
+#------------------------------------------------------------------------
+# MPI Status block (can be queried by user after each receive
+#------------------------------------------------------------------------
+
+class Status:
+  def __init__(self):
+    pass
+  
+  def set_values(self, status_tuple):
+    self.source = status_tuple[0] 
+    self.tag = status_tuple[1] 
+    self.error = status_tuple[2] 
+    self.length = status_tuple[3]
+    
+status = Status() #Initialise status object      
+
+#---------------------------------------------------------------------------
+# Communication functions
+#--------------------------------------------------------------------------
+
+
+def raw_send(x, destination, tag=0, vanilla=0):
+  """Wrapper for raw MPI send.
+     Send x to destination with tag.
+     
+     Automatically determine appropriate protocol
+     and call corresponding send function.
+     
+     The variable x can be any (picklable) type, but
+     Numeric variables and text strings will most efficient.
+     Setting vanilla = 1 forces vanilla mode for any type.
+
+  """
+
+  protocol = get_control_info(x, vanilla)[0]
+  if protocol == 'array':
+    send_array(x, destination, tag)  
+  elif protocol == 'string':
+    send_string(x, destination, tag)            
+  else:  
+    send_vanilla(x, destination, tag)
+
+      
+def raw_receive(x, source, tag=0, vanilla=0):
+  """Wrapper for raw MPI receive.
+     Receive something of same size as x from source with tag.
+     
+     Automatically determine appropriate protocol
+     and call corresponding receive function.
+     
+     The variable x can be any (picklable) type, but
+     Numeric variables and text strings will most efficient.
+     Setting vanilla = 1 forces vanilla mode for any type.
+  """
+
+  
+  protocol = get_control_info(x, vanilla)[0]
+  if protocol == 'array':
+    err, stat = receive_array(x, source, tag) 
+    if not err:
+      status.set_values(stat)
+    else:
+      raise 'receive_array failed with error code %d' %err  
+  elif protocol == 'string':
+    err, stat = receive_string(x, source, tag)
+    if not err:
+      status.set_values(stat)
+    else:
+      raise 'receive_string failed with error code %d' %err  
+  else:  
+    x = receive_vanilla(x, source, tag)
+
+  return x
+
+
+def send(x, destination, tag=0, vanilla=0):
+  """Wrapper for easy MPI send.
+     Send x to destination with tag.
+     
+     Automatically determine appropriate protocol
+     and call corresponding send function.
+     Also passes type and size information on as preceding message to
+     simplify the receive call.
+     
+     The variable x can be any (picklable) type, but
+     Numeric variables and text strings will most efficient.
+     Setting vanilla = 1 forces vanilla mode for any type.
+
+  """
+  import string
+
+  control_info = get_control_info(x, vanilla)
+  protocol = control_info[0]
+  
+  if protocol == 'array':
+    send_control_info(control_info, destination)
+    
+    send_array(x, destination, tag)    
+  elif protocol == 'string':
+    send_control_info(control_info, destination)    
+    
+    send_string(x, destination, tag)          
+  elif protocol == 'vanilla':
+    from cPickle import dumps     
+    s = dumps(x, 1)
+    control_info[2] = str(len(s))
+
+    send_control_info(control_info, destination)   
+    
+    send_string(s, destination, tag)
+  else:
+    raise "Unknown values for protocol: %s" %protocol    
+
+      
+def receive(source, tag=0):
+  """Wrapper for easy MPI receive.
+     Receive data from source with tag.
+     
+     Assumes preceding message containing protocol, type, size.
+     Create appropriate buffer and receive data.
+  """
+
+  control_info = receive_control_info(source)
+  
+  protocol = control_info[0]
+  typecode = control_info[1]
+  size =     control_info[2]  
+  
+  if protocol == 'array':
+    import Numeric
+    x = Numeric.zeros(size,typecode)
+    err, stat = receive_array(x, source, tag)    
+    if not err:
+      status.set_values(stat)
+    else:
+      raise 'receive_array failed with error code %d' %err  
+  elif protocol == 'string':
+    x = ' '*size
+    err, stat = receive_string(x, source, tag)       
+    if not err:
+      status.set_values(stat)
+    else:
+      raise 'receive_string failed with error code %d' %err  
+  elif protocol == 'vanilla':
+    from cPickle import loads 
+    s = ' '*size    
+    err, stat = receive_string(s, source, tag)
+    if not err:
+      status.set_values(stat)
+    else:
+      raise 'receive_string failed with error code %d' %err  
+    
+    x = loads(s)
+  else:
+    raise "Unknown values for protocol: %s" %protocol
+      
+  return x  
+
+def bcast(x, source, vanilla=0):
+  """Wrapper for MPI bcast.
+     Broadcast x from source.
+     
+     Automatically determine appropriate protocol
+     and call corresponding send function.
+     
+     The variable x can be any (picklable) type, but
+     Numeric variables and text strings will most efficient.
+     Setting vanilla = 1 forces vanilla mode for any type.
+
+  """
+
+  protocol = get_control_info(x, vanilla)[0]
+  if protocol == 'array':
+    bcast_array(x, source)    
+  elif protocol == 'string':
+    bcast_string(x, source)          
+  elif protocol == 'vanilla':
+    from cPickle import loads, dumps 
+    s = dumps(x, 1)
+    s = s + ' '*int(0.1*len(s)) #safety 
+    bcast_string(s, source)
+    x = loads(s)
+  else:
+    raise "Unknown values for protocol: %s" %protocol  
+    
+  return x         
+
+def raw_scatter(s, nums, d, source, vanilla=0):
+  """Wrapper for MPI scatter.
+     Scatter s in nums element to d (of the same nums size) from source.
+     
+     Automatically determine appropriate protocol
+     and call corresponding send function.
+     
+     The variable s can be any (picklable) type, but
+     Numeric variables and text strings will most efficient.
+     Setting vanilla = 1 forces vanilla mode for any type.
+
+  """
+
+  protocol = get_control_info(s, vanilla)[0]
+  if protocol == 'array':
+    scatter_array(s, nums, d, source)    
+  elif protocol == 'string':
+    scatter_string(s, nums, d, source)          
+  elif protocol == 'vanilla':
+    raise "Protocol: %s unsupported for scatter" %protocol
+  else:
+    raise "Unknown values for protocol: %s" %protocol  
+    
+  return d         
+
+
+def scatter(s, nums, source, vanilla=0):
+  """Wrapper for easy MPI Scatter receive.
+     Receive data from source with tag.
+     
+     Create appropriate buffer and receive data.
+  """
+
+  control_info = get_control_info(s)
+  
+  protocol = control_info[0]
+  typecode = control_info[1]
+  size =  nums   
+  
+  if protocol == 'array':
+    import Numeric
+    x = Numeric.zeros(size,typecode)
+    scatter_array(s, size, x, source)    
+  elif protocol == 'string':
+    x = ' '*size
+    scatter_string(s, size, x, source)          
+  elif protocol == 'vanilla':
+    raise "Protocol: %s unsupported for scatter" %protocol
+  else:
+    raise "Unknown values for protocol: %s" %protocol
+      
+  return x  
+
+def raw_gather(s, nums, d, source, vanilla=0):
+  """Wrapper for MPI gather.
+     Gather s in nums element to d (of the same nums size) from source.
+     
+     Automatically determine appropriate protocol
+     and call corresponding send function.
+     
+     The variable s can be any (picklable) type, but
+     Numeric variables and text strings will most efficient.
+     Setting vanilla = 1 forces vanilla mode for any type.
+
+  """
+
+  protocol = get_control_info(s, vanilla)[0]
+  if protocol == 'array':
+    gather_array(s, nums, d, source)    
+  elif protocol == 'string':
+    gather_string(s, nums, d, source)          
+  elif protocol == 'vanilla':
+    raise "Protocol: %s unsupported for gather" %protocol
+  else:
+    raise "Unknown values for protocol: %s" %protocol  
+    
+  return d         
+
+
+def gather(s, nums, source, vanilla=0):
+  """Wrapper for easy MPI Gather receive.
+     Receive data from source with tag.
+     
+     Create appropriate buffer and receive data.
+  """
+
+  control_info = get_control_info(s)
+  
+  protocol = control_info[0]
+  typecode = control_info[1]
+  s_size =  nums   
+  
+  if protocol == 'array':
+    import Numeric
+    x = Numeric.zeros(s_size * size(),typecode)
+    gather_array(s, s_size, x, source)    
+  elif protocol == 'string':
+    x = ' '*s_size*size()
+    gather_string(s, s_size, x, source)          
+  elif protocol == 'vanilla':
+    raise "Protocol: %s unsupported for gather" %protocol
+  else:
+    raise "Unknown values for protocol: %s" %protocol
+      
+  return x  
+
+
+def raw_reduce(s, d, nums, op, source, vanilla=0):
+  """Wrapper for MPI_Reduce.
+     Reduce s in nums element to d (of the same nums size) at source
+     applying operation op.
+     
+     Automatically determine appropriate protocol
+     and call corresponding send function.
+    
+  """
+
+  protocol = get_control_info(s, vanilla)[0]
+  if protocol == 'array':
+    reduce_array(s, d, nums, op, source)          
+  elif (protocol == 'vanilla' or protocol == 'string'):
+    raise "Protocol: %s unsupported for reduce" %protocol
+  else:
+    raise "Unknown values for protocol: %s" %protocol  
+    
+  return d         
+
+
+def reduce(s, nums, op, source, vanilla=0):
+  """Wrapper for easy MPI Gather receive.
+     Receive data from source with tag.
+     
+     Create appropriate buffer and receive data.
+  """
+
+  control_info = get_control_info(s)
+  
+  protocol = control_info[0]
+  typecode = control_info[1]
+  s_size =  nums   
+  
+  if protocol == 'array':
+    import Numeric
+    x = Numeric.zeros(s_size * size(),typecode)
+    reduce_array(s, x, s_size, op, source)    
+  elif (protocol == 'vanilla' or protocol == 'string'):
+    raise "Protocol: %s unsupported for reduce" %protocol
+  else:
+    raise "Unknown values for protocol: %s" %protocol
+      
+  return x  
+
+
+     
+
+#---------------------------------------------------------
+# AUXILIARY FUNCTIONS
+#---------------------------------------------------------
+def get_control_info(x, vanilla=0):
+  """Determine which protocol to use for communication:
+     (Numeric) arrays, strings, or vanilla based x's type.
+
+     There are three protocols:
+     'array':   Numeric arrays of type 'i', 'l', 'f', or 'd' can be communicated
+                with mpi.send_array and mpi.receive_array.
+     'string':  Text strings can be communicated with mpi.send_string and
+                mpi.receive_string.
+     'vanilla': All other types can be communicated using the scripts send_vanilla and
+                receive_vanilla provided that the objects can be serialised using
+                pickle (or cPickle). The latter mode is less efficient than the
+                first two but it can handle complex structures.
+
+     Rules:
+     If keyword argument vanilla == 1, vanilla is chosen regardless of 
+     x's type.
+     Otherwise if x is a string, the string protocol is chosen
+     If x is an array, the 'array' protocol is chosen provided that x has one
+     of the admissible typecodes.
+  """
+
+  protocol = 'vanilla'
+  typecode = ' '
+  size = '0'
+  
+  if not vanilla:
+    #if type(x).__name__ == 'string':  # OK in Python 2.1 but not 2.2
+    if type(x).__name__[0:3] == 'str': # Fixed by Gian Paolo Ciceri 10/2/2   
+      protocol = 'string'
+      typecode = 'c'
+      size = len(x)
+    elif type(x).__name__ == 'array':
+      try:
+        import Numeric
+
+        typecode = x.typecode() 
+        if typecode in ['i', 'l', 'f', 'd']:
+          protocol = 'array'
+          size = len(x)
+        else:    
+          print "WARNING (pypar.py): Numeric object type %s is not supported."\
+                %(x.typecode())
+          print "Only types 'i', 'l', 'f', 'd' are supported,",
+          print "Reverting to vanilla mode."
+          protocol = 'vanilla'
+  
+      except:
+        print "WARNING (pypar.py): Numeric module could not be imported,",
+        print "reverting to vanilla mode"
+        protocol = 'vanilla'
+
+
+  control_info = [protocol, typecode, str(size)]
+
+  return control_info
+
+
+
+#----------------------------------------------
+
+def send_control_info(control_info, destination):
+  """Send control info to destination
+  """
+  import string
+  
+  msg = string.join(control_info,',')
+  send_string(msg, destination, control_tag)
+
+  
+def receive_control_info(source):
+  """Receive control info from source
+  """
+  import string
+  
+  msg = ' '*control_data_max_size
+  err, stat = receive_string(msg, source, control_tag)
+  if err:
+    raise Exception
+  #NB: Do not set status block here  
+
+  control_info = msg.split(',')
+  assert len(control_info) == 3
+  control_info[2] = int(control_info[2])
+
+  return control_info
+
+
+
+
+#
+# Used only by raw communication
+#
+def send_vanilla(x, destination, tag=0):
+  from cPickle import dumps
+  from mpi import send_string as send
+  
+  s=dumps(x, 1)
+  send(s, destination, tag)     
+  return len(s)
+
+
+def receive_vanilla(x, source, tag=0):
+  from cPickle import loads, dumps
+  from mpi import receive_string as receive
+
+
+  #Create buffer of the right size
+  #(assuming that x is similar to sent x).
+
+  if options['vanilla_bufsize']:
+    s = ' '*options['vanilla_bufsize']
+  else:  
+    s = dumps(x, 1)
+    s = s + ' '*int(0.1*len(s)) #safety
+                  
+  receive(s, source, tag)
+
+  return loads(s)
+   
+
+#----------------------- 
+# ADM
+#----------------------- 
+
+def check_C_extension(sequential_allowed):
+  """Verify existence of mpi.so.
+  """   
+  
+
+  try:
+    if sequential_allowed:  
+      import mpi
+      # This will crash on systems like the Alpha Server or the Sun
+      # if program is run sequentially      
+    else:  
+      # A more general test suitable for the Alpha Server is
+      fid = open('mpi.so', 'r')
+      fid.close()
+      # On the other hand, when using pypar as a package, we do 
+      # not have access to mpi.so so we need link from parent directory to
+      # pypar/mpi.so or pypar/mpi.c
+  except:
+    try:
+      import compile
+      compile.compile('mpi.c', 'mpicc', verbose = 0)
+    except:
+      raise "ERROR: Please compile C extension mpi.c - python install.py"
+  
+
+
+#----------------------------------------------------------------------------
+# Initialise module
+#----------------------------------------------------------------------------
+
+# Determine if MPI program is allowed to run sequentially
+# Attempting to check this automatically may case some systems to hang.
+#
+if sys.platform in ['osf1V5', 'sunos5']:  #Compaq AlphaServer or Sun
+  sequential_allowed = 0
+else:  
+  sequential_allowed = 1
+
+#print sequential_allowed  
+
+# Attempt to compile mpi.so if necessary
+#
+
+check_C_extension(sequential_allowed) 
+
+
+#Check if MPI module can be initialised
+#
+# Attempt to import and initialise mpi.so
+# If this fails, define a rudimentary interface suitable for
+# sequential execution.
+#
+#    
+
+#
+
+
+if sys.platform == 'osf1V5':
+  error = os.system('python -c "import mpi" >/dev/null 2>/dev/null') 
+  
+  # On the sun we can't do this test - even in the parallel case...
+  #
+  #if sys.platform == 'osf1V5':
+  #  error = os.system('python -c "import mpi" >/dev/null 2>/dev/null') 
+  #else:
+  #  error = 0  
+
+  # The check is performed in a separate shell.
+  # Reason: The Alpha server or the Sun cannot recover from a 
+  # try: 
+  #   import mpi
+  # if mpi.so isn't there or if the program is run as sequential.
+  # On LAM/Linux, this test may cause system to hang.
+else:
+  error = 0  
+
+
+if error:
+  print "WARNING: MPI library could not be initialised - running sequentially"
+  parallel = 0
+
+  # Define rudimentary functions to keep sequential programs happy
+  
+  def size(): return 1
+  def rank(): return 0
+
+  def Get_processor_name():
+    import os
+    try:
+      hostname = os.environ['HOST']
+    except:
+      try:  
+        hostname = os.environ['HOSTNAME']  
+      except:
+        hostname = 'Unknown'  
+
+    return hostname
+      
+
+  def Abort():
+    import sys
+    sys.exit()
+
+  def Finalize(): pass
+  
+  def Barrier(): pass  
+
+  def Wtime():
+    import time
+    return time.time()
+else:    
+  #Import the C-extension and initialise MPI
+  #
+  parallel = 1
+  from mpi import size, rank, Barrier, Wtime, Get_processor_name, Finalize, Abort, \
+                  send_string, receive_string,\
+                  send_array, receive_array, \
+                  bcast_string, bcast_array, \
+                  scatter_string, scatter_array, \
+                  gather_string, gather_array, \
+                  reduce_array, \
+                  MPI_ANY_TAG as any_tag, MPI_ANY_SOURCE as any_source, \
+                  mpi_MAX, mpi_MIN, mpi_SUM, mpi_PROD, mpi_LAND, mpi_BAND, \
+                  mpi_LOR, mpi_BOR, mpi_LXOR, mpi_BXOR, mpi_MAXLOC, mpi_MINLOC, mpi_REPLACE
+
+		  
+  print "Proc %d: MPI initialised OK" %rank()    
+		  
+
+
+
+
+
Index: trunk/anuga_core/source/pypar_dist/supporting_material/contrib/schiotz
===================================================================
--- trunk/anuga_core/source/pypar_dist/supporting_material/contrib/schiotz	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/supporting_material/contrib/schiotz	(revision 8494)
@@ -0,0 +1,33 @@
+Dear Ole Nielsen,
+
+I saw your announcement of Pypar on comp.lang.python.announce.  In our
+programs, we are using Konrad Hinsen's Scientific Python module.  I
+have a small technical question.  You write that one advantage of your
+approach is, that one does not need a special version of the Python
+executable, just the pypar module.  In Scientific Python it is
+necessary to modify Python because MPI requires that MPI_Init is
+called before the program looks at its command line arguments, and
+Python needs to look at its command line arguments to figure out which
+script to run.  How do you bypass this problem?  I know there are MPI
+implementations where this is not a problem as MPI only adds extra
+stuff to the command-line on the non-master nodes, but there are some
+implementation (MPICH, for example) where the whole command line is
+replaced with MPI stuff and the command line is only restored when
+MPI_Init is called.
+
+I am very interested to hear if you have found an elegant way to solve
+this, as it is slightly annoying to have to have an extra version of
+the Python executable.
+
+
+Best regards
+
+Jakob Schiotz
+
+
+-- 
+Jakob Schiotz, Ph.D.             !  Email: schiotz@fysik.dtu.dk
+CAMP, Department of Physics      !  WWW: http://www.fysik.dtu.dk/~schiotz/
+Technical University of Denmark  !  Phone: (+45) 45 25 32 28
+DK-2800 Lyngby, Denmark          !  Fax:   (+45) 45 93 23 99
+
Index: trunk/anuga_core/source/pypar_dist/supporting_material/contrib/schiotz.reply
===================================================================
--- trunk/anuga_core/source/pypar_dist/supporting_material/contrib/schiotz.reply	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/supporting_material/contrib/schiotz.reply	(revision 8494)
@@ -0,0 +1,61 @@
+
+Hej Jakob
+
+(I assume you understand  danish)
+
+> I saw your announcement of Pypar on comp.lang.python.announce.  In our
+> programs, we are using Konrad Hinsen's Scientific Python module.  I
+> have a small technical question.  You write that one advantage of your
+> approach is, that one does not need a special version of the Python
+> executable, just the pypar module.  In Scientific Python it is
+> necessary to modify Python because MPI requires that MPI_Init is
+> called before the program looks at its command line arguments, and
+> Python needs to look at its command line arguments to figure out which
+> script to run.  
+
+Jeg har aldrig rigtigt forstaaet hvorfor det skulle vaere noedvendigt.
+Derfor gik jeg igang med at implementere et interface hvor 
+Python oversetteren forbliver uaendret. 
+MPI_init har selvfoelgelig ikke adgang til commandline parametre 
+- saa det er prisen for min tilgang.
+
+> How do you bypass this problem?  I know there are MPI
+> implementations where this is not a problem as MPI only adds extra
+> stuff to the command-line on the non-master nodes, but there are some
+> implementation (MPICH, for example) where the whole command line is
+> replaced with MPI stuff and the command line is only restored when
+> MPI_Init is called.
+
+Det er muligt at pypar vil have problemer med det.
+Jeg har imidlertid testet pypar under LAM/Mpi (linux), 
+Native MPI for DEC Alpha og MPICH for Sun enterprise.
+Jeg kunne ikke umiddelbart faa ideen til at virke under MPICH/Linux - maaske
+pga det problem du beskriver. 
+
+> I am very interested to hear if you have found an elegant way to solve
+> this, 
+
+Jeg har ikke fundet en elegant loesning - maaske snarere en vej udenom:
+Problemet er rent teoretisk for de implementationer jeg bruger 
+og kan ignoreres.
+
+
+> as it is slightly annoying to have to have an extra version of
+> the Python executable.
+
+Ja - det var det jeg taenkte.
+
+Hvis du har lyst kan du hjaelpe med at udvikle pypar ideen ?
+
+Mvh
+Ole
+
+-------------------------------------------------------------------  
+Ole Nielsen                    | Email: Ole.Nielsen@anu.edu.au
+-------------------------------------------------------------------  
+School of Mathematical Sciences| Phone: +61 2 6125 3873 (Direct)
+Australian National University | Fax:   +61 2 6125 5549
+Canberra ACT 0200              | 
+Australia                      | 
+-------------------------------------------------------------------  
+
Index: trunk/anuga_core/source/pypar_dist/supporting_material/contrib/schiotz2
===================================================================
--- trunk/anuga_core/source/pypar_dist/supporting_material/contrib/schiotz2	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/supporting_material/contrib/schiotz2	(revision 8494)
@@ -0,0 +1,66 @@
+ > 
+ > Hej Jakob
+ > 
+ > (I assume you understand  danish)
+
+Ja.  Jeg var også lige ved at skrive til dig på dansk, men så opdagede
+jeg at du er i Australien (edu.au), ikke Århus Universitet (au.dk) som
+jeg først havde læst din adresse som.  Og jeg kender adskillige amerikanere
+med svenske navne, som ikke taler svensk, så for en sikkerheds skyld...
+
+ > 
+ > > I saw your announcement of Pypar on comp.lang.python.announce.  In our
+ > > programs, we are using Konrad Hinsen's Scientific Python module.  I
+ > > have a small technical question.  You write that one advantage of your
+ > > approach is, that one does not need a special version of the Python
+ > > executable, just the pypar module.  In Scientific Python it is
+ > > necessary to modify Python because MPI requires that MPI_Init is
+ > > called before the program looks at its command line arguments, and
+ > > Python needs to look at its command line arguments to figure out which
+ > > script to run.  
+ > 
+ > Jeg har aldrig rigtigt forstaaet hvorfor det skulle vaere noedvendigt.
+ > Derfor gik jeg igang med at implementere et interface hvor 
+ > Python oversetteren forbliver uaendret. 
+ > MPI_init har selvfoelgelig ikke adgang til commandline parametre 
+ > - saa det er prisen for min tilgang.
+
+Jeg gjorde noget tilsvarende engang, dog uden Python bindings.  Jeg
+SWIGede et parallelt molekyldynamikprogram, så det kunne styres fra et
+Python script.  MPI_Init blev kaldt fra modulet.  Det virkede fint
+under AIX, men så fik vi en klynge Compaq Alpha workstations med
+MPICH, hvor resultatet var en kryptisk fejlmeddelelse ("10452: file not
+found", eller noget i den dur).  Efter et stykke tid fandt jeg ud af,
+at Python blev kaldt på alle nodes med en kommandoline a la 
+	python 10452 asrv 6F71AA864EF433
+og det kan man jo godt forstå ikke virkede. :-)
+MPI_Init rekonstruerede den oprindelige kommandolinie.
+
+Så jeg er bange for, at princippet bag Pypar ikke er generelt
+anvendeligt.  Men det gør det jo ikke mindre praktisk, der hvor det
+virker.
+
+	[ ... ]
+ > > as it is slightly annoying to have to have an extra version of
+ > > the Python executable.
+ > 
+ > Ja - det var det jeg taenkte.
+ > 
+ > Hvis du har lyst kan du hjaelpe med at udvikle pypar ideen ?
+
+Helst ikke, jeg er ret presset for tid, og kan ikke desværre nok ikke
+bruge modulet på vores egne maskiner.  Jeg skrev et par ekstra
+funktioner ind i Scientific.MPI for et års tid siden, og gruppen her
+bruger nu Scientific.MPI såvel i Python koden som i C++ modulerne.
+
+Med venlig hilsen
+
+Jakob
+
+
+-- 
+Jakob Schiøtz, Ph.D.             !  Email: schiotz@fysik.dtu.dk
+CAMP, Department of Physics      !  WWW: http://www.fysik.dtu.dk/~schiotz/
+Technical University of Denmark  !  Phone: (+45) 45 25 32 28
+DK-2800 Lyngby, Denmark          !  Fax:   (+45) 45 93 23 99
+
Index: trunk/anuga_core/source/pypar_dist/supporting_material/contrib/testpypar.py
===================================================================
--- trunk/anuga_core/source/pypar_dist/supporting_material/contrib/testpypar.py	(revision 8494)
+++ trunk/anuga_core/source/pypar_dist/supporting_material/contrib/testpypar.py	(revision 8494)
@@ -0,0 +1,364 @@
+#!/usr/bin/env python
+# Test of MPI module 'pypar' for Python
+# 
+# Run as 
+#   python pypartest.py
+# or 
+#   mpirun -np 2 pypartest.py
+# (perhaps try number of processors more than 2)
+#
+# To verify bandwidth of your architecture please run pytiming (and ctiming) 
+#
+# OMN, FEB 2002
+
+try:
+  import Numeric
+except:
+  raise 'Module Numeric must be present to run pypar'
+
+
+print "Importing pypar"
+import pypar
+methods = dir(pypar)
+assert 'Abort' in methods
+assert 'Finalize' in methods
+assert 'Get_processor_name' in methods
+assert 'Wtime' in methods
+assert 'rank' in methods
+assert 'raw_receive' in methods
+assert 'raw_send' in methods
+assert 'receive' in methods
+assert 'send' in methods
+assert 'bcast' in methods
+assert 'size' in methods
+
+print "Module pypar imported OK"
+#pypar.Barrier()
+
+
+myid =    pypar.rank()
+numproc = pypar.size()
+node =    pypar.Get_processor_name()
+
+print "I am processor %d of %d on node %s" %(myid, numproc, node)
+pypar.Barrier()
+
+
+if numproc > 1:
+  # Test simple raw communication (arrays, strings and general)
+  #
+  N = 17 #Number of elements
+  
+  if myid == 0:
+    # Integer arrays
+    #
+    A = Numeric.array(range(N))
+    B = Numeric.zeros(N)    
+    pypar.raw_send(A,1)
+    pypar.raw_receive(B,numproc-1)
+    
+    assert Numeric.allclose(A, B)
+    print "Raw communication of numeric integer arrays OK"
+
+    # Real arrays
+    #
+    A = Numeric.array(range(N)).astype('f')
+    B = Numeric.zeros(N).astype('f')    
+    pypar.raw_send(A,1)
+    pypar.raw_receive(B,numproc-1)
+    
+    assert Numeric.allclose(A, B)    
+    print "Raw communication of numeric real arrays OK"
+
+    # Strings (< 256 characters)
+    #
+    A = "and now to something completely different !"
+    B = " "*len(A)
+    pypar.raw_send(A,1)
+    pypar.raw_receive(B,numproc-1)
+    
+    assert A == B
+    print "Raw communication of strings OK"
+    
+    # A more general structure
+    #
+    A = ['ABC', (1,2,3.14), {8: 'Monty'}, Numeric.array([13.45, 1.2])]
+    B = ['   ', (0,0,0.0), {0: '     '}, Numeric.zeros(2).astype('f')]    
+    pypar.raw_send(A,1)
+    B = pypar.raw_receive(B,numproc-1)
+    
+    assert A == B
+    print "Raw communication of general structures OK"
+    
+  else:  
+    # Integers
+    #
+    X = Numeric.zeros(N)
+    pypar.raw_receive(X, myid-1)  
+    pypar.raw_send(X, (myid+1)%numproc)
+  
+    # Floats
+    #
+    X = Numeric.zeros(N).astype('f')
+    pypar.raw_receive(X, myid-1)  
+    pypar.raw_send(X, (myid+1)%numproc)    
+
+    # Strings
+    #
+    X = " "*256
+    pypar.raw_receive(X, myid-1)  
+    pypar.raw_send(X.strip(), (myid+1)%numproc)    
+
+    # General
+    #
+    X = ['   ', (0,0,0.0), {0: '     '}, Numeric.zeros(2).astype('f')]
+    X = pypar.raw_receive(X, myid-1)  
+    pypar.raw_send(X, (myid+1)%numproc)    
+    
+
+  # Test easy communication  - without buffers (arrays, strings and general)
+  #
+  N = 17 #Number of elements
+  
+  if myid == 0:
+    # Integer arrays
+    #
+    A = Numeric.array(range(N))
+
+    pypar.send(A,1)
+    B = pypar.receive(numproc-1)
+    
+
+    assert Numeric.allclose(A, B)
+    print "Simplified communication of numeric integer arrays OK"
+
+    # Real arrays
+    #
+    A = Numeric.array(range(N)).astype('f')
+    pypar.send(A,1)
+    B=pypar.receive(numproc-1)
+    
+    assert Numeric.allclose(A, B)    
+    print "Simplified communication of numeric real arrays OK"
+
+    # Strings
+    #
+    A = "and now to something completely different !"
+    pypar.send(A,1)
+    B=pypar.receive(numproc-1)
+    
+    assert A == B
+    print "Simplified communication of strings OK"
+    
+    # A more general structure
+    #
+    A = ['ABC', (1,2,3.14), {8: 'Monty'}, Numeric.array([13.45, 1.2])]
+    pypar.send(A,1)
+    B = pypar.receive(numproc-1)
+    
+    assert A == B
+    print "Simplified communication of general structures OK"
+    
+  else:  
+    # Integers
+    #
+    X=pypar.receive(myid-1)  
+    pypar.send(X, (myid+1)%numproc)
+  
+    # Floats
+    #
+    X=pypar.receive(myid-1)  
+    pypar.send(X, (myid+1)%numproc)    
+
+    # Strings
+    #
+    X=pypar.receive(myid-1)  
+    pypar.send(X, (myid+1)%numproc)    
+
+    # General
+    #
+    X = pypar.receive(myid-1)  
+    pypar.send(X, (myid+1)%numproc)    
+
+
+  # Test broadcast  - with buffers (arrays, strings and general)
+  #
+  N = 17 #Number of elements
+      
+  testString = 'test' + str(myid)
+  pypar.bcast(testString, 0)
+  assert testString == 'test0'
+  
+  testString = 'test' + str(myid)
+  pypar.bcast(testString, numproc-1)
+  assert testString == 'test' + str(numproc-1)
+  
+  if myid == 0:
+    print "Broadcast communication of strings OK"
+  
+  testArray = myid * Numeric.array(range(N))
+  pypar.bcast(testArray, 1)
+  assert Numeric.allclose(testArray, 1 * testArray)
+  
+  if myid == 0:    
+    print "Broadcast communication of numeric integer array OK"
+
+
+  testArray = myid * Numeric.array(range(N)).astype('f')
+  pypar.bcast(testArray, 1)
+  assert Numeric.allclose(testArray, 1 * testArray)
+      
+  if myid == 0:
+    print "Broadcast communication of numeric real array OK"
+    
+  testGeneral = ['ABC', myid, (1,2,3), {8: 'Monty'}, Numeric.array([13.45, 1.2])]
+  
+  testGeneral = pypar.bcast(testGeneral, 1)
+  
+  assert testGeneral ==  ['ABC', 1, (1,2,3), {8: 'Monty'}, Numeric.array([13.45, 1.2])]
+  
+  if myid == 0:
+    print "Broadcast communication of general structures OK"
+  
+
+
+
+
+
+  # Test scatter  - with/without buffers (arrays, strings)
+  #
+  N = 17 #Number of elements
+      
+  testString = 'test' + str(myid)
+  s_size = 1   
+  X = ' '*s_size
+  pypar.raw_scatter(testString, s_size, X, 0)
+  
+  Y = pypar.scatter(testString, s_size, 0)
+      
+  if myid == 0:
+    assert X == 't'
+    assert Y == 't'
+    print "Scatter communication of strings OK"
+
+  testArray = Numeric.array(range(N))
+  s_size = 1   
+  X = Numeric.zeros(s_size)
+  pypar.raw_scatter(testArray, s_size, X, 0)
+  
+  Y = pypar.scatter(testArray, s_size, 0)
+  
+
+  if myid == 0:
+    assert X == [0]
+    assert Y == [0]
+    print "Scatter communication of numeric integer array OK"
+
+
+  testArray = Numeric.array(range(N)).astype('f')
+  s_size = 1   
+  X = Numeric.zeros(s_size).astype('f')
+  pypar.raw_scatter(testArray, s_size, X, 0)
+    
+  Y = pypar.scatter(testArray, s_size, 0)
+  
+  if myid == 0:
+    assert X == [0.0]
+    assert Y == [0.0]
+    print "Scatter communication of numeric real array OK"
+
+
+
+  # Test gather  - with/without buffers (arrays, strings)
+  #
+  N = 17 #Number of elements
+      
+  testString = 'AB'
+  s_size = 2 # to help test
+  X = ' '*(s_size*numproc)
+  pypar.raw_gather(testString, s_size, X, 0, 0)
+
+  if myid == 0:
+    assert X == 'AB' * numproc
+
+  Y =  pypar.gather(testString, s_size, 0) 
+  #print myid, Y
+  
+  if myid == 0:
+    assert X == 'AB' * numproc
+    assert Y == 'AB' * numproc
+    print "Gather communication of strings OK"
+  
+
+  testArray = Numeric.array(range(N))
+  s_size = N   
+  X = Numeric.zeros(s_size*numproc)
+  pypar.raw_gather(testArray, s_size, X, 0, 0)
+
+  Y = pypar.gather(testArray, s_size, 0)
+  
+  if myid == 0:
+    for i in range(numproc):       
+      assert Numeric.allclose(testArray, X[(i * s_size): ((i+1)*s_size)])
+    print "Gather communication of numeric integer array OK"
+    
+    
+  testArray = Numeric.array(range(N)).astype('f')
+  s_size = N   
+  X = Numeric.zeros(s_size*numproc).astype('f')
+  pypar.raw_gather(testArray, s_size, X, 0, 0)
+  
+  Y = pypar.gather(testArray, s_size, 0)
+    
+  if myid == 0:
+    for i in range(numproc):       
+      assert Numeric.allclose(testArray, X[(i * s_size): ((i+1)*s_size)])
+    print "Gather communication of numeric real array OK"
+    
+
+  # Test reduce  - with/without buffers (arrays, strings)
+  #
+  N = 17 #Number of elements
+      
+  testArray = Numeric.array(range(N))
+  s_size = N # to help test
+  X = Numeric.zeros(s_size)
+  pypar.raw_reduce(testArray, X, s_size, pypar.mpi_SUM, 0, 0)
+  if myid == 0:
+    print 'SUM', X
+  pypar.raw_reduce(testArray, X, s_size, pypar.mpi_MAX, 0, 0)
+  if myid == 0:
+    print 'MAX', X
+  pypar.raw_reduce(testArray, X, s_size, pypar.mpi_MIN, 0, 0)
+  if myid == 0:
+    print 'MIN', X
+  pypar.raw_reduce(testArray, X, s_size, pypar.mpi_PROD, 0, 0)
+  if myid == 0:
+    print 'PROD', X
+  pypar.raw_reduce(testArray, X, s_size, pypar.mpi_LAND, 0, 0)
+  if myid == 0:
+    print 'LAND', X
+  pypar.raw_reduce(testArray, X, s_size, pypar.mpi_BAND, 0, 0)
+  if myid == 0:
+    print 'BAND', X
+  pypar.raw_reduce(testArray, X, s_size, pypar.mpi_LOR, 0, 0)
+  if myid == 0:
+    print 'LOR', X
+  pypar.raw_reduce(testArray, X, s_size, pypar.mpi_BOR, 0, 0)
+  if myid == 0:
+    print 'BOR', X
+  pypar.raw_reduce(testArray, X, s_size, pypar.mpi_LXOR, 0, 0)
+  if myid == 0:
+    print 'LXOR', X
+  pypar.raw_reduce(testArray, X, s_size, pypar.mpi_BXOR, 0, 0)
+  if myid == 0:
+    print 'BXOR', X
+  pypar.raw_reduce(testArray, X, s_size, pypar.mpi_MAXLOC, 0, 0)  
+  if myid == 0:
+    print 'MAXLOC', X
+  pypar.raw_reduce(testArray, X, s_size, pypar.mpi_MINLOC, 0, 0)
+  if myid == 0:
+    print 'MINLOC', X
+  pypar.raw_reduce(testArray, X, s_size, pypar.mpi_REPLACE, 0, 0)
+  if myid == 0:
+    print 'REPLACE', X
