| 1 | ------------------------------------------------ |
|---|
| 2 | PyPAR - Parallel Python, efficient and scalable |
|---|
| 3 | parallelism using the message passing interface (MPI). |
|---|
| 4 | |
|---|
| 5 | Author: Ole Nielsen (2001, 2002, 2003) |
|---|
| 6 | Email: Ole.Nielsen@anu.edu.au |
|---|
| 7 | Version: See pypar.__version__ |
|---|
| 8 | Date: See pypar.__date__ |
|---|
| 9 | |
|---|
| 10 | Major contributions by |
|---|
| 11 | Gian Paolo Ciceri (gp.ciceri@acm.org) |
|---|
| 12 | Prabhu Ramachandran (prabhu@aero.iitm.ernet.in) |
|---|
| 13 | |
|---|
| 14 | Minor but important contributions by |
|---|
| 15 | Doug Orr (doug@arbor.net) |
|---|
| 16 | Michal Kaukic (mike@frcatel.fri.utc.sk) |
|---|
| 17 | |
|---|
| 18 | ------------------------------------------------ |
|---|
| 19 | |
|---|
| 20 | The python module pypar.py and the C-extension mpi.c |
|---|
| 21 | implements scalable parallelism on distributed and shared |
|---|
| 22 | memory architectures using the essential subset of |
|---|
| 23 | the Message Passing Interface (MPI) standard. |
|---|
| 24 | |
|---|
| 25 | FEATURES |
|---|
| 26 | |
|---|
| 27 | - Python interpreter is not modified: |
|---|
| 28 | Parallel python programs need only import the pypar module. |
|---|
| 29 | |
|---|
| 30 | - Easy installation: This is essentially about compiling and linking |
|---|
| 31 | the C-extension with the local MPI installation. A distutils setup file |
|---|
| 32 | is included. |
|---|
| 33 | |
|---|
| 34 | - Flexibility: Pypar allows communication of general Python objects |
|---|
| 35 | of any type. |
|---|
| 36 | |
|---|
| 37 | - Intuitive API: |
|---|
| 38 | The user need only specify what to send and to which processor. |
|---|
| 39 | Pypar takes care of details about |
|---|
| 40 | data types and MPI specifics such as tags, communicators and buffers. |
|---|
| 41 | Receiving is analogous. |
|---|
| 42 | |
|---|
| 43 | - Efficiency: |
|---|
| 44 | Full bandwidth of C-MPI programs is achieved for consecutive Numerical |
|---|
| 45 | arrays. Latency is less than twice that of pure C-MPI programs. |
|---|
| 46 | Test programs to verify this are included (pytiming, ctiming.c) |
|---|
| 47 | |
|---|
| 48 | - Lightweight: |
|---|
| 49 | Pypar consists of just two files: mpiext.c and pypar.py |
|---|
| 50 | |
|---|
| 51 | See the DOC file for instructions on how to program with pypar. |
|---|
| 52 | |
|---|
| 53 | |
|---|
| 54 | |
|---|
| 55 | PRE-REQUISITES (on all nodes) |
|---|
| 56 | |
|---|
| 57 | Python 2.0 or later |
|---|
| 58 | Numeric Python (incl RandomArray) matching the Python installation |
|---|
| 59 | Native MPI C library |
|---|
| 60 | Native C compiler |
|---|
| 61 | |
|---|
| 62 | Pypar has been tested on the following platforms |
|---|
| 63 | MPI on DEC Alpha |
|---|
| 64 | LAM/MPI (6.5.6, 7.0) on Linux (Debian Woody/Cid and Red Hat 7.1) |
|---|
| 65 | MPICH and LAM 7.0 on Solaris (Sun Enterprise) |
|---|
| 66 | MPICH on Linux |
|---|
| 67 | MPICH on Windows (NT/2000) |
|---|
| 68 | MPICH for Darwin (Mac OS) |
|---|
| 69 | LAM/MPI with Linux on Opteron 64 bit |
|---|
| 70 | |
|---|
| 71 | |
|---|
| 72 | INSTALL |
|---|
| 73 | |
|---|
| 74 | UNIX PLATFORMS |
|---|
| 75 | Type |
|---|
| 76 | python setup.py install |
|---|
| 77 | |
|---|
| 78 | This should install pypar and its extension in the default |
|---|
| 79 | site-packages directory on your system. |
|---|
| 80 | |
|---|
| 81 | If you wish to install it in your home directory use |
|---|
| 82 | |
|---|
| 83 | python setup.py install --prefix=~ |
|---|
| 84 | |
|---|
| 85 | WINDOWS (NT/2000, MPICH) |
|---|
| 86 | To build, you need: |
|---|
| 87 | 1) MPICH (http://www-unix.mcs.anl.gov/mpi/mpich/) |
|---|
| 88 | 2) MinGW (http://www.mingw.org). Tested with GCC v 2.95-3.6. |
|---|
| 89 | 3) Set MPICH_DIR to the appropriate directory |
|---|
| 90 | 4) Build using 'python setup.py build --compiler=mingw32' |
|---|
| 91 | 5) Install using 'python setup.py install'. |
|---|
| 92 | |
|---|
| 93 | --- |
|---|
| 94 | If you encountered any installation problems, read on. |
|---|
| 95 | Do not worry about messages about unresolved symbols. This is normal |
|---|
| 96 | for shared libraries on some machines. |
|---|
| 97 | |
|---|
| 98 | To compile C-extension mpi.c requires either a mpi-aware c compiler |
|---|
| 99 | such as mpicc or a standard c compiler with mpi libraries linked in. |
|---|
| 100 | See local MPI installation for details, possibly edit Makefile and |
|---|
| 101 | type make. |
|---|
| 102 | See installation notes below about compiling under mpi. |
|---|
| 103 | |
|---|
| 104 | |
|---|
| 105 | TESTING |
|---|
| 106 | Pypar comes with a number of tests and demos available in the |
|---|
| 107 | examples directory - please try these to verify the installation. |
|---|
| 108 | |
|---|
| 109 | |
|---|
| 110 | RUNNING PYTHON MPI JOBS |
|---|
| 111 | |
|---|
| 112 | Pypar runs in exactly the same way as MPI programs written in |
|---|
| 113 | C or Fortran. |
|---|
| 114 | |
|---|
| 115 | E.g. to run the enclosed demo script (demo.py) on 4 processors, |
|---|
| 116 | enter a command similar to |
|---|
| 117 | |
|---|
| 118 | mpirun -np 4 python demo.py |
|---|
| 119 | |
|---|
| 120 | Consult your MPI distribution for exact syntax of mpirun. |
|---|
| 121 | Sometimes it is called prun and often parallel jobs will be |
|---|
| 122 | submitted in batch mode. |
|---|
| 123 | |
|---|
| 124 | You can also execute demo.py as a stand-alone executable python script |
|---|
| 125 | |
|---|
| 126 | mpirun -np 4 demo.py |
|---|
| 127 | |
|---|
| 128 | |
|---|
| 129 | Enclosed is a script to estimate the communication speed of your system. |
|---|
| 130 | Execute |
|---|
| 131 | |
|---|
| 132 | mpirun -np 4 pytiming |
|---|
| 133 | |
|---|
| 134 | Care has been taken in pypar to achieve the same bandwidth and almost as |
|---|
| 135 | good communication latency as corresponding C/MPI programs. |
|---|
| 136 | Please compile and run ctiming.c to see the reference values: |
|---|
| 137 | |
|---|
| 138 | make ctiming |
|---|
| 139 | mpirun -np 4 ctiming |
|---|
| 140 | |
|---|
| 141 | Note that timings might fluctuate from run to run due to variable |
|---|
| 142 | system load. |
|---|
| 143 | |
|---|
| 144 | An example of a master-slave program using pypar is available in demo2.py. |
|---|
| 145 | |
|---|
| 146 | |
|---|
| 147 | INSTALLATION NOTES (If all else fails) |
|---|
| 148 | |
|---|
| 149 | Most MPI implementations provide a script or an executable called |
|---|
| 150 | "mpicc" which compiles C programs using MPI and does |
|---|
| 151 | not require any explicitly mentioned libraries. |
|---|
| 152 | If such a script exists, but with a different name, change |
|---|
| 153 | the name in the beginning of compile.py. If no such script exists, put |
|---|
| 154 | the name of your C compiler in that place and add all required linking |
|---|
| 155 | options yourself. |
|---|
| 156 | |
|---|
| 157 | For example, on an Alpha server it would look something like |
|---|
| 158 | |
|---|
| 159 | cc -c mpi.c -I/opt/Python-2.1/include/python2.1/ |
|---|
| 160 | cc -shared mpi.o -o mpi.so -lmpi -lelan |
|---|
| 161 | |
|---|
| 162 | or using the wrapper |
|---|
| 163 | |
|---|
| 164 | mpicc -c mpi.c -I/opt/Python-2.1/include/python2.1/ |
|---|
| 165 | mpicc -shared mpi.o -o mpi.so |
|---|
| 166 | |
|---|
| 167 | On Linux (using LAM-MPI) it is |
|---|
| 168 | |
|---|
| 169 | mpicc -c mpi.c -I/opt/Python-2.1/include/python2.1/ |
|---|
| 170 | mpicc -shared mpi.o -o mpi.so |
|---|
| 171 | |
|---|
| 172 | Start processors using |
|---|
| 173 | lamboot -v lamhosts |
|---|
| 174 | |
|---|
| 175 | |
|---|
| 176 | HOW DOES PYPAR WORK? |
|---|
| 177 | |
|---|
| 178 | Pypar works as follows |
|---|
| 179 | 1 mpirun starts P python processes as specified by mpirun -np P |
|---|
| 180 | 2 each python process imports pypar which in turn imports a shared |
|---|
| 181 | library (python extension mpiext.so) that has been statically linked |
|---|
| 182 | to the C MPI libraries using e.g. mpicc (or cc -lmpi) |
|---|
| 183 | 3 The Python extension proceeds to call MPI_init with any commandline |
|---|
| 184 | parameters passed in by mpirun (As far as I remember MPICH uses this |
|---|
| 185 | to identify rank and size, whereas LAM doesn't) |
|---|
| 186 | 4 The supported MPI calls are made available to Python through the |
|---|
| 187 | pypar interface |
|---|
| 188 | |
|---|
| 189 | If pypar is invoked sequentially it supports a rudimentary interface |
|---|
| 190 | with e.g. rank() == 0 and size() == 1 |
|---|
| 191 | |
|---|
| 192 | |
|---|
| 193 | |
|---|
| 194 | DOCUMENTATION |
|---|
| 195 | |
|---|
| 196 | See the file DOC for an introduction to pypar. |
|---|
| 197 | See also examples demo.py and pytiming |
|---|
| 198 | as well as documentation in pypar.py. |
|---|
| 199 | |
|---|
| 200 | |
|---|
| 201 | HISTORY |
|---|
| 202 | version 1.9.1 (15 Dec 2003) |
|---|
| 203 | Consistent naming (all lower case, no abbreviations) |
|---|
| 204 | version 1.9 (3 Dec 2003) |
|---|
| 205 | Obsoleted raw forms of communication as they were confusing. |
|---|
| 206 | The semantics of send, receive, scatter, gather etc is now that |
|---|
| 207 | one can optionally specify a buffer to use if desired. |
|---|
| 208 | Bypass forms added. |
|---|
| 209 | |
|---|
| 210 | version 1.8.2 (16 November 2003) |
|---|
| 211 | Fixed scatter, gather and reduce calls to automatically |
|---|
| 212 | figure out buffer lengths. Also tested scatter and gather for |
|---|
| 213 | complex and multidimensional data. |
|---|
| 214 | version 1.8.1 (13 November 2003) |
|---|
| 215 | Added direct support (i.e. not using the vanilla protocol) |
|---|
| 216 | for multidimensional Numeric arrays of type Complex. |
|---|
| 217 | version 1.8 (11 November 2003) |
|---|
| 218 | Changed status object to be (optionally) returned from receive and |
|---|
| 219 | raw_receive rather maintaining a global state where one instance of |
|---|
| 220 | the status object is modified. |
|---|
| 221 | This was suggested by the audience at presentation at Department of |
|---|
| 222 | Computer Science, Australian National University, 18 June 2003. |
|---|
| 223 | |
|---|
| 224 | version 1.7 (7 November 2003) |
|---|
| 225 | Added support for Numerical arrays of arbitrary dimension |
|---|
| 226 | in send, receive and bcast. |
|---|
| 227 | The need for this obvious functionality was pointed by Michal Kaukic. |
|---|
| 228 | |
|---|
| 229 | version 1.6.5 (30 August 2003) |
|---|
| 230 | Added NULL commandline argument as required by LAM 7.0 and |
|---|
| 231 | identified by Doug Orr and Dave Reed. |
|---|
| 232 | version 1.6.4 (10 Jan 2003) |
|---|
| 233 | Comments and review of installation |
|---|
| 234 | version 1.6.3 (10 Jan 2003) |
|---|
| 235 | Minor issues and clean-up |
|---|
| 236 | version 1.6.2 (29 Oct 2002) |
|---|
| 237 | Added Windows platform to installation as contributed by |
|---|
| 238 | Simon Frost |
|---|
| 239 | version 1.6.0 (18 Oct 2002) |
|---|
| 240 | Changed installation to distutils as contributed by |
|---|
| 241 | Prabhu Ramachandran |
|---|
| 242 | |
|---|
| 243 | version 1.5 (30 April 2002) |
|---|
| 244 | Got pypar to work with MPICH/Linux and cleaned up initialisation |
|---|
| 245 | version 1.4 (4 March 2002) |
|---|
| 246 | Fixed-up and ran testpypar on 22 processors on Sun |
|---|
| 247 | version 1.3 (21 February 2002) |
|---|
| 248 | Added gather and reduce fixed up testpypar.py |
|---|
| 249 | |
|---|
| 250 | version 1.2.2, 1.2.3 (17 February 2002) |
|---|
| 251 | Minor fixes in distribution |
|---|
| 252 | version 1.2.1 (16 February 2002) |
|---|
| 253 | Status block, MPI_ANY_TAG, MPI_ANY_SOURCE exported |
|---|
| 254 | Version 1.2 (15 February 2002) |
|---|
| 255 | Scatter added by Gian Paolo Ciceri |
|---|
| 256 | Version 1.1 (14 February 2002) |
|---|
| 257 | Bcast added by Gian Paolo Ciceri |
|---|
| 258 | Version 1.0.2 (10 February 2002) |
|---|
| 259 | Modified by Gian Paulo Ciceri to allow pypar run under Python 2.2 |
|---|
| 260 | Version 1.0.1 (8 February 2002) |
|---|
| 261 | Modified to install on SUN enterprise systems under Mpich |
|---|
| 262 | Version 1.0 (7 February 2002) |
|---|
| 263 | First public release for Python 2.1 (OMN) |
|---|
| 264 | |
|---|
| 265 | TODO |
|---|
| 266 | max_tag is set to 32767. This works for Linux/LAM |
|---|
| 267 | I couldn't use MPI_TAG_UB as it returned 0. |
|---|
| 268 | I would like a general solution. |
|---|
| 269 | |
|---|
| 270 | |
|---|
| 271 | Scatter needs to send buffer specified on all processes |
|---|
| 272 | even though it is ignored by all non-root processes. |
|---|
| 273 | How could we overcome that? |
|---|
| 274 | Similar problem for gather |
|---|
| 275 | |
|---|
| 276 | Gather an scatter: One should be able to specify along which axis |
|---|
| 277 | arrays should be concatenated. |
|---|
| 278 | |
|---|
| 279 | |
|---|
| 280 | KNOWN BUGS |
|---|
| 281 | Scatter for the moment works only properly when the amount of data is a |
|---|
| 282 | multiple of the number of processors (as does the underlying MPI_Scatter). |
|---|
| 283 | I am working on a more general scatter (and gather) which will |
|---|
| 284 | distribute data as evenly as possible for all amounts of data. |
|---|
| 285 | |
|---|
| 286 | |
|---|
| 287 | LICENSE |
|---|
| 288 | This program is free software; you can redistribute it and/or modify |
|---|
| 289 | it under the terms of the GNU General Public License as published by |
|---|
| 290 | the Free Software Foundation; either version 2 of the License, or |
|---|
| 291 | (at your option) any later version. |
|---|
| 292 | |
|---|
| 293 | This program is distributed in the hope that it will be useful, |
|---|
| 294 | but WITHOUT ANY WARRANTY; without even the implied warranty of |
|---|
| 295 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
|---|
| 296 | GNU General Public License (http://www.gnu.org/copyleft/gpl.html) |
|---|
| 297 | for more details. |
|---|
| 298 | |
|---|
| 299 | You should have received a copy of the GNU General Public License |
|---|
| 300 | along with this program; if not, write to the Free Software |
|---|
| 301 | Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 |
|---|
| 302 | |
|---|
| 303 | |
|---|
| 304 | Contact address: Ole.Nielsen@anu.edu.au |
|---|
| 305 | |
|---|
| 306 | ============================================================================= |
|---|
| 307 | |
|---|
| 308 | ACKNOWLEDGEMENTS |
|---|
| 309 | This work was supported by School of Mathematical Sciences at |
|---|
| 310 | the Australian National University and funded by Australian |
|---|
| 311 | Partnership for Advanced Computing (APAC). |
|---|
| 312 | Many thanks go to |
|---|
| 313 | |
|---|
| 314 | Gian Paolo Ciceri (gp.ciceri@acm.org) |
|---|
| 315 | for fixing pypar to run under Python 2.2 and for adding all the |
|---|
| 316 | collective communication stuff from version 1.1 and onwards. |
|---|
| 317 | Prabhu Ramachandran (prabhu@aero.iitm.ernet.in) |
|---|
| 318 | for making a proper distutils installation procedure |
|---|
| 319 | Simon D. W. Frost (sdfrost@ucsd.edu) |
|---|
| 320 | for testing pypar under Windows and adding installation procedure |
|---|
| 321 | Jakob Schiotz (schiotz@fysik.dtu.dk) and Steven Farcy (steven@artabel.net) |
|---|
| 322 | for pointing out initial problems with pypar and MPICH. |
|---|
| 323 | Markus Hegland (Markus.Hegland@anu.edu.au) |
|---|
| 324 | for supporting the work. |
|---|
| 325 | Doug Orr (doug@arbor.net) |
|---|
| 326 | for identifying and fixing problem with LAM 7.0 requiring |
|---|
| 327 | last commandline argumnet to be NULL. |
|---|
| 328 | Dave Reed (dreed@capital.edu) |
|---|
| 329 | for pointing out problem with LAM 7.0 and for testing the fix. |
|---|
| 330 | Matthew Wakefield (matthew.wakefield@anu.edu.au) |
|---|
| 331 | for pointing out and fixing support for Mac OS X (darwin) |
|---|
| 332 | David Brown (David.L.Brown@kla-tencor.com) |
|---|
| 333 | for contributing to the Windows/Cygwin installation procedure |
|---|
| 334 | Michal Kaukic (mike@frcatel.fri.utc.sk) |
|---|
| 335 | for pointing out the need for multi dimensional arrays and |
|---|
| 336 | for suggesting an implementation in the 2D case. |
|---|
| 337 | |
|---|
| 338 | |
|---|
| 339 | |
|---|
| 340 | |
|---|
| 341 | |
|---|
| 342 | |
|---|
| 343 | |
|---|
| 344 | |
|---|
| 345 | |
|---|
| 346 | |
|---|
| 347 | |
|---|
| 348 | |
|---|
| 349 | |
|---|
| 350 | |
|---|
| 351 | |
|---|
| 352 | |
|---|
| 353 | |
|---|
