Index: inundation/parallel/documentation/parallel.tex
===================================================================
--- inundation/parallel/documentation/parallel.tex	(revision 3245)
+++ inundation/parallel/documentation/parallel.tex	(revision 3315)
@@ -44,5 +44,10 @@
 \end{figure}
 
-Figure \ref{fig:mergrid4} shows the Merimbula grid partitioned over four processors. Note that one submesh may comprise several unconnected mesh partitions. Table \ref{tbl:mer4} gives the node distribution over the four processors while Table \ref{tbl:mer8} shows the distribution over eight processors. These results imply that Pymetis gives a reasonably well balanced partition of the mesh.
+Figure \ref{fig:mergrid4} shows the Merimbula grid partitioned over
+four processors. Note that one submesh may comprise several
+unconnected mesh partitions. Table \ref{tbl:mer4} gives the node
+distribution over the four processors while Table \ref{tbl:mer8}
+shows the distribution over eight processors. These results imply
+that Pymetis gives a reasonably well balanced partition of the mesh.
 
 \begin{figure}[hbtp]
@@ -52,10 +57,10 @@
 
   \centerline{ \includegraphics[scale = 0.75]{figures/mermesh4d.eps}
-  \includegraphics[scale = 0.75]{figures/mermesh4b.eps}} 
+  \includegraphics[scale = 0.75]{figures/mermesh4b.eps}}
   \caption{The Merimbula grid partitioned over 4 processors using Metis.}
  \label{fig:mergrid4}
 \end{figure}
 
-\begin{table} 
+\begin{table}
 \caption{Running an 4-way test of
 {\tt run_parallel_sw_merimbula_metis.py}}\label{tbl:mer4}
@@ -81,12 +86,23 @@
 \end{table}
 
-The number of submeshes found by Pymetis is equal to the number of processors; Submesh $p$ will be assigned to Processor $p$.
+The number of submeshes found by Pymetis is equal to the number of
+processors; Submesh $p$ will be assigned to Processor $p$.
 
 \subsection {Building the Ghost Layer}\label{sec:part2}
-The function {\tt build_submesh.py} is the work-horse and is responsible for
-setting up the communication pattern as well as assigning the local numbering scheme for the submeshes.
-
-Consider the example subpartitioning given in Figure \ref{fig:subdomain}. During the \code{evolve} calculations Triangle 2 in Submesh 0 will need to access its neighbour Triangle 3 stored in Submesh 1. The standard approach to this problem is to add an extra layer of triangles, which we call ghost triangles. The ghost triangles
-are read-only, they should not be updated during the calculations, they are only there to hold any extra information that a processor may need to complete its calculations. The ghost triangle values are updated through communication calls. Figure \ref{fig:subdomaing} shows the submeshes with the extra layer of ghost triangles.
+The function {\tt build_submesh.py} is the work-horse and is
+responsible for setting up the communication pattern as well as
+assigning the local numbering scheme for the submeshes.
+
+Consider the example subpartitioning given in Figure
+\ref{fig:subdomain}. During the \code{evolve} calculations Triangle
+2 in Submesh 0 will need to access its neighbour Triangle 3 stored
+in Submesh 1. The standard approach to this problem is to add an
+extra layer of triangles, which we call ghost triangles. The ghost
+triangles are read-only, they should not be updated during the
+calculations, they are only there to hold any extra information that
+a processor may need to complete its calculations. The ghost
+triangle values are updated through communication calls. Figure
+\ref{fig:subdomaing} shows the submeshes with the extra layer of
+ghost triangles.
 
 \begin{figure}[hbtp]
@@ -99,17 +115,30 @@
 \begin{figure}[hbtp]
   \centerline{ \includegraphics[scale = 0.6]{figures/subdomainghost.eps}}
-  \caption{An example subpartitioning with ghost triangles. The numbers in brackets shows the local numbering scheme that is calculated and stored with the mesh, but not implemented until the local mesh is built. See Section \ref{sec:part4}. }
+  \caption{An example subpartitioning with ghost triangles. The numbers
+in brackets shows the local numbering scheme that is calculated and
+stored with the mesh, but not implemented until the local mesh is
+built. See Section \ref{sec:part4}. }
  \label{fig:subdomaing}
 \end{figure}
 
-When partitioning the mesh we introduce new, dummy, boundary edges. For example, Triangle 2 in Submesh 1 from Figure \ref{fig:subdomaing} originally shared an edge with Triangle 1, but after partitioning that edge becomes a boundary edge. These new boundary edges are are tagged as \code{ghost} and should, in general, be assigned a type of \code{None}. The following piece of code taken from {\tt run_parallel_advection.py} shows an example.  
-{\small \begin{verbatim}  
-T = Transmissive_boundary(domain)
-domain.default_order = 2
-domain.set_boundary( {'left': T, 'right': T, 'bottom': T, 'top': T, 'ghost': Non
-e} )
+When partitioning the mesh we introduce new, dummy, boundary edges.
+For example, Triangle 2 in Submesh 1 from Figure
+\ref{fig:subdomaing} originally shared an edge with Triangle 1, but
+after partitioning that edge becomes a boundary edge. These new
+boundary edges are are tagged as \code{ghost} and should, in
+general, be assigned a type of \code{None}. The following piece of
+code taken from {\tt run_parallel_advection.py} shows an example.
+{\small \begin{verbatim} T = Transmissive_boundary(domain)
+domain.default_order = 2 domain.set_boundary( {'left': T, 'right':
+T, 'bottom': T, 'top': T, 'ghost': Non e} )
 \end{verbatim}}
 
-Looking at Figure \ref{fig:subdomaing} we see that after each \code{evolve} step Processor 0  will have to send the updated values for Triangle 2 and Triangle 4 to Processor 1, and similarly Processor 1 will have to send the updated values for Triangle 3 and Triangle 5 (recall that Submesh $p$ will be assigned to Processor $p$). The \code{build_submesh} function builds a dictionary that defines the communication pattern.
+Looking at Figure \ref{fig:subdomaing} we see that after each
+\code{evolve} step Processor 0  will have to send the updated values
+for Triangle 2 and Triangle 4 to Processor 1, and similarly
+Processor 1 will have to send the updated values for Triangle 3 and
+Triangle 5 (recall that Submesh $p$ will be assigned to Processor
+$p$). The \code{build_submesh} function builds a dictionary that
+defines the communication pattern.
 
 Finally, the ANUGA code assumes that the triangles (and nodes etc.) are numbered consecutively starting from 0. Consequently, if Submesh 1 in Figure \ref{fig:subdomaing} was passed into the \code{evolve} calculations it would crash. The \code{build_submesh} function determines a local numbering scheme for each submesh, but it does not actually update the numbering, that is left to \code{build_local}.
@@ -118,7 +147,12 @@
 \subsection {Sending the Submeshes}\label{sec:part3}
 
-All of the functions described so far must be run in serial on Processor 0. The next step is to start the parallel computation by spreading the submeshes over the processors. The communication is carried out by
-\code{send_submesh} and \code{rec_submesh} defined in {\tt build_commun.py}.
-The \code{send_submesh} function should be called on Processor 0 and sends the Submesh $p$ to Processor $p$, while \code{rec_submesh} should be called by Processor $p$ to receive Submesh $p$ from Processor 0. 
+All of the functions described so far must be run in serial on
+Processor 0. The next step is to start the parallel computation by
+spreading the submeshes over the processors. The communication is
+carried out by \code{send_submesh} and \code{rec_submesh} defined in
+{\tt build_commun.py}. The \code{send_submesh} function should be
+called on Processor 0 and sends the Submesh $p$ to Processor $p$,
+while \code{rec_submesh} should be called by Processor $p$ to
+receive Submesh $p$ from Processor 0.
 
 As an aside, the order of communication is very important. If someone was to modify the \code{send_submesh} routine the corresponding change must be made to the \code{rec_submesh} routine.
@@ -141,9 +175,9 @@
 \section{Some Example Code}
 
-Chapter \ref{chap:code} gives full listings of some example codes. 
+Chapter \ref{chap:code} gives full listings of some example codes.
 
 The first example in Section \ref{subsec:codeRPA} solves the advection equation on a
 rectangular mesh. A rectangular mesh is highly structured so a coordinate based decomposition can be used and the partitioning is simply done by calling the
-routine \code{parallel_rectangle} as shown below. 
+routine \code{parallel_rectangle} as shown below.
 \begin{verbatim}
 #######################
@@ -158,5 +192,5 @@
 \end{verbatim}
 
-Most simulations will not be done on a rectangular mesh, and the approach to subpartitioning the mesh is different to the one described above, however this example may be of interest to those who want to measure the parallel efficiency of the code on their machine. A rectangular mesh should give a good load balance and is therefore an important first test problem.  
+Most simulations will not be done on a rectangular mesh, and the approach to subpartitioning the mesh is different to the one described above, however this example may be of interest to those who want to measure the parallel efficiency of the code on their machine. A rectangular mesh should give a good load balance and is therefore an important first test problem.
 
 
@@ -181,5 +215,5 @@
     # Build the mesh that should be assigned to each processor.
     # This includes ghost nodes and the communication pattern
-    
+
     submesh = build_submesh(nodes, triangles, boundary, quantities, \
                             triangles_per_proc)
@@ -208,5 +242,5 @@
 \newpage
 \begin{itemize}
-\item 
+\item
 These first few lines of code read in and define the (global) mesh. The \code{Set_Stage} function sets the initial conditions. See the code in \ref{subsec:codeRPMM} for the definition of \code{Set_Stage}.
 \begin{verbatim}
@@ -222,11 +256,11 @@
 \end{verbatim}
 
-\item The next step is to build a boundary layer of ghost triangles and define the communication pattern. This step is implemented by \code{build_submesh} as discussed in Section \ref{sec:part2}. The \code{submesh} variable contains a copy of the submesh for each processor. 
-\begin{verbatim}	
+\item The next step is to build a boundary layer of ghost triangles and define the communication pattern. This step is implemented by \code{build_submesh} as discussed in Section \ref{sec:part2}. The \code{submesh} variable contains a copy of the submesh for each processor.
+\begin{verbatim}
     submesh = build_submesh(nodes, triangles, boundary, quantities, \
                             triangles_per_proc)
 \end{verbatim}
 
-\item The actual parallel communication starts when the submesh partitions are sent to the processors by calling \code{send_submesh}. 
+\item The actual parallel communication starts when the submesh partitions are sent to the processors by calling \code{send_submesh}.
 \begin{verbatim}
     for p in range(1, numprocs):
@@ -242,5 +276,5 @@
 \end{verbatim}
 
-Note that the submesh is not received by, or sent to, Processor 0. Rather     \code{hostmesh = extract_hostmesh(submesh)} simply extracts the mesh that has been assigned to Processor 0. Recall \code{submesh} contains the list of submeshes to be assigned to each processor. This is described further in Section \ref{sec:part3}. 
+Note that the submesh is not received by, or sent to, Processor 0. Rather     \code{hostmesh = extract_hostmesh(submesh)} simply extracts the mesh that has been assigned to Processor 0. Recall \code{submesh} contains the list of submeshes to be assigned to each processor. This is described further in Section \ref{sec:part3}.
 %The \code{build_local_mesh} renumbers the nodes
 \begin{verbatim}
Index: inundation/parallel/documentation/report.tex
===================================================================
--- inundation/parallel/documentation/report.tex	(revision 3245)
+++ inundation/parallel/documentation/report.tex	(revision 3315)
@@ -55,7 +55,14 @@
 
 \begin{abstract}
-This document describes work done by the authors as part of a consultancy with GA during 2005-2006. The paper serves as both a report for GA and a user manual.
+This document describes work done by the authors as part of a
+consultancy with GA during 2005-2006. The paper serves as both a
+report for GA and a user manual.
 
-The report contains a description of how the code was parallelised and it lists efficiency results for a few example runs. It also gives  some examples codes showing how to run the Merimbula test problem in parallel and talks about more technical aspects such as compilation issues and batch scripts for submitting compute jobs on parallel machines.
+The report contains a description of how the code was parallelised
+and it lists efficiency results for a few example runs. It also
+gives some examples codes showing how to run the Merimbula test
+problem in parallel and talks about more technical aspects such as
+compilation issues and batch scripts for submitting compute jobs on
+parallel machines.
 \end{abstract}
 
Index: inundation/parallel/documentation/results.tex
===================================================================
--- inundation/parallel/documentation/results.tex	(revision 3245)
+++ inundation/parallel/documentation/results.tex	(revision 3315)
@@ -3,6 +3,10 @@
 
 
-To evaluate the performance of the code on a parallel machine we ran some examples on a cluster of four nodes connected with PathScale InfiniPath HTX. 
-Each node has two AMD Opteron 275 (Dual-core 2.2 GHz Processors) and 4 GB of main memory. The system achieves 60 Gigaflops with the Linpack benchmark, which is about 85\% of peak performance.
+To evaluate the performance of the code on a parallel machine we ran
+some examples on a cluster of four nodes connected with PathScale
+InfiniPath HTX. Each node has two AMD Opteron 275 (Dual-core 2.2 GHz
+Processors) and 4 GB of main memory. The system achieves 60
+Gigaflops with the Linpack benchmark, which is about 85\% of peak
+performance.
 
 For each test run we evaluate the parallel efficiency as
@@ -10,11 +14,20 @@
 E_n = \frac{T_1}{nT_n} 100,
 \]
-where $T_n = \max_{0\le i < n}\{t_i\}$, $n$ is the total number of processors (submesh) and $t_i$ is the time required to run the {\tt evolve} code on processor $i$.  Note that $t_i$ does not include the time required to build and subpartition the mesh etc., it only includes the time required to do the evolve calculations (eg. \code{domain.evolve(yieldstep = 0.1, finaltime = 3.0)}). 
+where $T_n = \max_{0\le i < n}\{t_i\}$, $n$ is the total number of
+processors (submesh) and $t_i$ is the time required to run the {\tt
+evolve} code on processor $i$.  Note that $t_i$ does not include the
+time required to build and subpartition the mesh etc., it only
+includes the time required to do the evolve calculations (eg.
+\code{domain.evolve(yieldstep = 0.1, finaltime = 3.0)}).
 
 \section{Advection, Rectangular Domain}
 
-The first example looked at the rectangular domain example given in Section \ref{subsec:codeRPA}, except we changed the finaltime time to 1.0 (\code{domain.evolve(yieldstep = 0.1, finaltime = 1.0)}).
+The first example looked at the rectangular domain example given in
+Section \ref{subsec:codeRPA}, except we changed the finaltime time
+to 1.0 (\code{domain.evolve(yieldstep = 0.1, finaltime = 1.0)}).
 
-For this particular example we can control the mesh size by changing the parameters \code{N} and \code{M} given in the following section of code taken from
+For this particular example we can control the mesh size by changing
+the parameters \code{N} and \code{M} given in the following section
+of code taken from
  Section \ref{subsec:codeRPA}.
 
@@ -33,10 +46,22 @@
 \end{verbatim}
 
-Tables \ref{tbl:rpa40}, \ref{tbl:rpa80} and \ref{tbl:rpa160} show the efficiency results for different values of \code{N} and \code{M}. The examples where $n \le 4$ were run on one Opteron node containing 4 processors, the $n = 8$ example was run on 2 nodes (giving a total of 8 processors). The communication within a node is faster than the communication across nodes, so we would expect to see a decrease in efficiency when we jump from 4 to 8 nodes. Furthermore, as \code{N} and \code{M} are increased the ratio of exterior to interior triangles decreases, which in-turn decreases the amount of communication relative the amount of computation  and thus the efficiency should increase. 
+Tables \ref{tbl:rpa40}, \ref{tbl:rpa80} and \ref{tbl:rpa160} show
+the efficiency results for different values of \code{N} and
+\code{M}. The examples where $n \le 4$ were run on one Opteron node
+containing 4 processors, the $n = 8$ example was run on 2 nodes
+(giving a total of 8 processors). The communication within a node is
+faster than the communication across nodes, so we would expect to
+see a decrease in efficiency when we jump from 4 to 8 nodes.
+Furthermore, as \code{N} and \code{M} are increased the ratio of
+exterior to interior triangles decreases, which in-turn decreases
+the amount of communication relative the amount of computation  and
+thus the efficiency should increase.
 
 The efficiency results shown here are competitive.
 
-\begin{table} 
-\caption{Parallel Efficiency Results for the Advection Problem on a Rectangular Domain with {\tt N} = 40, {\tt M} = 40.\label{tbl:rpa40}}
+\begin{table}
+\caption{Parallel Efficiency Results for the Advection Problem on a
+Rectangular Domain with {\tt N} = 40, {\tt M} =
+40.\label{tbl:rpa40}}
 \begin{center}
 \begin{tabular}{|c|c c|}\hline
@@ -50,6 +75,8 @@
 \end{table}
 
-\begin{table} 
-\caption{Parallel Efficiency Results for the Advection Problem on a Rectangular Domain with {\tt N} = 80, {\tt M} = 80.\label{tbl:rpa80}}
+\begin{table}
+\caption{Parallel Efficiency Results for the Advection Problem on a
+Rectangular Domain with {\tt N} = 80, {\tt M} =
+80.\label{tbl:rpa80}}
 \begin{center}
 \begin{tabular}{|c|c c|}\hline
@@ -64,6 +91,8 @@
 
 
-\begin{table} 
-\caption{Parallel Efficiency Results for the Advection Problem on a Rectangular Domain with {\tt N} = 160, {\tt M} = 160.\label{tbl:rpa160}}
+\begin{table}
+\caption{Parallel Efficiency Results for the Advection Problem on a
+Rectangular Domain with {\tt N} = 160, {\tt M} =
+160.\label{tbl:rpa160}}
 \begin{center}
 \begin{tabular}{|c|c c|}\hline
@@ -79,17 +108,18 @@
 
 %Another way of measuring the performance of the code on a parallel machine is to increase the problem size as the number of processors are increased so that the number of triangles per processor remains roughly the same.  We have not carried out measurements of this kind as we usually have static grids and it is not possible to increase the number of triangles.
-  
+
 \section{Advection, Merimbula Mesh}
 
-We now look at another advection example, except this time the mesh comes from the Merimbula test problem. That is, we ran the code given in Section
-\ref{subsec:codeRPMM}, except the final time was reduced to 10000
-(\code{finaltime = 10000}). The results are given in Table \ref{tbl:rpm}.
-These are good efficiency results, especially considering the structure of the
-Merimbula mesh. 
+We now look at another advection example, except this time the mesh
+comes from the Merimbula test problem. That is, we ran the code
+given in Section \ref{subsec:codeRPMM}, except the final time was
+reduced to 10000 (\code{finaltime = 10000}). The results are given
+in Table \ref{tbl:rpm}. These are good efficiency results,
+especially considering the structure of the Merimbula mesh.
 %Note that since we are solving an advection problem the amount of calculation
 %done on each triangle is relatively low, when we more to other problems that
 %involve more calculations we would expect the computation to communication ratio to increase and thus get an increase in efficiency.
 
-\begin{table} 
+\begin{table}
 \caption{Parallel Efficiency Results for the Advection Problem on the
   Merimbula Mesh.\label{tbl:rpm}}
@@ -120,5 +150,5 @@
 Processor 0 spent about 3.8 times more doing the \code{update_boundary}
 calculations than Processor 7. This load imbalance reduced the parallel
-efficiency. 
+efficiency.
 
 Before doing the shallow equation calculations on a larger number of
@@ -126,6 +156,6 @@
 optimised as much as possible to reduce the effect of the load imbalance.
 
- 
-\begin{table} 
+
+\begin{table}
 \caption{Parallel Efficiency Results for the Shallow Water Equation on the
   Merimbula Mesh.\label{tbl:rpsm}}
