1NetPIPE Network Protocol Independent Performance Evaluator, Release 2.4 2Copyright 1997, 1998, 1999 Iowa State University Research Foundation, Inc. 3 4$Id: README,v 1.1 2003/02/05 15:44:54 robbiew Exp $ 5 6This program is free software; you can redistribute it and/or modify 7it under the terms of the GNU General Public License as published by 8the Free Software Foundation. You should have received a copy of the 9GNU General Public License along with this program; if not, write to the 10Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. 11 12The URL for this document: 13 14ftp://ftp.scl.ameslab.gov/pub/netpipe/README 15 16Getting NetPIPE 17--------------- 18 19The NetPIPE implementation in C can be found at: 20 21ftp://ftp.scl.ameslab.gov/pub/netpipe/netpipe-2.4.tar.gz 22 23The source code for NetPIPE 2.4 is provided as a gzipped tar archive, 24which can be uncompressed with "gunzip netpipe-2.4.tar.gz" (or "gzip 25-d netpipe-2.4.tar.gz"), and then extracted from the uncompressed 26archive with the command "tar xvf netpipe-2.4.tar". If you do not 27have the gzip program, it can be obtained as: 28 29ftp://prep.ai.mit.edu/pub/gnu/gzip-1.2.4.tar 30 31Similarly, the NetPIPE implementation in Java can be found at: 32 33ftp://ftp.scl.ameslab.gov/pub/netpipe-Java-1.0.tar.gz 34 35The instructions that follow apply to the C implementation of 36NetPIPE. 37 38What is NetPIPE? 39---------------- 40 41NetPIPE is a protocol independent performance tool that encapsulates 42the best of ttcp and netperf and visually represents the network 43performance under a variety of conditions. By taking the end-to-end 44application view of a network, NetPIPE clearly shows the overhead 45associated with different protocol layers. Netpipe answers such 46questions as: how soon will a given data block of size k arrive at its 47destination? Which network and protocol will transmit size k blocks 48the fastest? What is a given network's effective maximum throughput 49and saturation level? Does there exist a block size k for which the 50throughput is maximized? How much communication overhead is due to the 51network communication protocol layer(s)? How quickly will a small (< 1 52kbyte) control message arrive, and which network and protocol are best 53for this purpose? 54 55For a paper fully describing NetPIPE and sample investigation of 56network performance issues using NetPIPE, see 57http://www.scl.ameslab.gov/netpipe/paper/full.html. 58 59Building NetPIPE 60---------------- 61 62NetPIPE is provided with protocol-specific shims for TCP (using the 63Berkeley sockets interface), MPI, and PVM. If you do not have MPI or 64PVM, don't worry; TCP is the typical shim used. It should be easy to 65write new interfaces for other protocols based on the examples shown 66by the TCP, MPI and PVM interfaces. 67 68NetPIPE requires an ANSI C compiler. 69 70Review the provided Makefile and change any necessary settings, such 71as the CFLAGS compiler flags, required extra libraries, and MPI or PVM 72library & include file pathnames if you have these communication 73libraries. If you want to turn on getrusage calls to get CPU time 74required for communication, add "-DHAVE_GETRUSAGE" to the CFLAGS line 75in the Makefile. 76 77Compile NetPIPE with the desired communication interface by using the 78command "make TCP", "make MPI", or "make PVM" as appropriate, 79corresponding to the executable files NPtcp, NPmpi, or NPpvm 80respectively. 81 82Consult the appropriate section below for details on running NetPIPE 83over TCP, MPI, or PVM, and the following section on interpreting the 84results. 85 86Running NPtcp 87------------- 88 89For TCP, run a NetPIPE receiver on one computer by issuing the command 90"NPtcp -r". Run a NetPIPE sender on another computer by issuing the 91command "NPtcp -t -h <receiver's address> -o <output file> -P" and any 92other options as appropriate (each option affects only the process on 93which it is specified -- options are not negotiated between the 94transmitter and the receiver): 95 96 -A: specify buffers alignment e.g. "-A 4096" 97 98 -a: asynchronous receive (a.k.a. preposted receive) 99 This option currently has no effect on TCP 100 101 -b: specify send and receive TCP buffer sizes e.g. "-b 32768" 102 103 -h: specify hostname of receiver e.g. "-h mumblehost" 104 105 -i: specify increment step size e.g. "-i 64" 106 Default is exponential increment calculated at runtime 107 108 -l: lower bound (start value for block size) e.g. "-l 1" 109 110 -O: specify buffer offset e.g. "-O 127" 111 112 -o: specify output filename e.g. "-o output.txt" 113 114 -P: print real-time results on stdout 115 116 -p: specify port e.g. "-p 5150" 117 118 -s: stream option (default mode is "ping pong") 119 If this option is used, it must be specified on both 120 the sending and receiving processes 121 122 -u: upper bound (stop value for block size) e.g. "-u 1048576" 123 124Running NPmpi 125------------- 126 127For MPI, how you run NPmpi may depend on the MPI implementation you 128are using. Assuming you are using the "p4" device (for a cluster of 129individual systems interconnected using TCP/IP) in the Argonne MPICH 130implementation, you could run NPmpi one of two ways. 131 132 If your system's default machine file begins with the two 133 names of the systems you want to test, use "mpirun -np 2 134 NPmpi", followed by any of the NetPIPE options listed below. 135 136 Otherwise, create a file that contains the host names of the 137 two systems you want to include in the test, one host name on 138 each line (assume the file is named "machines.p4"). Then, use 139 the command "mpirun -machinefile machines.p4 -np 2 NPmpi", 140 followed by any of the NetPIPE options listed below. 141 142To find out how to run NPmpi using any other implementation of MPI, 143please consult the implementation's documentation. 144 145The NetPIPE options for MPI are: 146 147 -A: specify buffers alignment e.g. "-A 4096" 148 149 -a: asynchronous receive (a.k.a. preposted receive) 150 May not have any effect, depending on your MPI 151 implementation 152 153 -i: specify increment step size e.g. "-i 64" 154 Default is exponential increment calculated at runtime 155 156 -l: lower bound (start value for block size) e.g. "-l 1" 157 158 -O: specify buffer offset e.g. "-O 127" 159 160 -o: specify output filename e.g. "-o output.txt" 161 162 -P: print real-time results on stdout 163 164 -s: stream option (default mode is "ping pong") 165 If this option is used, it must be specified on both 166 the sending and receiving processes 167 168 -u: upper bound (stop value for block size) e.g. "-u 1048576" 169 170Running NPpvm 171------------- 172 173First, start PVM with the command "pvm" on one machine and a second 174machine with the PVM command "add <othermachine>", where 175<othermachine> is the name of the other computer to include in the 176test. Exit the PVM command line interface. Start the receiver 177process on one of the machines with the command "NPpvm -r". Finally, 178start the transmitter process on the other machine with the command 179"NPpvm -t -o <output file> -P" and any other options as appropriate 180(each option affects only the process on which it is specified -- 181options are not negotiated between the transmitter and the receiver): 182 183 -A: specify buffers alignment e.g. "-A 4096" 184 185 -a: asynchronous receive (a.k.a. preposted receive) 186 This option has no effect on PVM 187 188 -i: specify increment step size e.g. "-i 64" 189 Default is exponential increment calculated at runtime 190 191 -l: lower bound (start value for block size) e.g. "-l 1" 192 193 -O: specify buffer offset e.g. "-O 127" 194 195 -o: specify output filename e.g. "-o output.txt" 196 197 -P: print real-time results on stdout 198 199 -s: stream option (default mode is "ping pong") 200 If this option is used, it must be specified on both 201 the sending and receiving processes 202 203 -u: upper bound (stop value for block size) e.g. "-u 1048576" 204 205 206Interpreting the Results 207------------------------ 208 209NetPIPE's output file contains five columns: time to transfer the block, 210bits per second, bits in block, bytes in block, and variance. These 211columns may be graphed to represent and compare the network's 212performance. For example, the "network signature" graph can be 213created by graphing the time column versus the bits per second column 214(see the NetPIPE report at the URL above for the details why this 215graph is important and how to interpret it). The more traditional 216"throughput versus block size" graph can be created by 217graphing the bytes column versus the bits per second column. 218 219See http://www.scl.ameslab.gov/Projects/ClusterCookbook/nprun.html for 220a detailed tutorial on running NetPIPE and graphing the results. 221 222Help 223---- 224 225NetPIPE is currently maintained by Guy Helmer. Email 226"ghelmer@scl.ameslab.gov" or call 515-294-9469 for help or 227suggestions. 228 229Changes 230------- 231 232version 2.4 (12/16/99) 233 * Add getrusage calls to get CPU time used by communication if 234 HAVE_GETRUSAGE is defined (be aware that no studies have been 235 conducted to test the accuracy of results across different systems) 236 * Use "unsigned int" instead of "unsigned long" to communicate 32-bit 237 integers in TCP.c (this solves interoperability problems between 238 Compaq/DEC Alphas and most other systems) 239 * Add dummy "echo" commands after TCP, MPI, and PVM targets in the 240 Makefile. Some implementations of make(1) (such as those found 241 in Linux distributions) interpret the targets with no following 242 statements as a rule to do something silly like 243 "cc -O -o TCP TCP.c" after the dependency is satisfied. 244 245version 2.3 (9/24/98) 246 * Add PVM interface contributed by Clark E. Dorman <dorman@s3i.com> 247 248 * Revamp README file with instructions for NPmpi and NPpvm, and 249 clarify some instructions for NPtcp 250 251version 2.2 (8/21/98): 252 * Carefully check all return values from write(2) and read(2) 253 system calls in TCP.c. Handle short reads properly. Make the Sync() 254 function transmit and receive a useful string which can be 255 checked for validity. 256 257 * Correct the overloading of SendTime() and RecvTime() functions 258 by breaking out SendRepeat() and RecvRepeat() as separate 259 functions. 260 261 * Handle systems whose accept(2) system call does not carry socket 262 options over from the listening socket. In particular, set the 263 TCP_NODELAY flag and socket buffers on an accepted socket. 264