Bandwidth testing: Difference between revisions

From genomewiki
Jump to navigationJump to search
m (more udp results)
(fix kolossus udp numbers)
 
(16 intermediate revisions by the same user not shown)
Line 6: Line 6:
* 1 mB/s = 1,000,000 bytes per second
* 1 mB/s = 1,000,000 bytes per second
* 1 gB/s = 1,000,000,000 bytes per second
* 1 gB/s = 1,000,000,000 bytes per second
Memory:
* 1 Kb = 1024^2 bytes = 1,048,576 bytes
* 1 Gb = 1024^3 bytes = 1,073,741,824 bytes
* 1 Tb = 1024^4 bytes = 1,099,511,627,776 bytes


==Machines tested==
==Machines tested==


<TABLE BORDER=1><TR><TH>machine<BR>name</TH><TH>processor<BR>speed</TH><TH>memSize</TH><TH>10 files<BR>of size</TH><TH>dd /dev/zero<BR>to /dev/shm</TH><TH>cp /dev/shm<BR>to /tmp</TH><TH>cp /dev/shm<BR>to hive</TH><TH>rsync /dev/shm<BR>to /dev/shm</TH><TH>rsync /dev/shm<BR>via rsh</TH><TH>UDT via<BR>UPD</TH></TR>
<TABLE BORDER=1><TR><TH>machine<BR>name</TH><TH>processor<BR>speed</TH><TH>memSize</TH><TH>10 files<BR>of size</TH><TH>dd /dev/zero<BR>to /dev/shm</TH><TH>cp /dev/shm<BR>to /tmp</TH><TH>cp /dev/shm<BR>to hive</TH><TH>rsync /dev/shm<BR>to /dev/shm</TH><TH>rsync /dev/shm<BR>via rsh</TH><TH>UDT via<BR>UPD</TH></TR>
<TR><TH ALIGN=LEFT>hgwdev</TH><TD>Opteron<BR>1150 Mhz</TD><TD>32 Gb</TD><TD>1 gB</TD><TD>800 mB/s</TD><TD>287 mB/s</TD><TD>54 mB/s</TD><TD>1638 mB/s<BR>20X rsync</TD><TD>n/a</TD><TD>4070 MB/s<BR>10X send</TD></TR>
<TR><TH ALIGN=LEFT>hgwdev</TH><TD>Opteron<BR>1150 Mhz</TD><TD>32 Gb</TD><TD>1 gB</TD><TD>5278 mB/s<BR>10X dd</TD><TD>480 mB/s<BR>2X cp</TD><TD>58 mB/s</TD><TD>1638 mB/s<BR>20X rsync</TD><TD>n/a</TD><TD>400 mB/s<BR>10X send</TD></TR>
<TR><TH ALIGN=LEFT>kolossus</TH><TD>Intel<BR>2261 Mhz</TD><TD>1 Tb</TD><TD>1 gB</TD><TD>1980 mB/s</TD><TD>192 mB/s</TD><TD>41 mB/s</TD><TD>120 mB/s<BR>4X rsync</TD><TD>115 mB/s</TD><TD>1135 mB/s</TD></TR>
<TR><TH ALIGN=LEFT>kolossus</TH><TD>Intel<BR>2261 Mhz</TD><TD>1 Tb</TD><TD>1 gB</TD><TD>6700 mB/s<BR>10X dd</TD><TD>600 mB/s<BR>2X cp</TD><TD>58 mB/s</TD><TD>120 mB/s<BR>4X rsync</TD><TD>115 mB/s</TD><TD>116 mB/s<BR>6X send</TD></TR>
<TR><TH ALIGN=LEFT>okazaki</TH><TD>Intel<BR>1600 Mhz</TD><TD>2 Gb</TD><TD>75 mB</TD><TD>700 mB/s</TD><TD>90 mB/s</TD><TD>11.3 mB/s</TD><TD>11.5 mB/s</TD><TD>11.6 mB/s</TD><TD>10.7 mB/s<BR>4X send</TD></TR>
<TR><TH ALIGN=LEFT>okazaki</TH><TD>Intel<BR>1600 Mhz</TD><TD>2 Gb</TD><TD>75 mB</TD><TD>700 mB/s</TD><TD>540 mB/s<BR>6X cp</TD><TD>11.3 mB/s</TD><TD>11.5 mB/s</TD><TD>11.6 mB/s</TD><TD>10.7 mB/s<BR>4X send</TD></TR>
<TR><TH ALIGN=LEFT>mitzie</TH><TD>Intel<BR>400 Mhz</TD><TD>512 Kb</TD><TD>18 mB</TD><TD>113 mB/s</TD><TD>25 mB/s</TD><TD>n/a</TD><TD> 37 kB/s up<BR>down</TD><TD>n/a</TD></TR>
<TR><TH ALIGN=LEFT>mitzie</TH><TD>Intel<BR>400 Mhz</TD><TD>512 Kb</TD><TD>18 mB</TD><TD>113 mB/s</TD><TD>25 mB/s</TD><TD>n/a</TD><TD>54.4 kB/s up<BR>316 kB/s down</TD><TD>n/a</TD></TR>
<TR><TH ALIGN=LEFT>cloud0</TH><TD>Opteron<BR>2000 Mhz</TD><TD>7.5 Gb</TD><TD>288 mB</TD><TD>940 mB/s<BR>4X dd</TD><TD>78 mB/s</TD><TD>n/a</TD><TD>16 mB/s<BR>2X rsync</TD><TD>43 mB/s<BR>3X rsync</TD><TD>46 mB/s<BR>3X send</TD></TR>
<TR><TH ALIGN=LEFT>altair</TH><TD>Opteron<BR>2300 Mhz</TD><TD>32 Gb</TD><TD>1 gB</TD><TD>5554 mB/s<BR>8X dd</TD><TD>600 mB/s<BR>2X cp</TD><TD>100 mB/s<BR>2X cp</TD><TD>200 to 1000 kB/s<BR>big variation</TD><TD>n/a</TD><TD>9 mB/s<BR>4X send</TD></TR>
</TABLE>
</TABLE>


Line 20: Line 28:
Sizing files based on available /dev/shm space.  Largest file size limited to 1 gB.
Sizing files based on available /dev/shm space.  Largest file size limited to 1 gB.


The /dev/zero to /dev/shm write is done ten times with a dd command:
The /dev/zero to /dev/shm write is done a dd command:
   dd if=/dev/zero of=/dev/shm/memToMem.$I.zero bs=200000 count=$ddCount
   dd if=/dev/zero of=/dev/shm/memToMem.$I.zero bs=200000 count=$ddCount
Running multiple copies of that simultaneously until the best speed was obtained.


The /dev/shm copy to /tmp is done ten times with the cp command:
The /dev/shm copy to /tmp is done the cp command:
  cp -p /dev/shm/memToMem.$I.zero ./memToFile.$I.zero
  cp -p /dev/shm/memToMem.$I.zero ./memToFile.$I.zero
Running multiple copies of that simultaneously until the best speed was obtained.  It only took several (2 to 3) multiple instances before they began to interfere with each other and slow down the effective rate.


These 10X copies were repeated three timesThe slowest measurement of the three times was recorded.
The /dev/shm copy to the hive filesystem was done with the cp command:
cp -p /dev/shm/memToMem.$I.zero ./memToFile.$I.zero
Running multiple copies of that simultaneously did '''not improve''' the speed on
the machine where the copies were running.  '''However''', running copies from
multiple machines at the same time to the same hive directory did '''not interfere'''
with each otherThey were independent of the other activity.
 
The altair /dev/shm to hive filesystem copy was performed to the /genome/ filesystem.


The rsync /dev/shm to /dev/shm was to and from hgwdev.  Multiple instances of rsync
The rsync /dev/shm to /dev/shm was to and from hgwdev.  Multiple instances of rsync
Line 32: Line 49:


The machine mitzie is at the end of a Santa Cruz DSL line and over a WiFi
The machine mitzie is at the end of a Santa Cruz DSL line and over a WiFi
network connection across the driveway to the garage.
network connection across the driveway to the garage.  The WiFi link is faster than the DSL line.
 
The UDT via UPD transfers are with the udp transport software from: [http://udt.sourceforge.net/ UDT].

Latest revision as of 23:56, 21 May 2010

Units

For these measurements:

  • 1 kB/s = 1,000 bytes per second
  • 1 mB/s = 1,000,000 bytes per second
  • 1 gB/s = 1,000,000,000 bytes per second

Memory:

  • 1 Kb = 1024^2 bytes = 1,048,576 bytes
  • 1 Gb = 1024^3 bytes = 1,073,741,824 bytes
  • 1 Tb = 1024^4 bytes = 1,099,511,627,776 bytes

Machines tested

machine
name
processor
speed
memSize10 files
of size
dd /dev/zero
to /dev/shm
cp /dev/shm
to /tmp
cp /dev/shm
to hive
rsync /dev/shm
to /dev/shm
rsync /dev/shm
via rsh
UDT via
UPD
hgwdevOpteron
1150 Mhz
32 Gb1 gB5278 mB/s
10X dd
480 mB/s
2X cp
58 mB/s1638 mB/s
20X rsync
n/a400 mB/s
10X send
kolossusIntel
2261 Mhz
1 Tb1 gB6700 mB/s
10X dd
600 mB/s
2X cp
58 mB/s120 mB/s
4X rsync
115 mB/s116 mB/s
6X send
okazakiIntel
1600 Mhz
2 Gb75 mB700 mB/s540 mB/s
6X cp
11.3 mB/s11.5 mB/s11.6 mB/s10.7 mB/s
4X send
mitzieIntel
400 Mhz
512 Kb18 mB113 mB/s25 mB/sn/a54.4 kB/s up
316 kB/s down
n/a
cloud0Opteron
2000 Mhz
7.5 Gb288 mB940 mB/s
4X dd
78 mB/sn/a16 mB/s
2X rsync
43 mB/s
3X rsync
46 mB/s
3X send
altairOpteron
2300 Mhz
32 Gb1 gB5554 mB/s
8X dd
600 mB/s
2X cp
100 mB/s
2X cp
200 to 1000 kB/s
big variation
n/a9 mB/s
4X send

Procedure

Sizing files based on available /dev/shm space. Largest file size limited to 1 gB.

The /dev/zero to /dev/shm write is done a dd command:

 dd if=/dev/zero of=/dev/shm/memToMem.$I.zero bs=200000 count=$ddCount

Running multiple copies of that simultaneously until the best speed was obtained.

The /dev/shm copy to /tmp is done the cp command:

cp -p /dev/shm/memToMem.$I.zero ./memToFile.$I.zero

Running multiple copies of that simultaneously until the best speed was obtained. It only took several (2 to 3) multiple instances before they began to interfere with each other and slow down the effective rate.

The /dev/shm copy to the hive filesystem was done with the cp command:

cp -p /dev/shm/memToMem.$I.zero ./memToFile.$I.zero

Running multiple copies of that simultaneously did not improve the speed on the machine where the copies were running. However, running copies from multiple machines at the same time to the same hive directory did not interfere with each other. They were independent of the other activity.

The altair /dev/shm to hive filesystem copy was performed to the /genome/ filesystem.

The rsync /dev/shm to /dev/shm was to and from hgwdev. Multiple instances of rsync from kolossus to hgwdev seemed to add together for a total bandwidth higher than a single instance of rsync.

The machine mitzie is at the end of a Santa Cruz DSL line and over a WiFi network connection across the driveway to the garage. The WiFi link is faster than the DSL line.

The UDT via UPD transfers are with the udp transport software from: UDT.