Bandwidth testing: Difference between revisions

From genomewiki
Jump to navigationJump to search
(more altair stats)
(fix kolossus udp numbers)
 
(6 intermediate revisions by the same user not shown)
Line 16: Line 16:


<TABLE BORDER=1><TR><TH>machine<BR>name</TH><TH>processor<BR>speed</TH><TH>memSize</TH><TH>10 files<BR>of size</TH><TH>dd /dev/zero<BR>to /dev/shm</TH><TH>cp /dev/shm<BR>to /tmp</TH><TH>cp /dev/shm<BR>to hive</TH><TH>rsync /dev/shm<BR>to /dev/shm</TH><TH>rsync /dev/shm<BR>via rsh</TH><TH>UDT via<BR>UPD</TH></TR>
<TABLE BORDER=1><TR><TH>machine<BR>name</TH><TH>processor<BR>speed</TH><TH>memSize</TH><TH>10 files<BR>of size</TH><TH>dd /dev/zero<BR>to /dev/shm</TH><TH>cp /dev/shm<BR>to /tmp</TH><TH>cp /dev/shm<BR>to hive</TH><TH>rsync /dev/shm<BR>to /dev/shm</TH><TH>rsync /dev/shm<BR>via rsh</TH><TH>UDT via<BR>UPD</TH></TR>
<TR><TH ALIGN=LEFT>hgwdev</TH><TD>Opteron<BR>1150 Mhz</TD><TD>32 Gb</TD><TD>1 gB</TD><TD>5278 mB/s<BR>10X dd</TD><TD>480 mB/s<BR>2X cp</TD><TD>58 mB/s</TD><TD>1638 mB/s<BR>20X rsync</TD><TD>n/a</TD><TD>4070 mB/s<BR>10X send</TD></TR>
<TR><TH ALIGN=LEFT>hgwdev</TH><TD>Opteron<BR>1150 Mhz</TD><TD>32 Gb</TD><TD>1 gB</TD><TD>5278 mB/s<BR>10X dd</TD><TD>480 mB/s<BR>2X cp</TD><TD>58 mB/s</TD><TD>1638 mB/s<BR>20X rsync</TD><TD>n/a</TD><TD>400 mB/s<BR>10X send</TD></TR>
<TR><TH ALIGN=LEFT>kolossus</TH><TD>Intel<BR>2261 Mhz</TD><TD>1 Tb</TD><TD>1 gB</TD><TD>6700 mB/s<BR>10X dd</TD><TD>600 mB/s<BR>2X cp</TD><TD>58 mB/s</TD><TD>120 mB/s<BR>4X rsync</TD><TD>115 mB/s</TD><TD>1135 mB/s</TD></TR>
<TR><TH ALIGN=LEFT>kolossus</TH><TD>Intel<BR>2261 Mhz</TD><TD>1 Tb</TD><TD>1 gB</TD><TD>6700 mB/s<BR>10X dd</TD><TD>600 mB/s<BR>2X cp</TD><TD>58 mB/s</TD><TD>120 mB/s<BR>4X rsync</TD><TD>115 mB/s</TD><TD>116 mB/s<BR>6X send</TD></TR>
<TR><TH ALIGN=LEFT>okazaki</TH><TD>Intel<BR>1600 Mhz</TD><TD>2 Gb</TD><TD>75 mB</TD><TD>700 mB/s</TD><TD>540 mB/s<BR>6X cp</TD><TD>11.3 mB/s</TD><TD>11.5 mB/s</TD><TD>11.6 mB/s</TD><TD>10.7 mB/s<BR>4X send</TD></TR>
<TR><TH ALIGN=LEFT>okazaki</TH><TD>Intel<BR>1600 Mhz</TD><TD>2 Gb</TD><TD>75 mB</TD><TD>700 mB/s</TD><TD>540 mB/s<BR>6X cp</TD><TD>11.3 mB/s</TD><TD>11.5 mB/s</TD><TD>11.6 mB/s</TD><TD>10.7 mB/s<BR>4X send</TD></TR>
<TR><TH ALIGN=LEFT>mitzie</TH><TD>Intel<BR>400 Mhz</TD><TD>512 Kb</TD><TD>18 mB</TD><TD>113 mB/s</TD><TD>25 mB/s</TD><TD>n/a</TD><TD>54.4 kB/s up<BR>316 kB/s down</TD><TD>n/a</TD></TR>
<TR><TH ALIGN=LEFT>mitzie</TH><TD>Intel<BR>400 Mhz</TD><TD>512 Kb</TD><TD>18 mB</TD><TD>113 mB/s</TD><TD>25 mB/s</TD><TD>n/a</TD><TD>54.4 kB/s up<BR>316 kB/s down</TD><TD>n/a</TD></TR>
<TR><TH ALIGN=LEFT>cloud0</TH><TD>Opteron<BR>2000 Mhz</TD><TD>7.5 Gb</TD><TD>288 mB</TD><TD>940 mB/s<BR>4X dd</TD><TD>78 mB/s</TD><TD>n/a</TD><TD>16 mB/s<BR>2X rsync</TD><TD>43 mB/s<BR>3X rsync</TD><TD>46 mB/s<BR>3X send</TD></TR>
<TR><TH ALIGN=LEFT>cloud0</TH><TD>Opteron<BR>2000 Mhz</TD><TD>7.5 Gb</TD><TD>288 mB</TD><TD>940 mB/s<BR>4X dd</TD><TD>78 mB/s</TD><TD>n/a</TD><TD>16 mB/s<BR>2X rsync</TD><TD>43 mB/s<BR>3X rsync</TD><TD>46 mB/s<BR>3X send</TD></TR>
<TR><TH ALIGN=LEFT>altair</TH><TD>Opteron<BR>2300 Mhz</TD><TD>32 Gb</TD><TD>1 gB</TD><TD>5554 mB/s<BR>8X dd</TD><TD>600 mB/s<BR>2X cp</TD><TD>100 mB/s<BR>2X cp</TD><TD>pending</TD><TD>n/a</TD><TD>tbd</TD></TR>
<TR><TH ALIGN=LEFT>altair</TH><TD>Opteron<BR>2300 Mhz</TD><TD>32 Gb</TD><TD>1 gB</TD><TD>5554 mB/s<BR>8X dd</TD><TD>600 mB/s<BR>2X cp</TD><TD>100 mB/s<BR>2X cp</TD><TD>200 to 1000 kB/s<BR>big variation</TD><TD>n/a</TD><TD>9 mB/s<BR>4X send</TD></TR>
</TABLE>
</TABLE>


Line 42: Line 42:
multiple machines at the same time to the same hive directory did '''not interfere'''
multiple machines at the same time to the same hive directory did '''not interfere'''
with each other.  They were independent of the other activity.
with each other.  They were independent of the other activity.
The altair /dev/shm to hive filesystem copy was performed to the /genome/ filesystem.


The rsync /dev/shm to /dev/shm was to and from hgwdev.  Multiple instances of rsync
The rsync /dev/shm to /dev/shm was to and from hgwdev.  Multiple instances of rsync
Line 48: Line 50:
The machine mitzie is at the end of a Santa Cruz DSL line and over a WiFi
The machine mitzie is at the end of a Santa Cruz DSL line and over a WiFi
network connection across the driveway to the garage.  The WiFi link is faster than the DSL line.
network connection across the driveway to the garage.  The WiFi link is faster than the DSL line.
The UDT via UPD transfers are with the udp transport software from: [http://udt.sourceforge.net/ UDT].

Latest revision as of 23:56, 21 May 2010

Units

For these measurements:

  • 1 kB/s = 1,000 bytes per second
  • 1 mB/s = 1,000,000 bytes per second
  • 1 gB/s = 1,000,000,000 bytes per second

Memory:

  • 1 Kb = 1024^2 bytes = 1,048,576 bytes
  • 1 Gb = 1024^3 bytes = 1,073,741,824 bytes
  • 1 Tb = 1024^4 bytes = 1,099,511,627,776 bytes

Machines tested

machine
name
processor
speed
memSize10 files
of size
dd /dev/zero
to /dev/shm
cp /dev/shm
to /tmp
cp /dev/shm
to hive
rsync /dev/shm
to /dev/shm
rsync /dev/shm
via rsh
UDT via
UPD
hgwdevOpteron
1150 Mhz
32 Gb1 gB5278 mB/s
10X dd
480 mB/s
2X cp
58 mB/s1638 mB/s
20X rsync
n/a400 mB/s
10X send
kolossusIntel
2261 Mhz
1 Tb1 gB6700 mB/s
10X dd
600 mB/s
2X cp
58 mB/s120 mB/s
4X rsync
115 mB/s116 mB/s
6X send
okazakiIntel
1600 Mhz
2 Gb75 mB700 mB/s540 mB/s
6X cp
11.3 mB/s11.5 mB/s11.6 mB/s10.7 mB/s
4X send
mitzieIntel
400 Mhz
512 Kb18 mB113 mB/s25 mB/sn/a54.4 kB/s up
316 kB/s down
n/a
cloud0Opteron
2000 Mhz
7.5 Gb288 mB940 mB/s
4X dd
78 mB/sn/a16 mB/s
2X rsync
43 mB/s
3X rsync
46 mB/s
3X send
altairOpteron
2300 Mhz
32 Gb1 gB5554 mB/s
8X dd
600 mB/s
2X cp
100 mB/s
2X cp
200 to 1000 kB/s
big variation
n/a9 mB/s
4X send

Procedure

Sizing files based on available /dev/shm space. Largest file size limited to 1 gB.

The /dev/zero to /dev/shm write is done a dd command:

 dd if=/dev/zero of=/dev/shm/memToMem.$I.zero bs=200000 count=$ddCount

Running multiple copies of that simultaneously until the best speed was obtained.

The /dev/shm copy to /tmp is done the cp command:

cp -p /dev/shm/memToMem.$I.zero ./memToFile.$I.zero

Running multiple copies of that simultaneously until the best speed was obtained. It only took several (2 to 3) multiple instances before they began to interfere with each other and slow down the effective rate.

The /dev/shm copy to the hive filesystem was done with the cp command:

cp -p /dev/shm/memToMem.$I.zero ./memToFile.$I.zero

Running multiple copies of that simultaneously did not improve the speed on the machine where the copies were running. However, running copies from multiple machines at the same time to the same hive directory did not interfere with each other. They were independent of the other activity.

The altair /dev/shm to hive filesystem copy was performed to the /genome/ filesystem.

The rsync /dev/shm to /dev/shm was to and from hgwdev. Multiple instances of rsync from kolossus to hgwdev seemed to add together for a total bandwidth higher than a single instance of rsync.

The machine mitzie is at the end of a Santa Cruz DSL line and over a WiFi network connection across the driveway to the garage. The WiFi link is faster than the DSL line.

The UDT via UPD transfers are with the udp transport software from: UDT.