Bandwidth testing
Units
For these measurements:
- 1 kB/s = 1,000 bytes per second
- 1 mB/s = 1,000,000 bytes per second
- 1 gB/s = 1,000,000,000 bytes per second
Memory:
- 1 Kb = 1024^2 bytes = 1,048,576 bytes
- 1 Gb = 1024^3 bytes = 1,073,741,824 bytes
- 1 Tb = 1024^4 bytes = 1,099,511,627,776 bytes
Machines tested
| machine name | processor speed | memSize | 10 files of size | dd /dev/zero to /dev/shm | cp /dev/shm to /tmp | cp /dev/shm to hive | rsync /dev/shm to /dev/shm | rsync /dev/shm via rsh | UDT via UPD |
|---|---|---|---|---|---|---|---|---|---|
| hgwdev | Opteron 1150 Mhz | 32 Gb | 1 gB | 5278 mB/s 10X dd | 480 mB/s 2X cp | 58 mB/s | 1638 mB/s 20X rsync | n/a | 400 mB/s 10X send |
| kolossus | Intel 2261 Mhz | 1 Tb | 1 gB | 6700 mB/s 10X dd | 600 mB/s 2X cp | 58 mB/s | 120 mB/s 4X rsync | 115 mB/s | 116 mB/s 6X send |
| okazaki | Intel 1600 Mhz | 2 Gb | 75 mB | 700 mB/s | 540 mB/s 6X cp | 11.3 mB/s | 11.5 mB/s | 11.6 mB/s | 10.7 mB/s 4X send |
| mitzie | Intel 400 Mhz | 512 Kb | 18 mB | 113 mB/s | 25 mB/s | n/a | 54.4 kB/s up 316 kB/s down | n/a | |
| cloud0 | Opteron 2000 Mhz | 7.5 Gb | 288 mB | 940 mB/s 4X dd | 78 mB/s | n/a | 16 mB/s 2X rsync | 43 mB/s 3X rsync | 46 mB/s 3X send |
| altair | Opteron 2300 Mhz | 32 Gb | 1 gB | 5554 mB/s 8X dd | 600 mB/s 2X cp | 100 mB/s 2X cp | 200 to 1000 kB/s big variation | n/a | 9 mB/s 4X send |
Procedure
Sizing files based on available /dev/shm space. Largest file size limited to 1 gB.
The /dev/zero to /dev/shm write is done a dd command:
dd if=/dev/zero of=/dev/shm/memToMem.$I.zero bs=200000 count=$ddCount
Running multiple copies of that simultaneously until the best speed was obtained.
The /dev/shm copy to /tmp is done the cp command:
cp -p /dev/shm/memToMem.$I.zero ./memToFile.$I.zero
Running multiple copies of that simultaneously until the best speed was obtained. It only took several (2 to 3) multiple instances before they began to interfere with each other and slow down the effective rate.
The /dev/shm copy to the hive filesystem was done with the cp command:
cp -p /dev/shm/memToMem.$I.zero ./memToFile.$I.zero
Running multiple copies of that simultaneously did not improve the speed on the machine where the copies were running. However, running copies from multiple machines at the same time to the same hive directory did not interfere with each other. They were independent of the other activity.
The altair /dev/shm to hive filesystem copy was performed to the /genome/ filesystem.
The rsync /dev/shm to /dev/shm was to and from hgwdev. Multiple instances of rsync from kolossus to hgwdev seemed to add together for a total bandwidth higher than a single instance of rsync.
The machine mitzie is at the end of a Santa Cruz DSL line and over a WiFi network connection across the driveway to the garage. The WiFi link is faster than the DSL line.
The UDT via UPD transfers are with the udp transport software from: UDT.
