File system performance
Testing filesystem performance read/write numbers of files into overloaded directory with up to 7 million files.
- start with empty directory
- writing test uses kent source library trash directory functions to obtain unique filenames in the directory
- start with fileCount = 1,024
- begin loop
- writing fileCount number of files of approximate size 16,384 bytes (bell curve of sizes)
- then randomly reads fileCount/2 number of files
- increment fileCount by 10,000
- after first time through this loop
- second inner loop repeats this entire procedure from 1,024 number of files up to outer loop count
- repeat outer loop
The reading test procedure:
- read in file names for entire directory
- looping for fileCount
- randomly select a file from the list
- stat the file to obtain its size
- read the file into a memory buffer
- close file handle
- repeat loop
Recording the time for obtaining the list of file names for the directory, and the time for reading/writing fileCount number of files, as well as counting the number of bytes read/written.
/dev/shm/ tmpfs in memory filesystem
The null model test. This is the RAM memory filesystem '/dev/shm/'. Performance remains constant despite numbers of files in the directory.
Writing performance declines rapidly, reading performance remains constant.
Both read and write performance drop off between one and two million files in the directory.
Both read and write performance are mostly stable despite high file count.