UdcFuse

The udc (Url Data Cache - kent/src/lib/udc.c) module is the URL random access and sparse-file caching mechanism underlying the bigBed and bigWig custom track implementation. Each bigBed/bigWig custom track's track line includes the bigDataUrl parameter which is set to the URL of the user's bigBed/bigWig file, e.g. "track name=myBB type=bigBed dataUrl=http://my.edu/myBigBed.bb".

Similar to bigBed/bigWig, the BAM alignment format (binary compressed flavor of SAM)is indexed for random access which makes it suitable for track display. The samtools-C library includes code to do HTTP and FTP random access using the BAM index, so it is easy to implement basic custom track support by simply passing the bigDataUrl to samtools-C access functions.

However, samtools-C lacks SSL (https/ftps) and caching. For each access, the entire index file is downloaded to a file in the current directory. SSL is a valuable feature for users who want to display unpublished data (one alpha-tester is waiting for SSL support), and the lack of caching slows down the genome browser display (constant 4sec track load time for a 1000 Genomes test BAM file with smaller-than-average index file size).

MarkD made the most excellent suggestion to place udc underneath the file handles used by samtools-C, as a Filesystem in Userspace (FUSE) module. FUSE provides an efficient kernel interface for userspace code to implement a fully functional file system. udcFuse is a userspace program built on FUSE that mounts a filesystem that is actually a wrapper on udc functionality. Paths within the udcFuse-mounted filesystem can be passed to samtools, which will treat them as local files. File system accesses (from samtools-C, ls, cd etc) to the udcFuse-mounted filesystem will result in calls to udcFuse FUSE method bindings, which will call udc methods. The udcFuse filesystem will be read-only, and will simply reflect the state of udc's local cache of the files.

udcFuse filesystem structure

A udcFuse filesystem is created by executing the udcFuse program:

udcFuse mountPoint [udcCacheDir]

mountPoint is an empty directory with permissions that don't exclude the user running udcFuse. In practice I expect this to be $TMPDIR/udcFuse. Actual files are stored in udc's local cache directory, so I expect little or no space to be taken up by mountPoint.

The optional udcCacheDir specifies a non-default location for udc's local cache directory. (Default is in udc.c: /tmp/udcCache)

The directory structure beneath mountPoint will mirror the directory structure of udc's local cache directory:

mountPoint/urlProtocol/urlHost/urlRestOfPath

samtools-C will be passed filenames underneath mountPoint. For example, if a custom track's bigDataUrl is http://my.edu/myBam.bam, samtools-C will be passed /mountPoint/http/my.edu/myBam.bam as if it were a local file that already existed. When samtools opens a file handle on that path, udcFuse code will reconstruct the URL, open a udcFile object on the URL, and store the udcFile for later use when samtools-C wants to seek and read.

File system accesses will be mapped onto udc methods where possible, and mapped onto accesses to the corresponding udc cache directories and files otherwise.

Implementation considerations

• Unlike CGIs and utils, this is meant to run for a long time -- don't leak mem, don't errAbort.
• udc cache filenames are qEncode'd to avoid special characters in URLs. Code that invokes samtools etc. needs to do the same encoding on URLs as udc, to get the same filenames.
• The caller must initiate the udc local caching, because udcFuse is a read-only reflection of what is already in the udc cache directory structure.

FUSE methods

Almost all FUSE methods return 0 on success, and negative system error number on failure. E.g. if a path does not exist, we return -ENOENT. If we're failing because a system call failed, rwe eturn -errno.

To implement a read-only thin wrapper on udc cache files, we need to populate only a few of the many FUSE methods.

getattr

static int udcfs_getattr(const char *path, struct stat *stbuf)

This is called before almost every other function (see http://sourceforge.net/apps/mediawiki/fuse/index.php?title=FuseInvariants), e.g. to test that a path actually exists before it tries to open or read it. We can simply fill stbuf with stat() of corresponding udc cache path, masking off any write permissions.

readdir

static int udcfs_readdir(const char *path, void *buf, fuse_fill_dir_t filler,
                         off_t offset, struct fuse_file_info *fi)

opendir() the corresponding udc cache directory, call readdir() on it in a loop, pass each readdir's d_name to FUSE's filler callback function, and closedir().

open

static int udcfs_open(const char *path, struct fuse_file_info *fi)

Convert the given path to a URL for udc and call udcFileMayOpen on the url. If successful, store the udcFile handle in fuse_file_handle fi's fh field, which is meant just for storing such a handle.

read

static int udcfs_read(const char *path, char *buf, size_t size, off_t offset,
                      struct fuse_file_info *fi)

This is an exception to the return value convention. Instead of -errno, the return value is the number of bytes actually read. Get udcFile handle from fi->fh where it was stored in udcfs_open, udcSeek to specified offset, and return the number of bytes read by udcRead.

release

static int udcfs_release(const char *path, struct fuse_file_info *fi)

Call udcFileClose on fi->fh.