Browser Track Construction
General comments
The file system layout in the examples here is typical for the UCSC genome browser build. The consistency of the file directory hierarchy allows automatic scripts to perform many of the functions. Therefore, we keep required files in the same location for each browser build to allow the tools to find what they need.
All browser builds are kept under one single directory hierarchy which is on a file system that is also shared with the cluster computer system. This allows work to take place on a large memory work horse system, and also allow the same files to be used during cluster runs for that type of processing.
For example, keeping all browser builds in /data/genomes/ with each specific browser build in a directory of the database name, for example,
- /data/genomes/hg19/
- /data/genomes/mm10/
- /data/genomes/ricCom1/
2bit file construction
The usual genome browser construction at UCSC starts with the official released files from the NCBI FTP site:
ftp://ftp.ncbi.nih.gov/genbank/genomes/Eukaryotes/
Download the NCBI files into:
- /data/genomes/ricCom1/genbank/
rsync works with the NCBI FTP site:
rsync -a -P rsync://ftp.ncbi.nlm.nih.gov/genbank/genomes/Eukaryotes/plants/Ricinus_communis/JCVI_RCG_1.1/ /data/genomes/ricCom1/genbank/
This is a typical unplaced scaffold assembly with one additional non-nuclear chrCp plastid. There is a UCSC script in the source tree:
unplacedScaffolds.pl
which processes the genbank files into UCSC style fasta and AGP files. The actual work in the script is simple, it removes the .1 from the accession identifiers in the fasta and AGP files. The bulk of the script merely maintains the naming scheme and file system hierarchy for follow-on tool processing. It creates the directory /data/genomes/ricCom1/ucsc/ and leaves the files ricCom1.ucsc.fa.gz and ricCom1.ucsc.agp files there.
The extra non-nuclear plastid is manually added to the unplaced AGP file and the chrCp fasta sequence included with the unplaced fasta file to construct the .2bit file:
cd /data/genomes/ricCom1/ucsc faToTwoBit ricCom1.ucsc.fa.gz chrCp.fa ../ricCom1.unmasked.2bit cd /data/genomes/ricCom1 twoBitInfo ricCom1.unmasked.2bit stdout | sort -k2nr > chrom.sizes
The chrom.sizes file is useful for later processing.
Repeat Masking
Repeat masking is necessary for almost all other track construction. UCSC uses Repeat Masker for the repeat masking track, TRF for simple repeats, and the NCBI WindowMasker for the WIndow Masker tracks.
The procedure is usually:
- break up the genome sequence into manageable pieces
- cluster run to mask each piece
- reassemble the results to original assembly coordinates
A choice is to be made whether the Repeat Masker or Window Masker result is used to mask the sequence. If Repeat Masker has repeats in the RM libraries for this species and can mask a good amount of the genome, it can be used for the masking. If RM only masks a small percent of the genome, then UCSC uses Window Masker to mask the sequence. In this Ricinus communis example, RM only found %3 in repeats, where Window Masker found almost %45 in repeats. Thus the Window Masker result, plus the TRF repeats of period less than or equal to 12 are used to mask the sequence resulting in %45.95 of the sequence masked. UCSC uses the larger masking result to avoid problems in processing during pair-wise alignments. When there isn't enough masking is aligned sequences, the repeats overwhelm the alignment results.
Please note specific details for running each of the masking operations:
