Browser Mirrors: Difference between revisions
No edit summary |
No edit summary |
||
| Line 4: | Line 4: | ||
==Partial Mirrors== | ==Partial Mirrors== | ||
A complete mirror of all assemblies requires a large amount of disk space (currently on the order of a terabyte). However, it is not too difficult to set things up so that only a portion of assemblies are mirrored. The following scripts are used for this purpose at Cornell (http://genome-mirror.bscb.cornell.edu). | A complete mirror of all assemblies requires a large amount of disk space (currently on the order of a terabyte). However, it is not too difficult to set things up so that only a portion of assemblies are mirrored. The following scripts and auxiliary files are used for this purpose at Cornell (http://genome-mirror.bscb.cornell.edu). | ||
* [[Media:doDownloads.sh]] - Script for downloading selected files from hgdownload.cse.ucsc.edu | * [[Media:doDownloads.sh]] - Script for downloading selected files from hgdownload.cse.ucsc.edu | ||
| Line 10: | Line 10: | ||
* [[Media:databases]] - File identifying which databases to mirror. | * [[Media:databases]] - File identifying which databases to mirror. | ||
* [[Media:gbdb.exclude]] - File identifying directories to be excluded when rsyncing /gbdb. | * [[Media:gbdb.exclude]] - File identifying directories to be excluded when rsyncing /gbdb. | ||
These programs are run nightly via cron, using the following crontab entry: | |||
<blockquote> | |||
<code> | |||
0 0 * * * /usr/data/mirror-download/doAll.sh | |||
</code> | |||
</blockquote> | |||
where doAll.sh is a simple wrapper for doDownloads.sh and doUpdateDb.sh, viz. | |||
<blockquote> | |||
<code> | |||
#!/bin/bash -e | |||
# do downloads and updates | |||
# for use with cron | |||
echo "#####################################" | |||
/usr/data/mirror-download/doDownloads.sh | |||
echo "#####################################" | |||
/usr/data/mirror-download/doUpdateDb.sh | |||
echo "#####################################" | |||
echo "Successfully updated mirror." | |||
</code> | |||
</blockquote> | |||
Revision as of 15:08, 10 August 2006
This page contains information for users interested in mirroring the UCSC Genome Browser on their own servers. See also http://genome.ucsc.edu/mirror.html
Partial Mirrors
A complete mirror of all assemblies requires a large amount of disk space (currently on the order of a terabyte). However, it is not too difficult to set things up so that only a portion of assemblies are mirrored. The following scripts and auxiliary files are used for this purpose at Cornell (http://genome-mirror.bscb.cornell.edu).
- Media:doDownloads.sh - Script for downloading selected files from hgdownload.cse.ucsc.edu
- Media:doUpdateDb.sh - Script for updating local mysql database with downloaded files.
- Media:databases - File identifying which databases to mirror.
- Media:gbdb.exclude - File identifying directories to be excluded when rsyncing /gbdb.
These programs are run nightly via cron, using the following crontab entry:
0 0 * * * /usr/data/mirror-download/doAll.sh
where doAll.sh is a simple wrapper for doDownloads.sh and doUpdateDb.sh, viz.
- !/bin/bash -e
- do downloads and updates
- for use with cron
echo "#####################################" /usr/data/mirror-download/doDownloads.sh
echo "#####################################" /usr/data/mirror-download/doUpdateDb.sh
echo "#####################################" echo "Successfully updated mirror."
