Source tree compilation on Debian/Ubuntu

The UCSC source tree includes tools for the command line and the CGIs for a local UCSC genome browser mirror. The installation is not easy, see below for a script that automates the process.

Please note the instructions and scripts in the source tree that can build the kent source tree on any Linux system: src/product

These commands compile the UCSC genome browser and all source tools. They have been tested on Debian Lenny (64 and 32bit), Ubuntu Maverick and Ubuntu 13.04. They will put all binaries in /usr/local, all cgi-bins in /usr/lib/cgi-bin and all html-like things into /var/www/genome.

In theory, once you have installed the samtools library, you can just copy-paste this script into a command line

# we need to be root to install packages and put stuff into /usr/local
su 
# install required packages from repository
# please not that these won't install the samtools library to handle bam files
apt-get install mysql-server-5.0 apache2 libmysqlclient15-dev libpng12-dev libssl-dev openssl mysql-client-5.5  mysql-client-core-5.5
# set variables for compilation
export MACHTYPE=$(uname -m)
export MYSQLINC=`mysql_config --include | sed -e 's/^-I//g'`
export MYSQLLIBS=`mysql_config --libs`

DIRS='SCRIPTS=/usr/local/bin CGI_BIN=/usr/lib/cgi-bin DOCUMENTROOT=/var/www/genome BINDIR=/usr/local/bin'
# this does not seem to be necessary anymore
#ENCODE_PIPELINE_BIN=/usr/local/bin'

# download
cd /usr/local
wget http://hgdownload.soe.ucsc.edu/admin/jksrc.zip
unzip jksrc.zip
mkdir -p /var/www/genome/

# compile libraries
cd kent/src/lib
make  

cd ../jkOwnLib
make 

# compile browser
cd ..
make $DIRS

# set permissions
cd /home/data/www
chown apache:apache -R *

Add the parameter HG_WARN=-w to DIRS if you don't want the compilation to stop on error messages. Dirty, but it works. :-)

The fastest way ever to get a genome browser up and running on Ubuntu, Fedora, Centos, OSX

Just tested this with ubuntu 14 LTS, centos 6, centos 7, fedora 20 and OSX 10.10 (on OSX, requires Xcode and macports):

 wget https://github.com/maximilianh/browserInstall/blob/master/browserInstall.sh
 bash installBrowser.sh

Then connect with a firefox to localhost. Done.

If you look into the /usr/local/apache/cgi-bin/hg.conf file, you can see that we use the public mysql server through the internet. This will be very slow unless you're in San Francisco. So let's copy the complete genome of C. Briggsae + all annotations:

 bash browserInstall.sh cb1

Now select the nematodes clade, the C. Briggsae 2002 Assembly. It should be very fast.

You can mirror hg19 like this ("bash browserInstall.sh hg19"), but remember that it will take a lot of space, ~2.2 TB.

Run a genome browser mirror on your local machine, compiled from source

I prefer to place the CGIs in the same location where UCSC has them, that's /usr/local/apache. This path does not conform to Ubuntu's specifications but has the advantage that we don't need to change any make variable in the UCSC tree.

 sudo apt-get install ptrace apache2
 sudo mkdir -p /usr/local/apache/{htdocs,cgi-bin,trash}
 # apache needs to be able to create images
 sudo chown www-data:www-data /usr/local/apache/trash
 # the cgis create links to images in /trash
 cd /usr/local/apache/htdocs 
 sudo ln -s ../trash
 # setup the custom trash dir in the same way
 sudo mkdir -p /usr/local/apache/trash/customTrash
 sudo chown www-data:www-data /usr/local/apache/trash/customTrash
 # make cgi-bin writable by ourselves
 sudo chown $USER:$USER /usr/local/apache/cgi-bin
 # setup apache
 sudo a2enmod include
 wget http://hgwdev.soe.ucsc.edu/~max/gbInstall/000-default -O /etc/apache2/sites-enabled/000-default
 sudo service apache2 restart
 
 # setup CGIs

 # setup mysql, set the root password to "browser"
 apt-get install mysql-server
 mysql -u root -p (you chose the password during installation of the package) -e 'create database customTrash; create database hgcentral;

 # configure the CGIs
 cd /usr/local/apache/cgi-bin
 wget 'http://genome-source.soe.ucsc.edu/gitlist/kent.git/raw/master/src/product/ex.hg.conf' -O hg.conf
 vim hg.conf

 Set the mysql connection to the public mysql server:

 db.host=genome-mysql.soe.ucsc.edu
 db.user=genomep
 db.password=password
 customTracks.host=localhost
 customTracks.user=root
 customTracks.password=<enter the password you defined during mysql installation here>
 customTracks.useAll=yes
 customTracks.tmpdir=/usr/local/apache/trash/customTrash
 central.db=hgcentral
 central.host=localhost
 central.user=root
 central.password=<your password again>
 central.domain=<you internet domain>

 # alternative: use the public mysql server and my configuration, should work if your root password is "browser"
 wget http://hgwdev.soe.ucsc.edu/~max/gbInstall/hg.conf.public -O /usr/local/apache/cgi-bin/hg.conf 

 # copy just the cgi bin scripts
 cd /usr/local/apache/cgi-bin
 wget http://hgwdev.soe.ucsc.edu/~max/gbInstall/cgiBinUbuntu1304_64.tgz -O - | tar xvz  

 OR: compile everything:
 # set variables
 export MYSQLLIBS="-lmysqlclient -lz"
 export MACHTYPE=$(uname -m)
 export MYSQLINC=/usr/include/mysql
 cd ~/kent
 make alpha

Download Hiram's install scripts to some place... how?... and run them.