Source tree compilation on Debian/Ubuntu: Difference between revisions

From genomewiki
Jump to navigationJump to search
(changed cse to soe and genome-source fixes)
 
(13 intermediate revisions by one other user not shown)
Line 1: Line 1:
These binaries can also be used to run a local UCSC genome browser mirror.
The UCSC source tree includes tools for the command line and the CGIs for a local UCSC genome browser mirror. The installation is not easy, see below for a script that automates the process.
   
   
Please note the instructions and scripts in the source tree that can build the kent source tree on
Please note the instructions and scripts in the source tree that can build the kent source tree on
any Linux system:
any Linux system:
[http://genome-source.cse.ucsc.edu/gitweb/?p=kent.git;a=tree;f=src/product;hb=HEAD src/product]
[http://genome-source.soe.ucsc.edu/gitlist/kent.git/tree/master/src/product;hb=HEAD src/product]


These commands compile the UCSC genome browser and all source tools. They have been tested on Debian Lenny (64 and 32bit), Ubuntu Maverick and Ubuntu 13.04. They will put all binaries in /usr/local, all cgi-bins in /usr/lib/cgi-bin and all html-like things into /var/www/genome.
These commands compile the UCSC genome browser and all source tools. They have been tested on Debian Lenny (64 and 32bit), Ubuntu Maverick and Ubuntu 13.04. They will put all binaries in /usr/local, all cgi-bins in /usr/lib/cgi-bin and all html-like things into /var/www/genome.
Line 16: Line 16:
# set variables for compilation
# set variables for compilation
export MACHTYPE=$(uname -m)
export MACHTYPE=$(uname -m)
export MYSQLINC=`mysql_config --include`
export MYSQLINC=`mysql_config --include | sed -e 's/^-I//g'`
export MYSQLLIBS=`mysql_config --libs`
export MYSQLLIBS=`mysql_config --libs`


Line 25: Line 25:
# download
# download
cd /usr/local
cd /usr/local
wget http://hgdownload.cse.ucsc.edu/admin/jksrc.zip
wget http://hgdownload.soe.ucsc.edu/admin/jksrc.zip
unzip jksrc.zip
unzip jksrc.zip
mkdir -p /var/www/genome/
mkdir -p /var/www/genome/
Line 47: Line 47:
Add the parameter HG_WARN=-w to DIRS if you don't want the compilation to stop on error messages. Dirty, but it works. :-)
Add the parameter HG_WARN=-w to DIRS if you don't want the compilation to stop on error messages. Dirty, but it works. :-)


== The fastest way ever to get a genome browser up and running on Ubuntu, Fedora, Centos, OSX ==
Just tested this with ubuntu 14 LTS, centos 6, centos 7, fedora 20 and OSX 10.10 (on OSX, requires Xcode and macports):


== Run a genome browser mirror on your local machine ==
  wget https://github.com/maximilianh/browserInstall/blob/master/browserInstall.sh
  bash installBrowser.sh
 
Then connect with a firefox to localhost. Done.
 
If you look into the /usr/local/apache/cgi-bin/hg.conf file, you can see that we use the public mysql server through the internet. This will be very slow unless you're in San Francisco. So let's copy the complete genome of C. Briggsae + all annotations:
 
  bash browserInstall.sh cb1
 
Now select the nematodes clade, the C. Briggsae 2002 Assembly. It should be very fast.
 
You can mirror hg19 like this ("bash browserInstall.sh hg19"), but remember that it will take a lot of space, ~2.2 TB.
 
 
== Run a genome browser mirror on your local machine, compiled from source ==
I prefer to place the CGIs in the same location where UCSC has them, that's /usr/local/apache. This path does not conform to Ubuntu's specifications but has the advantage that we don't need to change any make variable in the UCSC tree.
I prefer to place the CGIs in the same location where UCSC has them, that's /usr/local/apache. This path does not conform to Ubuntu's specifications but has the advantage that we don't need to change any make variable in the UCSC tree.


Line 65: Line 81:
   # setup apache
   # setup apache
   sudo a2enmod include
   sudo a2enmod include
   sudo vim /etc/apache2/sites-enabled/000-default
   wget http://hgwdev.soe.ucsc.edu/~max/gbInstall/000-default -O /etc/apache2/sites-enabled/000-default
 
Paste these lines (use the vim command :set paste to easier pasting) and remove the old ones:
 
      DocumentRoot /usr/local/apache/htdocs
        <Directory />
                Options FollowSymLinks +Includes
                XBitHack on
                AllowOverride None
        </Directory>
        ScriptAlias /cgi-bin /usr/local/apache/cgi-bin
        <Directory "/usr/local/apache/cgi-bin">
                XBitHack on
                AllowOverride None
                Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch +Includes FollowSymLinks
                Order allow,deny
                Allow from all
        </Directory>
 
   sudo service apache2 restart
   sudo service apache2 restart
 
  # setup CGIs


   mysql -u root -p (you chose the password during installation of the package)
  # setup mysql, set the root password to "browser"
  create database customTrash
  apt-get install mysql-server
  create database hgcentral
   mysql -u root -p (you chose the password during installation of the package) -e 'create database customTrash; create database hgcentral;
  quit


  # configure the CGIs
   cd /usr/local/apache/cgi-bin
   cd /usr/local/apache/cgi-bin
   wget 'http://genome-source.cse.ucsc.edu/gitweb/?p=kent.git;a=blob_plain;f=src/product/ex.hg.conf;hb=HEAD' -O hg.conf
   wget 'http://genome-source.soe.ucsc.edu/gitlist/kent.git/raw/master/src/product/ex.hg.conf' -O hg.conf
   vim hg.conf
   vim hg.conf


Set the mysql connection to the public mysql server:


   db.host=genome-mysql.cse.ucsc.edu
  Set the mysql connection to the public mysql server:
 
   db.host=genome-mysql.soe.ucsc.edu
   db.user=genomep
   db.user=genomep
   db.password=password
   db.password=password
Line 111: Line 112:
   central.domain=<you internet domain>
   central.domain=<you internet domain>


Now compile everything:
  # alternative: use the public mysql server and my configuration, should work if your root password is "browser"
  wget http://hgwdev.soe.ucsc.edu/~max/gbInstall/hg.conf.public -O /usr/local/apache/cgi-bin/hg.conf
 
  # copy just the cgi bin scripts
  cd /usr/local/apache/cgi-bin
  wget http://hgwdev.soe.ucsc.edu/~max/gbInstall/cgiBinUbuntu1304_64.tgz -O - | tar xvz 
 
  OR: compile everything:
   # set variables
   # set variables
   export MYSQLLIBS="-lmysqlclient -lz"
   export MYSQLLIBS="-lmysqlclient -lz"

Latest revision as of 07:32, 1 September 2018

The UCSC source tree includes tools for the command line and the CGIs for a local UCSC genome browser mirror. The installation is not easy, see below for a script that automates the process.

Please note the instructions and scripts in the source tree that can build the kent source tree on any Linux system: src/product

These commands compile the UCSC genome browser and all source tools. They have been tested on Debian Lenny (64 and 32bit), Ubuntu Maverick and Ubuntu 13.04. They will put all binaries in /usr/local, all cgi-bins in /usr/lib/cgi-bin and all html-like things into /var/www/genome.

In theory, once you have installed the samtools library, you can just copy-paste this script into a command line

# we need to be root to install packages and put stuff into /usr/local
su 
# install required packages from repository
# please not that these won't install the samtools library to handle bam files
apt-get install mysql-server-5.0 apache2 libmysqlclient15-dev libpng12-dev libssl-dev openssl mysql-client-5.5  mysql-client-core-5.5
# set variables for compilation
export MACHTYPE=$(uname -m)
export MYSQLINC=`mysql_config --include | sed -e 's/^-I//g'`
export MYSQLLIBS=`mysql_config --libs`

DIRS='SCRIPTS=/usr/local/bin CGI_BIN=/usr/lib/cgi-bin DOCUMENTROOT=/var/www/genome BINDIR=/usr/local/bin'
# this does not seem to be necessary anymore
#ENCODE_PIPELINE_BIN=/usr/local/bin'

# download
cd /usr/local
wget http://hgdownload.soe.ucsc.edu/admin/jksrc.zip
unzip jksrc.zip
mkdir -p /var/www/genome/

# compile libraries
cd kent/src/lib
make  

cd ../jkOwnLib
make 

# compile browser
cd ..
make $DIRS

# set permissions
cd /home/data/www
chown apache:apache -R *

Add the parameter HG_WARN=-w to DIRS if you don't want the compilation to stop on error messages. Dirty, but it works. :-)

The fastest way ever to get a genome browser up and running on Ubuntu, Fedora, Centos, OSX

Just tested this with ubuntu 14 LTS, centos 6, centos 7, fedora 20 and OSX 10.10 (on OSX, requires Xcode and macports):

 wget https://github.com/maximilianh/browserInstall/blob/master/browserInstall.sh
 bash installBrowser.sh

Then connect with a firefox to localhost. Done.

If you look into the /usr/local/apache/cgi-bin/hg.conf file, you can see that we use the public mysql server through the internet. This will be very slow unless you're in San Francisco. So let's copy the complete genome of C. Briggsae + all annotations:

 bash browserInstall.sh cb1

Now select the nematodes clade, the C. Briggsae 2002 Assembly. It should be very fast.

You can mirror hg19 like this ("bash browserInstall.sh hg19"), but remember that it will take a lot of space, ~2.2 TB.


Run a genome browser mirror on your local machine, compiled from source

I prefer to place the CGIs in the same location where UCSC has them, that's /usr/local/apache. This path does not conform to Ubuntu's specifications but has the advantage that we don't need to change any make variable in the UCSC tree.

 sudo apt-get install ptrace apache2
 sudo mkdir -p /usr/local/apache/{htdocs,cgi-bin,trash}
 # apache needs to be able to create images
 sudo chown www-data:www-data /usr/local/apache/trash
 # the cgis create links to images in /trash
 cd /usr/local/apache/htdocs 
 sudo ln -s ../trash
 # setup the custom trash dir in the same way
 sudo mkdir -p /usr/local/apache/trash/customTrash
 sudo chown www-data:www-data /usr/local/apache/trash/customTrash
 # make cgi-bin writable by ourselves
 sudo chown $USER:$USER /usr/local/apache/cgi-bin
 # setup apache
 sudo a2enmod include
 wget http://hgwdev.soe.ucsc.edu/~max/gbInstall/000-default -O /etc/apache2/sites-enabled/000-default
 sudo service apache2 restart
 
 # setup CGIs
 # setup mysql, set the root password to "browser"
 apt-get install mysql-server
 mysql -u root -p (you chose the password during installation of the package) -e 'create database customTrash; create database hgcentral;
 # configure the CGIs
 cd /usr/local/apache/cgi-bin
 wget 'http://genome-source.soe.ucsc.edu/gitlist/kent.git/raw/master/src/product/ex.hg.conf' -O hg.conf
 vim hg.conf


 Set the mysql connection to the public mysql server:
 db.host=genome-mysql.soe.ucsc.edu
 db.user=genomep
 db.password=password
 customTracks.host=localhost
 customTracks.user=root
 customTracks.password=<enter the password you defined during mysql installation here>
 customTracks.useAll=yes
 customTracks.tmpdir=/usr/local/apache/trash/customTrash
 central.db=hgcentral
 central.host=localhost
 central.user=root
 central.password=<your password again>
 central.domain=<you internet domain>
 # alternative: use the public mysql server and my configuration, should work if your root password is "browser"
 wget http://hgwdev.soe.ucsc.edu/~max/gbInstall/hg.conf.public -O /usr/local/apache/cgi-bin/hg.conf 
 # copy just the cgi bin scripts
 cd /usr/local/apache/cgi-bin
 wget http://hgwdev.soe.ucsc.edu/~max/gbInstall/cgiBinUbuntu1304_64.tgz -O - | tar xvz  
 OR: compile everything:
 # set variables
 export MYSQLLIBS="-lmysqlclient -lz"
 export MACHTYPE=$(uname -m)
 export MYSQLINC=/usr/include/mysql
 cd ~/kent
 make alpha

Download Hiram's install scripts to some place... how?... and run them.