Source tree compilation on Debian/Ubuntu: Difference between revisions

From genomewiki
Jump to navigationJump to search
No edit summary
(changed cse to soe and genome-source fixes)
 
(51 intermediate revisions by 3 users not shown)
Line 1: Line 1:
This worked on my Debian Lenny am64 machine to compile the UCSC source tree. It will put all binaries in /usr/local, all cgi-bins in /usr/lib/cgi-bin and all html-like things into /var/www/genome.
The UCSC source tree includes tools for the command line and the CGIs for a local UCSC genome browser mirror. The installation is not easy, see below for a script that automates the process.
Please note the instructions and scripts in the source tree that can build the kent source tree on
any Linux system:
[http://genome-source.soe.ucsc.edu/gitlist/kent.git/tree/master/src/product;hb=HEAD src/product]
 
These commands compile the UCSC genome browser and all source tools. They have been tested on Debian Lenny (64 and 32bit), Ubuntu Maverick and Ubuntu 13.04. They will put all binaries in /usr/local, all cgi-bins in /usr/lib/cgi-bin and all html-like things into /var/www/genome.
 
In theory, once you have installed the samtools library, you can just copy-paste this script into a command line
<pre>
<pre>
# we need to be root to install packages and put stuff into /usr/local
su  
su  
aptitude install mysql-server-5.0 apache2 libmysqlclient15-dev  
# install required packages from repository
# libmysqlclient.a does not contain SSL symbols, so use shared library
# please not that these won't install the samtools library to handle bam files
export MYSQLLIBS="-lmysqlclient -lz"
apt-get install mysql-server-5.0 apache2 libmysqlclient15-dev libpng12-dev libssl-dev openssl mysql-client-5.5  mysql-client-core-5.5
# set variables for compilation
export MACHTYPE=$(uname -m)
export MACHTYPE=$(uname -m)
export MYSQLINC=/usr/include/mysql
export MYSQLINC=`mysql_config --include | sed -e 's/^-I//g'`
DIRS='SCRIPTS=/usr/local/bin CGI_BIN=/usr/lib/cgi-bin DOCUMENTROOT=/var/www/genome BINDIR=/usr/local/bin ENCODE_PIPELINE_BIN=/usr/local/bin'
export MYSQLLIBS=`mysql_config --libs`
 
DIRS='SCRIPTS=/usr/local/bin CGI_BIN=/usr/lib/cgi-bin DOCUMENTROOT=/var/www/genome BINDIR=/usr/local/bin'
# this does not seem to be necessary anymore
#ENCODE_PIPELINE_BIN=/usr/local/bin'


# download
# download
cd /usr/local
cd /usr/local
wget http://hgdownload.cse.ucsc.edu/admin/jksrc.zip
wget http://hgdownload.soe.ucsc.edu/admin/jksrc.zip
unzip jksrc.zip
unzip jksrc.zip
mkdir -p /var/www/genome/
mkdir -p /var/www/genome/


# compile libraries
# compile libraries
cd kent/src/libs
cd kent/src/lib
make compile $DIRS
make 
 
cd ../jkOwnLib
make  


# compile browser
# compile browser
cd ../hg
cd ..
make $DIRS
make $DIRS
# set permissions
cd /home/data/www
chown apache:apache -R *
</pre>
</pre>
Add the parameter HG_WARN=-w to DIRS if you don't want the compilation to stop on error messages. Dirty, but it works. :-)
== The fastest way ever to get a genome browser up and running on Ubuntu, Fedora, Centos, OSX ==
Just tested this with ubuntu 14 LTS, centos 6, centos 7, fedora 20 and OSX 10.10 (on OSX, requires Xcode and macports):
  wget https://github.com/maximilianh/browserInstall/blob/master/browserInstall.sh
  bash installBrowser.sh
Then connect with a firefox to localhost. Done.
If you look into the /usr/local/apache/cgi-bin/hg.conf file, you can see that we use the public mysql server through the internet. This will be very slow unless you're in San Francisco. So let's copy the complete genome of C. Briggsae + all annotations:
  bash browserInstall.sh cb1
Now select the nematodes clade, the C. Briggsae 2002 Assembly. It should be very fast.
You can mirror hg19 like this ("bash browserInstall.sh hg19"), but remember that it will take a lot of space, ~2.2 TB.
== Run a genome browser mirror on your local machine, compiled from source ==
I prefer to place the CGIs in the same location where UCSC has them, that's /usr/local/apache. This path does not conform to Ubuntu's specifications but has the advantage that we don't need to change any make variable in the UCSC tree.
  sudo apt-get install ptrace apache2
  sudo mkdir -p /usr/local/apache/{htdocs,cgi-bin,trash}
  # apache needs to be able to create images
  sudo chown www-data:www-data /usr/local/apache/trash
  # the cgis create links to images in /trash
  cd /usr/local/apache/htdocs
  sudo ln -s ../trash
  # setup the custom trash dir in the same way
  sudo mkdir -p /usr/local/apache/trash/customTrash
  sudo chown www-data:www-data /usr/local/apache/trash/customTrash
  # make cgi-bin writable by ourselves
  sudo chown $USER:$USER /usr/local/apache/cgi-bin
  # setup apache
  sudo a2enmod include
  wget http://hgwdev.soe.ucsc.edu/~max/gbInstall/000-default -O /etc/apache2/sites-enabled/000-default
  sudo service apache2 restart
 
  # setup CGIs
  # setup mysql, set the root password to "browser"
  apt-get install mysql-server
  mysql -u root -p (you chose the password during installation of the package) -e 'create database customTrash; create database hgcentral;
  # configure the CGIs
  cd /usr/local/apache/cgi-bin
  wget 'http://genome-source.soe.ucsc.edu/gitlist/kent.git/raw/master/src/product/ex.hg.conf' -O hg.conf
  vim hg.conf
  Set the mysql connection to the public mysql server:
  db.host=genome-mysql.soe.ucsc.edu
  db.user=genomep
  db.password=password
  customTracks.host=localhost
  customTracks.user=root
  customTracks.password=<enter the password you defined during mysql installation here>
  customTracks.useAll=yes
  customTracks.tmpdir=/usr/local/apache/trash/customTrash
  central.db=hgcentral
  central.host=localhost
  central.user=root
  central.password=<your password again>
  central.domain=<you internet domain>
  # alternative: use the public mysql server and my configuration, should work if your root password is "browser"
  wget http://hgwdev.soe.ucsc.edu/~max/gbInstall/hg.conf.public -O /usr/local/apache/cgi-bin/hg.conf
  # copy just the cgi bin scripts
  cd /usr/local/apache/cgi-bin
  wget http://hgwdev.soe.ucsc.edu/~max/gbInstall/cgiBinUbuntu1304_64.tgz -O - | tar xvz 
  OR: compile everything:
  # set variables
  export MYSQLLIBS="-lmysqlclient -lz"
  export MACHTYPE=$(uname -m)
  export MYSQLINC=/usr/include/mysql
  cd ~/kent
  make alpha
Download Hiram's install scripts to some place... how?... and run them.
[[Category:User Developed Scripts]]
[[Category:Installation]]

Latest revision as of 07:32, 1 September 2018

The UCSC source tree includes tools for the command line and the CGIs for a local UCSC genome browser mirror. The installation is not easy, see below for a script that automates the process.

Please note the instructions and scripts in the source tree that can build the kent source tree on any Linux system: src/product

These commands compile the UCSC genome browser and all source tools. They have been tested on Debian Lenny (64 and 32bit), Ubuntu Maverick and Ubuntu 13.04. They will put all binaries in /usr/local, all cgi-bins in /usr/lib/cgi-bin and all html-like things into /var/www/genome.

In theory, once you have installed the samtools library, you can just copy-paste this script into a command line

# we need to be root to install packages and put stuff into /usr/local
su 
# install required packages from repository
# please not that these won't install the samtools library to handle bam files
apt-get install mysql-server-5.0 apache2 libmysqlclient15-dev libpng12-dev libssl-dev openssl mysql-client-5.5  mysql-client-core-5.5
# set variables for compilation
export MACHTYPE=$(uname -m)
export MYSQLINC=`mysql_config --include | sed -e 's/^-I//g'`
export MYSQLLIBS=`mysql_config --libs`

DIRS='SCRIPTS=/usr/local/bin CGI_BIN=/usr/lib/cgi-bin DOCUMENTROOT=/var/www/genome BINDIR=/usr/local/bin'
# this does not seem to be necessary anymore
#ENCODE_PIPELINE_BIN=/usr/local/bin'

# download
cd /usr/local
wget http://hgdownload.soe.ucsc.edu/admin/jksrc.zip
unzip jksrc.zip
mkdir -p /var/www/genome/

# compile libraries
cd kent/src/lib
make  

cd ../jkOwnLib
make 

# compile browser
cd ..
make $DIRS

# set permissions
cd /home/data/www
chown apache:apache -R *

Add the parameter HG_WARN=-w to DIRS if you don't want the compilation to stop on error messages. Dirty, but it works. :-)

The fastest way ever to get a genome browser up and running on Ubuntu, Fedora, Centos, OSX

Just tested this with ubuntu 14 LTS, centos 6, centos 7, fedora 20 and OSX 10.10 (on OSX, requires Xcode and macports):

 wget https://github.com/maximilianh/browserInstall/blob/master/browserInstall.sh
 bash installBrowser.sh

Then connect with a firefox to localhost. Done.

If you look into the /usr/local/apache/cgi-bin/hg.conf file, you can see that we use the public mysql server through the internet. This will be very slow unless you're in San Francisco. So let's copy the complete genome of C. Briggsae + all annotations:

 bash browserInstall.sh cb1

Now select the nematodes clade, the C. Briggsae 2002 Assembly. It should be very fast.

You can mirror hg19 like this ("bash browserInstall.sh hg19"), but remember that it will take a lot of space, ~2.2 TB.


Run a genome browser mirror on your local machine, compiled from source

I prefer to place the CGIs in the same location where UCSC has them, that's /usr/local/apache. This path does not conform to Ubuntu's specifications but has the advantage that we don't need to change any make variable in the UCSC tree.

 sudo apt-get install ptrace apache2
 sudo mkdir -p /usr/local/apache/{htdocs,cgi-bin,trash}
 # apache needs to be able to create images
 sudo chown www-data:www-data /usr/local/apache/trash
 # the cgis create links to images in /trash
 cd /usr/local/apache/htdocs 
 sudo ln -s ../trash
 # setup the custom trash dir in the same way
 sudo mkdir -p /usr/local/apache/trash/customTrash
 sudo chown www-data:www-data /usr/local/apache/trash/customTrash
 # make cgi-bin writable by ourselves
 sudo chown $USER:$USER /usr/local/apache/cgi-bin
 # setup apache
 sudo a2enmod include
 wget http://hgwdev.soe.ucsc.edu/~max/gbInstall/000-default -O /etc/apache2/sites-enabled/000-default
 sudo service apache2 restart
 
 # setup CGIs
 # setup mysql, set the root password to "browser"
 apt-get install mysql-server
 mysql -u root -p (you chose the password during installation of the package) -e 'create database customTrash; create database hgcentral;
 # configure the CGIs
 cd /usr/local/apache/cgi-bin
 wget 'http://genome-source.soe.ucsc.edu/gitlist/kent.git/raw/master/src/product/ex.hg.conf' -O hg.conf
 vim hg.conf


 Set the mysql connection to the public mysql server:
 db.host=genome-mysql.soe.ucsc.edu
 db.user=genomep
 db.password=password
 customTracks.host=localhost
 customTracks.user=root
 customTracks.password=<enter the password you defined during mysql installation here>
 customTracks.useAll=yes
 customTracks.tmpdir=/usr/local/apache/trash/customTrash
 central.db=hgcentral
 central.host=localhost
 central.user=root
 central.password=<your password again>
 central.domain=<you internet domain>
 # alternative: use the public mysql server and my configuration, should work if your root password is "browser"
 wget http://hgwdev.soe.ucsc.edu/~max/gbInstall/hg.conf.public -O /usr/local/apache/cgi-bin/hg.conf 
 # copy just the cgi bin scripts
 cd /usr/local/apache/cgi-bin
 wget http://hgwdev.soe.ucsc.edu/~max/gbInstall/cgiBinUbuntu1304_64.tgz -O - | tar xvz  
 OR: compile everything:
 # set variables
 export MYSQLLIBS="-lmysqlclient -lz"
 export MACHTYPE=$(uname -m)
 export MYSQLINC=/usr/include/mysql
 cd ~/kent
 make alpha

Download Hiram's install scripts to some place... how?... and run them.