User talk:Fubar

Some notes on installing a local mirror on an x86_64 CentOS linux box

'I hope these notes are useful for anyone trying to download and setup a local mirror'

I just finished a limited install on a x86_64 CentOS 4.3 machine. There are some gotcha's worth knowing about. I only grabbed hg18 because that's all I needed for my local mirror.

The instructions on this wiki at Browser_Installation were very helpful. When things got wierd, the various README files in JK's source were useful but sometimes you just need to look at the source. Before you start, understand that there is no single recipe that I could find (or write) to make this painless. You're undertaking a fairly complex task in setting up a local mirror. Most of the available instructions are slightly out of date or for a different environment than yours - the good folks at UCSC are doing a great job and they will help if you ask sensible questions after doing your own due diligence, but this is not a commodity software installation and should probably not be undertaken unless you really know your way around as root with linux/apache/mysql. This will likely not be a pleasant experience if it's your first undertaking as a system administrator. However, with some judicious googling I was able to fake it so you probably can too :)

Firstly, downloading the necessary datasets, especially gbdb/hg18 takes a very, very long time. There's a lot there!

Secondly, I gave up after a day of trying to get everything working anywhere but at the root of the website. All sorts of hardcoded references to ../trash and such like. Once I gave up and reverted to /var/www/html as the root, everything seemed to work better...

Thirdly, rsyncing the mysql directories directly works a treat - if you have mysql 4.x as CentOS does. I did not undergo the pain of downloading the raw SQL and rebuilding all those tables and indices. Be aware that the files become unavailable while they're being updated in the early morning, so you may find some of your rsync jobs end with 'file unavailable' errors - don't panic - re-starting the same rsync command will work fine and the successfully downloaded files won't be downloaded again. Don't forget to restart mysql /etc/mysqld restart as root after all the mysql database directories have been downloaded.

You must have all the development libraries for MySQL installed - use yum list and yum install as needed until you see something like this:

[root@meme rerla]# yum list mysql*
Setting up repositories
update                    100% |=========================|  951 B    00:00     
base                      100% |=========================| 1.1 kB    00:00     
addons                    100% |=========================|  951 B    00:00     
extras                    100% |=========================| 1.1 kB    00:00     
Reading repository metadata in from local files
Installed Packages
mysql.x86_64                             4.1.20-1.RHEL4.1       installed         
mysql-devel.x86_64                       4.1.20-1.RHEL4.1       installed       
mysql-server.x86_64                      4.1.20-1.RHEL4.1       installed       

The following environment settings (bash) should work for standard CentOS 4.4 fully patched systems - they worked for me.

export  MYSQLINC='/usr/include/mysql'
export MYSQLLIBS='/usr/lib64/mysql/libmysqlclient.a -lz -lcrypto -lssl -lm -lnsl'
export GLOBAL_CONFIG_FILE=/var/www/cgi-bin/hg.conf 
export HGCGI=/var/www/cgi-bin

If you have SE Linux enforcing on, you need to do something like:

chcon -R  -u system_u -r object_r -t httpd_sys_content_t /var/www

so apache can run cgi's and generally do useful things.

While rsync for the mysql tables directly is what I recommend, rsync of the executables did not work for me - there was a problem with the authentication scheme the exe mysql client was trying to use and setting old passwords on in /etc/my.cnf did not fix it. So I'd recommend doing your own compile from a fresh checkout of the source. Strongly recommend that you do a [cvs checkout] of the source - that fixed a couple of misfeatures I found using the jksrc.zip file. Turns out the download of that file is refreshed every 2 weeks or so - for new features such as genome graphs that I was particularly interested in, a fresh CVS checkout and recompile solved a lot of issues.

The build instructions described on that CVS page are different from the ones that worked for me - you'll probably want to do the checkout, then follow the build instructions at Browser_Installation. I think there are UCSC specific issues mixed up in some documentation.