Make your own virtual space

So you want to have your own copies of the binaries and your own hgwdev-user.cse.ucsc.edu browser? Here is a walkthrough:

Request a "virtual space" from the cluster-admins:

This includes, among other things, adding a block to the file /usr/local/apache/conf/httpd.conf

<VirtualHost 128.114.xx.yyy>
 ServerName hgwdev-myusername.cse.ucsc.edu
 ServerAlias hgwdev-myusername
 ScriptAlias /cgi-bin/ "/usr/local/apache/cgi-bin-myusername/"
</VirtualHost>

Prepare for the making of your CGIs

• If you don't already have one, set up your own personal Kent repository by following the instructions here: http://genomewiki.ucsc.edu/index.php/Getting_Started_With_Git#Setting_Up_Your_Own_Personal_Git_Kent_Repository

• Make sure your .bashrc or .tcshrc file (in your home directory) sets the build environment variables. It should include something like the following:

MYSQLINC=/usr/include/mysql
MYSQLLIBS="/usr/lib64/mysql/libmysqlclient.a -lz" 
MACHTYPE=x86_64
HGCGI=/usr/local/apache/cgi-bin
export MYSQLINC MYSQLLIBS MACHTYPE HGCGI

• Copy an hg.conf file from another user and place it here '/usr/local/apache/cgi-bin-username/'. It should contain at a minimum:

include ../cgi-bin/hg.conf
db.trackDb=trackDb_myusername

# Angie recommends that you do not add these lines unless you are using
# your own central database instead of hgcentraltest.
# Assuming you are using the default central.db (hgcentraltest):
# * If you use the default central.* settings from ../cgi-bin/hg.conf, then you can
#   seamlessly switch between genome-test and hgwdev-$USER -- your cookie session
#   will be the same on both.
# * If you change the central.* options, you will have separate cookies between
#   genome-test and hgwdev-$USER, which means your settings from one won't carry
#   over to the other unless you use the same hgsid when switching hosts.
central.host=localhost
central.domain=hgwdev-myusername.cse.ucsc.edu
central.cookie=hguid.hgwdev-myusername
backupcentral.domain=hgwdev-myusername.ucsc.edu 

Note that you can define your own background image by putting it in ~/public_html/ and adding

browser.background=http://hgwdev.cse.ucsc.edu/~myusername/yourimage.jpg 

Here is a generic [background image] file you can download and edit.

• Please note, in order for your browser to be fully functional, there needs to be a symlink to trash in your htdocs-myusername:

[myusername@hgwdev ~]$ cd /usr/local/apache/htdocs-myusername
[myusername@hgwdev htdocs-myusername]$ ln -s ../trash

• In order to load database tables and run CGIs on the command line for debugging, you will also need to create your own ~/.hg.conf . It must be readable and writable only by you because it contains a username and password that enables write access to the database. Here is one way to create it:

touch ~/.hg.conf
chmod 600 ~/.hg.conf
cat > ~/.hg.conf <<EOF
include /usr/local/apache/cgi-bin-myusername/hg.conf
db.user=uuuu
db.password=pppp
EOF

Instead of uuuu and pppp, use the actual write-access username and passwords -- ask cluster-admin or a colleague (best not to email them).

Populate my /usr/local/apache/cgi-bin-myusername/ directory for the first time by doing the following:

• From your home directory on hgwdev /cluster/home/myusername type the following things:

[myusername@hgwdev ~]$ cd kent
[myusername@hgwdev ~/kent]$ git pull
[myusername@hgwdev ~/kent]$ cd src
[myusername@hgwdev src]$ make libs
[myusername@hgwdev src]$ cd hg
[myusername@hgwdev hg]$ make cgi
[myusername@hgwdev hg]$ cd htdocs
[myusername@hgwdev htdocs]$ make user
[myusername@hgwdev htdocs]$ cd ../makeDb/trackDb
[myusername@hgwdev trackDb]$ make update

You now have your own compiled version of the CGIs.

Keep it current:

Here is a script (makeBinaries.csh) that you can copy and paste into your ~myusername/bin directory to make your binaries every night. You can also add a git pull to the script to ensure that your binaries are kept up to do date. Make sure to change the permissions (chmod 775) and if the script doesn't run check that the export PATH is set in your .bashrc file:

#!/bin/tcsh

if ($HOST != hgwdev) then
  echo "\n  this must be run from hgwdev!\n"
endif

cd ~/kent/src
make clean > /dev/null
cd ~/kent
git pull | mail -s 'Git pull report on kent tree' $USER
cd ~/kent/src
make libs
cd ~/kent/src/hg
make cgi
cd ~/kent/src/hg/makeDb/trackDb
make update

Automate it:

Create a crontab file. Note that you may have numerous crontab files. Each machine that you want cron jobs running on will need its own crontab file (assuming the jobs are different). Example, name your crontab file: hgwdev.cron for your hgwdev crontab. With your editor, add a line to your crontab file to run the command at the time you desire. For example, to run your daily build at 04:42 every weekday, the line would read:

42 04 * * 1-5 bin/makeBinaries.csh

See also: 'man 5 crontab' for a description of the fields used in these crontab lines.

When your crontab file is set correctly, hand it off to the cron system with the command:

$ crontab <yourCrontabFile>

Example: 'crontab hgwdev.cron'

To see what you have submitted to the cron system, ask it to display what it knows about your cron jobs with the -l argument:

$ crontab -l

Cron job tips: It is better to not run your jobs exactly at the top of the hour. They would be competing with system cron jobs that are running at times like the top of the hour. Better to run your job at odd minutes during the hour. Don't use the example time listed here, otherwise everyone's job will be running at 04:42 every day and competing with each other.

Any output to stdout or stderr by your cron job will be emailed to you via the cron system as it runs your job. If you would rather not see that output in email, construct your cron job shell script in such a way that it takes care of all of its stdout and stderr output. For example, it could create an organized by date hierarchy of log files for its output.