Gbib development

This page is for developers, it's not relevant for QA. The QA page is Genome_Browser_in_a_box_config.

Box-specific development can happen on the devbox, a development version of the browserbox with git, gcc etc. For testing during development, a special command on the box can pull in the CGIs that were compiled on the devbox. The devbox is necessary as otherwise the gbib image has to allow development which adds around 2gb of packages most users don't ever need. In addition, it avoids the potential problem of our developer passwords leaking out through tempfiles or the swapspace.

The browserbox-dev VM, alias "the devbox", listens to http on 1236 and for ssh on 1237.

On hgwdev, all virtual machines live under ~qateam/VirtualBox VMs/. The central virtualbox xml config file where they are registered is ~/.config/VirtualBox/VirtualBox.xml

To start the devbox, you need to login as qateam onto hgwdev.

From hgwdev as qateam, you can then connect to the box with "ssh devbox". This works, because of qateam's .ssh/config file:

 Host devbox
 Hostname localhost
 Port 1237
 User browser

You can start and stop browserbox-dev with the aliases devStart and devStop as qateam.

Setup your laptop

to connect to the devbox, add this to .ssh/config on your laptop:

 Host devtunnel
 Hostname hgwdev.soe.ucsc.edu
 User YOURUSERNAME
 # local forward to gbib running on hgwdev, using local port 4321 to avoid clashing with the gbib running on your own machine
 LocalForward 127.0.0.1:4321 127.0.0.1:1234
 # local forward to devbox running on hgwdev
 LocalForward 127.0.0.1:1236 127.0.0.1:1236

Commented walkthrough of a GBib specific bugfix using the devbox

open a new terminal on your laptop and login to hgwdev with the tunnel:

 ssh devtunnel

Now start browserbox-dev as qa:

 ssh qa
 devStart

Wait a little bit then ssh into the devbox:

 ssh devbox
 cd kent/src
 git remote set-url origin ssh://YOURUSERNAME@hgwdev.cse.ucsc.edu/data/git/kent.git/
 git pull
 git checkout <THE BRANCH YOU WANT TO WORK ON, e.g. beta or master>
 make docs
 make cgi -j8 COREDUMP=1

setup the tunnel on port 1236 to hgwdev from your laptop (do this in a 2nd terminal):

 ssh devtunnel

You can now test the CGIs on the devbox by going to this URL

 http://127.0.0.1:1236

You can browse around on this site. It's slightly less configured than gbib (colors, icons etc) but the CGIs are the same. As soon as you go to the URL that fails in your browser e.g.:

 http://127.0.0.1:1236/cgi-bin/hgFileUi?db=hg19&g=wgEncodeAwgSegmentation

.. there will be no error message, but the CGI crashed with "abort()" and it will core dump.

This should produce a new core file on the devbox:

 ll /data/trash/core/core
 -rw------- 1 www-data www-data 73338880 Oct  3 08:07 /data/trash/core/core

Run gdb with the core file (the sudo password is "browser"):

 sudo gdb /usr/local/apache/cgi-bin/hgFileUi /data/trash/core/core

A typical solution is to replace cloneString with hReplaceGbdb at places where files in /gbdb are opened. Fix the bug and test after recompile.

Then don't forget to checkout the master branch and commit there (!):

 git checkout master
 git add hg/lib/fileUi.c
 git commit
 git push
 

Now you can file a build patch with this commit id.

testing of alpha binaries on a real gbib

To do a more realistic test, the alpha binaries can also be put into the final browserbox.

To copy the alpha binaries into the browserbox: Copy them from the devbox to alpha distribution rsync server directory (this command is defined in qateam's .bashrc) by running this command as qateam on hgwdev while the devbox is running (this will ssh into the devbox and then copy the binaries onto hgwdev):

 boxUpdateAlpha

Now start the gbib running on hgwdev and pull the alpha binaries from the rsync server:

 boxStart # will show an error message if already running
 ssh box
 updateBrowser alpha

Or you can run on the gbib on your own machine and type:

 updateBrowser alpha

You can then test the alpha CGIs in the "real" gbib environment via 127.0.0.1:4321 (gbib on hgwdev) or 127.0.0.1:1234 (your own gbib).

Don't forget to start the ssh tunnel to hgwdev (ssh devtunel) if you're using the gbib on hgwdev.

Pushing your changes to hgwdev from the devbox

Earlier on this page, we did the git pull/push directly from the devbox onto hgwdev. Here are some more details on this.

Direct git, as above - easier: set username for kent repo:

 cd kent
 git remote set-url origin ssh://YOURUSERNAME@hgwdev.cse.ucsc.edu/data/git/kent.git/
 git push
 # You can also pull from the public repo
 git remote set-url origin git://genome-source.cse.ucsc.edu/kent.git

Indirect: simply rsync or scp to copy into your own homedir on hgwdev and commit there:

 rsync file USER@hgwdev:kent/PATH/FILE
 logout