|
|
| (One intermediate revision by one other user not shown) |
| Line 1: |
Line 1: |
| This page is for developers, it's not relevant for QA. The QA page is [[Genome_Browser_in_a_box_config]]. | | This page is no longer maintained. |
|
| |
|
| Box-specific development can happen on the devbox, a development version of the browserbox with git, gcc etc. For testing during development, a special command on the box can pull in the CGIs that were compiled on the devbox. The devbox is necessary as otherwise the gbib image has to allow development which adds around 2gb of packages most users don't ever need. In addition, it avoids the
| | See http://genomewiki.ucsc.edu/genecats/index.php/Gbib_development |
| potential problem of our developer passwords leaking out through tempfiles or the swapspace.
| |
| | |
| The browserbox-dev VM, alias "the devbox", listens to http on 1236 and for ssh on 1237.
| |
| | |
| On hgwdev, all virtual machines live under ~qateam/VirtualBox VMs/. The central virtualbox xml config file where they are registered is ~/.config/VirtualBox/VirtualBox.xml
| |
| | |
| To start the devbox, you need to login as qateam onto hgwdev.
| |
| | |
| From hgwdev as qateam, you can then connect to the box with "ssh devbox". This works, because of qateam's .ssh/config file:
| |
| | |
| Host devbox
| |
| Hostname localhost
| |
| Port 1237
| |
| User browser
| |
| | |
| You can start and stop browserbox-dev with the aliases devStart and devStop as qateam.
| |
| | |
| == Setup your laptop ==
| |
| to connect to the devbox, add this to .ssh/config on your laptop:
| |
| | |
| Host devtunnel
| |
| Hostname hgwdev.soe.ucsc.edu
| |
| User YOURUSERNAME
| |
| # local forward to gbib running on hgwdev, using local port 4321 to avoid clashing with the gbib running on your own machine
| |
| LocalForward 127.0.0.1:4321 127.0.0.1:1234
| |
| # local forward to devbox running on hgwdev
| |
| LocalForward 127.0.0.1:1236 127.0.0.1:1236
| |
| | |
| == Commented walkthrough of a GBib specific bugfix using the devbox ==
| |
| | |
| open a new terminal on your laptop and login to hgwdev with the tunnel:
| |
| ssh devtunnel
| |
| | |
| Now start browserbox-dev as qa:
| |
| ssh qa
| |
| devStart
| |
|
| |
| Wait a little bit then ssh into the devbox:
| |
| ssh devbox
| |
| cd kent/src
| |
| git remote set-url origin ssh://YOURUSERNAME@hgwdev.cse.ucsc.edu/data/git/kent.git/
| |
| git pull
| |
| git checkout <THE BRANCH YOU WANT TO WORK ON, e.g. beta or master>
| |
| make docs
| |
| make cgi -j8 COREDUMP=1
| |
| | |
| setup the tunnel on port 1236 to hgwdev from your laptop (do this in a 2nd terminal):
| |
| ssh devtunnel
| |
| | |
| You can now test the CGIs on the devbox by going to this URL
| |
| http://127.0.0.1:1236
| |
| | |
| You can browse around on this site. It's slightly less configured than gbib (colors, icons etc) but the CGIs are the same. As soon as you go to the URL that fails in your browser e.g.:
| |
| http://127.0.0.1:1236/cgi-bin/hgFileUi?db=hg19&g=wgEncodeAwgSegmentation
| |
| .. there will be no error message, but the CGI crashed with "abort()" and it will core dump.
| |
| | |
| This should produce a new core file on the devbox:
| |
| ll /data/trash/core/core
| |
| -rw------- 1 www-data www-data 73338880 Oct 3 08:07 /data/trash/core/core
| |
| | |
| Run gdb with the core file (the sudo password is "browser"):
| |
| sudo gdb /usr/local/apache/cgi-bin/hgFileUi /data/trash/core/core
| |
|
| |
| A typical solution is to replace cloneString with hReplaceGbdb at places where files in /gbdb are opened. Fix the bug and test after recompile.
| |
| | |
| Then don't forget to checkout the master branch and commit there (!):
| |
| git checkout master
| |
| git add hg/lib/fileUi.c
| |
| git commit
| |
| git push
| |
|
| |
| Now you can file a build patch with this commit id.
| |
| | |
| == testing of alpha binaries on a real gbib ==
| |
| | |
| To do a more realistic test, the alpha binaries can also be put into the final browserbox.
| |
| | |
| To copy the alpha binaries into the browserbox: Copy them from the devbox to alpha distribution rsync server directory (this command is defined in qateam's .bashrc) by running this command as qateam on hgwdev while the devbox is running (this will ssh into the devbox and then copy the binaries onto hgwdev):
| |
| boxUpdateAlpha
| |
| | |
| Now start the gbib running on hgwdev and pull the alpha binaries from the rsync server:
| |
| boxStart # will show an error message if already running
| |
| ssh box
| |
| updateBrowser alpha
| |
| | |
| Or you can run on the gbib on your own machine and type:
| |
| updateBrowser alpha
| |
| | |
| You can then test the alpha CGIs in the "real" gbib environment via 127.0.0.1:4321 (gbib on hgwdev) or 127.0.0.1:1234 (your own gbib).
| |
| | |
| Don't forget to start the ssh tunnel to hgwdev (ssh devtunel) if you're using the gbib on hgwdev.
| |
| | |
| == Gbib performance and internet latency ==
| |
| | |
| The Gbib is running on users' machines and users can be very far from UCSC, up to 200msec in TCP time
| |
| | |
| To simulate this situation, you can try the "netSlow" command on the gbib. The apache error log /var/log/apache2/error.log should tell you what the slow statements are, as the JKSQL_PROF profiling is enabled in hg.conf
| |
| | |
| If a track does a SELECT statment for each feature, this will make it very slow. As a remedy, you can
| |
| * copy the mysql table onto the gbib (mysqldump + scp + hgsql hg19< dumpfile)
| |
| ** weekly rsync will keep it updated then in the future, see the /root/updateBrowser shell script
| |
| * or modify the source code to reduce the number of select statements.
| |
|
| |
| == Pushing your changes to hgwdev from the devbox ==
| |
| | |
| Earlier on this page, we did the git pull/push directly from the devbox onto hgwdev. Here are some more details on this.
| |
|
| |
| Direct git, as above - easier: set username for kent repo:
| |
| cd kent
| |
| git remote set-url origin ssh://YOURUSERNAME@hgwdev.cse.ucsc.edu/data/git/kent.git/
| |
| git push
| |
| # You can also pull from the public repo
| |
| git remote set-url origin git://genome-source.cse.ucsc.edu/kent.git
| |
| | |
| | |
| Indirect: simply rsync or scp to copy into your own homedir on hgwdev and commit there:
| |
| rsync file USER@hgwdev:kent/PATH/FILE
| |
| logout
| |
