Gbib updates

The Gbib contains a script /root/updateBrowser.sh. It updates itself, the directories gbdb, cgi-bin, htdocs, mysql databases and a few other files across the gbib's filesystem (gbib "push" directory).

To main aim of this update process is to keep the CGIs, htdocs, gbdb and mysql files current with the version on hgdownload on all gbibs in the wild. A secondary aim is - during alpha or beta test time and only at UCSC by QA - to install the the alpha/beta cgi versions prepared on hgwdev, using the same script, but with a parameter, "alpha" or "beta".

The script is run from various places:

• manually, via the script /home/browser/updateBrowser, which uses sudo without a password prompt. You can specify a parameter here, "hgwdev" (see below)
• upon login on the main console, without a parameter. This is configured in .bashrc of ~browser.
• every 4 minutes via a cronjob, without a parameter. This cronjob has a delay of 120 secs to make sure the main console job makes it first. It also has a delay of a random number of seconds < 30 to spread out the hits to our rsync server.

The script goes through these steps:

1. It first checks if it's run as root. It stops if not.
2. It then checks if it's already running. If it seems to be already running more than twice, it stops (to see why the check is "running more than twice", see step 4).
3. It checks if there is an internet connection to hgdownload. It stops if not.
4. The script then compares the time of http://hgdownload.soe.ucsc.edu/gbib/lastUpdate via a HTTP HEAD request and checks if the URL is more recent than the time of the file /root/lastUpdateTime.flag. If not, it stops. (See below: after the update, it sets the time of /root/lastUpdateTime.flag to the current time)
5. It will then download a new copy of itself from the source server. This server is hgdownload by default, or hgwdev if run with parameter "alpha" or "beta". It will run this updated copy.
6. It checks and stops if a hgMirror job is running.
7. It updates the CGIs and htdocs either via rsync from hgdownload or via rysnc from hgwdev (see below)
8. It updates the /gbdb files
   1. note: /gbdb, mysql and hgcentral always come from hgdownload because cluster-admon doesn't want to have an open rsyncd on hgwdev/hgwbeta and QA might be working off campus
9. It updates the mysql files
10. It updates hgcentral
11. It rsyncs all other files (gbib.css, gbib icons, updated config files, if needed: new ubuntu packages, if needed: other gbib specific patches) from hgdownload (or hgwdev) from their "gbib/push" directories into the "/" directory of gbib
12. it modifies the main and the left side menu (remove visiGene, neandertal, galaxy, etc), and rewrites the main page (e.g. replace UCSC Genome browser with "UCSC Genome Browser in a Box")
13. it modifies the contacts page to have a link to UCSC, not to the gbib
14. it makes sure the tableList is not in our local copy of hgcentral (the table list in hgcentral might be outdated on hgdownload and we certainly don't need it, as this DB is always local)
15. it hides the conservation and retroAli tracks in hg19 trackDb
16. it fixes all mysql tables in hg19, hgFixed and hgcentral, as they are often in a crashed state on hgdownload (as the server's tables might not be flushed when the copy is made by Jorge's script)
17. it runs hgMirror to let it do "postRsyncUpdates" (these are part of hgMirror, because they also have to be run after every hgMirror download)
    1. hgMirror goes over all installed databases in mysql and gets a list of tables in there
    2. To improve performance of search, hide some tracks from hgTracks feature searchSpec (e.g. the dozens of outdated gencode tracks)
    3. Tries to find all conservation tracks and hides them
    4. To improve performance, hide some other tracks (e.g. intronEst)
    5. update the local tableList with protected tracks that exist only in gbib, but not on hgdownload. At the moment, this is only HGMD and OMIM.
18. It runs mysqlcheck on all mysql databases and tables (because they are often in a crashed state on hgdownload)
19. it touches /root/lastUpdateTime.flag

updateBrowser hgwdev USER RELEASE

• this command is meant to be used by QA or developers to rsync manually a certain release into the gbib
• note that this only copies the CGIs/htdocs, not gbdb or the public mysql server. These always stay.
• it requires two parameters: the hgwdev username and the release name
• release can be "alpha", "beta" or a username like "braney", which means the directories cgi-bin, cgi-bin-beta or cgi-bin-braney respectively. It also copies the corresponding htdocs files
• example: updateBrowser hgwdev max alpha will copy the alpha release into the current gbib. It will ask three times for the max@hgwdev password.
• example: updateBrowser hgwdev max braney will copy braney's CGIs into the current gbib, asking three times for the max@hgwdev password.
• to avoid having to type your password three times:
  • ssh-key -t dsa (enter, enter)
  • scpdsa USERNAME@dev
  • DO NOT DO THIS ON THE GBIB THAT IS RUNNING ON HGWDEV! The key might leak and anyone will be able to login to hgwdev!
• this command does not rsync gbdb, or mysql or anything else than the CGIs, the htdocs and the push files
• push files for alpha and beta mode come from the push directory on hgwdev, /usr/local/apache/htdocs/gbib/push, otherwise from htdownload's push directory.
• this command also touches the lastUpdate flag file. This means no updates will happen until next Sunday, but on the next Sunday, the CGIs and htdocs will get overwritten with the new versions.
  • This explains why if you have a beta candidate, after the beta week, the Sunday touch of hgdownload files will update the date of lastUpdate curl -i http://hgdownload.soe.ucsc.edu/gbib/lastUpdate and trigger your beta to update to the version on hgdownload.
  • When this happens, say you are looking at the beta candidate on the Monday right before the release (and after beta week), you will need to do updateBrowser hgwdev yourUserName beta to restore your GBiB to the beta CGI status.