Gbib updates

The Gbib contains a script /root/updateBrowser.sh. It updates itself, the directories gbdb, cgi-bin, htdocs, mysql databases and a few other files across the gbib's filesystem (gbib "push" directory).

To main aim of this update process is to keep the CGIs, htdocs, gbdb and mysql files current with the version on hgdownload on all gbibs in the wild. A secondary aim is - during alpha or beta test time and only at UCSC by QA - to install the the alpha/beta cgi versions prepared on hgwdev, using the same script, but with a parameter, "alpha" or "beta".

The script is run from various places:

• manually, via the script /home/browser/updateBrowser, which uses sudo without a password prompt. You can specify a parameter here, "beta" or "alpha" (see below)
• upon login on the main console, without a parameter. This is configured in .bashrc of ~browser.
• every 4 minutes via a cronjob, without a parameter. This cronjob has a delay of 120 secs to make sure the main console job makes it first. It also has a delay of a random number of seconds < 30 to spread out the hits to our rsync server.

The script goes through these steps:

1. It first checks if it's run as root. It stops if not.
2. It then checks if it's already running. If it seems to be already running more than twice, it stops (to see why the check is "running more than twice", see step 4).
3. The script then compares the time of http://hgdownload.cse.ucsc.edu/gbib/lastUpdate via a HTTP HEAD request and checks if the URL is more recent than the time of the file /root/lastUpdateTime.flag. If not, it stops. After a successful update, it sets the time of /root/lastUpdateTime.flag to the current time.
4. It will then download a new copy of itself from the source server. This server is hgdownload by default, or hgwdev if run with parameter "alpha" or "beta". It will run this updated copy.
5. It checks and stops if a hgMirror job is running.
6. It updates the CGIs and htdocs either via rsync from hgdownload or by downloading/extracting a tarball from hgwdev.
7. It updates the /gbdb files
   1. note: /gbdb, mysql and hgcentral always come from hgdownload because cluster-admon doesn't want to have an open rsyncd on hgwdev and QA might be working off campus
8. It updates the mysql files
9. It updates hgcentral
10. It copies all other files (gbib.css, gbib icons, updated config files, if needed: new ubuntu packages, if needed: other gbib specific patches) from hgdownload or hgwdev from their "gbib/push" directories into the "/" directory of gbib
11. it modifies the main and the left side menu (remove visiGene, neandertal, galaxy, etc), and rewrites the main page (e.g. replace UCSC Genome browser with "UCSC Genome Browser in a Box")
12. it modifies the contacts page to have a link to UCSC, not to the gbib
13. it makes sure the tableList is not in our local copy of hgcentral (the table list in hgcentral might be outdated on hgdownload and we certainly don't need it, as this DB is always local)
14. it hides the conservation and retroAli tracks in hg19 trackDb
15. it fixes all mysql tables in hg19, hgFixed and hgcentral, as they are often in a crashed state on hgdownload (as the server's tables might not be flushed when the copy is made by Jorge's script)
16. it runs hgMirror to let it do "postRsyncUpdates" (these are part of hgMirror, because they also have to be run after every hgMirror download)
    1. hgMirror goes over all installed databases in mysql and gets a list of tables in there
    2. To improve performance of search, hide some tracks from hgTracks feature searchSpec (e.g. the dozens of outdated gencode tracks)
    3. Tries to find all conservation tracks and hides them
    4. To improve performance, hide some other tracks (e.g. intronEst)
    5. update the local tableList with protected tracks that exist only in gbib, but not on hgdownload. At the moment, this is only HGMD and OMIM.
17. It runs mysqlcheck on all mysql databases and tables (because they are often in a crashes state on hgdownload)
18. it touches /root/lastUpdateTime.flag