Genome-preview machine

In early 2011, we created a new machine called genome-preview, which is a mirror of our development machine: genome-test.

It is here: http://genome-preview.soe.ucsc.edu/

It has these warnings on the home page:

WARNING: This is our preview site. This website is a weekly mirror of our internal development server for public access. Data and tools here are under construction, have not been quality reviewed, and are subject to change at any time. We provide this site for early access, with the warning that it is less available and stable than our public site. For high-quality reviewed annotations on our production server, visit http://genome.ucsc.edu.

In general, we do not push updates/files/tables/patches to this machine by hand. It is generally understood that it will sometimes be out of commission, as it is a mirror of a development machine.

The impetus behind this machine was to allow early access to the ENCODE data by ENCODE consortia members.

Automated Updates

The genome-preview machine is updated on a weekly basis like so: On Friday Mornings (5:27 am) a build crontab makes the CGIs for genome preview. On Friday nights (11:33 pm), the following stuff is copied from hgwdev:

• mysql tables (excluding mysql, customTrash, and other things)
• hgcentraltest tables (excluding userDb, sessionDb, namedSessionDb and hubStatus)
• CGIs
• htdocs

Most of the stuff on /gbdb comes from backups of the /hive/data and /hive/groups.

• /hive/data is backed up on Fridays at 5:22 am
• /hive/groups is backed up on Wednesdays at 5:22 am.

Note that this may cause some inconsistencies if new data is added on Thursday to /hive/groups and accessed by new CGIs, since the data will not be available on the backups until the following Wednesday.

Updating Genome-Preview with pushDevToPreview

PREPARATION from your hgwdev kent repository clean your tree and update your libraries.

If your libraries are out of date or your repository is not clean you can run into errors when making your CGIs. Be sure your branch is current. If you are seeing errors on the build process please ask for help.

NOW from your hgwdev kent repository complete the below steps to update CGIs/js/style files.

Please make sure that you do not have stuff that is not checked in and your git repo working directories are clean so you will not get merge conflicts etc.

Automated Method

Run one script to do it all. Be sure to have the genecats repo installed in your home dir. Update it first.

pushd ~/genecats
git pull
popd
~/genecats/qa/testTools/previewUpdate

Manual Method

cd ~/kent/src
git pull
make clean > /dev/null
cd ~/kent/src/hg/lib
make

Note: you'll first need to update the js files for preview at: ls -l /usr/local/apache/htdocs/js/preview/

cd ~/kent/src/hg/js
make USER=preview

Note: you'll first need to update the style files for hgwdev at: ls -lrt /usr/local/apache/htdocs/style

cd ~/kent/src/hg/htdocs/style
make alpha

Note: you'll first need to update the CGIs for preview at: ls -lrt /usr/local/apache/cgi-bin-preview/

cd ~/kent/src
USER=preview make cgi

Be sure to login as qateam on hgwdev before starting any of these operations.

 ssh qateam@hgwdev

If there is a need to sync the CGIs on hgwdev to genome-preview the pushDevToPreview script

pushDevToPreview -h
Usage: pushDevToPreview [-chiqvy] 

  -c | --check    : Don't do the push, just say what would be pushed.
  -h | --help     : Print this message.
  -i              : Interactive mode. Ask me before doing any command.
  -q | --quiet    : Quiet mode. Don't print anything unless necessary.
  -v | --verbose  : Increase verbosity.
  -y | --noprompt : Just do it, don't ask me.

Once all your files are updated, you should run the command with the -c --check option to see what will be updated: pushDevToPreview -c

If things look pretty good, then you are ready to run the command, with the -i interactive mode so you can see what commands are being run:

 pushDevToPreview -i

Pushing files from hgwdev to genome-preview

Do you wish to proceed? (y/[n]): y

Doing directory /usr/local/apache/cgi-bin-preview/ between hgwdev and genome-preview


The command to be executed is /usr/bin/rsync -rlptDi --delete --chown=qateam:protein --exclude=hg.conf*  /usr/local/apache/cgi-bin-preview/ genome-preview:/usr/local/apache/cgi-bin-preview/
...
...

Finally you'll check on genome-preview if the changes are updated.