Push Shepherd Responsibilities

Set Up Notifications

Before starting as push shepherd, you will need to make sure you are on the list for notifications:

• Email Build Meister to be added on email notifications for robots, makes, and buildTableDescrptions.pl output.
• Email Ann to be added as a watcher on Code Review parent ticket.

Week 1 (Preview Build 1)

• Create CGI Build Ticket (CGI chatter) in Redmine under the *UCSC* umbrella project:
  • Set the Status as "Preview 1".
  • Set the Target version.
  • Add the Build Meister and QA Team as watchers.

Week 2 (Preview Build 2)

• Change CGI Build Ticket status to "Preview 2".
• Make sure all build related issues are tested on dev before build happens on beta.
• Send a reminder to QA that trackDb changes should be documented by engineers DbDoc DbHub.

Week 3 (Final Build)

• Check the timestamps of the CGI binaries on hgwbeta: /usr/local/apache/cgi-bin/
• Change CGI Build Ticket status to "Final Build".
• Review error logs for the robots:

1. hgNear - sends an email with results
2. hgTables - sends an email with results
3. LiftOverTest - check by hand: /cluster/bin/build/scripts/logs/LiftOverTest-v$VERSION_NUMBER.log

• If there are any errors reported then go to the following location and check the logs: /cluster/bin/build/scripts/logs/
• Make sure of the following:
  • All related issues (Bugs, Features) have status of "CGI-ready" or "CGI-ready-open-issues".

A Few Days Before The Push

• Make sure of the following:
  • All Build Patch requests have been fulfilled (see Build Patch section below).
  • All Code Reviews are closed.
  • Everyone has reported their "done testing" messages (if any) in the CGI Build ticket.
• If necessary, send out a status update (or reminder) email 1-2 days before release day.
• The day before the push, send an email notice to browser-staff letting them know that tomorrow is a push day. Something along these lines:

   Just a reminder that tomorrow is a CGI push day. If you have big code changes included in 
   this release please be available in case something goes wrong with the push of your changes. 
   QA will be starting the push around 1:30pm.

Day of Push

• Send a reminder to push-request that today is CGI push day.

Push to hgw0 only

• Log on to hgw0 *before* sending push request: ssh qateam@hgw0
• View apache error log at:

  $ cd /usr/local/apache/logs/
  $ less -s error_log

• use shift + g to go to bottom of page
• use shift + f to follow incoming errors

*Update*: To view the error log *without* Hiram's CGI_TIME entries (for background info see: http://redmine.soe.ucsc.edu/issues/10081):

  $ tail -f error_log | grep -v CGI_TIME

• Send out push request.
• After push has gone through, monitor error log while QA does their testing:
  • See this wiki page for examples of errors/warnings that can be ignored here
  • If there are any unusual errors, use whois $ip_address or host $ip_address to see where errors are coming from, or vi error_log to determine if error occurred before pushing to hgw0.
• Wait to hear from QA on how their CGIs look on hgw0 (usually done via Skype chat with 'done testing, everything looks fine' messages).

Push to hgwN only

• SSH to hgwN machine before sending out push request.
• Send out push request.
• Monitor error log for a short while to make sure no new errors occur under the load.

Push to rest of the RR, hgwbeta-public, and euronode

• Send out push request.
• Check hgTracks title version on each machine to ensure the CGI's have made it to the world.
• Send email to Build Meister to let him/her know the CGIs are on the RR (send wrap-up email to Build Meister).

Day after Push

• Make sure all related issues have been closed out.
• Close Build Patch tickets.
• Change CGI Build Ticket status to "Released".

Build Patches

The general rule is that we don't approve Build Patches within 24 hours of pushing the CGIs. We can make exceptions, but this depends on which CGIs are affected and how much QA will have to re-test. Our Build Patch process can be found here.

tableDescriptions

Along with the buildmeister, make sure the source of the errors from buildTableDescrptions.pl are tracked down and corrected. Usually the person who checked in the change that is causing the problem needs to be notified.

The type of error that shows up is usually either kind of cryptic, like this:

   hgsql error for hg17.tableDescriptions.sql at /cluster/home/build/kent/src/test/buildTableDescriptions.pl line 449.

or kind of straightforward, like this:

   Duplicate autoSql def for table bed12UniProtMut (/cluster/home/build/kent/src/hg/utils/uniprotMutations/bed12UniProtAnnot.as vs. /cluster/home/build/kent/src/hg/utils/uniprotMutations/bed12UniProtMut.as) at /cluster/home/build/kent/src/test/buildTableDescriptions.pl line 203, <F> line 24.

(someone entered a second set of table descriptions for bed12UniProtMut, and it says what files the problem is in).