Push Shepherd Responsibilities

From Genecats
Jump to navigationJump to search

The role of Push Shepherd is important, timely, and process-oriented. The Push Shepherd should be on alert for all build-related emails especially during "beta week", and the days around the release. The Push Shepherd is responsible for ensuring the smooth and timely progression of the release candidate from dev to beta to the RR so that the other QAers can concentrate on their testing.

Example task checklist

Set Up Notifications

Before starting as push shepherd, you will need to make sure you are on the list for notifications:

  • Email Ann to be added as a watcher on Code Review parent ticket.

Week 1 (Preview Build 1)

  • Create CGI Build Ticket (CGI chatter) in Redmine under the *UCSC* umbrella project:
    • Set the Status as "Preview 1".
    • Set the Target version.
    • Add the Build Meister and QA Team as watchers.

Week 2 (Preview Build 2)

  • Change CGI Build Ticket status to "Preview 2".
  • Make sure all build-related issues are tested on dev before build happens on beta.
  • Review error logs for the robots:
  1. hgTables - sends an email with results
  • Send a reminder to QA that trackDb changes should be documented by engineers DbDoc DbHub.

Week 3 (Final Build)

  • Check the timestamps of the CGI binaries on hgwbeta: /usr/local/apache/cgi-bin/
  • Change CGI Build Ticket status to "Final Build".
  • Review error logs for the robots:
  1. hgNear - sends an email with results
  2. hgTables - sends an email with results
  3. LiftOverTest - check by hand: /cluster/bin/build/scripts/logs/LiftOverTest-v$VERSION_NUMBER.log
  • If there are any errors reported then go to the following location and check the logs: /cluster/bin/build/scripts/logs/
  • Make sure of the following:
    • All related issues (Bugs, Features) have status of "CGI-ready" or "CGI-ready-open-issues".

A Few Days Before The Push

  • Make sure of the following:
    • All Build Patch requests have been fulfilled (see Build Patch section below).
    • All Code Reviews are closed.
    • Everyone has reported their "done testing" messages (if any) in the CGI Build ticket.
  • If necessary, send out a status update (or reminder) email 1-2 days before release day.
  • The day before the push, send an email notice to browser-staff letting them know that tomorrow is a push day. Something along these lines:
   Just a reminder that tomorrow is a CGI push day. If you have big code changes included in 
   this release please be available in case something goes wrong with the push of your changes. 
   QA will be starting the push around 1:30pm.
  • Query the QA Team to see if there are any extra files that need to be pushed in conjunction with the software.
  • Consider if there are features that should not sit alone too long on only one of the RR machines. That is, sometimes there are features that need to be present on all of the RR machines (e.g. new login features or new hg.conf settings) and will actually cause problems if they only exist on one of the machines.

Day of Push

  • Send a reminder to push-request that today is CGI push day.

Create a Skype Chat around 1:15pm

It should include QA members individually, possibly Ann and Brian. It should be named "vNNN CGI Release"

Push to hgw0 only

  • Log on to hgw0 *before* sending push request: ssh qateam@hgw0
  • View apache error log at:
  $ cd /usr/local/apache/logs/
  $ less -s error_log
  • use shift + g to go to bottom of page
  • use shift + f to follow incoming errors

*Update*: To view the error log *without* Hiram's CGI_TIME entries (for background info see: http://redmine.soe.ucsc.edu/issues/10081):

  $ tail -F logs/error_log | grep -v "CGI_TIME\|trackLog"
  • Send out push request.
  • After push has gone through, monitor error log while QA does their testing:
    • See this wiki page for examples of errors/warnings that can be ignored here
    • If there are any unusual errors, use whois $ip_address or host $ip_address to see where errors are coming from, or vi error_log to determine if error occurred before pushing to hgw0.
  • Wait to hear from QA on how their CGIs look on hgw0 (usually done via Skype chat with 'done testing, everything looks fine' messages).

Push to hgwN only

  • SSH to hgwN machine before sending out push request.
  • Send out push request.
  • Monitor error log for a short while to make sure no new errors occur under the load.

Push to rest of the RR, hgwbeta-public, and euronode

  • Send out push request.
  • Check hgTracks title version on each machine to ensure the CGI's have made it to the world.
  • Send email to Build Meister to let him/her know the CGIs are on the RR (send wrap-up email to Build Meister).

Day after Push

  • Make sure all related issues have been closed out.
  • Close Build Patch tickets.
  • Change CGI Build Ticket status to "Released".

Build Patches

The general rule is that we don't do Build Patches within the last 24 hours before a scheduled push of the CGIs. We can make exceptions, but this depends on which CGIs are affected and how much QA will have to re-test. In extreme circumstances, we can do the patch, then delay the CGI release by a day. Build Patches have a priority of Urgent; until some resolution is made, they should be a top priority for both QA and Developers. Our Build Patch process can be found here.

tableDescriptions

Along with the build meister, make sure the source of the errors from buildTableDescrptions.pl are tracked down and corrected. Usually the person who checked in the change that is causing the problem needs to be notified.

The type of error that shows up is usually either kind of cryptic, like this:

   hgsql error for hg17.tableDescriptions.sql at /cluster/home/build/kent/src/test/buildTableDescriptions.pl line 449.

or kind of straightforward, like this:

   Duplicate autoSql def for table bed12UniProtMut (/cluster/home/build/kent/src/hg/utils/uniprotMutations/bed12UniProtAnnot.as vs. /cluster/home/build/kent/src/hg/utils/uniprotMutations/bed12UniProtMut.as) at /cluster/home/build/kent/src/test/buildTableDescriptions.pl line 203, <F> line 24.

(someone entered a second set of table descriptions for bed12UniProtMut, and it says what files the problem is in).

Example push emails

=============================
1. Push v439 CGIs to hgw0
=============================

Pushers,

Please push these CGIs:

cartDump cartReset das gar hgApi hgBeacon hgBlat hgc hgCollection
hgConvert hgCustom hgEncodeApi hgEncodeDataVersions hgEncodeVocab
hgFileSearch hgFileUi hgGateway hgGene hgGeneGraph hgGenome
hgGtexTrackSettings hgHubConnect hgIntegrator hgLinkIn hgLiftOver hgLogin
hgMenubar hgMirror hgNear hgPal hgPcr hgPhyloPlace hgPublicSessions
hgRenderTracks hgSearch hgSession hgSuggest hgTables hgTracks hgTrackUi
hgUserSuggestion hgVai hgVisiGene hubApi phyloPng

and these configuration files:

/usr/local/apache/cgi-bin/all.joiner
/usr/local/apache/cgi-bin/extTools.ra
/usr/local/apache/cgi-bin/greatData/*
/usr/local/apache/cgi-bin/hgCgiData/*
/usr/local/apache/cgi-bin/hgGeneData/*
/usr/local/apache/cgi-bin/hgNearData/*
/usr/local/apache/cgi-bin/hgcData/*
/usr/local/apache/cgi-bin/loader/*
/usr/local/apache/cgi-bin/lsSnpPdbChimera.py
/usr/local/apache/cgi-bin/visiGeneData/*
/usr/local/apache/cgi-bin/pyLib/*

from hgwbeta -> hgw0 only


Please rsync --delete these directories on beta:

/usr/local/apache/htdocs/js/*
/usr/local/apache/htdocs/style/*

to the same location on hgw0.

Reason: releasing v439 CGIs to the test RR machine.

Thanks,
$you

=============================
2.Push v439 CGIs to hgw1
=============================

Pushers,

Please push these CGIs:

cartDump cartReset das gar hgApi hgBeacon hgBlat hgc hgCollection
hgConvert hgCustom hgEncodeApi hgEncodeDataVersions hgEncodeVocab
hgFileSearch hgFileUi hgGateway hgGene hgGeneGraph hgGenome
hgGtexTrackSettings hgHubConnect hgIntegrator hgLinkIn hgLiftOver hgLogin
hgMenubar hgMirror hgNear hgPal hgPcr hgPhyloPlace hgPublicSessions
hgRenderTracks hgSearch hgSession hgSuggest hgTables hgTracks hgTrackUi
hgUserSuggestion hgVai hgVisiGene hubApi phyloPng

and these configuration files:

/usr/local/apache/cgi-bin/all.joiner
/usr/local/apache/cgi-bin/extTools.ra
/usr/local/apache/cgi-bin/greatData/*
/usr/local/apache/cgi-bin/hgCgiData/*
/usr/local/apache/cgi-bin/hgGeneData/*
/usr/local/apache/cgi-bin/hgNearData/*
/usr/local/apache/cgi-bin/hgcData/*
/usr/local/apache/cgi-bin/loader/*
/usr/local/apache/cgi-bin/lsSnpPdbChimera.py
/usr/local/apache/cgi-bin/visiGeneData/*
/usr/local/apache/cgi-bin/pyLib/*

from hgwbeta -> hgw1 ONLY

Please rsync --delete these directories on beta:

/usr/local/apache/htdocs/js/*
/usr/local/apache/htdocs/style/*

to the same location on hgw1.

Reason: releasing v439 CGIs to one RR machine.

Thanks,
$you

=============================
3. Push v439 CGIs to RR
=============================

Pushers,

Please push these CGIs:

cartDump cartReset das gar hgApi hgBeacon hgBlat hgc hgCollection
hgConvert hgCustom hgEncodeApi hgEncodeDataVersions hgEncodeVocab
hgFileSearch hgFileUi hgGateway hgGene hgGeneGraph hgGenome
hgGtexTrackSettings hgHubConnect hgIntegrator hgLinkIn hgLiftOver hgLogin
hgMenubar hgMirror hgNear hgPal hgPcr hgPhyloPlace hgPublicSessions
hgRenderTracks hgSearch hgSession hgSuggest hgTables hgTracks hgTrackUi
hgUserSuggestion hgVai hgVisiGene hubApi phyloPng

and these configuration files:

/usr/local/apache/cgi-bin/all.joiner
/usr/local/apache/cgi-bin/extTools.ra
/usr/local/apache/cgi-bin/greatData/*
/usr/local/apache/cgi-bin/hgCgiData/*
/usr/local/apache/cgi-bin/hgGeneData/*
/usr/local/apache/cgi-bin/hgNearData/*
/usr/local/apache/cgi-bin/hgcData/*
/usr/local/apache/cgi-bin/loader/*
/usr/local/apache/cgi-bin/lsSnpPdbChimera.py
/usr/local/apache/cgi-bin/visiGeneData/*
/usr/local/apache/cgi-bin/pyLib/*

from hgwbeta -> hgwbeta-public AND hgw1-2 AND genome-euro/asia (note
that they are already on hgw0 and hgw1)

Please rsync --delete these directories on beta:

/usr/local/apache/htdocs/js/*
/usr/local/apache/htdocs/style/*

to the same location on the RR machines.

Please also sync to these other two directories to hgwbeta-public ONLY:

/usr/local/apache/htdocs/js-public/*
/usr/local/apache/htdocs/style-public/*

Reason: releasing v439 CGIs to hgwbeta-public, rest of the RR
machines, and genome-euro/asia

Thanks,
$you

=============================
4. v439 build wrap up
=============================

Hi Build Meister,

v430 CGIs are now on the RR and ready for the wrap-up.

Thanks,
$you

Location of older CGIs

  1. ssh qateam@hgwbeta
  2. ls /data/apache/
  3. see: cgi-bin-public.v###
  4. ls /data/apache/htdocs/
  5. see js-public.v### and style-public.v###