Push Shepherd Responsibilities

From Genecats
Revision as of 17:02, 21 February 2018 by Cath (talk | contribs) (Adding example checklist for others to copy, adding example push emails)
Jump to navigationJump to search

Example task checklist

Set Up Notifications

Before starting as push shepherd, you will need to make sure you are on the list for notifications:

  • Email Build Meister to be added on email notifications for robots, makes, and buildTableDescrptions.pl output.
  • Email Ann to be added as a watcher on Code Review parent ticket.

Week 1 (Preview Build 1)

  • Create CGI Build Ticket (CGI chatter) in Redmine under the *UCSC* umbrella project:
    • Set the Status as "Preview 1".
    • Set the Target version.
    • Add the Build Meister and QA Team as watchers.

Week 2 (Preview Build 2)

  • Change CGI Build Ticket status to "Preview 2".
  • Make sure all build related issues are tested on dev before build happens on beta.
  • Send a reminder to QA that trackDb changes should be documented by engineers DbDoc DbHub.

Week 3 (Final Build)

  • Check the timestamps of the CGI binaries on hgwbeta: /usr/local/apache/cgi-bin/
  • Change CGI Build Ticket status to "Final Build".
  • Review error logs for the robots:
  1. hgNear - sends an email with results
  2. hgTables - sends an email with results
  3. LiftOverTest - check by hand: /cluster/bin/build/scripts/logs/LiftOverTest-v$VERSION_NUMBER.log
  • If there are any errors reported then go to the following location and check the logs: /cluster/bin/build/scripts/logs/
  • Make sure of the following:
    • All related issues (Bugs, Features) have status of "CGI-ready" or "CGI-ready-open-issues".

A Few Days Before The Push

  • Make sure of the following:
    • All Build Patch requests have been fulfilled (see Build Patch section below).
    • All Code Reviews are closed.
    • Everyone has reported their "done testing" messages (if any) in the CGI Build ticket.
  • If necessary, send out a status update (or reminder) email 1-2 days before release day.
  • The day before the push, send an email notice to browser-staff letting them know that tomorrow is a push day. Something along these lines:
   Just a reminder that tomorrow is a CGI push day. If you have big code changes included in 
   this release please be available in case something goes wrong with the push of your changes. 
   QA will be starting the push around 1:30pm.

Day of Push

  • Send a reminder to push-request that today is CGI push day.

Push to hgw0 only

  • Log on to hgw0 *before* sending push request: ssh qateam@hgw0
  • View apache error log at:
  $ cd /usr/local/apache/logs/
  $ less -s error_log
  • use shift + g to go to bottom of page
  • use shift + f to follow incoming errors

*Update*: To view the error log *without* Hiram's CGI_TIME entries (for background info see: http://redmine.soe.ucsc.edu/issues/10081): 

  $ tail -F logs/error_log | grep -v "CGI_TIME\|trackLog"
  • Send out push request.
  • After push has gone through, monitor error log while QA does their testing:
    • See this wiki page for examples of errors/warnings that can be ignored here
    • If there are any unusual errors, use whois $ip_address or host $ip_address to see where errors are coming from, or vi error_log to determine if error occurred before pushing to hgw0.
  • Wait to hear from QA on how their CGIs look on hgw0 (usually done via Skype chat with 'done testing, everything looks fine' messages).

Push to hgwN only

  • SSH to hgwN machine before sending out push request.
  • Send out push request.
  • Monitor error log for a short while to make sure no new errors occur under the load.

Push to rest of the RR, hgwbeta-public, and euronode

  • Send out push request.
  • Check hgTracks title version on each machine to ensure the CGI's have made it to the world.
  • Send email to Build Meister to let him/her know the CGIs are on the RR (send wrap-up email to Build Meister).

Day after Push

  • Make sure all related issues have been closed out.
  • Close Build Patch tickets.
  • Change CGI Build Ticket status to "Released".

Build Patches

The general rule is that we don't approve Build Patches within 24 hours of pushing the CGIs. We can make exceptions, but this depends on which CGIs are affected and how much QA will have to re-test. Our Build Patch process can be found here.

tableDescriptions

Along with the buildmeister, make sure the source of the errors from buildTableDescrptions.pl are tracked down and corrected. Usually the person who checked in the change that is causing the problem needs to be notified.

The type of error that shows up is usually either kind of cryptic, like this: 

   hgsql error for hg17.tableDescriptions.sql at /cluster/home/build/kent/src/test/buildTableDescriptions.pl line 449.

or kind of straightforward, like this: 

   Duplicate autoSql def for table bed12UniProtMut (/cluster/home/build/kent/src/hg/utils/uniprotMutations/bed12UniProtAnnot.as vs. /cluster/home/build/kent/src/hg/utils/uniprotMutations/bed12UniProtMut.as) at /cluster/home/build/kent/src/test/buildTableDescriptions.pl line 203, <F> line 24.

(someone entered a second set of table descriptions for bed12UniProtMut, and it says what files the problem is in).

Example push emails

1. Push v361 CGIs to hgw0

Pushers,

Please push these CGIs:

cartDump cartReset das hgApi hgBeacon hgBlat hgc hgConvert hgCustom
hgEncodeApi hgEncodeDataVersions hgEncodeVocab hgFileSearch hgFileUi
hgGateway hgGene hgGeneGraph hgGenome hgGtexTrackSettings hgHubConnect
hgIntegrator hgLinkIn hgLiftOver hgLogin hgMenubar hgMirror hgNear
hgPal hgPcr hgPublicSessions hgRenderTracks hgSession hgSuggest
hgTables hgTracks hgTrackUi hgUserSuggestion hgVai hgVisiGene phyloPng

      and these configuration files:

/usr/local/apache/cgi-bin/all.joiner
/usr/local/apache/cgi-bin/extTools.ra
/usr/local/apache/cgi-bin/greatData/*
/usr/local/apache/cgi-bin/hgCgiData/*
/usr/local/apache/cgi-bin/hgGeneData/*
/usr/local/apache/cgi-bin/hgNearData/*
/usr/local/apache/cgi-bin/hgcData/*
/usr/local/apache/cgi-bin/loader/*
/usr/local/apache/cgi-bin/lsSnpPdbChimera.py
/usr/local/apache/cgi-bin/visiGeneData/*
/usr/local/apache/cgi-bin/pyLib/*

from hgwbeta -> hgw0 only


Please rsync --delete these directories on beta:

/usr/local/apache/htdocs/js/*
/usr/local/apache/htdocs/style/*

to the same location on hgw0.

Reason: releasing v361 CGIs to the test RR machine.

Thanks,
$you

2.Push v361 CGIs to hgw5

Pushers,

Please push these CGIs:

cartDump cartReset das hgApi hgBeacon hgBlat hgc hgConvert hgCustom
hgEncodeApi hgEncodeDataVersions hgEncodeVocab hgFileSearch hgFileUi
hgGateway hgGene hgGeneGraph hgGenome hgGtexTrackSettings hgHubConnect
hgIntegrator hgLinkIn hgLiftOver hgLogin hgMenubar hgMirror hgNear
hgPal hgPcr hgPublicSessions hgRenderTracks hgSession hgSuggest
hgTables hgTracks hgTrackUi hgUserSuggestion hgVai hgVisiGene phyloPng

      and these configuration files:

/usr/local/apache/cgi-bin/all.joiner
/usr/local/apache/cgi-bin/extTools.ra
/usr/local/apache/cgi-bin/greatData/*
/usr/local/apache/cgi-bin/hgCgiData/*
/usr/local/apache/cgi-bin/hgGeneData/*
/usr/local/apache/cgi-bin/hgNearData/*
/usr/local/apache/cgi-bin/hgcData/*
/usr/local/apache/cgi-bin/loader/*
/usr/local/apache/cgi-bin/lsSnpPdbChimera.py
/usr/local/apache/cgi-bin/visiGeneData/*
/usr/local/apache/cgi-bin/pyLib/*

from hgwbeta -> hgw5 ONLY

Please rsync --delete these directories on beta:

/usr/local/apache/htdocs/js/*
/usr/local/apache/htdocs/style/*

to the same location on hgw5.

Reason: releasing v361 CGIs to one RR machine.

Thanks,
$you

3.
Push v361 CGIs to RR

Pushers,

Please push these CGIs:

cartDump cartReset das hgApi hgBeacon hgBlat hgc hgConvert hgCustom
hgEncodeApi hgEncodeDataVersions hgEncodeVocab hgFileSearch hgFileUi
hgGateway hgGene hgGeneGraph hgGenome hgGtexTrackSettings hgHubConnect
hgIntegrator hgLinkIn hgLiftOver hgLogin hgMenubar hgMirror hgNear
hgPal hgPcr hgPublicSessions hgRenderTracks hgSession hgSuggest
hgTables hgTracks hgTrackUi hgUserSuggestion hgVai hgVisiGene phyloPng

      and these configuration files:

/usr/local/apache/cgi-bin/all.joiner
/usr/local/apache/cgi-bin/extTools.ra
/usr/local/apache/cgi-bin/greatData/*
/usr/local/apache/cgi-bin/hgCgiData/*
/usr/local/apache/cgi-bin/hgGeneData/*
/usr/local/apache/cgi-bin/hgNearData/*
/usr/local/apache/cgi-bin/hgcData/*
/usr/local/apache/cgi-bin/loader/*
/usr/local/apache/cgi-bin/lsSnpPdbChimera.py
/usr/local/apache/cgi-bin/visiGeneData/*
/usr/local/apache/cgi-bin/pyLib/*

from hgwbeta -> hgwbeta-public AND hgw1-6 AND genome-euro/asia (note
that they are already on hgw0 and hgw5)

Please rsync --delete these directories on beta:

/usr/local/apache/htdocs/js/*
/usr/local/apache/htdocs/style/*

to the same location on the RR machines.

Please also sync to these other two directories to hgwbeta-public ONLY:

/usr/local/apache/htdocs/js-public/*
/usr/local/apache/htdocs/style-public/*

Reason: releasing v361 CGIs to hgwbeta-public, rest of the RR
machines, and genome-euro/asia

Thanks,
$you

4. v361 build wrap up

Hi Hiram,

v361 CGIs are now on the RR and ready for wrap-up.

Thanks,
$you
Retrieved from "https://genomewiki.ucsc.edu/genecats/index.php?title=Push_Shepherd_Responsibilities&oldid=4612"