Monitoring Tasks Notes

This page is intended to document procedures and notes for various QA monitoring tasks.

Cronjob: Results from checkGbibMd5.sh

Hiram generally requests the push of vXXX gBiB Store Push (push request example). This push is usually done soon after the Tuesday release of CGIs to the RR/world. However, if this push is not done before this script runs, then you may see the md5 mismatch. It is recommended to wait until the end of the week (and possibly wait until the end of Friday following the release) to give Hiram time to do the push. Check the push request group for the latest store push. If the md5sums still do not match after the push of the GBiB to the store, then let Hiram know.

The /cluster/home/qateam/bin/scripts/checkGbibMd5.sh script checks for matching 'modify' times between beta's hgTracks and RR's hgTracks:

ssh qateam@hgwbeta stat -c %y /usr/local/apache/cgi-bin/hgTracks
2017-03-20 10:31:02.000000000 -0700

ssh qateam@hgw1 stat -c %y /usr/local/apache/cgi-bin/hgTracks
2017-03-20 10:31:02.000000000 -0700

If those dates match, the md5sums are compared between the local gbibBeta with the version on the store, which should also match, but in this case they don't match.

md5sum /usr/local/apache/htdocs/gbib/gbibBeta.zip | awk '{print $1}'
14e5f65fd19ecf43af05a313f884d26c

curl -s https://genome-store.ucsc.edu/media/products/gbib.zip | md5sum | awk '{print $1}'
0a90536be4f4e2fdb3e2d865762db818

Cronjob: Results from checkMetaAday.csh

This is one of the monitoring tasks that looks at metaData such as in hgcentral on hgwdev, hgwbeta, and the RR and notifies people with an email that they must investigate to see if there are differences. Differences are then corrected. The output is put into genecats: http://genecats.cse.ucsc.edu/qa/test-results/metadata/ where each database gets an entry.

Email Example

checkMetaAday.csh mm8

database = mm8

 0 dbDb.mm8.hgcentralbetaOnly
 0 dbDb.mm8.hgcentralOnly
 1 dbDb.mm8.common

 0 blatServers.mm8.hgcentralbetaOnly
 0 blatServers.mm8.hgcentralOnly
 2 blatServers.mm8.common

 0 defaultDb.mm8.hgcentralbetaOnly
 0 defaultDb.mm8.hgcentralOnly
 0 defaultDb.mm8.common

 0 genomeClade.mm8.hgcentralbetaOnly
 0 genomeClade.mm8.hgcentralOnly
 1 genomeClade.mm8.common

 0 liftOverChain.mm8.hgcentralbetaOnly
 1 liftOverChain.mm8.hgcentralOnly   <---This should be a zero. There is "1" row difference. Need to fix. 
52 liftOverChain.mm8.common

  details in
 http://genecats.cse.ucsc.edu/qa/test-results/metadata/details

Process Example

Problem: The file below is in rr but not in beta

ornAna1 mm8 /gbdb/ornAna1/liftOver/ornAna1ToMm8.over.chain.gz 0.1 0 0 Y 1 N

Solution:​ Add row to beta liftOverChain table​

First, double check: rr

hgsql -h genome-centdb -Ne "SELECT * FROM liftOverChain WHERE fromDb = 'ornAna1' AND toDb = 'mm8'" hgcentral +---------+-----+---------------------------------------------------+-----+---+---+---+---+---+ | ornAna1 | mm8 | /gbdb/ornAna1/liftOver/ornAna1ToMm8.over.chain.gz | 0.1 | 0 | 0 | Y | 1 | N | +---------+-----+---------------------------------------------------+-----+---+---+---+---+---+

beta

hgsql -h mysqlbeta -Ne "SELECT * FROM liftOverChain WHERE fromDb = 'ornAna1' AND toDb = 'mm8'" hgcentralbeta No results. No entries match. Need to move​ the row that exists in the rr table into​ ​the beta​ table​.

First, make a file of the row which is on rr but not on beta. hgsql -h genome-centdb -Ne "SELECT * FROM liftOverChain WHERE fromDb = 'ornAna1' AND toDb = 'mm8'" hgcentral > chain.dev

Next, load the file into beta:

hgsql -h mysqlbeta -e "LOAD DATA LOCAL INFILE 'chain.dev' INTO TABLE liftOverChain" hgcentralbeta

Look in beta to see if the contents of the file make it into the table: hgsql -h mysqlbeta -Ne "SELECT * FROM liftOverChain WHERE fromDb = 'ornAna1' AND toDb = 'mm8'" hgcentralbeta +---------+-----+---------------------------------------------------+-----+---+---+---+---+---+ | ornAna1 | mm8 | /gbdb/ornAna1/liftOver/ornAna1ToMm8.over.chain.gz | 0.1 | 0 | 0 | Y | 1 | N | +---------+-----+---------------------------------------------------+-----+---+---+---+---+---+

Looks good. CheckMetaAday again, for both ornAna1 and mm8

checkMetaAday.csh mm8

database = mm8

 0 dbDb.mm8.hgcentralbetaOnly
 0 dbDb.mm8.hgcentralOnly
 1 dbDb.mm8.common

 0 blatServers.mm8.hgcentralbetaOnly
 0 blatServers.mm8.hgcentralOnly
 2 blatServers.mm8.common

 0 defaultDb.mm8.hgcentralbetaOnly
 0 defaultDb.mm8.hgcentralOnly
 0 defaultDb.mm8.common

 0 genomeClade.mm8.hgcentralbetaOnly
 0 genomeClade.mm8.hgcentralOnly
 1 genomeClade.mm8.common

 0 liftOverChain.mm8.hgcentralbetaOnly
 0 liftOverChain.mm8.hgcentralOnly  <---Looks good!
 53 liftOverChain.mm8.common <--up by 1, good!

SLA Monitoring & Reporting

In Jan 2017, these data were migrated to a Google Spreadsheet:

• SLA Outage Report Worksheet.
• Current manager of this spreadsheet: Alexa Lakey
• Please contact the current manager of the spreadsheet to enter/modify outages.
• The spreadsheet has a "README" tab to explain the spreadsheet format and best practices.

The old (unused) way to report outages was by adding it to an html table, SLA.html.