Monitoring Tasks

From Genecats
Revision as of 20:41, 31 July 2017 by Brianlee (talk | contribs)
Jump to navigationJump to search

Please add to the documentation page, Monitoring Tasks Notes to add procedures and notes for specific jobs, then link back to the job in the list below.

task frequency NOW who April '16 -Feb '17 previous previous PROPOSED
from hgwdev qateam cron
Respond to items on the broken links report: Check to make sure that broken links reported are actually broken. If so, fix the link if it is clear what the new link should be and push to the RR. If not, contact the appropriate person at the web site in question to find out what the new link should be. daily ChrisV Cath, Matt Matt Matt
Respond to the trackDbGlobal report: Make public and request a push of trackDb and friends for any assemblies that show up as changed in the trackDbGlobal report. weekly Jairo Cath Matt Luvina
receive output from syncAday.csh; alert; if anything is out of sync (checks that tables are in sync between beta and RR for database of the day) daily Cath ChrisV Brian Brian
receive output from checkMetaAday.csh; alert; if anything is out of sync (compares hgcentralbeta to hgcentral for db of the day) daily Cath Cath, Matt Matt Matt
receive output from realTime.csh (previously gbLoaded) cron job and alert braney if anything seems very outdated daily, problems every 3-6 months Jairo Cath, Luvina Steve, Jonathan Steve, Jonathan
receive output from showProcesslist.csh daily, no action Cath, Brian Jairo, Matt Steve, Jonathan Steve, Jonathan
get output from checkTableStatus.csh on Mondays (this is a report on whether the genbank table dumps are up-to-date...if we could get rid of our dependence on those, we could get rid of this cron job) weekly Cath Brian, Jonathan Steve, Jonathan Steve, Jonathan
get an email from backupCentral.csh each month that keeps a dump of hgcentral tables (there's a redmine ticket on whether we should change this one: http://redmine.soe.ucsc.edu/issues/7834) monthly Brian Matt, Brian Steve, Jonathan Steve, Jonathan
receive an email on additions and subtractions to files in /gbdb on hgnfs1 daily Matt ChrisL, Luvina Steve, Jonathan Steve, Jonathan
receive an email from cron if qateam DailySandboxUpdate fails daily Matt, Brian Matt, Brian Steve, Jonathan Steve, Jonathan
email request to build doPublicCrawl public.ix and public.ixx hub search files and push them. runs quarterly Jairo ChrisL
increment databaseAday, /cluster/bin/scripts/databaseAday.csh problems only ChrisL, Matt, Brian Brian, Jonathan Steve, Jonathan Steve, Jonathan
checkHgFindSpec -checkTermRegex, /cluster/bin/scripts/runCheckHgFindSpec go problems only ChrisL, Matt, Brian Brian, Jonathan Steve, Jonathan Steve, Jonathan
check to see if there are new files to go with the CGI push /cluster/home/qateam/bin/scripts/findCgiFiles.csh go problems only ChrisL, Matt, Brian Brian, Jonathan Steve, Jonathan Steve, Jonathan
get output from checkGenomeMysql daily Brian ChrisL, Matt Steve, Jonathan Steve, Jonathan
[checkGbibMd5.sh] compares the md5sum of gbibBeta.zip on dev with the sum for gbib.zip on genome-store, complaining if they don't match. runs weekly Jairo, Cath Brian, Cath, Jonathan Brian, Steve, Jonathan
Check for unigene update for human SAGE track daily Cath, Braney Luvina, Braney Steve, Braney
look for broken symlinks in /gbdb: daily Browser Staff Browser Staff Browser Staff
look for broken symlinks in in (very large) visiGene directory twice yearly ChrisL Cath, Galt Steve, Galt
look for broken symlinks in htdocs-hgdownload/goldenpath daily Browser Staff Browser Staff Browser Staff
Check the source tree for duplicate case-independent filenames. These wreck havoc with git. daily Browser Staff Browser Staff Browser Staff
Check to see what kind of browsers and o/s people are using to access the public website weekly Ann, Jairo, ChrisL Ann, ChrisL, Brian Ann, Steve
sends a reminder to check the list of folks who have mirror sites. This list is created by a script of Hiram's and located here: http://genome-test.cse.ucsc.edu/admin/tools/useCount.html. The information comes from the phoneHome program in the CGIs. Sometimes companies set up mirrors, which is not OK. monthly Matt, Brian Brian, Luvina Matt Pauline
run /cluster/bin/scripts/sessionStats.csh monthly Angie, Brian, Bob Angie, Brian, Bob Angie, Brian, Bob, Steve
ask Galt/Jim about blat updates 3x per year ChrisV,Cath Cath, Brian Steve, Jonathan Pauline, Steve, Jonathan
check that the ga4gh beacon server is running. Email Max if there is an issue. daily Jairo Cath Ann Ann
check that blat servers are running ok ongoing Ann, Brian, Galt, Matt, Erich, ChrisL, ChrisV, Jairo, Cath Ann, Brian, Donna, Galt, Luvina, Matt, Gary, Ann, Brian, Donna, Galt, Luvina, Steve, Gary
receive output from hgTracksRandom cron; alert admins to problems as needed many: cath, chmalee, kuhn, brianlee, braney, ann,, jcasper, matt, jairo, chrisv many: cath, chmalee, kuhn, brianlee, braney, luvina, gary, ann,, jcasper, matt many: kuhn, katrina, brianlee, braney, luvina, gary, ann, steve, jcasper, matt many: kuhn, pauline, katrina, brianlee, braney, luvina, gary, ann, steve, jcasper, matt
check that beta builds alright 3x week Brian, Matt Brian, ChrisL,Braney Steve, Jonathan, Braney
hubPublicCheck complains when the hubPublic table doesn't have the same strings as the actual hub.txt file. (NOTE: To run only on Beta hubs requires these declarations: export HGDB_CONF=$HOME/.hg.conf.beta and SHELL=/bin/sh). runs daily, complains rarely Matt Matt Brian Brian
Curl of http://genomewiki.ucsc.edu/index.php/Assembly_Hubs to test Public Wiki is accessible, output only if no response. hourly ChrisV Brian Brian
Cronjob reports checkUrlsInTable hg38 wgEncodeCell and cv.ra (likely infrequent results) weekly ChrisV ChrisL Brian Brian
qaAutoTrack -script that does some auto-QA on the autopushed tracks. daily Matt
from build cron
receive cron output for tableDescriptions build errors rare Matt Luvina Steve, Luvina Pauline, Matt
from galt's cron
run ReleaseLog script (this requires a copy of the kent repository) daily Galt Galt Galt Galt
back up the qapushq every 30 minutes M-F Galt Galt Galt Galt
non-cron jobs
Check for MLQs that may have slipped through the cracks with Google Groups to check for unanswered Q's. Check to make sure that the last response for every ticket came from one of us. If not, look it up in redmine to ensure that A) there is indeed a redmine ticket for it, and B) it has been responded to. If not, notify the appropriate MLM with as much information about the MLQ as possible. weekly Cath Luvina Luvina Luvina
Monitor & report UCSC Genome Browser outages. See wiki procedures hopefully, not much! Alexa/ChrisV Cath, Luvina Steve, Jonathan Steve, Jonathan
update crontab in genecats/qa/crontabs/*.crontab (and log in as qateam and restart the qateam crontab with updates) as needed Brian Brian Jonathan Pauline, Jonathan
update UCSC Entrez LinkOut when necessary. Entrez LinkOut sends requests for changes (and statistics) to the browser-qa email address rarely ChrisL Matt Matt Bob
Check genome-preview machine is still running biweekly Jairo Brian Brian Brian
Updated Facebook with Announcements As needed Alexa/Team Brian/Alexa/Team Brian Brian
Update Twitter with Announcements As needed Alexa/Team Brian/Alexa/Team Ann
otto cron jobs (/hive/data/outside/otto)
omimWrapper.sh checks for new OMIM data and downloads and loads it into hgwdev if so runs daily, complains bi-weekly jcasper jcasper braney braney
omimUploadWrapper.sh uploads the OMIM tables to OMIM runs daily, complains rarely jcasper jcasper braney braney
decipherWrapper.sh checks for new DECIPHER data and downloads and loads it into hgwdev if so runs daily, complains monthly jcasper jcasper braney braney
iscaWrapper.sh checks for new ISCA data and downloads and loads it into hgwdev if so runs daily, complains monthly jcasper jcasper braney braney
gwasWrapper.sh checks for new GWAS data and downloads and loads it into hgwdev if so runs weekly, complains quarterly jcasper jcasper braney braney
geneReviewsWrapper.sh checks for new GeneReviews data and downloads and loads it into hgwdev if so runs weekly, complains quarterly jcasper jcasper braney braney
otto cron jobs (/hive/data/outside/otto)
~jcasper/bin/scripts/updateHgConf.sh - checks for changes to hg.conf files and checks them in to the kent tree runs weekly jcasper jcasper nobody nobody
~jcasper/bin/backup_hgconfprivate.sh - checks most recent versions of hg.conf.private files into a private repository (~jcasper/hgconf_private/) runs twice a week jcasper jcasper nobody nobody