Monitoring Tasks: Difference between revisions

From Genecats
Jump to navigationJump to search
No edit summary
(Adding the Hub Public Coordinator role to Monitoring Tasks)
 
(80 intermediate revisions by 8 users not shown)
Line 1: Line 1:
Please add to the documentation page, [[Monitoring_Tasks_Notes|Monitoring Tasks Notes]] to add procedures and notes for specific jobs, then link back to the job in the list below. 
<table border="1">
<table border="1">


Line 4: Line 5:
<th>task</th>
<th>task</th>
<th>frequency</th>
<th>frequency</th>
<th>who NOW</th>
<th> Until Aug '18</th>
<th>who Aug '14 - Aug '15-Apr'16</th>
<th>Until Sep '19</th>
<th>who Aug '13 - Jul '14</th>
<th>Now</th>
<th>before</th>
</tr>
</tr>


<tr>
<tr>
<td colspan="6" style="background-color:gray"><b>from [http://genomewiki.cse.ucsc.edu/genecats/index.php/Updating_QA_crontabs hgwdev] qateam cron</b></td>
<td colspan="7" style="background-color:gray"><b>from [http://genomewiki.cse.ucsc.edu/genecats/index.php/Updating_QA_crontabs hgwdev] qateam cron</b></td>
</tr>
 
<tr>
<td>ask Galt/Jim about blat updates</td>
<td>3x per year</td>
<td>Cath, Brian </td>
<td>Steve, Jonathan</td>
<td>Pauline, Steve, Jonathan</td>
<td>Pauline, Steve, Jonathan</td>
</tr>
</tr>


Line 26: Line 17:
<td>Respond to items on the broken links report: Check to make sure that broken links reported are actually broken.  If so, fix the link if it is clear what the new link should be and push to the RR.  If not, contact the appropriate person at the web site in question to find out what the new link should be.</td>
<td>Respond to items on the broken links report: Check to make sure that broken links reported are actually broken.  If so, fix the link if it is clear what the new link should be and push to the RR.  If not, contact the appropriate person at the web site in question to find out what the new link should be.</td>
<td>daily</td>
<td>daily</td>
<td>Cath, Matt</td>
<td>Jairo</td>
<td>Matt</td>
<td>Jairo</td>
<td>Matt</td>
<td>Jairo</td>
<td>Steve</td>
</tr>
</tr>


Line 35: Line 25:
<td>Respond to the trackDbGlobal report: Make public and request a push of trackDb and friends for any assemblies that show up as changed in the trackDbGlobal report.</td>
<td>Respond to the trackDbGlobal report: Make public and request a push of trackDb and friends for any assemblies that show up as changed in the trackDbGlobal report.</td>
<td>weekly</td>
<td>weekly</td>
<td>Jairo</td>
<td>Conner</td>
<td>Lou</td>
</tr>
<tr>
<td>'''Retired (See #23998)''' receive output from syncAday.csh; alert; if anything is out of sync (checks that tables are in sync between beta and RR for database of the day) </td>
<td>daily</td>
<td>Cath</td>
<td>Cath</td>
<td>Matt</td>
<td>Lou</td>
<td>Luvina</td>
<td>Retired</td>
<td>Steve</td>
</tr>
</tr>


<tr>
<tr>
<td>receive output from hgTracksRandom cron; alert admins to problems</td>
<td>'''Retired (See #23998)''' receive output from [http://genomewiki.ucsc.edu/genecats/index.php/Monitoring_Tasks_Notes#Cronjob:_Results_from_checkMetaAday.csh checkMetaAday.csh]; alert;  if anything is out of sync (compares hgcentralbeta to hgcentral for db of the day) </td>
<td>as needed</td>
<td>daily</td>
<td>many: cath, christopher, kuhn, brianlee, braney, luvina, gary, ann,, jcasper, matt</td>
<td>Cath</td>
<td>many: kuhn, katrina, brianlee, braney, luvina, gary, ann, steve, jcasper, matt</td>
<td>Lou</td>
<td>many: kuhn, pauline, katrina, brianlee, braney, luvina, gary, ann, steve, jcasper, matt</td>
<td>Retired</td>
<td>many: kuhn, rhead, pauline, katrina, brianlee, braney, luvina, gary, ann, steve, jcasper</td>
</tr>
</tr>


<tr>
<tr>
<td>receive output from gbLoaded cron job and alert braney if anything seems very outdated</td>
<td> Receive output of checkSyncAndMetaDataAll.py which is a combination of the two previous cronjobs syncAday.csh and checkMetaAday.csh. This cron refines the hgcentralbeta/hgcentral sync output as well as the hgwbeta/hgw1/hgw2 table syncs, and runs on all databases. See [http://redmine.soe.ucsc.edu/issues/23998 #23998] for more info. </td>
<td>daily, problems every 3-6 months</td>
<td>weekly</td>
<td>Cath, Luvina</td>
<td></td>
<td>Steve, Jonathan</td>
<td>Lou, Conner</td>
<td>Steve, Jonathan</td>
<td>Lou</td>
<td>Brooke, Steve, Jonathan</td>
</tr>
</tr>


<tr>
<tr>
<td>receive output from showProcesslist.csh</td>
<td>receive output from [http://genomewiki.ucsc.edu/genecats/index.php/Monitoring_Tasks_Notes#Cronjob:_Results_from_realTime.csh_.28previously_known_as_gbLoaded.29 realTime.csh] (previously gbLoaded) cron job and alert braney if anything seems very outdated <br> No output for xenTro3, fr1, fr2, fr3, eboVir3, dm2</td>
<td>daily, no action</td>
<td>daily, problems every 3-6 months</td>
<td>Jairo, Matt </td>
<td>Jairo</td>
<td>Steve, Jonathan</td>
<td>Jairo</td>
<td>Steve, Jonathan</td>
<td>Daniel</td>
<td>Brooke, Steve, Jonathan</td>
</tr>
</tr>


<tr>
<tr>
<td>get output from checkTableStatus.csh on Mondays (this is a report on whether the genbank table dumps are up-to-date...if we could get rid of our dependence on those, we could get rid of this cron job)</td>
<td>get output from checkTableStatus.csh on Mondays (this is a report on whether the genbank table dumps are up-to-date...{Read how this cronjob is important for updateTimes.csh [http://genomewiki.ucsc.edu/genecats/index.php/Monitoring_Tasks_Notes#Cronjob:_Results_from_checkTableStatus.csh_.22_TABLE_STATUS_dump.22_emails here]</td>
<td>weekly</td>
<td>weekly</td>
<td>Brian, Jonathan</td>
<td>Cath</td>
<td>Steve, Jonathan</td>
<td>Lou</td>
<td>Steve, Jonathan</td>
<td>Gerardo</td>
<td>Brooke, Steve, Jonathan</td>
</tr>
</tr>


Line 80: Line 75:
<td>get an email from backupCentral.csh each month that keeps a dump of hgcentral tables (there's a redmine ticket on whether we should change this one: http://redmine.soe.ucsc.edu/issues/7834)</td>
<td>get an email from backupCentral.csh each month that keeps a dump of hgcentral tables (there's a redmine ticket on whether we should change this one: http://redmine.soe.ucsc.edu/issues/7834)</td>
<td>monthly</td>
<td>monthly</td>
<td>Matt, Brian </td>
<td>Brian</td>
<td>Steve, Jonathan</td>
<td>Daniel</td>
<td>Steve, Jonathan</td>
<td>Gerardo</td>
<td>Brooke, Steve, Jonathan</td>
</tr>
</tr>


Line 89: Line 83:
<td>receive an email on additions and subtractions to files in /gbdb on hgnfs1</td>
<td>receive an email on additions and subtractions to files in /gbdb on hgnfs1</td>
<td>daily</td>
<td>daily</td>
<td>Christopher, Luvina </td>
<td>Brian</td>
<td>Steve, Jonathan</td>
<td>Conner</td>
<td>Steve, Jonathan</td>
<td>Lou</td>
<td>Brooke, Steve, Jonathan</td>
</tr>
</tr>


<tr>
<tr>
<td>receive an email from cron if qateam DailySandboxUpdate fails</td>
<td>receive an email from cron if qateam DailySandboxUpdate fails, '''note:''' emails defined in /cluster/home/qateam/bin/scripts/DailySandboxUpdate</td>
<td>daily</td>
<td>daily</td>
<td>Matt, Brian</td>
<td>ChrisL, Brian</td>
<td>Steve, Jonathan</td>
<td>Jairo Brian Lou</td>
<td>Steve, Jonathan</td>
<td>Jairo, Brian, Lou</td>
<td>Brooke, Steve, Jonathan</td>
</tr>
</tr>


<tr>
<tr>
<td>receive output from checkMetaAday.csh; alert;  if anything is out of sync (compares hgcentralbeta to hgcentral for db of the day) [[CheckMetaADay.csh_monitoring_task | Instructions & Example]]</td>
<td>email request to check [https://en.wikipedia.org/wiki/UCSC_Genome_Browser#Genomes Genomes section of the Wikipedia Page] and make other changes or updates if needed.</td>
<td>daily</td>
<td>runs quarterly</td>
<td>Cath, Matt</td>
<td></td>
<td>Matt</td>
<td>Bob,Conner</td>
<td>Matt</td>
<td>Bob,Gerardo</td>
<td>Luvina</td>
</tr>
 
<tr>
<td>email request to check trackDbLibrary.shtml and trackDbHub and run hubCheck -settings -level=required</td>
<td>runs monthly</td>
<td></td>
<td></td>
<td>BrianL</td>
</tr>
</tr>


<tr>
<tr>
<td>receive output from syncAday.csh; alert; if anything is out of sync (checks that tables are in sync between beta and RR for database of the day) </td>
<td>'''Retired (See #27425)''' email request to check [http://genomewiki.ucsc.edu/genecats/index.php/Public_Hub_QA#Push_to_the_RR doPublicCrawl] public.ix and public.ixx hub search files and push them.</td>
<td>daily</td>
<td>runs quarterly</td>
<td>ChrisV</td>
<td>Jairo</td>
<td>Brian</td>
<td>Daniel</td>
<td>Brian</td>
<td>Gerardo,Brian</td>
<td>Luvina</td>
</tr>
</tr>


Line 126: Line 124:
<td> increment databaseAday, /cluster/bin/scripts/databaseAday.csh </td>
<td> increment databaseAday, /cluster/bin/scripts/databaseAday.csh </td>
<td>problems only</td>
<td>problems only</td>
<td>Brian, Jonathan</td>
<td>ChrisL, Cath, Brian</td>
<td>Steve, Jonathan</td>
<td>Jairo, Brian, Lou</td>
<td>Steve, Jonathan</td>
<td>Jairo, Brian, Lou</td>
<td>Brooke, Steve, Jonathan</td>
</tr>
</tr>


Line 135: Line 132:
<td>checkHgFindSpec -checkTermRegex,  /cluster/bin/scripts/runCheckHgFindSpec go</td>
<td>checkHgFindSpec -checkTermRegex,  /cluster/bin/scripts/runCheckHgFindSpec go</td>
<td>problems only</td>
<td>problems only</td>
<td>Brian, Jonathan</td>
<td>ChrisL, Brian, Cath</td>
<td>Steve, Jonathan</td>
<td>Jairo, Brian, Lou</td>
<td>Steve, Jonathan</td>
<td>Jairo, Brian, Lou</td>
<td>Brooke, Steve, Jonathan</td>
</tr>
</tr>


Line 144: Line 140:
<td>check to see if there are new files to go with the CGI push /cluster/home/qateam/bin/scripts/findCgiFiles.csh go</td>
<td>check to see if there are new files to go with the CGI push /cluster/home/qateam/bin/scripts/findCgiFiles.csh go</td>
<td>problems only</td>
<td>problems only</td>
<td>Brian, Jonathan</td>
<td>ChrisL, Cath, Brian</td>
<td>Steve, Jonathan</td>
<td>Jairo, Brian, Lou</td>
<td>Steve, Jonathan</td>
<td>Jairo, Brian, Lou</td>
<td>Brooke, Steve, Jonathan</td>
</tr>
</tr>




<tr>
<tr>
<td>get output from checkGenomeMysql</td>
<td>get output from checkGenomeMysql (discovers corrupt tables)</td>
<td>daily</td>
<td>daily</td>
<td>Christopher, Matt</td>
<td>Brian</td>
<td>Steve, Jonathan</td>
<td>Brian, Daniel</td>
<td>Steve, Jonathan</td>
<td>Daniel, Gerardo</td>
<td>Brooke, Steve, Jonathan</td>
</tr>
</tr>


<tr>
<tr>
<td>Check for unigene update for human SAGE track</td>
<td>[[http://genomewiki.ucsc.edu/genecats/index.php/Monitoring_Tasks_Notes#Cronjob:_Results_from_checkGbibMd5.sh checkGbibMd5.sh]] compares the md5sum of gbibBeta.zip on dev with the sum for gbib.zip on genome-store, complaining if they don't match.</td>
<td>daily</td>
<td>runs weekly</td>
<td>Luvina, Braney</td>
<td>Jairo, Cath</td>
<td>Steve, Braney</td>
<td>Jairo,Conner,Daniel</td>
<td></td>
<td>Lou,Gerardo,Daniel</td>
<td></td>
</tr>
</tr>


Line 174: Line 167:
<td>Browser Staff</td>
<td>Browser Staff</td>
<td>Browser Staff</td>
<td>Browser Staff</td>
<td></td>
<td>Browser Staff</td>
<td></td>
</tr>
</tr>


Line 181: Line 173:
<td>look for broken symlinks in in (very large) visiGene directory</td>
<td>look for broken symlinks in in (very large) visiGene directory</td>
<td>twice yearly</td>
<td>twice yearly</td>
<td>Cath, Galt</td>
<td>ChrisL</td>
<td>Steve, Galt</td>
<td>Conner</td>
<td></td>
<td>Brian</td>
<td></td>
</tr>
</tr>


Line 192: Line 183:
<td>Browser Staff</td>
<td>Browser Staff</td>
<td>Browser Staff</td>
<td>Browser Staff</td>
<td></td>
<td>Browser Staff</td>
<td></td>
</tr>
</tr>


Line 201: Line 191:
<td>Browser Staff</td>
<td>Browser Staff</td>
<td>Browser Staff</td>
<td>Browser Staff</td>
<td></td>
<td>Browser Staff</td>
<td></td>
</tr>
 
<tr>
<td>sends a reminder to check the list of folks who have mirror sites. This list is created by a script of Hiram's and located here: http://genome-test.gi.ucsc.edu/admin/tools/useCount.html. The information comes from the phoneHome program in the CGIs (see #22593 for login/password and discussion about how an initial 'registration.txt' file is created then moved to trash, where presence of that file prevents the request being repeated so only an initial phone call is made). Sometimes companies set up mirrors, which is not OK.</td>
<td>monthly</td>
<td>Ann [was: Matt, Brian]</td>
<td>Ann,Lou,Hiram</td>
<td>Ann,Jairo,Hiram</td>
</tr>
</tr>


<tr>
<tr>
<td>Check to see what kind of browsers and o/s people are using to access the public website</td>
<td>Monthly reminder to check for stale New pennantIcon tags</td>
<td>weekly</td>
<td>monthly</td>
<td>Ann, Christopher, Brian</td>
<td>Ann, Steve</td>
<td></td>
<td></td>
<td></td>
<td>Lou</td>
<td>Lou</td>
</tr>
</tr>


<tr>
<tr>
<td>run /cluster/bin/scripts/sessionStats.csh</td>
<td>run /cluster/bin/scripts/sessionStats.csh NOTE: this is separate to each machine (genome-asia & genome-euro) thus cronjobs are specific to each machine and jonathan owns asia</td>
<td>monthly</td>
<td>monthly</td>
<td>Angie, Brian, Bob</td>
<td>Angie, Brian, Bob</td>
<td>Angie, Brian, Bob, Steve</td>
<td>Angie, Brian, Bob</td>
<td></td>
<td>Angie, Lou, Bob</td>
<td></td>
</tr>
 
<tr>
<td>ask Galt/Jim about blat updates</td>
<td>3x per year</td>
<td>ChrisL,Cath</td>
<td>Daniel, Conner</td>
<td>Daniel, Gerardo</td>
</tr>
</tr>


Line 226: Line 230:
<td>check that the [https://genomicsandhealth.org/work-products-demonstration-projects/beacon-project-0 ga4gh] [http://genome.ucsc.edu/cgi-bin/hgBeacon beacon server] is running. Email Max if there is an issue. </td>
<td>check that the [https://genomicsandhealth.org/work-products-demonstration-projects/beacon-project-0 ga4gh] [http://genome.ucsc.edu/cgi-bin/hgBeacon beacon server] is running. Email Max if there is an issue. </td>
<td>daily</td>
<td>daily</td>
<td>Cath</td>
<td>Jairo</td>
<td>Ann</td>
<td>Conner</td>
<td>Ann</td>
<td>Daniel</td>
<td></td>
</tr>
</tr>


<tr>
<tr>
<td>check that blat servers are running ok</td>
<td>check that blat servers are running ok [http://genomewiki.ucsc.edu/genecats/index.php/Monitoring_Tasks_Notes#check_that_blat_servers_are_running_ok link to error log checking]</td>
<td>ongoing</td>
<td>ongoing</td>
<td>Ann, Brian, Donna, Galt, Luvina, Matt, Gary,</td>
<td>Ann, Brian, Galt, Matt, Erich, ChrisL, Jairo, Cath</td>
<td>Ann, Brian, Donna, Galt, Luvina, Steve, Gary</td>
<td>Ann, Galt, Erich, Matt, Brian, Conner, Daniel, Lou, Jairo</td>
<td></td>
<td>Clay, Galt, Erich, Matt, Brian, Gerardo, Daniel, Lou, Jairo</td>
<td></td>
</tr>
 
<tr>
<td>receive output from hgTracksRandom cron; alert admins to problems</td>
<td>as needed</td>
<td>many: cath, chmalee, kuhn, brianlee, braney, ann,, jcasper, matt, jairo</td>
<td>Bob,Ann, Galt, Braney, Jonathan, Christopher, Erich, Matt, Brian, Conner, Daniel, Lou, Jairo</td>
<td>Clay, Bob, Braney, Jonathan, Erich, Matt, Brian, Gerardo, Daniel, Lou, Jairo</td>
</tr>
</tr>


Line 244: Line 254:
<td>check that beta builds alright</td>
<td>check that beta builds alright</td>
<td>3x week</td>
<td>3x week</td>
<td>Brian, Christopher,Braney</td>
<td>Brian, ChrisL</td>
<td>Steve, Jonathan, Braney</td>
<td>Brian, Braney, Jairo, Lou</td>
<td></td>
<td>Brian, Braney, Jairo, Lou</td>
<td></td>
</tr>
 
<tr>
<td>'''Retired (See #27779)''' hubPublicCheck complains when the hubPublic table doesn't have the same strings as the actual hub.txt file. '' (NOTE: To run only on Beta hubs requires these declarations: '''export HGDB_CONF=$HOME/.hg.conf.beta'''  and '''SHELL=/bin/sh''')''.</td>
<td>runs daily, complains moderately</td>
<td>ChrisL</td>
<td>Daniel, Braney, Christopher</td>
<td>Daniel, Braney, Christopher</td>
</tr>
</tr>


<tr>
<tr>
<td>hubPublicCheck complains when the hubPublic table doesn't have the same strings as the actual hub.txt file. '' (NOTE: To run only on Beta hubs requires these declarations: '''export HGDB_CONF=$HOME/.hg.conf.beta'''  and '''SHELL=/bin/sh''')''.</td>
<td>Curl of http://genomewiki.ucsc.edu/index.php/Assembly_Hubs to test Public Wiki is accessible, output only if no response.</td>
<td>runs daily, complains rarely</td>
<td>hourly</td>
<td>Matt</td>
<td>Brian</td>
<td>Brian</td>
<td>Brian</td>
<td>braney</td>
<td>Conner</td>
<td>Daniel</td>
</tr>
</tr>


<tr>
<tr>
<td>email request to build [http://genomewiki.ucsc.edu/genecats/index.php/Public_Hub_QA#Push_to_the_RR doPublicCrawl] public.ix and public.ixx hub search files and push them.</td>
<td>qaAutoTrack -script that does some auto-QA on the autopushed tracks.</td>
<td>runs quarterly</td>
<td>daily</td>
<td>Christopher</td>
<td>Jario</td>
<td></td>
<td>Jairo</td>
<td></td>
<td>Jairo</td>
<td></td>
</tr>
</tr>


<tr>
<tr>
<td>Cronjob reports checkUrlsInTable hg38 wgEncodeCell  and cv.ra (likely infrequent results) </td>
<td>checkAllStaticLinks.csh script that checks static pages.</td>
<td>weekly</td>
<td>weekly</td>
<td>Christopher</td>
<td></td>
<td>Brian</td>
<td>Daniel,Conner,Brian</td>
<td>Brian</td>
<td>Daniel,Gerardo,Brian</td>
<td>Brian</td>
</tr>
</tr>


<tr>
<tr>
<td>sends a reminder to check the list of folks who have mirror sites. This list is created by a script of Hiram's and located here: http://genome-test.cse.ucsc.edu/admin/tools/useCount.html. The information comes from the phoneHome program in the CGIs. Sometimes companies set up mirrors, which is not OK.</td>
<td>assemblyStatsCron.py pulls and reports GB usage stats over the last month from the RR+mirrors.</td>
<td>monthly</td>
<td>monthly</td>
<td>Brian, Luvina</td>
<td></td>
<td>Matt</td>
<td>QA Team</td>
<td>Pauline</td>
<td>QA Team</td>
<td>Ann</td>
</tr>
</tr>


<tr>
<tr>
<td>checkGbibMd5.sh compares the md5sum of gbibBeta.zip on dev with the sum for gbib.zip on genome-store, complaining if they don't match (and it's not during final build)</td>
<td>HTTPs expiration check runs genecats/qa/testTools/checkHTTPSdir/checkHTTPS to report any expiring certificates</td>
<td>runs weekly on Thursday</td>
<td>weekly on Tuesday</td>
<td>Brian, Cath, Jonathan</td>
<td></td>
<td>Brian, Steve, Jonathan</td>
<td></td>
<td>-</td>
<td>QA Team, cluster admin</td>
<td>-</td>
</tr>
</tr>


<tr>
<tr>
<td>Curl of http://genomewiki.ucsc.edu/index.php/Assembly_Hubs to test Public Wiki is accessible, output only if no response.</td>
<td>Hub Public Coordinator maintains error-free Public Hubs hubs and keeps the hubs updated. Fully automated to update labels/titles, the search index, and contact hub provider when hub is down. Monitor the three cronjobs: hubPublicAutoUpdate, hubSearchUpdate, and hubPublicMail. More details can be found [http://genomewiki.ucsc.edu/genecats/index.php/Public_Hub_QA#Hub_Public_Coordinator_Role here].</td>
<td>hourly</td>
<td>daily</td>
<td>Brian</td>
<td></td>
<td>Brian</td>
<td></td>
<td>-</td>
<td>Daniel, Gerardo</td>
<td>-</td>
</tr>
</tr>


<tr>
<tr>
<td colspan="6" style="background-color:gray"><b>from build cron</b></td>
<td colspan="7" style="background-color:gray"><b>from build cron </b></td>
</tr>
</tr>




<tr>
<tr>
<td>receive cron output for tableDescriptions build errors</td>
<td>receive cron output for tableDescriptions build errors, AutoPush completed</td>
<td>rare</td>
<td>rare</td>
<td>Luvina</td>
<td>Cath</td>
<td>Steve, Luvina</td>
<td>QA Team</td>
<td>Pauline, Matt</td>
<td>QA Team</td>
<td>Brooke, Pauline</td>
</tr>
</tr>


<tr>
<tr>
<td colspan="6" style="background-color:gray"><b>from galt's cron</b></td>
<td colspan="7" style="background-color:gray"><b>from galt's cron</b></td>
</tr>
</tr>


Line 326: Line 335:
<td>run ReleaseLog script (this requires a copy of the kent repository)</td>
<td>run ReleaseLog script (this requires a copy of the kent repository)</td>
<td>daily</td>
<td>daily</td>
<td>Galt</td>
<td>Galt</td>
<td>Galt</td>
<td>Galt</td>
<td>Galt</td>
Line 333: Line 341:


<tr>
<tr>
<td>back up the qapushq</td>
<td colspan="7" style="background-color:gray"><b>non-cron jobs</b></td>
<td>every 30 minutes M-F</td>
<td>Galt</td>
<td>Galt</td>
<td>Galt</td>
<td>Galt</td>
</tr>
 
<tr>
<td colspan="6" style="background-color:gray"><b>non-cron jobs</b></td>
</tr>
</tr>


<tr>
<tr>
<td>Check old forum (http://redmine.soe.ucsc.edu/forum/index.php) for MLQs that may have slipped through the cracks: Check to make sure that the last response for every ticket came from one of us.  If not, look it up in redmine to ensure that A) there is indeed a redmine ticket for it, and B) it has been responded to.  If not, notify the appropriate MLM with as much information about the MLQ as possible.</td>
<td>Check for MLQs that may have slipped through the cracks with Google Groups to check for unanswered Q's. Check to make sure that the last response for every ticket came from one of us.  If not, look it up in redmine to ensure that A) there is indeed a redmine ticket for it, and B) it has been responded to.  If not, notify the appropriate MLM with as much information about the MLQ as possible.</td>
<td>weekly</td>
<td>weekly</td>
<td>Luvina</td>
<td>Cath</td>
<td>Luvina</td>
<td>Lou</td>
<td>Luvina</td>
<td>Daniel</td>
<td>Steve</td>
</tr>
</tr>


<tr>
<tr>
<td>update sla.html whenever we have RR downtime. [http://genomewiki.ucsc.edu/genecats/index.php/Static_Page_Protocol#Editing_SLA.html See wiki procedures]</td>
<td>Monitor & report UCSC Genome Browser outages (SLA). [http://genomewiki.ucsc.edu/genecats/index.php/Monitoring_Tasks_Notes#SLA_Monitoring_.26_Reporting See wiki procedures]</td>
<td>hopefully, not much!</td>
<td>hopefully, not much!</td>
<td>Cath, Luvina</td>
<td>Cath</td>
<td>Steve, Jonathan</td>
<td>Conner</td>
<td>Steve, Jonathan</td>
<td>Gerardo</td>
<td>Brooke, Pauline</td>
</tr>
</tr>


Line 367: Line 364:
<td>as needed</td>
<td>as needed</td>
<td>Brian</td>
<td>Brian</td>
<td>Jonathan</td>
<td>Brian</td>
<td>Pauline, Jonathan</td>
<td>Brian</td>
<td>Brooke, Pauline</td>
</tr>
</tr>


Line 375: Line 371:
<td>update UCSC Entrez LinkOut when necessary. Entrez LinkOut sends requests for changes (and statistics) to the browser-qa email address</td>
<td>update UCSC Entrez LinkOut when necessary. Entrez LinkOut sends requests for changes (and statistics) to the browser-qa email address</td>
<td>rarely</td>
<td>rarely</td>
<td>Matt</td>
<td>ChrisL,Matt</td>
<td>Matt</td>
<td>Daniel</td>
<td>Bob</td>
<td>Gerardo</td>
<td>Ann</td>
</tr>
</tr>


<tr>
<tr>
<td>Check genome-preview machine is still running</td>
<td>Check genome-preview machine is still running well</td>
<td>biweekly</td>
<td>weekly</td>
<td>Brian</td>
<td>Jairo</td>
<td>Brian</td>
<td>Conner</td>
<td>Brian</td>
<td>Daniel</td>
<td>Brian</td>
</tr>
</tr>


<tr>
<tr>
<td>Updated Facebook with Announcements</td>
<td>Updated Facebook with Announcements</td>  
<td>As needed</td>
<td>As needed</td>
<td>Brian</td>
<td>Team</td>
<td>Brian</td>
<td>Team</td>
<td></td>
<td>Team</td>
<td></td>
</tr>
</tr>


<tr>
<tr>
<td>Update Twitter with Announcements</td>
<td>Update Twitter with Announcements</td>    
<td>As needed</td>
<td>As needed</td>
<td>Brian</td>
<td>Team</td>
<td>Ann</td>
<td>Team</td>
<td></td>
<td>Team</td>
<td></td>
</tr>
</tr>




<td colspan="6" style="background-color:gray"><b>otto cron jobs (/hive/data/outside/otto) </b></td>
<td colspan="7" style="background-color:gray"><b>otto cron jobs (/hive/data/outside/otto) </b></td>


<tr>
<tr>
Line 415: Line 407:
<td>runs daily, complains bi-weekly</td>
<td>runs daily, complains bi-weekly</td>
<td>jcasper</td>
<td>jcasper</td>
<td>braney</td>
<td>ChrisL</td>
<td>braney</td>
<td>ChrisL</td>
<td>braney</td>
</tr>
</tr>


Line 424: Line 415:
<td>runs daily, complains rarely</td>
<td>runs daily, complains rarely</td>
<td>jcasper</td>
<td>jcasper</td>
<td>braney</td>
<td>ChrisL</td>
<td>braney</td>
<td>ChrisL</td>
<td>braney</td>
</tr>
</tr>


Line 433: Line 423:
<td>runs daily, complains monthly</td>
<td>runs daily, complains monthly</td>
<td>jcasper</td>
<td>jcasper</td>
<td>braney</td>
<td>ChrisL</td>
<td>braney</td>
<td>ChrisL</td>
<td>braney</td>
</tr>
</tr>


Line 442: Line 431:
<td>runs daily, complains monthly</td>
<td>runs daily, complains monthly</td>
<td>jcasper</td>
<td>jcasper</td>
<td>braney</td>
<td>ChrisL</td>
<td>braney</td>
<td>ChrisL</td>
<td>braney</td>
</tr>
</tr>


Line 451: Line 439:
<td>runs weekly, complains quarterly</td>
<td>runs weekly, complains quarterly</td>
<td>jcasper</td>
<td>jcasper</td>
<td>braney</td>
<td>ChrisL</td>
<td>braney</td>
<td>ChrisL</td>
<td>braney</td>
</tr>
</tr>


Line 460: Line 447:
<td>runs weekly, complains quarterly</td>
<td>runs weekly, complains quarterly</td>
<td>jcasper</td>
<td>jcasper</td>
<td>braney</td>
<td>ChrisL</td>
<td>braney</td>
<td>ChrisL</td>
<td>braney</td>
</tr>
</tr>


<td colspan="6" style="background-color:gray"><b>otto cron jobs (/hive/data/outside/otto) </b></td>
<td colspan="7" style="background-color:gray"><b>otto cron jobs (/hive/data/outside/otto) </b></td>


<tr>
<tr>
Line 471: Line 457:
<td>runs weekly</td>
<td>runs weekly</td>
<td>jcasper</td>
<td>jcasper</td>
<td>nobody</td>
<td>jcasper</td>
<td>nobody</td>
<td>jcasper</td>
<td>nobody</td>
</tr>
</tr>


Line 480: Line 465:
<td>runs twice a week</td>
<td>runs twice a week</td>
<td>jcasper</td>
<td>jcasper</td>
<td>nobody</td>
<td>jcasper</td>
<td>nobody</td>
<td>jcasper</td>
<td>nobody</td>
</tr>
</tr>


Line 488: Line 472:


[[Category:Browser QA]]
[[Category:Browser QA]]
[[Category: Browser QA Monitoring Tasks]]

Latest revision as of 20:59, 25 March 2022

Please add to the documentation page, Monitoring Tasks Notes to add procedures and notes for specific jobs, then link back to the job in the list below.

task frequency Until Aug '18 Until Sep '19 Now
from hgwdev qateam cron
Respond to items on the broken links report: Check to make sure that broken links reported are actually broken. If so, fix the link if it is clear what the new link should be and push to the RR. If not, contact the appropriate person at the web site in question to find out what the new link should be. daily Jairo Jairo Jairo
Respond to the trackDbGlobal report: Make public and request a push of trackDb and friends for any assemblies that show up as changed in the trackDbGlobal report. weekly Jairo Conner Lou
Retired (See #23998) receive output from syncAday.csh; alert; if anything is out of sync (checks that tables are in sync between beta and RR for database of the day) daily Cath Lou Retired
Retired (See #23998) receive output from checkMetaAday.csh; alert; if anything is out of sync (compares hgcentralbeta to hgcentral for db of the day) daily Cath Lou Retired
Receive output of checkSyncAndMetaDataAll.py which is a combination of the two previous cronjobs syncAday.csh and checkMetaAday.csh. This cron refines the hgcentralbeta/hgcentral sync output as well as the hgwbeta/hgw1/hgw2 table syncs, and runs on all databases. See #23998 for more info. weekly Lou, Conner Lou
receive output from realTime.csh (previously gbLoaded) cron job and alert braney if anything seems very outdated
No output for xenTro3, fr1, fr2, fr3, eboVir3, dm2		 daily, problems every 3-6 months Jairo Jairo Daniel
get output from checkTableStatus.csh on Mondays (this is a report on whether the genbank table dumps are up-to-date...{Read how this cronjob is important for updateTimes.csh here weekly Cath Lou Gerardo
get an email from backupCentral.csh each month that keeps a dump of hgcentral tables (there's a redmine ticket on whether we should change this one: http://redmine.soe.ucsc.edu/issues/7834) monthly Brian Daniel Gerardo
receive an email on additions and subtractions to files in /gbdb on hgnfs1 daily Brian Conner Lou
receive an email from cron if qateam DailySandboxUpdate fails, note: emails defined in /cluster/home/qateam/bin/scripts/DailySandboxUpdate daily ChrisL, Brian Jairo Brian Lou Jairo, Brian, Lou
email request to check Genomes section of the Wikipedia Page and make other changes or updates if needed. runs quarterly Bob,Conner Bob,Gerardo
email request to check trackDbLibrary.shtml and trackDbHub and run hubCheck -settings -level=required runs monthly BrianL
Retired (See #27425) email request to check doPublicCrawl public.ix and public.ixx hub search files and push them. runs quarterly Jairo Daniel Gerardo,Brian
increment databaseAday, /cluster/bin/scripts/databaseAday.csh problems only ChrisL, Cath, Brian Jairo, Brian, Lou Jairo, Brian, Lou
checkHgFindSpec -checkTermRegex, /cluster/bin/scripts/runCheckHgFindSpec go problems only ChrisL, Brian, Cath Jairo, Brian, Lou Jairo, Brian, Lou
check to see if there are new files to go with the CGI push /cluster/home/qateam/bin/scripts/findCgiFiles.csh go problems only ChrisL, Cath, Brian Jairo, Brian, Lou Jairo, Brian, Lou
get output from checkGenomeMysql (discovers corrupt tables) daily Brian Brian, Daniel Daniel, Gerardo
[checkGbibMd5.sh] compares the md5sum of gbibBeta.zip on dev with the sum for gbib.zip on genome-store, complaining if they don't match. runs weekly Jairo, Cath Jairo,Conner,Daniel Lou,Gerardo,Daniel
look for broken symlinks in /gbdb: daily Browser Staff Browser Staff Browser Staff
look for broken symlinks in in (very large) visiGene directory twice yearly ChrisL Conner Brian
look for broken symlinks in htdocs-hgdownload/goldenpath daily Browser Staff Browser Staff Browser Staff
Check the source tree for duplicate case-independent filenames. These wreck havoc with git. daily Browser Staff Browser Staff Browser Staff
sends a reminder to check the list of folks who have mirror sites. This list is created by a script of Hiram's and located here: http://genome-test.gi.ucsc.edu/admin/tools/useCount.html. The information comes from the phoneHome program in the CGIs (see #22593 for login/password and discussion about how an initial 'registration.txt' file is created then moved to trash, where presence of that file prevents the request being repeated so only an initial phone call is made). Sometimes companies set up mirrors, which is not OK. monthly Ann [was: Matt, Brian] Ann,Lou,Hiram Ann,Jairo,Hiram
Monthly reminder to check for stale New pennantIcon tags monthly Lou Lou
run /cluster/bin/scripts/sessionStats.csh NOTE: this is separate to each machine (genome-asia & genome-euro) thus cronjobs are specific to each machine and jonathan owns asia monthly Angie, Brian, Bob Angie, Brian, Bob Angie, Lou, Bob
ask Galt/Jim about blat updates 3x per year ChrisL,Cath Daniel, Conner Daniel, Gerardo
check that the ga4gh beacon server is running. Email Max if there is an issue. daily Jairo Conner Daniel
check that blat servers are running ok link to error log checking ongoing Ann, Brian, Galt, Matt, Erich, ChrisL, Jairo, Cath Ann, Galt, Erich, Matt, Brian, Conner, Daniel, Lou, Jairo Clay, Galt, Erich, Matt, Brian, Gerardo, Daniel, Lou, Jairo
receive output from hgTracksRandom cron; alert admins to problems as needed many: cath, chmalee, kuhn, brianlee, braney, ann,, jcasper, matt, jairo Bob,Ann, Galt, Braney, Jonathan, Christopher, Erich, Matt, Brian, Conner, Daniel, Lou, Jairo Clay, Bob, Braney, Jonathan, Erich, Matt, Brian, Gerardo, Daniel, Lou, Jairo
check that beta builds alright 3x week Brian, ChrisL Brian, Braney, Jairo, Lou Brian, Braney, Jairo, Lou
Retired (See #27779) hubPublicCheck complains when the hubPublic table doesn't have the same strings as the actual hub.txt file. (NOTE: To run only on Beta hubs requires these declarations: export HGDB_CONF=$HOME/.hg.conf.beta and SHELL=/bin/sh). runs daily, complains moderately ChrisL Daniel, Braney, Christopher Daniel, Braney, Christopher
Curl of http://genomewiki.ucsc.edu/index.php/Assembly_Hubs to test Public Wiki is accessible, output only if no response. hourly Brian Conner Daniel
qaAutoTrack -script that does some auto-QA on the autopushed tracks. daily Jario Jairo Jairo
checkAllStaticLinks.csh script that checks static pages. weekly Daniel,Conner,Brian Daniel,Gerardo,Brian
assemblyStatsCron.py pulls and reports GB usage stats over the last month from the RR+mirrors. monthly QA Team QA Team
HTTPs expiration check runs genecats/qa/testTools/checkHTTPSdir/checkHTTPS to report any expiring certificates weekly on Tuesday QA Team, cluster admin
Hub Public Coordinator maintains error-free Public Hubs hubs and keeps the hubs updated. Fully automated to update labels/titles, the search index, and contact hub provider when hub is down. Monitor the three cronjobs: hubPublicAutoUpdate, hubSearchUpdate, and hubPublicMail. More details can be found here. daily Daniel, Gerardo
from build cron
receive cron output for tableDescriptions build errors, AutoPush completed rare Cath QA Team QA Team
from galt's cron
run ReleaseLog script (this requires a copy of the kent repository) daily Galt Galt Galt
non-cron jobs
Check for MLQs that may have slipped through the cracks with Google Groups to check for unanswered Q's. Check to make sure that the last response for every ticket came from one of us. If not, look it up in redmine to ensure that A) there is indeed a redmine ticket for it, and B) it has been responded to. If not, notify the appropriate MLM with as much information about the MLQ as possible. weekly Cath Lou Daniel
Monitor & report UCSC Genome Browser outages (SLA). See wiki procedures hopefully, not much! Cath Conner Gerardo
update crontab in genecats/qa/crontabs/*.crontab (and log in as qateam and restart the qateam crontab with updates) as needed Brian Brian Brian
update UCSC Entrez LinkOut when necessary. Entrez LinkOut sends requests for changes (and statistics) to the browser-qa email address rarely ChrisL,Matt Daniel Gerardo
Check genome-preview machine is still running well weekly Jairo Conner Daniel
Updated Facebook with Announcements As needed Team Team Team
Update Twitter with Announcements As needed Team Team Team
otto cron jobs (/hive/data/outside/otto)
omimWrapper.sh checks for new OMIM data and downloads and loads it into hgwdev if so runs daily, complains bi-weekly jcasper ChrisL ChrisL
omimUploadWrapper.sh uploads the OMIM tables to OMIM runs daily, complains rarely jcasper ChrisL ChrisL
decipherWrapper.sh checks for new DECIPHER data and downloads and loads it into hgwdev if so runs daily, complains monthly jcasper ChrisL ChrisL
iscaWrapper.sh checks for new ISCA data and downloads and loads it into hgwdev if so runs daily, complains monthly jcasper ChrisL ChrisL
gwasWrapper.sh checks for new GWAS data and downloads and loads it into hgwdev if so runs weekly, complains quarterly jcasper ChrisL ChrisL
geneReviewsWrapper.sh checks for new GeneReviews data and downloads and loads it into hgwdev if so runs weekly, complains quarterly jcasper ChrisL ChrisL
otto cron jobs (/hive/data/outside/otto)
~jcasper/bin/scripts/updateHgConf.sh - checks for changes to hg.conf files and checks them in to the kent tree runs weekly jcasper jcasper jcasper
~jcasper/bin/backup_hgconfprivate.sh - checks most recent versions of hg.conf.private files into a private repository (~jcasper/hgconf_private/) runs twice a week jcasper jcasper jcasper
Retrieved from "https://genomewiki.ucsc.edu/genecats/index.php?title=Monitoring_Tasks&oldid=5860"