Monitoring Tasks: Difference between revisions

From Genecats
Jump to navigationJump to search
No edit summary
(Adding the Hub Public Coordinator role to Monitoring Tasks)
 
(32 intermediate revisions by 5 users not shown)
Line 5: Line 5:
<th>task</th>
<th>task</th>
<th>frequency</th>
<th>frequency</th>
<th>who Mar 17' to Aug '18</th>
<th> Until Aug '18</th>
<th>who April '16 -Feb '17</th>
<th>Until Sep '19</th>
<th>_</th>
<th>Now</th>
<th>_</th>
</tr>
<th>NOW</th>
 
</tr>uni<tr>
<tr>
<td colspan="7" style="background-color:gray"><b>from [http://genomewiki.cse.ucsc.edu/genecats/index.php/Updating_QA_crontabs hgwdev] qateam cron</b></td>
<td colspan="7" style="background-color:gray"><b>from [http://genomewiki.cse.ucsc.edu/genecats/index.php/Updating_QA_crontabs hgwdev] qateam cron</b></td>
</tr>
</tr>
Line 18: Line 18:
<td>daily</td>
<td>daily</td>
<td>Jairo</td>
<td>Jairo</td>
<td>Cath, Matt</td>
<td>Jairo</td>
<td></td>
<td></td>
<td>Jairo</td>
<td>Jairo</td>
</tr>
</tr>
Line 28: Line 26:
<td>weekly</td>
<td>weekly</td>
<td>Jairo</td>
<td>Jairo</td>
<td>Cath</td>
<td></td>
<td></td>
<td>Conner</td>
<td>Conner</td>
<td>Lou</td>
</tr>
</tr>




<tr>
<tr>
<td>receive output from syncAday.csh; alert; if anything is out of sync (checks that tables are in sync between beta and RR for database of the day) </td>
<td>'''Retired (See #23998)''' receive output from syncAday.csh; alert; if anything is out of sync (checks that tables are in sync between beta and RR for database of the day) </td>
<td>daily</td>
<td>daily</td>
<td>Cath</td>
<td>Cath</td>
<td>ChrisV</td>
<td></td>
<td></td>
<td>Lou</td>
<td>Lou</td>
<td>Retired</td>
</tr>
</tr>


<tr>
<tr>
<td>receive output from [http://genomewiki.ucsc.edu/genecats/index.php/Monitoring_Tasks_Notes#Cronjob:_Results_from_checkMetaAday.csh checkMetaAday.csh]; alert;  if anything is out of sync (compares hgcentralbeta to hgcentral for db of the day) </td>
<td>'''Retired (See #23998)''' receive output from [http://genomewiki.ucsc.edu/genecats/index.php/Monitoring_Tasks_Notes#Cronjob:_Results_from_checkMetaAday.csh checkMetaAday.csh]; alert;  if anything is out of sync (compares hgcentralbeta to hgcentral for db of the day) </td>
<td>daily</td>
<td>daily</td>
<td>Cath</td>
<td>Cath</td>
<td>Cath, Matt</td>
<td>Lou</td>
<td></td>
<td>Retired</td>
</tr>
 
<tr>
<td> Receive output of checkSyncAndMetaDataAll.py which is a combination of the two previous cronjobs syncAday.csh and checkMetaAday.csh. This cron refines the hgcentralbeta/hgcentral sync output as well as the hgwbeta/hgw1/hgw2 table syncs, and runs on all databases. See [http://redmine.soe.ucsc.edu/issues/23998 #23998] for more info. </td>
<td>weekly</td>
<td></td>
<td></td>
<td>Lou, Conner</td>
<td>Lou</td>
<td>Lou</td>
</tr>
</tr>
Line 59: Line 60:
<td>daily, problems every 3-6 months</td>
<td>daily, problems every 3-6 months</td>
<td>Jairo</td>
<td>Jairo</td>
<td>Cath, Luvina</td>
<td></td>
<td></td>
<td>Jairo</td>
<td>Jairo</td>
<td>Daniel</td>
</tr>
</tr>


Line 69: Line 68:
<td>weekly</td>
<td>weekly</td>
<td>Cath</td>
<td>Cath</td>
<td>Brian, Jonathan</td>
<td></td>
<td></td>
<td>Lou</td>
<td>Lou</td>
<td>Gerardo</td>
</tr>
</tr>


Line 79: Line 76:
<td>monthly</td>
<td>monthly</td>
<td>Brian</td>
<td>Brian</td>
<td>Matt, Brian </td>
<td></td>
<td></td>
<td>Daniel</td>
<td>Daniel</td>
<td>Gerardo</td>
</tr>
</tr>


Line 89: Line 84:
<td>daily</td>
<td>daily</td>
<td>Brian</td>
<td>Brian</td>
<td>ChrisL, Luvina </td>
<td></td>
<td></td>
<td>Conner</td>
<td>Conner</td>
<td>Lou</td>
</tr>
</tr>


Line 99: Line 92:
<td>daily</td>
<td>daily</td>
<td>ChrisL, Brian</td>
<td>ChrisL, Brian</td>
<td>Matt, Brian</td>
<td>Jairo Brian Lou</td>
<td>Jairo, Brian, Lou</td>
</tr>
 
<tr>
<td>email request to check [https://en.wikipedia.org/wiki/UCSC_Genome_Browser#Genomes Genomes section of the Wikipedia Page] and make other changes or updates if needed.</td>
<td>runs quarterly</td>
<td></td>
<td>Bob,Conner</td>
<td>Bob,Gerardo</td>
</tr>
 
<tr>
<td>email request to check trackDbLibrary.shtml and trackDbHub and run hubCheck -settings -level=required</td>
<td>runs monthly</td>
<td></td>
<td></td>
<td></td>
<td></td>
<td>Jairo Brian Lou</td>
<td>BrianL</td>
</tr>
</tr>


<tr>
<tr>
<td>email request to build [http://genomewiki.ucsc.edu/genecats/index.php/Public_Hub_QA#Push_to_the_RR doPublicCrawl] public.ix and public.ixx hub search files and push them.</td>
<td>'''Retired (See #27425)''' email request to check [http://genomewiki.ucsc.edu/genecats/index.php/Public_Hub_QA#Push_to_the_RR doPublicCrawl] public.ix and public.ixx hub search files and push them.</td>
<td>runs quarterly</td>
<td>runs quarterly</td>
<td>Jairo</td>
<td>Jairo</td>
<td>ChrisL</td>
<td></td>
<td></td>
<td>Daniel</td>
<td>Daniel</td>
<td>Gerardo,Brian</td>
</tr>
</tr>


<tr>
<tr>
Line 119: Line 125:
<td>problems only</td>
<td>problems only</td>
<td>ChrisL, Cath, Brian</td>
<td>ChrisL, Cath, Brian</td>
<td>Brian, Jonathan</td>
<td>Jairo, Brian, Lou</td>
<td></td>
<td></td>
<td>Jairo, Brian, Lou</td>
<td>Jairo, Brian, Lou</td>
</tr>
</tr>
Line 129: Line 133:
<td>problems only</td>
<td>problems only</td>
<td>ChrisL, Brian, Cath</td>
<td>ChrisL, Brian, Cath</td>
<td>Brian, Jonathan</td>
<td>Jairo, Brian, Lou</td>
<td></td>
<td></td>
<td>Jairo, Brian, Lou</td>
<td>Jairo, Brian, Lou</td>
</tr>
</tr>
Line 139: Line 141:
<td>problems only</td>
<td>problems only</td>
<td>ChrisL, Cath, Brian</td>
<td>ChrisL, Cath, Brian</td>
<td>Brian, Jonathan</td>
<td>Jairo, Brian, Lou</td>
<td></td>
<td></td>
<td>Jairo, Brian, Lou</td>
<td>Jairo, Brian, Lou</td>
</tr>
</tr>
Line 147: Line 147:


<tr>
<tr>
<td>get output from checkGenomeMysql</td>
<td>get output from checkGenomeMysql (discovers corrupt tables)</td>
<td>daily</td>
<td>daily</td>
<td>Brian</td>
<td>Brian</td>
<td>ChrisL, Matt</td>
<td></td>
<td></td>
<td>Brian, Daniel</td>
<td>Brian, Daniel</td>
<td>Daniel, Gerardo</td>
</tr>
</tr>


Line 160: Line 158:
<td>runs weekly</td>
<td>runs weekly</td>
<td>Jairo, Cath</td>
<td>Jairo, Cath</td>
<td>Brian, Cath, Jonathan</td>
<td></td>
<td></td>
<td>Jairo,Conner,Daniel</td>
<td>Jairo,Conner,Daniel</td>
<td>Lou,Gerardo,Daniel</td>
</tr>
</tr>


Line 171: Line 167:
<td>Browser Staff</td>
<td>Browser Staff</td>
<td>Browser Staff</td>
<td>Browser Staff</td>
<td></td>
<td></td>
<td>Browser Staff</td>
<td>Browser Staff</td>
</tr>
</tr>
Line 180: Line 174:
<td>twice yearly</td>
<td>twice yearly</td>
<td>ChrisL</td>
<td>ChrisL</td>
<td>Cath, Galt</td>
<td></td>
<td></td>
<td>Conner</td>
<td>Conner</td>
<td>Brian</td>
</tr>
</tr>


Line 191: Line 183:
<td>Browser Staff</td>
<td>Browser Staff</td>
<td>Browser Staff</td>
<td>Browser Staff</td>
<td></td>
<td></td>
<td>Browser Staff</td>
<td>Browser Staff</td>
</tr>
</tr>
Line 201: Line 191:
<td>Browser Staff</td>
<td>Browser Staff</td>
<td>Browser Staff</td>
<td>Browser Staff</td>
<td></td>
<td></td>
<td>Browser Staff</td>
<td>Browser Staff</td>
</tr>
</tr>


<tr>
<tr>
<td>Check to see what kind of browsers and o/s people are using to access the public website</td>
<td>sends a reminder to check the list of folks who have mirror sites. This list is created by a script of Hiram's and located here: http://genome-test.gi.ucsc.edu/admin/tools/useCount.html. The information comes from the phoneHome program in the CGIs (see #22593 for login/password and discussion about how an initial 'registration.txt' file is created then moved to trash, where presence of that file prevents the request being repeated so only an initial phone call is made). Sometimes companies set up mirrors, which is not OK.</td>
<td>weekly</td>
<td>monthly</td>
<td>Ann, Jairo, ChrisL</td>
<td>Ann [was: Matt, Brian]</td>
<td>Ann, ChrisL, Brian</td>
<td>Ann,Lou,Hiram</td>
<td></td>
<td>Ann,Jairo,Hiram</td>
<td></td>
<td>Ann, Jairo, Daniel</td>
</tr>
</tr>


<tr>
<tr>
<td>sends a reminder to check the list of folks who have mirror sites. This list is created by a script of Hiram's and located here: http://genome-test.gi.ucsc.edu/admin/tools/useCount.html. The information comes from the phoneHome program in the CGIs. Sometimes companies set up mirrors, which is not OK.</td>
<td>Monthly reminder to check for stale New pennantIcon tags</td>
<td>monthly</td>
<td>monthly</td>
<td>Ann [was: Matt, Brian]</td>
<td>Brian, Luvina</td>
<td></td>
<td></td>
<td></td>
<td>Lou</td>
<td>Ann,Lou,Hiram</td>
<td>Lou</td>
</tr>
</tr>


<tr>
<tr>
<td>run /cluster/bin/scripts/sessionStats.csh</td>
<td>run /cluster/bin/scripts/sessionStats.csh NOTE: this is separate to each machine (genome-asia & genome-euro) thus cronjobs are specific to each machine and jonathan owns asia</td>
<td>monthly</td>
<td>monthly</td>
<td>Angie, Brian, Bob</td>
<td>Angie, Brian, Bob</td>
<td>Angie, Brian, Bob</td>
<td>Angie, Brian, Bob</td>
<td></td>
<td>Angie, Lou, Bob</td>
<td></td>
<td>Angie, Brian, Bob</td>
</tr>
</tr>


Line 240: Line 223:
<td>3x per year</td>
<td>3x per year</td>
<td>ChrisL,Cath</td>
<td>ChrisL,Cath</td>
<td>Cath, Brian </td>
<td></td>
<td></td>
<td>Daniel, Conner</td>
<td>Daniel, Conner</td>
<td>Daniel, Gerardo</td>
</tr>
</tr>


Line 250: Line 231:
<td>daily</td>
<td>daily</td>
<td>Jairo</td>
<td>Jairo</td>
<td>Cath</td>
<td></td>
<td></td>
<td>Conner</td>
<td>Conner</td>
<td>Daniel</td>
</tr>
</tr>


<tr>
<tr>
<td>check that blat servers are running ok</td>
<td>check that blat servers are running ok [http://genomewiki.ucsc.edu/genecats/index.php/Monitoring_Tasks_Notes#check_that_blat_servers_are_running_ok link to error log checking]</td>
<td>ongoing</td>
<td>ongoing</td>
<td>Ann, Brian, Galt, Matt, Erich, ChrisL, Jairo, Cath</td>
<td>Ann, Brian, Galt, Matt, Erich, ChrisL, Jairo, Cath</td>
<td>Ann, Brian, Donna, Galt, Luvina, Matt, Gary,</td>
<td></td>
<td></td>
<td>Ann, Galt, Erich, Matt, Brian, Conner, Daniel, Lou, Jairo</td>
<td>Ann, Galt, Erich, Matt, Brian, Conner, Daniel, Lou, Jairo</td>
<td>Clay, Galt, Erich, Matt, Brian, Gerardo, Daniel, Lou, Jairo</td>
</tr>
</tr>


Line 270: Line 247:
<td>as needed</td>
<td>as needed</td>
<td>many: cath, chmalee, kuhn, brianlee, braney, ann,, jcasper, matt, jairo</td>
<td>many: cath, chmalee, kuhn, brianlee, braney, ann,, jcasper, matt, jairo</td>
<td>many: cath, chmalee, kuhn, brianlee, braney, luvina, gary, ann,, jcasper, matt</td>
<td></td>
<td></td>
<td>Bob,Ann, Galt, Braney, Jonathan, Christopher, Erich, Matt, Brian, Conner, Daniel, Lou, Jairo</td>
<td>Bob,Ann, Galt, Braney, Jonathan, Christopher, Erich, Matt, Brian, Conner, Daniel, Lou, Jairo</td>
<td>Clay, Bob, Braney, Jonathan, Erich, Matt, Brian, Gerardo, Daniel, Lou, Jairo</td>
</tr>
</tr>


Line 280: Line 255:
<td>3x week</td>
<td>3x week</td>
<td>Brian, ChrisL</td>
<td>Brian, ChrisL</td>
<td>Brian, ChrisL,Braney</td>
<td>Brian, Braney, Jairo, Lou</td>
<td></td>
<td></td>
<td>Brian, Braney, Jairo, Lou</td>
<td>Brian, Braney, Jairo, Lou</td>
</tr>
</tr>


<tr>
<tr>
<td>hubPublicCheck complains when the hubPublic table doesn't have the same strings as the actual hub.txt file. '' (NOTE: To run only on Beta hubs requires these declarations: '''export HGDB_CONF=$HOME/.hg.conf.beta'''  and '''SHELL=/bin/sh''')''.</td>
<td>'''Retired (See #27779)''' hubPublicCheck complains when the hubPublic table doesn't have the same strings as the actual hub.txt file. '' (NOTE: To run only on Beta hubs requires these declarations: '''export HGDB_CONF=$HOME/.hg.conf.beta'''  and '''SHELL=/bin/sh''')''.</td>
<td>runs daily, complains moderately</td>
<td>runs daily, complains moderately</td>
<td>ChrisL</td>
<td>ChrisL</td>
<td>Matt</td>
<td>Daniel, Braney, Christopher</td>
<td></td>
<td></td>
<td>Daniel, Braney, Christopher</td>
<td>Daniel, Braney, Christopher</td>
</tr>
</tr>
Line 300: Line 271:
<td>hourly</td>
<td>hourly</td>
<td>Brian</td>
<td>Brian</td>
<td>Brian</td>
<td></td>
<td></td>
<td>Conner</td>
<td>Conner</td>
<td>Daniel</td>
</tr>
</tr>


Line 310: Line 279:
<td>daily</td>
<td>daily</td>
<td>Jario</td>
<td>Jario</td>
<td></td>
<td>Jairo</td>
<td></td>
<td></td>
<td>Jairo</td>
<td>Jairo</td>
</tr>
</tr>
Line 320: Line 287:
<td>weekly</td>
<td>weekly</td>
<td></td>
<td></td>
<td>Daniel,Conner,Brian</td>
<td>Daniel,Gerardo,Brian</td>
</tr>
<tr>
<td>assemblyStatsCron.py pulls and reports GB usage stats over the last month from the RR+mirrors.</td>
<td>monthly</td>
<td></td>
<td></td>
<td>QA Team</td>
<td>QA Team</td>
</tr>
<tr>
<td>HTTPs expiration check runs genecats/qa/testTools/checkHTTPSdir/checkHTTPS to report any expiring certificates</td>
<td>weekly on Tuesday</td>
<td></td>
<td></td>
<td></td>
<td></td>
<td>Daniel,Conner,Brian</td>
<td>QA Team, cluster admin</td>
</tr>
</tr>


<tr>
<td>Hub Public Coordinator maintains error-free Public Hubs hubs and keeps the hubs updated. Fully automated to update labels/titles, the search index, and contact hub provider when hub is down. Monitor the three cronjobs: hubPublicAutoUpdate, hubSearchUpdate, and hubPublicMail. More details can be found [http://genomewiki.ucsc.edu/genecats/index.php/Public_Hub_QA#Hub_Public_Coordinator_Role here].</td>
<td>daily</td>
<td></td>
<td></td>
<td>Daniel, Gerardo</td>
</tr>


<tr>
<tr>
<td colspan="7" style="background-color:gray"><b>from build cron</b></td>
<td colspan="7" style="background-color:gray"><b>from build cron </b></td>
</tr>
</tr>




<tr>
<tr>
<td>receive cron output for tableDescriptions build errors</td>
<td>receive cron output for tableDescriptions build errors, AutoPush completed</td>
<td>rare</td>
<td>rare</td>
<td>Cath</td>
<td>Cath</td>
<td>Luvina</td>
<td>QA Team</td>
<td></td>
<td>QA Team</td>
<td></td>
<td>?</td>
</tr>
</tr>


Line 351: Line 337:
<td>Galt</td>
<td>Galt</td>
<td>Galt</td>
<td>Galt</td>
<td></td>
<td></td>
<td>Galt</td>
<td>Galt</td>
</tr>
</tr>
Line 364: Line 348:
<td>weekly</td>
<td>weekly</td>
<td>Cath</td>
<td>Cath</td>
<td>Luvina</td>
<td></td>
<td></td>
<td>Lou</td>
<td>Lou</td>
<td>Daniel</td>
</tr>
</tr>


<tr>
<tr>
<td>Monitor & report UCSC Genome Browser outages. [http://genomewiki.ucsc.edu/genecats/index.php/Monitoring_Tasks_Notes#SLA_Monitoring_.26_Reporting See wiki procedures]</td>
<td>Monitor & report UCSC Genome Browser outages (SLA). [http://genomewiki.ucsc.edu/genecats/index.php/Monitoring_Tasks_Notes#SLA_Monitoring_.26_Reporting See wiki procedures]</td>
<td>hopefully, not much!</td>
<td>hopefully, not much!</td>
<td>Cath</td>
<td>Cath</td>
<td>Cath, Luvina</td>
<td></td>
<td></td>
<td>Conner</td>
<td>Conner</td>
<td>Gerardo</td>
</tr>
</tr>


Line 385: Line 365:
<td>Brian</td>
<td>Brian</td>
<td>Brian</td>
<td>Brian</td>
<td></td>
<td></td>
<td>Brian</td>
<td>Brian</td>
</tr>
</tr>
Line 394: Line 372:
<td>rarely</td>
<td>rarely</td>
<td>ChrisL,Matt</td>
<td>ChrisL,Matt</td>
<td>Matt</td>
<td></td>
<td></td>
<td>Daniel</td>
<td>Daniel</td>
<td>Gerardo</td>
</tr>
</tr>


Line 404: Line 380:
<td>weekly</td>
<td>weekly</td>
<td>Jairo</td>
<td>Jairo</td>
<td>Brian</td>
<td></td>
<td></td>
<td>Conner</td>
<td>Conner</td>
<td>Daniel</td>
</tr>
</tr>


Line 414: Line 388:
<td>As needed</td>
<td>As needed</td>
<td>Team</td>
<td>Team</td>
<td>Brian/Alexa/Team</td>
<td>Team</td>
<td></td>
<td></td>
<td>Team</td>
<td>Team</td>
</tr>
</tr>
Line 424: Line 396:
<td>As needed</td>
<td>As needed</td>
<td>Team</td>
<td>Team</td>
<td>Brian/Alexa/Team</td>
<td>Team</td>
<td></td>
<td></td>
<td>Team</td>
<td>Team</td>
</tr>
</tr>
Line 437: Line 407:
<td>runs daily, complains bi-weekly</td>
<td>runs daily, complains bi-weekly</td>
<td>jcasper</td>
<td>jcasper</td>
<td>jcasper</td>
<td>ChrisL</td>
<td></td>
<td>ChrisL</td>
<td></td>
<td>jcasper</td>
</tr>
</tr>


Line 447: Line 415:
<td>runs daily, complains rarely</td>
<td>runs daily, complains rarely</td>
<td>jcasper</td>
<td>jcasper</td>
<td>jcasper</td>
<td>ChrisL</td>
<td></td>
<td>ChrisL</td>
<td></td>
<td>jcasper</td>
</tr>
</tr>


Line 457: Line 423:
<td>runs daily, complains monthly</td>
<td>runs daily, complains monthly</td>
<td>jcasper</td>
<td>jcasper</td>
<td>jcasper</td>
<td>ChrisL</td>
<td></td>
<td>ChrisL</td>
<td></td>
<td>jcasper</td>
</tr>
</tr>


Line 467: Line 431:
<td>runs daily, complains monthly</td>
<td>runs daily, complains monthly</td>
<td>jcasper</td>
<td>jcasper</td>
<td>jcasper</td>
<td>ChrisL</td>
<td></td>
<td>ChrisL</td>
<td></td>
<td>jcasper</td>
</tr>
</tr>


Line 477: Line 439:
<td>runs weekly, complains quarterly</td>
<td>runs weekly, complains quarterly</td>
<td>jcasper</td>
<td>jcasper</td>
<td>jcasper</td>
<td>ChrisL</td>
<td></td>
<td>ChrisL</td>
<td></td>
<td>jcasper</td>
</tr>
</tr>


Line 487: Line 447:
<td>runs weekly, complains quarterly</td>
<td>runs weekly, complains quarterly</td>
<td>jcasper</td>
<td>jcasper</td>
<td>jcasper</td>
<td>ChrisL</td>
<td></td>
<td>ChrisL</td>
<td></td>
<td>jcasper</td>
</tr>
</tr>


Line 500: Line 458:
<td>jcasper</td>
<td>jcasper</td>
<td>jcasper</td>
<td>jcasper</td>
<td></td>
<td></td>
<td>jcasper</td>
<td>jcasper</td>
</tr>
</tr>
Line 510: Line 466:
<td>jcasper</td>
<td>jcasper</td>
<td>jcasper</td>
<td>jcasper</td>
<td></td>
<td></td>
<td>jcasper</td>
<td>jcasper</td>
</tr>
</tr>

Latest revision as of 20:59, 25 March 2022

Please add to the documentation page, Monitoring Tasks Notes to add procedures and notes for specific jobs, then link back to the job in the list below.

task frequency Until Aug '18 Until Sep '19 Now
from hgwdev qateam cron
Respond to items on the broken links report: Check to make sure that broken links reported are actually broken. If so, fix the link if it is clear what the new link should be and push to the RR. If not, contact the appropriate person at the web site in question to find out what the new link should be. daily Jairo Jairo Jairo
Respond to the trackDbGlobal report: Make public and request a push of trackDb and friends for any assemblies that show up as changed in the trackDbGlobal report. weekly Jairo Conner Lou
Retired (See #23998) receive output from syncAday.csh; alert; if anything is out of sync (checks that tables are in sync between beta and RR for database of the day) daily Cath Lou Retired
Retired (See #23998) receive output from checkMetaAday.csh; alert; if anything is out of sync (compares hgcentralbeta to hgcentral for db of the day) daily Cath Lou Retired
Receive output of checkSyncAndMetaDataAll.py which is a combination of the two previous cronjobs syncAday.csh and checkMetaAday.csh. This cron refines the hgcentralbeta/hgcentral sync output as well as the hgwbeta/hgw1/hgw2 table syncs, and runs on all databases. See #23998 for more info. weekly Lou, Conner Lou
receive output from realTime.csh (previously gbLoaded) cron job and alert braney if anything seems very outdated
No output for xenTro3, fr1, fr2, fr3, eboVir3, dm2
daily, problems every 3-6 months Jairo Jairo Daniel
get output from checkTableStatus.csh on Mondays (this is a report on whether the genbank table dumps are up-to-date...{Read how this cronjob is important for updateTimes.csh here weekly Cath Lou Gerardo
get an email from backupCentral.csh each month that keeps a dump of hgcentral tables (there's a redmine ticket on whether we should change this one: http://redmine.soe.ucsc.edu/issues/7834) monthly Brian Daniel Gerardo
receive an email on additions and subtractions to files in /gbdb on hgnfs1 daily Brian Conner Lou
receive an email from cron if qateam DailySandboxUpdate fails, note: emails defined in /cluster/home/qateam/bin/scripts/DailySandboxUpdate daily ChrisL, Brian Jairo Brian Lou Jairo, Brian, Lou
email request to check Genomes section of the Wikipedia Page and make other changes or updates if needed. runs quarterly Bob,Conner Bob,Gerardo
email request to check trackDbLibrary.shtml and trackDbHub and run hubCheck -settings -level=required runs monthly BrianL
Retired (See #27425) email request to check doPublicCrawl public.ix and public.ixx hub search files and push them. runs quarterly Jairo Daniel Gerardo,Brian
increment databaseAday, /cluster/bin/scripts/databaseAday.csh problems only ChrisL, Cath, Brian Jairo, Brian, Lou Jairo, Brian, Lou
checkHgFindSpec -checkTermRegex, /cluster/bin/scripts/runCheckHgFindSpec go problems only ChrisL, Brian, Cath Jairo, Brian, Lou Jairo, Brian, Lou
check to see if there are new files to go with the CGI push /cluster/home/qateam/bin/scripts/findCgiFiles.csh go problems only ChrisL, Cath, Brian Jairo, Brian, Lou Jairo, Brian, Lou
get output from checkGenomeMysql (discovers corrupt tables) daily Brian Brian, Daniel Daniel, Gerardo
[checkGbibMd5.sh] compares the md5sum of gbibBeta.zip on dev with the sum for gbib.zip on genome-store, complaining if they don't match. runs weekly Jairo, Cath Jairo,Conner,Daniel Lou,Gerardo,Daniel
look for broken symlinks in /gbdb: daily Browser Staff Browser Staff Browser Staff
look for broken symlinks in in (very large) visiGene directory twice yearly ChrisL Conner Brian
look for broken symlinks in htdocs-hgdownload/goldenpath daily Browser Staff Browser Staff Browser Staff
Check the source tree for duplicate case-independent filenames. These wreck havoc with git. daily Browser Staff Browser Staff Browser Staff
sends a reminder to check the list of folks who have mirror sites. This list is created by a script of Hiram's and located here: http://genome-test.gi.ucsc.edu/admin/tools/useCount.html. The information comes from the phoneHome program in the CGIs (see #22593 for login/password and discussion about how an initial 'registration.txt' file is created then moved to trash, where presence of that file prevents the request being repeated so only an initial phone call is made). Sometimes companies set up mirrors, which is not OK. monthly Ann [was: Matt, Brian] Ann,Lou,Hiram Ann,Jairo,Hiram
Monthly reminder to check for stale New pennantIcon tags monthly Lou Lou
run /cluster/bin/scripts/sessionStats.csh NOTE: this is separate to each machine (genome-asia & genome-euro) thus cronjobs are specific to each machine and jonathan owns asia monthly Angie, Brian, Bob Angie, Brian, Bob Angie, Lou, Bob
ask Galt/Jim about blat updates 3x per year ChrisL,Cath Daniel, Conner Daniel, Gerardo
check that the ga4gh beacon server is running. Email Max if there is an issue. daily Jairo Conner Daniel
check that blat servers are running ok link to error log checking ongoing Ann, Brian, Galt, Matt, Erich, ChrisL, Jairo, Cath Ann, Galt, Erich, Matt, Brian, Conner, Daniel, Lou, Jairo Clay, Galt, Erich, Matt, Brian, Gerardo, Daniel, Lou, Jairo
receive output from hgTracksRandom cron; alert admins to problems as needed many: cath, chmalee, kuhn, brianlee, braney, ann,, jcasper, matt, jairo Bob,Ann, Galt, Braney, Jonathan, Christopher, Erich, Matt, Brian, Conner, Daniel, Lou, Jairo Clay, Bob, Braney, Jonathan, Erich, Matt, Brian, Gerardo, Daniel, Lou, Jairo
check that beta builds alright 3x week Brian, ChrisL Brian, Braney, Jairo, Lou Brian, Braney, Jairo, Lou
Retired (See #27779) hubPublicCheck complains when the hubPublic table doesn't have the same strings as the actual hub.txt file. (NOTE: To run only on Beta hubs requires these declarations: export HGDB_CONF=$HOME/.hg.conf.beta and SHELL=/bin/sh). runs daily, complains moderately ChrisL Daniel, Braney, Christopher Daniel, Braney, Christopher
Curl of http://genomewiki.ucsc.edu/index.php/Assembly_Hubs to test Public Wiki is accessible, output only if no response. hourly Brian Conner Daniel
qaAutoTrack -script that does some auto-QA on the autopushed tracks. daily Jario Jairo Jairo
checkAllStaticLinks.csh script that checks static pages. weekly Daniel,Conner,Brian Daniel,Gerardo,Brian
assemblyStatsCron.py pulls and reports GB usage stats over the last month from the RR+mirrors. monthly QA Team QA Team
HTTPs expiration check runs genecats/qa/testTools/checkHTTPSdir/checkHTTPS to report any expiring certificates weekly on Tuesday QA Team, cluster admin
Hub Public Coordinator maintains error-free Public Hubs hubs and keeps the hubs updated. Fully automated to update labels/titles, the search index, and contact hub provider when hub is down. Monitor the three cronjobs: hubPublicAutoUpdate, hubSearchUpdate, and hubPublicMail. More details can be found here. daily Daniel, Gerardo
from build cron
receive cron output for tableDescriptions build errors, AutoPush completed rare Cath QA Team QA Team
from galt's cron
run ReleaseLog script (this requires a copy of the kent repository) daily Galt Galt Galt
non-cron jobs
Check for MLQs that may have slipped through the cracks with Google Groups to check for unanswered Q's. Check to make sure that the last response for every ticket came from one of us. If not, look it up in redmine to ensure that A) there is indeed a redmine ticket for it, and B) it has been responded to. If not, notify the appropriate MLM with as much information about the MLQ as possible. weekly Cath Lou Daniel
Monitor & report UCSC Genome Browser outages (SLA). See wiki procedures hopefully, not much! Cath Conner Gerardo
update crontab in genecats/qa/crontabs/*.crontab (and log in as qateam and restart the qateam crontab with updates) as needed Brian Brian Brian
update UCSC Entrez LinkOut when necessary. Entrez LinkOut sends requests for changes (and statistics) to the browser-qa email address rarely ChrisL,Matt Daniel Gerardo
Check genome-preview machine is still running well weekly Jairo Conner Daniel
Updated Facebook with Announcements As needed Team Team Team
Update Twitter with Announcements As needed Team Team Team
otto cron jobs (/hive/data/outside/otto)
omimWrapper.sh checks for new OMIM data and downloads and loads it into hgwdev if so runs daily, complains bi-weekly jcasper ChrisL ChrisL
omimUploadWrapper.sh uploads the OMIM tables to OMIM runs daily, complains rarely jcasper ChrisL ChrisL
decipherWrapper.sh checks for new DECIPHER data and downloads and loads it into hgwdev if so runs daily, complains monthly jcasper ChrisL ChrisL
iscaWrapper.sh checks for new ISCA data and downloads and loads it into hgwdev if so runs daily, complains monthly jcasper ChrisL ChrisL
gwasWrapper.sh checks for new GWAS data and downloads and loads it into hgwdev if so runs weekly, complains quarterly jcasper ChrisL ChrisL
geneReviewsWrapper.sh checks for new GeneReviews data and downloads and loads it into hgwdev if so runs weekly, complains quarterly jcasper ChrisL ChrisL
otto cron jobs (/hive/data/outside/otto)
~jcasper/bin/scripts/updateHgConf.sh - checks for changes to hg.conf files and checks them in to the kent tree runs weekly jcasper jcasper jcasper
~jcasper/bin/backup_hgconfprivate.sh - checks most recent versions of hg.conf.private files into a private repository (~jcasper/hgconf_private/) runs twice a week jcasper jcasper jcasper