Assembly QA Part 4 RR Steps: Difference between revisions

From Genecats
Jump to navigationJump to search
 
(41 intermediate revisions by 5 users not shown)
Line 19: Line 19:


You can use  
You can use  
  hgwdev > updateTimesDb.sh -d $db
  updateTimesDb.sh -d $db
to compare table update times between hgwdev and hgwbeta. Everything should have the same update times except


to compare table update times between hgwdev and hgwbeta. Everything but hgFindSpec, history, tableDescriptions, trackDb and the genbank tables should have the same update times.  
hgFindSpec, history, tableDescriptions, trackDb and the genbank tables
author, cds, chainRBest*, chainSyn*, cell, description, development, gbCdnaInfo, gbExtFile, gbLoaded
gbMiscDiff, gbSeq, gbWarn, geneName, imageClone, keyword, library, mrnaClone, netRBest, netSyn,
organism, productName, refLink, refSeqStatus, refSeqSummary, sex, source, tissue
 
Many of these were moved to hgFixed (https://genome.ucsc.edu/FAQ/FAQdownloads.html#download35)


To see all of the tables in the assembly that are related to genbank do this:
To see all of the tables in the assembly that are related to genbank do this:
  hgwdev > hgsql -Ne 'show tables' $db | egrep -f /cluster/data/genbank/etc/genbank.tbls
  hgsql -Ne 'show tables' $db | egrep -f /cluster/data/genbank/etc/genbank.tbls


====<span style="color:dodgerblue">RR: Request push: rsync complete database====
====<span style="color:dodgerblue">RR: Request push: rsync complete database====
Line 31: Line 37:
*Announce you're gonna do a make public to Browser QA  
*Announce you're gonna do a make public to Browser QA  
*Make sure you have done a "make public" for your db.
*Make sure you have done a "make public" for your db.
* Request an rsync of the entire database from mysqlbeta to mysqlrr/euro/asia.
* Request an rsync of the entire database from hgwbeta to mysqlrr/euro/asia.
* Request drop of trackDb_public and hgFindSpec_public from mysqlrr/euro/asia
* Request drop of trackDb_public and hgFindSpec_public from mysqlrr/euro/asia
* Request push of trackDb and friends
* Request push of trackDb and friends
Line 91: Line 97:
hostname        mysqlrr
hostname        mysqlrr
</pre>
</pre>
'''Alternative''' (If inherited .hg.conf)
ssh qateam@hgw0
[qateam@hgw0 /home/qateam] HGDB_CONF=`pwd`/.hg.local.conf /home/qateam/bin/x86_64/hgsql -e 'show tables;' $db
----


'''How to access non-public sql-rr on genome-euro and genome-asia'''
'''How to access non-public sql-rr on genome-euro and genome-asia'''
Line 149: Line 164:
* Push chain/chainLink/net tables for each database.  
* Push chain/chainLink/net tables for each database.  
* See this [https://groups.google.com/a/soe.ucsc.edu/d/msg/push-request/eXHprKJRN-c/y4jnzq53AwAJ example push request ].
* See this [https://groups.google.com/a/soe.ucsc.edu/d/msg/push-request/eXHprKJRN-c/y4jnzq53AwAJ example push request ].
<b> To be kind to the pushers, send separate emails for each database!</b>


====<span style="color:dodgerblue">RR: Push done for chain nets for other assemblies ====
====<span style="color:dodgerblue">RR: Push done for chain nets for other assemblies ====
Line 157: Line 174:


* Push hgdownload files to RR
* Push hgdownload files to RR
* These files are listed in your Redmine File List.
* These files are listed in your Redmine File List, but DO NOT include /gbdb/ files in this list.
* Double-check the README & md5sum.txt file.   
* Double-check the README & md5sum.txt file.   


Line 169: Line 186:


====<span style="color:dodgerblue">RR: Push complete for downloads ====
====<span style="color:dodgerblue">RR: Push complete for downloads ====
</span>
====<span style="color:dodgerblue">RR: Push /gbdb files to the RR ====
</span>
Push the /gbdb files from hgwbeta to hgnfs1/euro/asia. Note that you CANNOT use * characters in these pushes. Expand them with ls before pushing
* [https://groups.google.com/a/soe.ucsc.edu/d/msg/push-request/Ac66Jjg9E-Q/ohDvrCBFAwAJ Example request]
====<span style="color:dodgerblue">RR: Push complete for /gbdb files ====
</span>
</span>


====<span style="color:dodgerblue">RR: Add or update symlink htdocs:hgdownload currentGenomes====
====<span style="color:dodgerblue">RR: Add or update symlink htdocs:hgdownload currentGenomes====
</span>  
</span>  
Your assembly should have a symlink in /usr/local/apache/htdocs-hgdownload/goldenPath/currentGenomes, you probably need to add it.  
Your assembly should have a symlink in /usr/local/apache/htdocs-hgdownload/goldenPath/currentGenomes, you probably need to update or add the symlink on /usr/local/apache/htdocs-hgdownload/goldenPath/currentGenomes so that it points to the most recent assembly for your organism. If your assembly is a new version and does not have a multiple alignment, you can skip the symlink step.
 
Update or add a symlink to
 
hgwdev


/usr/local/apache/htdocs-hgdownload/goldenPath/currentGenomes so that it points to the most recent assembly for your organism.  
NOTE that autocomplete adds a /, which will make your rm statement fail.  If you tab complete, remove the / at the end of the statement, otherwise it'll fail silently and your pushers won't be happy.


   rm Name_of_symlink
   rm Name_of_symlink
Line 187: Line 209:
<pre>
<pre>
/usr/local/apache/htdocs-hgdownload/goldenPath/currentGenomes
/usr/local/apache/htdocs-hgdownload/goldenPath/currentGenomes
ln -sf ../manPen1 Manis_pentadactyla
ln -s ../manPen1 Manis_pentadactyla
</pre>
</pre>


Line 230: Line 252:
Check for .txt.gz and .sql files on hgdownload:/usr/local/apache/htdocs/goldenPath/$db/database
Check for .txt.gz and .sql files on hgdownload:/usr/local/apache/htdocs/goldenPath/$db/database


To get a link for the database directory, you can copy paste this into your terminal, and then copy paste that into your web browser.
<pre>echo http://hgdownload.soe.ucsc.edu/goldenPath/$db/database/</pre>
E.g.,
E.g.,
http://hgdownload.soe.ucsc.edu/goldenPath/manPen1/database/
http://hgdownload.soe.ucsc.edu/goldenPath/manPen1/database/
Line 244: Line 268:
From hgwdev:
From hgwdev:


  * mysql -h genome-mysql -A -u genome $db
  mysql -h genome-mysql -A -u genome $db
  * mysql -h genome-mysql-euro -A -u genome $db
  mysql -h genome-mysql-euro -A -u genome $db


====<span style="color:dodgerblue">RR: Review copyHgcentral steps ====
====<span style="color:dodgerblue">RR: Review copyHgcentral steps ====
Line 252: Line 276:
You can copy items from hgcentraltest to hgcentral with the copyHgcentral script.  For the usage statement, run:
You can copy items from hgcentraltest to hgcentral with the copyHgcentral script.  For the usage statement, run:


  hgwdev > copyHgcentral -h
  copyHgcentral -h


* The copyHgcentral script must be run in test mode first.   
* The copyHgcentral script must be run in test mode first.   
Line 261: Line 285:


<pre>
<pre>
hgwdev > copyHgcentral test $db all beta rr
copyHgcentral test $db all beta rr
</pre>
</pre>


====<span style="color:dodgerblue">RR: hgCentral:dbDb set active equal to 1====
</span>
*Go to hgcentral and see what the 'active' field is set to (0=not visible, 1 = visible for gateway assembly version drop-down.)
<pre>
hgwdev > hgsql -h genome-centdb hgcentral
mysql > UPDATE dbDb SET active = 1 WHERE name = "$db";
</pre>


====<span style="color:dodgerblue">RR: copyHgcentral test db blatServers beta rr ====
====<span style="color:dodgerblue">RR: copyHgcentral test db blatServers beta rr ====
Line 280: Line 295:
</span>
</span>
<pre>
<pre>
hgwdev > copyHgcentral test $db blatServers beta rr
copyHgcentral test $db blatServers beta rr
</pre>
</pre>
<pre>
<pre>
hgwdev > copyHgcentral execute $db blatServers beta rr
copyHgcentral execute $db blatServers beta rr


</pre>
</pre>
Line 294: Line 309:


select * from blatServers where db='manPen1';
select * from blatServers where db='manPen1';
or
hgsql -h genome-centdb -e "select * from blatServers where db='felCat9' " hgcentral
</pre>
</pre>


Line 302: Line 321:


<pre>
<pre>
hgwdev > copyHgcentral test $db dbDb beta rr
copyHgcentral test $db dbDb beta rr
</pre>
</pre>
<pre>
<pre>
hgwdev > copyHgcentral execute $db dbDb beta rr
copyHgcentral execute $db dbDb beta rr
</pre>
</pre>


Line 318: Line 337:
select * from dbDb where name='manPen1' \G;
select * from dbDb where name='manPen1' \G;
</pre>
</pre>
====<span style="color:dodgerblue">RR: hgCentral:dbDb set active equal to 1====
</span>
'''Note'''- This step should be completed after copying hgcentralbeta to hgcentral**
*Go to hgcentral and see what the 'active' field is set to (0=not visible, 1 = visible for gateway assembly version drop-down.)
<pre>
hgsql -h genome-centdb hgcentral
UPDATE dbDb SET active = 1 WHERE name = "$db";
</pre>
Or without going into SQL, followed by a check that active=1
<pre>
hgsql -h genome-centdb -e "update dbDb set active = 1 where name = '$db'" hgcentral
hgsql -h genome-centdb -e "select * from dbDb where name = '$db'" hgcentral
</pre>


====<span style="color:dodgerblue">RR: copyHgcentral test db defaultDb beta rr ====
====<span style="color:dodgerblue">RR: copyHgcentral test db defaultDb beta rr ====
Line 325: Line 364:


<pre>
<pre>
hgwdev > copyHgcentral test $db defaultDb beta rr
copyHgcentral test $db defaultDb beta rr
</pre>
</pre>
<pre>
<pre>
hgwdev > copyHgcentral execute $db defaultDb beta rr
copyHgcentral execute $db defaultDb beta rr
</pre>
</pre>


Line 344: Line 383:
====<span style="color:dodgerblue">RR: make your new assembly the default ====
====<span style="color:dodgerblue">RR: make your new assembly the default ====


'''NOTE:'''This step should be added to the start of the copyhgCentral part, ad the update command should be done on central, then copied to beta, then to the rr the command then would be:
'''NOTE:'''This step should be added to the start of the copyhgCentral part, and the update command should be done on central, then copied to beta, then to the rr the command then would be:
<pre>
<pre>
hgsql hgcentraltest
hgsql hgcentraltest
Line 358: Line 397:
For human and mouse, likely the previous assembly will remain default for some time, check with management.
For human and mouse, likely the previous assembly will remain default for some time, check with management.
<pre>
<pre>
hgsql -h genome-centdb hgcentral
hgsql -h genome-centdb -e "update defaultDb SET name='$db' where name = 'galGal4'" hgcentral
update defaultDb SET name="susScr11" where name="susScr3";
hgsql -h genome-centdb -e "select * from defaultDb where genome='Chicken';" hgcentral
</pre>
</pre>


Line 383: Line 422:


<pre>
<pre>
hgwdev > copyHgcentral test $db genomeClade beta rr
copyHgcentral test $db genomeClade beta rr
</pre>
</pre>
<pre>
<pre>
hgwdev > copyHgcentral execute $db genomeClade beta rr
copyHgcentral execute $db genomeClade beta rr
</pre>
</pre>


Line 399: Line 438:
* More details on the [http://genomewiki.ucsc.edu/genecats/index.php/Chains_and_Nets_QA Chain and Net QA wiki page].
* More details on the [http://genomewiki.ucsc.edu/genecats/index.php/Chains_and_Nets_QA Chain and Net QA wiki page].
<pre>
<pre>
  hgsql -h mysqlbeta -Ne "SELECT * FROM liftOverChain WHERE fromDb = '$db' OR toDb = '$db'" hgcentralbeta > chain.beta  
  hgsql -h hgwbeta -Ne "SELECT * FROM liftOverChain WHERE fromDb = '$db' OR toDb = '$db'" hgcentralbeta > chain.beta  
</pre>
</pre>
Check public mysql, load if not present and recheck:
Check public mysql, load if not present and recheck:
Line 434: Line 473:


<pre>
<pre>
checkMetaData.csh manPen1 hgwbeta rr
checkMetaData.csh $db hgwbeta rr


database = manPen1
database = manPen1
Line 459: Line 498:
</pre>
</pre>


====<span style="color:dodgerblue">RR: Request push of gateway image====
====<span style="color:dodgerblue">RR: Request push of gateway image if not already there====
</span>
</span>


Line 531: Line 570:
* Add the new assembly to ~/kent/src/hg/makeDb/genbank/etc/rr.dbs in alphabetical order.   
* Add the new assembly to ~/kent/src/hg/makeDb/genbank/etc/rr.dbs in alphabetical order.   
* Be sure to save, git add, git commit, and git push the file.
* Be sure to save, git add, git commit, and git push the file.
***For now ignore this step, Hiram & Braney said that the update to align.dbs on beta is sufficient and will turn on Genbank updates


====<span style="color:dodgerblue">RR: GenBank updates: make libs and run make====
====<span style="color:dodgerblue">RR: GenBank updates: make libs and run make====
Line 551: Line 587:
* To see whether updates have run (the Monday after the *.dbs file was updated), check the update times of the table 'gbLoaded'.  
* To see whether updates have run (the Monday after the *.dbs file was updated), check the update times of the table 'gbLoaded'.  


hgwdev > updateTimes.csh $db gbLoaded verbose
updateTimes.csh $db gbLoaded verbose


For example, you'll see updates for dev/beta/rr/euro/asia):
For example, you'll see updates for dev/beta/rr/euro/asia):
Line 603: Line 639:
====<span style="color:dodgerblue">RR: Edit newsarch:html and add:commit:git:push====
====<span style="color:dodgerblue">RR: Edit newsarch:html and add:commit:git:push====
</span>
</span>
Edit in your personal kent directory, make beta, check work, add, commit, push


/usr/local/apache/htdocs/goldenPath/newsarch.html
You'll be editing:
~/htdocs/goldenPath/newsarch.html
And pushing:
/usr/local/apache/htdocs/goldenPath/newsarch.html


====<span style="color:dodgerblue">RR: Edit indexNews:html and add:commit:push====
====<span style="color:dodgerblue">RR: Edit indexNews:html and add:commit:push====
</span>
</span>
Edit in your personal kent directory, make beta, check work, add, commit, push


/usr/local/apache/htdocs/indexNews.html
You'll be editing:
~/htdocs/indexNews.html
And push requesting:
/usr/local/apache/htdocs/indexNews.html


====<span style="color:dodgerblue">RR: Edit credits:html and add:commit:push====
====<span style="color:dodgerblue">RR: Edit credits:html and add:commit:push====
</span>
</span>
/usr/local/apache/htdocs/goldenPath/credits.html
Edit in your personal kent directory, make beta, check work, add, commit, push
 
You'll be editing:
~/htdocs/goldenPath/credits.html
And push requesting:
/usr/local/apache/htdocs/goldenPath/credits.html


It's a good idea to add a link to the NCBI credits, because there is so much detail there.  
It's a good idea to add a link to the NCBI credits, because there is so much detail there.  
Line 620: Line 669:
====<span style="color:dodgerblue">RR: FAQreleases.html and add, commit, git push====
====<span style="color:dodgerblue">RR: FAQreleases.html and add, commit, git push====
</span>
</span>
Edit in your personal kent directory, make beta, check work, add, commit, push


/usr/local/apache/htdocs/FAQ/FAQreleases.html
You'll be editing:
~/htdocs/FAQ/FAQreleases.html
You'll be push requesting:
/usr/local/apache/htdocs/FAQ/FAQreleases.html


https://genome.ucsc.edu/FAQ/FAQreleases.html
https://genome.ucsc.edu/FAQ/FAQreleases.html


====<span style="color:dodgerblue">RR: Post Google Groups announcement====
</span>
See [https://groups.google.com/a/soe.ucsc.edu/forum/#!topic/genome-announce/rO4q4coTv8g announcement example].


====<span style="color:dodgerblue">RR: Request push for static docs====
====<span style="color:dodgerblue">RR: Request push for static docs====
Line 636: Line 685:
====<span style="color:dodgerblue">RR: Push complete for static docs and image and downloads ====
====<span style="color:dodgerblue">RR: Push complete for static docs and image and downloads ====
</span>
</span>
====<span style="color:dodgerblue">RR: Post Google Group, Facebook, and Twitter announcements====
</span>
See [https://groups.google.com/a/soe.ucsc.edu/forum/#!topic/genome-announce/rO4q4coTv8g announcement example]. Be sure to get it proofread.
Also post a FB and Twitter update.  See info [http://genomewiki.ucsc.edu/genecats/index.php/Facebook_update here].


====<span style="color:dodgerblue">RR: Write release log text in Redmine and close====
====<span style="color:dodgerblue">RR: Write release log text in Redmine and close====
Line 649: Line 705:


<pre>
<pre>
cat /hive/data/genomes/xenLae2/pushQ/redmine.xenLae2.releaseLog.txt | sed -n -e 'H;${x;s/\n/, /g;s/^,//;p;}' stdout
cat /hive/data/genomes/$db/pushQ/redmine.$db.releaseLog.txt | sed -n -e 'H;${x;s/\n/, /g;s/^,//;p;}'
</pre>
</pre>


Line 671: Line 727:


Notes: Be sure that the 'Release Log URL' starts with just two dots (../cgi-bin/ not ../../cgi-bin/) and that if you have multiple assemblies (hg19, hg38, mm10) for your data, then we should avoid a URL as it will populate the same link to all database sections
Notes: Be sure that the 'Release Log URL' starts with just two dots (../cgi-bin/ not ../../cgi-bin/) and that if you have multiple assemblies (hg19, hg38, mm10) for your data, then we should avoid a URL as it will populate the same link to all database sections
====<span style="color:dodgerblue">RR: Check track data in GBiB====
</span>
Check your data tracks in the GBiB
*GBiB pulls on data from hgdownload and the public MySQL server. Until the assembly is available in those places, it won't be visible in the GBiB.





Latest revision as of 22:22, 5 August 2021

These steps were revised in 2017, but you can also see the old steps: Releasing an assembly (old steps)


Navigation Menu

Home Page
Assembly QA Part 1: DEV Steps
Assembly QA Part 2: Track Steps
Assembly QA Part 3: BETA Steps
Assembly QA Part 4: RR Steps


RR: Check that tables do not need to be pushed again

You can use

updateTimesDb.sh -d $db

to compare table update times between hgwdev and hgwbeta. Everything should have the same update times except

hgFindSpec, history, tableDescriptions, trackDb and the genbank tables author, cds, chainRBest*, chainSyn*, cell, description, development, gbCdnaInfo, gbExtFile, gbLoaded gbMiscDiff, gbSeq, gbWarn, geneName, imageClone, keyword, library, mrnaClone, netRBest, netSyn, organism, productName, refLink, refSeqStatus, refSeqSummary, sex, source, tissue

Many of these were moved to hgFixed (https://genome.ucsc.edu/FAQ/FAQdownloads.html#download35)

To see all of the tables in the assembly that are related to genbank do this:

hgsql -Ne 'show tables' $db | egrep -f /cluster/data/genbank/etc/genbank.tbls

RR: Request push: rsync complete database

  • Announce you're gonna do a make public to Browser QA
  • Make sure you have done a "make public" for your db.
  • Request an rsync of the entire database from hgwbeta to mysqlrr/euro/asia.
  • Request drop of trackDb_public and hgFindSpec_public from mysqlrr/euro/asia
  • Request push of trackDb and friends
  • See an example push request

RR: Push complete for rsync database

RR: Double check tables on mysqlrr, euro, asia from rsync push

  • QA does not have write access to mysqlrr.
  • Tables are populated to mysqlrr first, then mirrored to public servers (genome-mysql and genome-euro-mysql)
  • QA can can set up read-only access to mysqrrr.

You can view the database contents on mysqlrr by creating a new .hg.conf file in your home dir using the read-only configuration. You can copy the needed config lines from the .hg.conf file on any of the RR machines in cgi-bin/hg.conf.private

Steps to create your own .hg.conf file which will allow you to query mysqlrr:

  1. ssh qateam@hgw0
  2. cat cgi-bin/hg.conf.private | grep -A6 "db.host is"
  3. note the contents of the above grep results
  4. exit

In your HOME directory on hgwdev, make up a file:

  • $HOME/.hg.mysqlrr.conf

then, add the following settings to the file, which you found in the above grep step:

db.host=mysqlrr
db.user=<userName>
db.password=<passWord>
db.trackDb=trackDb


next, change the permissions of your file:

chmod 600 .hg.mysqlrr.conf

Now you should be able to query mysqlrr with a query like this:

$ HGDB_CONF=$HOME/.hg.mysqlrr.conf hgsql -e 'show tables;' hg38Patch11
+-----------------------+
| Tables_in_hg38Patch11 |
+-----------------------+
| chromInfo             |
+-----------------------+

If you are ever in doubt about which SQL server you are working with:

$ HGDB_CONF=$HOME/.hg.mysqlrr.conf hgsql -e 'show variables;' hg38Patch11 | grep hostname
hostname        mysqlrr

Alternative (If inherited .hg.conf)

ssh qateam@hgw0

[qateam@hgw0 /home/qateam] HGDB_CONF=`pwd`/.hg.local.conf /home/qateam/bin/x86_64/hgsql -e 'show tables;' $db



How to access non-public sql-rr on genome-euro and genome-asia

genome-euro ssh qateam@genome-euro Last login: Thu Nov 9 21:11:50 2017 from hgwdev.gi.ucsc.edu [qateam@genome-euro /home/qateam]

  • hgsql -e "show databases like 'susScr%'";
  • hgsql -e "show tables" xenLae2;

genome-asia

  1. From dev: ssh cath@genome-asia.ucsc.edu
  2. Now you will be in [cath@genomebrowsermirror ~]$
  3. Now, do this special step only for asia:
  • sudo /bin/su - qateam

Then you can query mysqlrr on genome-asia like this:

[qateam@asiaNode /home/qateam] hgsql -e 'show databases like "%atch%";'
+-------------------+
| Database (%atch%) |
+-------------------+
| hg19Patch10       |
| hg38Patch11       |
| mm10Patch4        |
+-------------------+

RR: Check for Ensembl tracks

Review the Ensembl QA wiki for special procedures related to Ensembl tracks.

RR: Check for correct row in hgFixed.trackVersion for Ensembl tracks

  • Make sure the database hgFixed has the correct line/version for Ensembl for your assembly.
  • Never push the entire trackVersion table:
Be careful not to push any lines that are not ready to go the rr, there may be a row staged with a newer version that hasn't been released yet.
Note: Documentation needed on how to push rows. It is possible that the engineer will do this in advance, QA should check and then ask the engineer if the needed row is not already on the beta and rr table.
Grab rows for each Ensembl track using
  hgsql -Ne "select * from trackVersion where db='oviAri4'" hgFixed > oviAri4.trackVersion
Insert rows into trackVersion table in hgFixed by the 'load infile' command from beta steps
  • Once a line has been added to the trackVersion table on hgFixed, a normal push request for the table can be made.

RR: Request push: chain nets for other assemblies

  • At the start of the RR steps, you asked for an rsync of your database from beta>rr/euro/asia, so you have already pushed the chain/nets within your database. Now we need to do this for any other databases (aka, other assemblies that your assembly has chain/net alignments to).
  • Email BrowserQA with a make public notification (with no text, only a subject), wait a few minutes, and then make public from trackDb.
  make public DBS="$db $db2 $db3"
  • Push trackDb & friends for any other databases
  • Push chain/chainLink/net tables for each database.
  • See this example push request .

To be kind to the pushers, send separate emails for each database!

RR: Push done for chain nets for other assemblies

RR: Request push: hgdownload files

  • Push hgdownload files to RR
  • These files are listed in your Redmine File List, but DO NOT include /gbdb/ files in this list.
  • Double-check the README & md5sum.txt file.
  • Data files destined for hgdownload are organized on hgwdev at:
/usr/local/apache/htdocs-hgdownload/goldenPath/*

RR: Push complete for downloads

RR: Push /gbdb files to the RR

Push the /gbdb files from hgwbeta to hgnfs1/euro/asia. Note that you CANNOT use * characters in these pushes. Expand them with ls before pushing

RR: Push complete for /gbdb files

RR: Add or update symlink htdocs:hgdownload currentGenomes

Your assembly should have a symlink in /usr/local/apache/htdocs-hgdownload/goldenPath/currentGenomes, you probably need to update or add the symlink on /usr/local/apache/htdocs-hgdownload/goldenPath/currentGenomes so that it points to the most recent assembly for your organism. If your assembly is a new version and does not have a multiple alignment, you can skip the symlink step.

NOTE that autocomplete adds a /, which will make your rm statement fail. If you tab complete, remove the / at the end of the statement, otherwise it'll fail silently and your pushers won't be happy.

 rm Name_of_symlink
 ln -s ../$db Name_of_symlink

e.g.,

/usr/local/apache/htdocs-hgdownload/goldenPath/currentGenomes
ln -s ../manPen1 Manis_pentadactyla

More about symlinks on stackoverflow

RR: Request push of symlink for currentGenomes

  • Request a push of the symlink from hgwdev to hgdownload.
  • This is for ftp users who only want to go to the most recent assembly for an organism.
  • After it is pushed, check that it is functioning correctly on the current genomes ftp page.

See example push request.

RR: Push complete for currentGenomes symlink

This step is only for an assembly that is the NOT the 1st for the organism. A $db1 does not need this step.

RR: check symlink on ftp site

ftp://hgdownload.soe.ucsc.edu

The path is /goldenPath/currentGenomes/$db

ftp://hgdownload.soe.ucsc.edu/goldenPath/currentGenomes/

RR: Request push: start dump:autodump

See this example push request.

Notes:

genome-mysql syncs with hgdownload every night, so when you requested the autodump, then genome-mysql will automatically sync that night (anything new). If the autodump was completed at least 1 day ago, your new assembly should be available on genome-mysql, and a push request is not needed. If you do not want to wait 1 day for the nightly sync, you can request that the admins "make the $db database available on genome-mysql."

In the past, we would also request, "Links and​ ​permissions should be made for user, "genome" and "genomep".​ ​(Jorge says to follow​ ​the instructions in the wiki page for​ ​"Mirror_Server".)​"​ This information is no longer needed in the push request.

RR: Check that autodump is working

Check for .txt.gz and .sql files on hgdownload:/usr/local/apache/htdocs/goldenPath/$db/database

To get a link for the database directory, you can copy paste this into your terminal, and then copy paste that into your web browser.

echo http://hgdownload.soe.ucsc.edu/goldenPath/$db/database/

E.g., http://hgdownload.soe.ucsc.edu/goldenPath/manPen1/database/

RR: Check public genome-mysql

This is the public MySQL server - check that all tables are present. Tables/database will appear here nightly after they are on mysqlrr. For euro, updates are pushed out once per week, on the weekend.

Check for

  • your assembly
  • your 'other organism' assemblies for any chain/nets.


From hgwdev:

mysql -h genome-mysql -A -u genome $db
mysql -h genome-mysql-euro -A -u genome $db

RR: Review copyHgcentral steps

You can copy items from hgcentraltest to hgcentral with the copyHgcentral script. For the usage statement, run:

copyHgcentral -h
  • The copyHgcentral script must be run in test mode first.
  • Test mode will show you the state of hgcentraltest, hgcentralbeta and hgcentral.
  • Once test mode has been run and reviewed, run execute mode to copy from hgcentralbeta to hgcentral.
  • Note that test mode generates output files which must be manually deleted afterward. Be sure to run copyHgcentral in hive or your home directory and not in a directory where temp files should not be.
  • Note that copyHgcentral can be run for "all" (blatServers, dbDb, defaultDb, genomeClade):
copyHgcentral test $db all beta rr


RR: copyHgcentral test db blatServers beta rr

Generates files, run in hive:

copyHgcentral test $db blatServers beta rr
copyHgcentral execute $db blatServers beta rr

You can also check on mysql:

hgsql -h genome-centdb

use hgcentral;

select * from blatServers where db='manPen1';

or

hgsql -h genome-centdb -e "select * from blatServers where db='felCat9' " hgcentral

RR: copyHgcentral test db dbDb beta rr

Generates files, run in hive:

copyHgcentral test $db dbDb beta rr
copyHgcentral execute $db dbDb beta rr

You can also check on mysql:

hgsql -h genome-centdb

use hgcentral;

select * from dbDb where name='manPen1' \G;

RR: hgCentral:dbDb set active equal to 1

Note- This step should be completed after copying hgcentralbeta to hgcentral**


  • Go to hgcentral and see what the 'active' field is set to (0=not visible, 1 = visible for gateway assembly version drop-down.)
hgsql -h genome-centdb hgcentral
UPDATE dbDb SET active = 1 WHERE name = "$db";

Or without going into SQL, followed by a check that active=1

hgsql -h genome-centdb -e "update dbDb set active = 1 where name = '$db'" hgcentral
hgsql -h genome-centdb -e "select * from dbDb where name = '$db'" hgcentral


RR: copyHgcentral test db defaultDb beta rr

Generates files, run in hive:

copyHgcentral test $db defaultDb beta rr
copyHgcentral execute $db defaultDb beta rr

You can also check on mysql:

hgsql -h genome-centdb

use hgcentral;

select * from defaultDb where name="manPen1"limit 1;

RR: make your new assembly the default

NOTE:This step should be added to the start of the copyhgCentral part, and the update command should be done on central, then copied to beta, then to the rr the command then would be:

hgsql hgcentraltest
update defaultDb SET name="susScr11" where name="susScr3";

Ignore commands above, just a note for the future.


If your organism has multiple assemblies, you should make your newest assembly the 'default' assembly - it will appear by default in the drop-down list for assemblies for that organism in the gateway page. There may be reasons to leave the older assembly as the default, for example, if the newer assembly is scaffold-only and has no chroms, or if the older assembly has more tracks. If unsure, ask the engineer and/or management which assembly should be default.

For human and mouse, likely the previous assembly will remain default for some time, check with management.

hgsql -h genome-centdb -e "update defaultDb SET name='$db' where name = 'galGal4'" hgcentral
hgsql -h genome-centdb -e "select * from defaultDb where genome='Chicken';" hgcentral

RR: copyHgcentral test db genomeClade beta rr

NOTE: This table probably will not need to be updated. It contains records like this:

mysql> select * from genomeClade order by rand() limit 5;
+-----------------+------------+----------+
| genome          | clade      | priority |
+-----------------+------------+----------+
| GRCh38.p2       | haplotypes |      134 |
| C. japonica     | worm       |       70 |
| Atlantic cod    | vertebrate |      125 |
| D. melanogaster | insect     |       10 |
| D. persimilis   | insect     |       55 |
+-----------------+------------+----------+

Generates files, run in hive:

copyHgcentral test $db genomeClade beta rr
copyHgcentral execute $db genomeClade beta rr

RR: copyHgcentral: liftOverChain: manual move

liftOverChain is not copied with the copyHgcentral script, it needs to be copied manually.

  • Only copy lines from liftOverChain on hgcentralbeta to hgcentral if there are liftOver files listed in the pushQ and if the assemblies they go to/from exist on the RR.
  • Check for lines in liftOverChain that should be in the pushQ, but aren't (e.g., the liftOver from a previous assembly).
  • Add lines related to your assembly, any previous versions of your organism, and any other organisms that are associated with liftOver files and your assembly.
  • More details on the Chain and Net QA wiki page.
 hgsql -h hgwbeta -Ne "SELECT * FROM liftOverChain WHERE fromDb = '$db' OR toDb = '$db'" hgcentralbeta > chain.beta 

Check public mysql, load if not present and recheck:

hgsql -h genome-centdb -Ne "SELECT * FROM liftOverChain WHERE fromDb = '$db' OR toDb = '$db'" hgcentral 

Example: hgsql -h genome-centdb -Ne "SELECT * FROM liftOverChain WHERE fromDb = 'manPen1' OR toDb = 'manPen1'" hgcentral

hgsql -h genome-centdb -e "LOAD DATA LOCAL INFILE 'chain.beta' INTO TABLE liftOverChain" hgcentral

TIP: An easy way to add liftOver rows!

  1. Run $ checkMetaData.csh $db hgwbeta RR
  2. This creates files in the dir in which you run it.
  3. Use the file "liftOverChain.$db.hgcentralbetaOnly" as your LOAD file:
hgsql -h genome-centdb -e "LOAD DATA LOCAL INFILE 'liftOverChain.susScr11.hgcentralbetaOnly' INTO TABLE liftOverChain" hgcentral

After loading, run checkMetaData.csh $db again to make sure there are no differences.

RR: checkMetaData

After completing copyHgcentral steps, run checkMetaData.csh $db hgwbeta rr

  • This checks that all of the metadata is the same on hgcentraltest, hgcentralbeta, and hgcentral.
  • Run this script in a temporary folder or hive; it creates some comparison files that can be deleted after the check.

Example output below.

  • The "table.$db.common" rows should not be zero (they have X rows in common that are the same).
  • The rows above "common" (hgcentralbetaOnly & hgcentralOnly) should be zero (they have zero differences between them). If they are not zero, it means that there are differences among the the row/s for that table, so you'll need to figure out what's different and sync them. One possibly problem is that you have just run, "copyHgcentral all" and dbDb active was set to 0, where the others are set to 1.
checkMetaData.csh $db hgwbeta rr

database = manPen1

  0 dbDb.manPen1.hgcentralbetaOnly
  0 dbDb.manPen1.hgcentralOnly
  1 dbDb.manPen1.common

  0 blatServers.manPen1.hgcentralbetaOnly
  0 blatServers.manPen1.hgcentralOnly
  2 blatServers.manPen1.common

  0 defaultDb.manPen1.hgcentralbetaOnly
  0 defaultDb.manPen1.hgcentralOnly
  1 defaultDb.manPen1.common

  0 genomeClade.manPen1.hgcentralbetaOnly
  0 genomeClade.manPen1.hgcentralOnly
  1 genomeClade.manPen1.common

  0 liftOverChain.manPen1.hgcentralbetaOnly
  0 liftOverChain.manPen1.hgcentralOnly
  4 liftOverChain.manPen1.common

RR: Request push of gateway image if not already there

Please push the following files from hgwbeta --> RR/euro/asia.

/usr/local/apache/htdocs/images/Manis_pentadactyla.jpg

RR: Push complete of gateway image

RR: Check all tracks

Check your tracks on the RR. Also check that all default tracks still display.

RR: Check search

Check that searches work (if not, you probably need to push the hgFindSpec_public table).

RR: Check Tools: LiftOver

Check LiftOver on rr.

RR: Check Tools: BLAT: dna

Copy some dna and check BLAT for a DNA search on rr.

RR: Check Tools: BLAT: protein

Copy some protein and check BLAT for a protein search on RR.

RR: Check Tools: isPCR

Check Tools > In Silico PCR on the RR. See how to test PCR details from DEV steps

RR: Check Table Browser

Check Table Browser functions, try exporting sequence, look at tables via TB, look at schemas.

RR: Check chain nets tracks for other assemblies

QA chain/nets for the other assemblies.

If this is an assembly update to an existing organism, you will need to remove the previous chain/net tables from mysqlrr/genome-euro/genome-asia, i.e.,

MySQL [mm10]> show tables like '%anTro5%';
+---------------------------+
| Tables_in_mm10 (%anTro5%) |
+---------------------------+
| chainPanTro5              |
| chainPanTro5Link          |
| netPanTro5                |
+---------------------------+

RR: QA everything on asia

RR: QA everything on euro

RR: Turn on GenBank updates add:commit:push

  • Once your assembly is listed in align.dbs see this section in BETA steps, turn on GenBank updates on the rr before 4:30 p.m.
  • Genbank updates run once a week on the rr, check for updated the Monday after starting genbank updates.
  • Add the new assembly to ~/kent/src/hg/makeDb/genbank/etc/rr.dbs in alphabetical order.
  • Be sure to save, git add, git commit, and git push the file.

RR: GenBank updates: make libs and run make

After committing the change, make sure your libs are up to date:

cd ~/kent/src ; make libs

then go ahead and run the make:

cd ~/kent/src/hg/makeDb/genbank/ 
git pull 
make install-server

RR: GenBank updates: check Genbank update times

  • Genbank updates run once a week on the RR.
  • To see whether updates have run (the Monday after the *.dbs file was updated), check the update times of the table 'gbLoaded'.

updateTimes.csh $db gbLoaded verbose

For example, you'll see updates for dev/beta/rr/euro/asia):

realTime.csh manPen1 gbLoaded verbose

gbLoaded
=============
dev  2017-04-29 11:48:19
beta 2017-04-29 11:48:19

rr   2017-04-20 11:26:10
euro 2017-04-20 20:26:10
asia (null) (null)

The update times will be out of sync between machines, but not by more than 24 hours or so if updates are running. The gbLoaded table will be updated regardless of whether changes to other GenBank tables were picked up. More genbank update instructions are available at Genbank updates.

The etc-update-server part of the make will cause the downloads mentioned below in the "Verify downloads" section to be created.

RR: Edit downloads:html and add:commit:push

  • git pull (since this is a different repo, be sure you have the latest version).
  • Add your assembly to downloads.html (~/hgdownload/downloads.html).
  • Remember to add any necessary links to other organisms/assemblies, such as "vs" files.
  • git add, commit, push.
  • ENCODE NOTE: If you are pushing ENCODE tracks, when using a second/third/fourth version of the data there is often a "releaseLatest" directory that has the latest files. Be sure that you are not pushing the entire releaseLatest directory, only the files from there.

RR: Push request for downloads:html

example push request

RR: Push request for downloads:html complete

RR: Check hgdownload files generated by Genbank process

Make sure that the files generated by Genbank that are mentioned in the bigZips/README file, (e.g. mrna.fa.gz) are present on hgdownload.

For example, Here is the README for manPen1 in the bigZips dir: http://hgdownload.soe.ucsc.edu/goldenPath/manPen1/bigZips/

And I can see the file xenoRefMrna.fa.gz.

RR: Edit newsarch:html and add:commit:git:push

Edit in your personal kent directory, make beta, check work, add, commit, push

You'll be editing:

~/htdocs/goldenPath/newsarch.html

And pushing:

/usr/local/apache/htdocs/goldenPath/newsarch.html

RR: Edit indexNews:html and add:commit:push

Edit in your personal kent directory, make beta, check work, add, commit, push

You'll be editing:

~/htdocs/indexNews.html

And push requesting:

/usr/local/apache/htdocs/indexNews.html

RR: Edit credits:html and add:commit:push

Edit in your personal kent directory, make beta, check work, add, commit, push

You'll be editing:

~/htdocs/goldenPath/credits.html

And push requesting:

/usr/local/apache/htdocs/goldenPath/credits.html

It's a good idea to add a link to the NCBI credits, because there is so much detail there. For example, on manPen1, there is a link to NCBI credits.

RR: FAQreleases.html and add, commit, git push

Edit in your personal kent directory, make beta, check work, add, commit, push

You'll be editing:

~/htdocs/FAQ/FAQreleases.html

You'll be push requesting:

/usr/local/apache/htdocs/FAQ/FAQreleases.html

https://genome.ucsc.edu/FAQ/FAQreleases.html


RR: Request push for static docs

When writing the announcement for the home (indexNews) page and the news (newsarch) page, please include the common name of the organism, in addition to the scientific name and assembly name. e.g. African clawed frog, xenLae2 (Xenopus laevis).

RR: Push complete for static docs and image and downloads

RR: Post Google Group, Facebook, and Twitter announcements

See announcement example. Be sure to get it proofread.

Also post a FB and Twitter update. See info here.

RR: Write release log text in Redmine and close

In Redmine

  • Write in the redmine release log field, see example on the release log
  • Mark the checkbox "Released to RR"
  • Change status to "Released."

Replace line break with comma: Reminder to add a missing close parenthesis. Example:

cat /hive/data/genomes/$db/pushQ/redmine.$db.releaseLog.txt | sed -n -e 'H;${x;s/\n/, /g;s/^,//;p;}'

RR: Next day: check release log

Helpful wiki pages: redmine instead of the pushQ process for Tracks in redmine

Jonathan's program scans Redmine for 4 things:

  • Status = Released
  • Released to RR = checked
  • Release Log Text = populated (that's enough to get an entry in the new RL)
  • If the program also finds something in Release Log URL, it will turn the text into a link.


Notes: Be sure that the 'Release Log URL' starts with just two dots (../cgi-bin/ not ../../cgi-bin/) and that if you have multiple assemblies (hg19, hg38, mm10) for your data, then we should avoid a URL as it will populate the same link to all database sections

RR: Check track data in GBiB

Check your data tracks in the GBiB

  • GBiB pulls on data from hgdownload and the public MySQL server. Until the assembly is available in those places, it won't be visible in the GBiB.


🔵 Done with RR steps? You're done! Go grab yourself some coffee and a delicious donut, you deserve it!