Assembly QA Part 4 RR Steps: Difference between revisions
mNo edit summary |
mNo edit summary |
||
Line 64: | Line 64: | ||
See this [https://groups.google.com/a/soe.ucsc.edu/forum/#!searchin/push-request/dump$20autodump%7Csort:date/push-request/RaJuwbp5NnI/nD7kEAYGAQAJ example push request]. | See this [https://groups.google.com/a/soe.ucsc.edu/forum/#!searchin/push-request/dump$20autodump%7Csort:date/push-request/RaJuwbp5NnI/nD7kEAYGAQAJ example push request]. | ||
====<span style="color:dodgerblue"> | ====<span style="color:dodgerblue">RR: Review copyHgcentral steps ==== | ||
</span> | </span> | ||
You can copy items from hgcentraltest to | You can copy items from hgcentraltest to hgcentral with the copyHgcentral script. For the usage statement, run: | ||
hgwdev > copyHgcentral -h | hgwdev > copyHgcentral -h | ||
Line 73: | Line 73: | ||
* The copyHgcentral script must be run in test mode first. | * The copyHgcentral script must be run in test mode first. | ||
* Test mode will show you the state of hgcentraltest, hgcentralbeta and hgcentral. | * Test mode will show you the state of hgcentraltest, hgcentralbeta and hgcentral. | ||
* Once test mode has been run and reviewed, run execute mode to copy from | * Once test mode has been run and reviewed, run execute mode to copy from hgcentralbeta to hgcentral. | ||
* Note that test mode generates output files which must be manually deleted afterward. Be sure to run copyHgcentral in hive or your home directory and not in a directory where temp files should not be. | * Note that test mode generates output files which must be manually deleted afterward. Be sure to run copyHgcentral in hive or your home directory and not in a directory where temp files should not be. | ||
* Note that copyHgcentral can be run for "all" (blatServers, dbDb, defaultDb, genomeClade): | * Note that copyHgcentral can be run for "all" (blatServers, dbDb, defaultDb, genomeClade): | ||
<pre> | <pre> | ||
hgwdev > copyHgcentral test $db all | hgwdev > copyHgcentral test $db all beta rr | ||
</pre> | </pre> | ||
====<span style="color:dodgerblue">Beta: copyHgcentral test $db blatServers | ====<span style="color:dodgerblue">Beta: copyHgcentral test $db blatServers beta rr ==== | ||
Generates files, run in hive: | Generates files, run in hive: | ||
Line 87: | Line 87: | ||
</span> | </span> | ||
<pre> | <pre> | ||
hgwdev > copyHgcentral test $db blatServers | hgwdev > copyHgcentral test $db blatServers beta rr | ||
</pre> | </pre> | ||
<pre> | <pre> | ||
hgwdev > copyHgcentral execute $db blatServers | hgwdev > copyHgcentral execute $db blatServers beta rr | ||
</pre> | </pre> | ||
You can also check on mysql: | |||
<pre> | |||
hgsql -h genome-centdb | |||
use hgcentral; | |||
use | |||
select * from blatServers where db='manPen1'; | select * from blatServers where db='manPen1'; | ||
Line 113: | Line 115: | ||
</pre> | </pre> | ||
You can also check on | </pre> | ||
You can also check on mysql: | |||
<pre> | <pre> | ||
use | hgsql -h genome-centdb | ||
use hgcentral; | |||
select * from dbDb where name='manPen1' \G; | select * from dbDb where name='manPen1' \G; | ||
Line 133: | Line 138: | ||
</pre> | </pre> | ||
You can also check on | </pre> | ||
You can also check on mysql: | |||
<pre> | <pre> | ||
use | hgsql -h genome-centdb | ||
use hgcentral; | |||
select * from defaultDb where name="manPen1"limit 1; | select * from defaultDb where name="manPen1"limit 1; | ||
Line 161: | Line 170: | ||
<pre> | <pre> | ||
hgwdev > copyHgcentral test $db genomeClade | hgwdev > copyHgcentral test $db genomeClade beta rr | ||
</pre> | </pre> | ||
<pre> | <pre> | ||
hgwdev > copyHgcentral execute $db genomeClade | hgwdev > copyHgcentral execute $db genomeClade beta rr | ||
</pre> | </pre> | ||
Line 172: | Line 181: | ||
liftOverChain is not copied with the copyHgcentral script, it needs to be copied manually. | liftOverChain is not copied with the copyHgcentral script, it needs to be copied manually. | ||
* Only copy lines from liftOverChain on | * Only copy lines from liftOverChain on hgcentralbeta to hgcentral if there are liftOver files listed in the pushQ and if the assemblies they go to/from exist on the RR. | ||
* Check for lines in liftOverChain that should be in the pushQ, but aren't (e.g., the liftOver from a previous assembly). | * Check for lines in liftOverChain that should be in the pushQ, but aren't (e.g., the liftOver from a previous assembly). | ||
* Add lines related to your assembly, any previous versions of your organism, and any other organisms that are associated with liftOver files and your assembly. | * Add lines related to your assembly, any previous versions of your organism, and any other organisms that are associated with liftOver files and your assembly. | ||
* More details on the [http://genomewiki.ucsc.edu/genecats/index.php/Chains_and_Nets_QA Chain and Net QA wiki page]. | * More details on the [http://genomewiki.ucsc.edu/genecats/index.php/Chains_and_Nets_QA Chain and Net QA wiki page]. | ||
<pre> | <pre> | ||
hgsql -Ne "SELECT * FROM liftOverChain WHERE fromDb = '$db' OR toDb = '$db'" | hgsql -Ne "SELECT * FROM liftOverChain WHERE fromDb = '$db' OR toDb = '$db'" hgcentralbeta > chain.dev | ||
</pre> | </pre> | ||
Check | Check public mysql, load if not present and recheck: | ||
<pre> | <pre> | ||
hgsql -h genome-centdb -Ne "SELECT * FROM liftOverChain WHERE fromDb = '$db' OR toDb = '$db'" hgcentral | |||
</pre> | |||
Example: | |||
hgsql -h genome-centdb -Ne "SELECT * FROM liftOverChain WHERE fromDb = 'manPen1' OR toDb = 'manPen1'" hgcentral | |||
<pre> | |||
hgsql -h genome-centdb -e "LOAD DATA LOCAL INFILE 'chain.dev' INTO TABLE liftOverChain" hgcentral | |||
</pre> | </pre> | ||
====<span style="color:dodgerblue"> | ====<span style="color:dodgerblue">RR: checkMetaData ==== | ||
</span> | </span> | ||
After completing copyHgcentral steps, run checkMetaData.csh $db | After completing copyHgcentral steps, run checkMetaData.csh $db | ||
* This checks that all of the metadata is the same on | * This checks that all of the metadata is the same on hgcentraltest, hgcentralbeta, and hgcentral. | ||
* Run this script in a temporary folder or hive; it creates some comparison files that can be deleted after the check. | * Run this script in a temporary folder or hive; it creates some comparison files that can be deleted after the check. | ||
Revision as of 22:11, 20 April 2017
This page is currently a draft in progress. For now, use Releasing an assembly instead.
Navigation Menu |
RR: Populate spreadsheet steps
- We need to create a checklist for your RR steps.
- You can add a new tab to track RR steps, or you can pick up where you left off on the same tab as your "dev" and "beta" steps.
- To populate the "wiki link" for each step, add this formula to cell A2 (or the row after your last "dev" step) in your new "track checklist" spreadsheet and drag the formula down:
- A2 (or the row after your last "beta" step)
=HYPERLINK("http://genomewiki.ucsc.edu/genecats/index.php/Assembly_QA_Part_4_RR_Steps#"&SUBSTITUTE(B2," ", "_"),"link")
- IMPORTANT: Drag the formula for "A" down the spreadsheet to populate the other rows.
- To populate the "track checklist steps," add this formula to cell B2 ((or the row after your last "dev" step) in your new "track checklist" spreadsheet. Do NOT drag the formula down.
- B2
=IMPORTXML("http://genomewiki.ucsc.edu/genecats/index.php/Assembly_QA_Part_4_RR_Steps#", "/html/body/div/div/div/div/div/h4/span/span")
This formula will populate all the rows below it with the wiki section titles. You do no need to drag this formula down.
RR: Push Request: rsync complete database
- Request an rsync of the entire database from mysqlbeta to mysqlrr/euro/asia.
- Request drop of trackDb_public and hgFindSpec_public from mysqlrr/euro/asia
- Request push of trackDb and friends
- See an example push request
RR: Push Downloads
If there are files associated with the track that are to be pushed to hgdownload, check to see that there is a README that makes sense and the files have an md5sum.txt file that goes with them and is correct. Check the file itself to make sure it is not corrupted and that it contains what is expected. If it is a gzipped file, you can do "zcat file.gz | head" and "zcat file.gz | tail" to look at it. Looking at the last part of the file can sometimes catch corruption that can't be seen by only looking at the first part.
Data files destined for hgdownload are organized on hgwdev at:
/usr/local/apache/htdocs-hgdownload/goldenPath/*
and can be viewed in a browser from http://hgdownload-test.cse.ucsc.edu/downloads.html. Non-data files (such as downloads.html) are in the "hgdownload" git repository. See the Static_Page_Protocol for instructions on checking out that repository. Push requests for downloads should look like something like this:
Please push files from here on hgwdev: /usr/local/apache/htdocs-hgdownload/goldenPath/$db/file To here on hgdownload: /usr/local/apache/htdocs/goldenPath/$db/file (in the path, "htdocs-hgdownload" should become "htdocs")
RR: Push Request: start dump/autodump
- Database rsync should be complete before doing this.
- Request start of dump/autodump for your assembly on rr/euro/asia.
See this example push request.
RR: Review copyHgcentral steps
You can copy items from hgcentraltest to hgcentral with the copyHgcentral script. For the usage statement, run:
hgwdev > copyHgcentral -h
- The copyHgcentral script must be run in test mode first.
- Test mode will show you the state of hgcentraltest, hgcentralbeta and hgcentral.
- Once test mode has been run and reviewed, run execute mode to copy from hgcentralbeta to hgcentral.
- Note that test mode generates output files which must be manually deleted afterward. Be sure to run copyHgcentral in hive or your home directory and not in a directory where temp files should not be.
- Note that copyHgcentral can be run for "all" (blatServers, dbDb, defaultDb, genomeClade):
hgwdev > copyHgcentral test $db all beta rr
Beta: copyHgcentral test $db blatServers beta rr
Generates files, run in hive:
hgwdev > copyHgcentral test $db blatServers beta rr
hgwdev > copyHgcentral execute $db blatServers beta rr
You can also check on mysql:
hgsql -h genome-centdb use hgcentral; select * from blatServers where db='manPen1';
Beta: copyHgcentral test $db dbDb dev beta
Generates files, run in hive:
hgwdev > copyHgcentral test $db dbDb dev beta
hgwdev > copyHgcentral execute $db dbDb dev beta
You can also check on mysql:
hgsql -h genome-centdb use hgcentral; select * from dbDb where name='manPen1' \G;
Beta: copyHgcentral test $db defaultDb dev beta
Generates files, run in hive:
hgwdev > copyHgcentral test $db defaultDb dev beta
hgwdev > copyHgcentral execute $db defaultDb dev beta
You can also check on mysql:
hgsql -h genome-centdb use hgcentral; select * from defaultDb where name="manPen1"limit 1;
Beta: copyHgcentral test $db genomeClade dev beta
NOTE: This table probably will not need to be updated. It contains records like this:
mysql> select * from genomeClade order by rand() limit 5; +-----------------+------------+----------+ | genome | clade | priority | +-----------------+------------+----------+ | GRCh38.p2 | haplotypes | 134 | | C. japonica | worm | 70 | | Atlantic cod | vertebrate | 125 | | D. melanogaster | insect | 10 | | D. persimilis | insect | 55 | +-----------------+------------+----------+
Generates files, run in hive:
hgwdev > copyHgcentral test $db genomeClade beta rr
hgwdev > copyHgcentral execute $db genomeClade beta rr
Beta: copyHgcentral: liftOverChain (manual move)
liftOverChain is not copied with the copyHgcentral script, it needs to be copied manually.
- Only copy lines from liftOverChain on hgcentralbeta to hgcentral if there are liftOver files listed in the pushQ and if the assemblies they go to/from exist on the RR.
- Check for lines in liftOverChain that should be in the pushQ, but aren't (e.g., the liftOver from a previous assembly).
- Add lines related to your assembly, any previous versions of your organism, and any other organisms that are associated with liftOver files and your assembly.
- More details on the Chain and Net QA wiki page.
hgsql -Ne "SELECT * FROM liftOverChain WHERE fromDb = '$db' OR toDb = '$db'" hgcentralbeta > chain.dev
Check public mysql, load if not present and recheck:
hgsql -h genome-centdb -Ne "SELECT * FROM liftOverChain WHERE fromDb = '$db' OR toDb = '$db'" hgcentral
Example: hgsql -h genome-centdb -Ne "SELECT * FROM liftOverChain WHERE fromDb = 'manPen1' OR toDb = 'manPen1'" hgcentral
hgsql -h genome-centdb -e "LOAD DATA LOCAL INFILE 'chain.dev' INTO TABLE liftOverChain" hgcentral
RR: checkMetaData
After completing copyHgcentral steps, run checkMetaData.csh $db
- This checks that all of the metadata is the same on hgcentraltest, hgcentralbeta, and hgcentral.
- Run this script in a temporary folder or hive; it creates some comparison files that can be deleted after the check.
RR: Check all tracks on the RR
Check your track on the RR. Check that searches work (if not, you probably need to push the hgFindSpec_public table). Also check that all default tracks still display. If you filled in the "Release Log URL" field in the push queue, check the next day to be sure that the link from the release log works as expected.
RRs: Edit downloads.html
Check to see if the downloads files ought to have a link from downloads.html. If so, add the link and push downloads.html (after the files are already pushed!). NOTE: If you are pushing ENCODE tracks, when using a second/third/fourth version of the data there is often a "releaseLatest" directory that has the latest files. Be sure that you are not pushing the entire releaseLatest directory, only the files from there. Be sure to add a helpful sentence in your push-request to tip off the admin about this unusual push.
RR: Push chain/nets for other assemblies
- At the start of the RR steps, you asked for an rsync of your database from beta>rr/euro/asia, so you have already pushed the chain/nets within your database. Now we need to do this for any other databases (aka, other assemblies that your assembly has chain/net alignments to).
- Push trackDb & friends for any other databases
- Push chain/chainLink/net tables for each database.
- See this example push request .
RR: Chain chain/nets tracks for other assemblies
QA chain/nets for the other assemblies.
RR: Write Google Groups announcement
See announcement example.
RR: Write release log text in Redmine & close
🔵 Done with RR steps? Go to Assembly QA Part 5: Post Release Steps