Assembly QA Part 3 BETA Steps: Difference between revisions

From Genecats
Jump to navigationJump to search
 
(125 intermediate revisions by 6 users not shown)
Line 1: Line 1:
<div style="border: 2px solid #A3B1BF; padding: .5em 1em 1em 1em; border-top: none; background-color: #FFFEFB; color: #000;">
<div style="border: 2px solid #A3B1BF; padding: .5em 1em 1em 1em; border-top: none;">


<span style="color:olive">See also: [[Releasing_an_assembly | Releasing an assembly (old steps)]] </span>
<span style="color:olive">These steps were revised in 2017, but you can also see the old steps: [[Releasing_an_assembly | Releasing an assembly (old steps)]] </span>


{| class=wikitable width=415 align=right
{| class=wikitable width=415 align=right
Line 22: Line 22:
====<span style="color:dodgerblue">Beta: Copy redmine table list to hive====
====<span style="color:dodgerblue">Beta: Copy redmine table list to hive====
</span>
</span>
Should have been done in Dev steps: Alignment files are to valid assemblies, if not copy file and table list to assembly hive directory as below.


In Redmine for your assembly, look at the field, "Table List." The engineer should have provided a path to redmine.$db.table.list
In Redmine for your assembly, look at the field, "Table List." The engineer should have provided a path to redmine.$db.table.list
E.g.,
E.g.,
/hive/data/genomes/manPen1/redmine5515/redmine.manPen1.table.list
/hive/data/genomes/manPen1/pushQ/redmine.manPen1.table.list


Your file contents should be something like this:
Your file contents should be something like this:
Line 41: Line 43:


<pre>
<pre>
cp /hive/data/genomes/manPen1/redmine5515/redmine.manPen1.table.list .
cp /hive/data/genomes/$db/pushQ/redmine.$db.table.list .
</pre>
</pre>


Line 47: Line 49:


<pre>
<pre>
cut -d'.' -f1 redmine.manPen1.table.list | sort -u
cut -d'.' -f1 redmine.$db.table.list | sort -u


hg38
hg38
Line 65: Line 67:
You'll be using this cleanTableList when doing your push.
You'll be using this cleanTableList when doing your push.


====<span style="color:dodgerblue">Beta: Remove seq and extFile tables from your HIVE table list====
Another way to do this:
 
<pre>
grep -Ev "(trackDb|hgFindSpec)" redmine.$db.table.list > tableList.hgFindSpecANDtrackDb.removed
</pre>
 
====<span style="color:dodgerblue">Beta: Special steps for extFile tables====
</span>
</span>


* If this is a new assembly and this is the first Conservation track, the extFile table probably doesn't exist on beta and likely only has a single entry on dev.  If this is the case, it is ok to just push the entire table from dev to beta.
* Include extFile table in push request when you are ready to push to the RR.
* Follow the steps above to remove seq or extFile tables from cleanTableList.  
* Follow the steps above to remove seq or extFile tables from cleanTableList.  
* NOTE: Do not push seq or extFile tables from dev to beta.  
* NOTE: Do not push seq or extFile tables from dev to beta.  
* You must use the copyExtSeqRows.csh script to move only the rows needed.  
* You must use the copyExtSeqRows.csh script to move only the rows needed.  
* More information can be found [[Conservation_Track_QA#Figure_out_extFile_and_seq_tables| on the conservation track wiki page]].


====<span style="color:dodgerblue">Beta: Make clean table lists for pushing====
* There may be more than one set of entries in the extFile table.  Make sure you only push the set that pertains to the actual files you are pushing to hgnfs1 (e.g. /gbdb/ponAbe2/multiz8way/anno/maf/*)
* These are the lines that will need to be added to beta:
mysql> select path from extFile where path like "%anno/maf%";
 
* See resource [[Conservation_Track_QA#Figure_out_extFile_and_seq_tables| on the conservation track wiki page]].
 
====<span style="color:dodgerblue">Beta: Make table PUSH lists for each assembly====
</span>
</span>


Line 80: Line 95:


Using the example above, make 3 files with a list of tables to push for each assembly,  
Using the example above, make 3 files with a list of tables to push for each assembly,  
* 1 for manPen1 with tables to push
* manPen1 with tables to push
<pre>
<pre>
cat redmine.manPen1.table.list| grep "mm10." | cut -d'.' -f2 > mm10.push
db="manPen1"
cat redmine.$db.table.list| grep "mm10." | cut -d'.' -f2 > mm10.push
</pre>
</pre>
* 1 for hg38 with tables to push
* hg38 with tables to push
<pre>
<pre>
cat redmine.manPen1.table.list| grep "hg38." | cut -d'.' -f2 > hg38.push
cat redmine.$db.table.list| grep "hg38." | cut -d'.' -f2 > hg38.push
</pre>
</pre>
* 1 for mm10 with tables to push
* mm10 with tables to push: NOTE: This will need additional cleaning in the next step.
<pre>
<pre>
cat redmine.manPen1.table.list| grep "manPen1." | cut -d'.' -f2 > manPen1.push
cat redmine.$db.table.list| grep "$db." | cut -d'.' -f2 > $db.push  
</pre>
</pre>


====<span style="color:dodgerblue">Beta: Compare cleanTableList with dev mySQL table list====
Since the above commands are specific to this example, you could have different patterns in your file list. Be sure to do comparison wc-l counts to be sure you didn't miss any tables.
 
====<span style="color:dodgerblue">Beta: Remove certain tables from your assembly push list====
</span>
</span>
It's worthwhile to compare the Redmin Table List with the tables that are on dev for your assembly database and also for any other databases listed (e.g., databases that have associated chain net files).  
 
Here is a list of tables NOT to push. see notes on this [https://docs.google.com/spreadsheets/d/1HTDscw5LLW4W5_d_B_rKO3C8XQ98ZwfVTIM0zqcLwSg/edit?usp=sharing Gspreadsheet]
* '''You may need to remove "extFile" on this exclude list depending on the outcome of the previous steps.'''


<pre>
<pre>
hgsql -Ne "show tables" manPen1 > devTablesAll
author
cds
cell
description
development
extFile
gbCdnaInfo
gbExtFile
gbMiscDiff
gbSeq
gbStatus
gbWarn
geneName
genscanSubopt
hgFindSpec_public or hgFindSpec_username
imageClone
keyword
library
mrnaClone
organism
productName
refLink
refSeqStatus
refSeqSummary
BJRgbStatus
sex
source
tissue
trackDb_public or trackDb_username
</pre>
 
Access this file hive, or if it's removed, you can put this list in a file, "tablesNotToPush" and use grep to remove them from your assembly.push table (e.g., manPen1.push ) that you made in the previous step.
 
<pre>
grep -vf /hive/data/staging/tablesNotToPush $db.push > $db.push1
sdiff $db.push $db.push1
</pre>
Do this for all three assembly push files and if your sdiff output shows the two files are the same, then continue with one more check.
 
====<span style="color:dodgerblue">Beta: Compare assembly.push list with dev mySQL table list====
</span>
It's worthwhile to compare the assembly.push list with the tables that are on dev for your assembly. For example, there may be tables added in dev since Hiram's redmine.$db.table.list was created that should not be pushed yet. If you find any discrepancies between the tables actually on dev, and the tables in your push list, ask the developer about it. 


cat devTablesAll | grep -v trackDb | grep -v hgFindSpec > devMySqlTableList
1. Get the tables in your dev assembly db.
<pre>
hgsql -Ne "show tables" $db > devTablesAll
</pre>
2. Remove the tables that you know won't be pushed:
<pre>
grep -vf /hive/data/staging/tablesNotToPush devTablesAll > devMySqlTableList
</pre>
</pre>


This command will show a side-by side difference between the tables on dev and the tables you will push.  
This command will show a side-by side difference between the tables on dev and the tables you will push.  
<pre>
<pre>
sdiff -s devMySqlTableList manPen1.push
sdiff -s devMySqlTableList $db.push
</pre>
</pre>
Examine which tables are missing, ask if you have questions.
Examine which tables are missing, ask if you have questions. Common differences for the sql table list include chainRBestHg38%, chainSyn%, gapOverlap, hgFindSpec_%, netRBest%, netSyn%, tandemDups, trackDb_%.  The important this is to not miss any super interesting tables in the push list.


====<span style="color:dodgerblue">Beta: Push all tables for your assembly to beta====
====<span style="color:dodgerblue">Beta: Push all tables for your assembly to beta====
</span>
</span>


Push all tables (EXCEPT seq, extFile, hgFindSpec and trackDb tables)'' from hgwdev to hgwbeta:
 
'''OPTION 3: Use the bigPush.sh script, push tables in a file'''
These commands also push the two other commonly chained assemblies (the next step).
<pre>
bigPush.sh $db $db.push
bigPush.sh mm10 mm10.push
bigPush.sh hg38 hg38.push
</pre>
 
Push the clean table lists that you created in the steps above from hgwdev to hgwbeta:


'''Option 1: Push 1 table at a time'''
'''Option 1: Push 1 table at a time'''
Line 118: Line 194:
This command will push one table from dev to beta for your database/assembly:
This command will push one table from dev to beta for your database/assembly:


   sudo mypush $db $table mysqlbeta
   sudo mypush $db $table hgwbeta


'''Option 2: Push tables in a loop'''
'''Option 2: Push tables in a loop'''


<pre>
<pre>
for table in $(cat yourTableList); do sudo mypush $db $tbl mysqlbeta; done
for table in $(cat yourTableList); do sudo mypush $db $table hgwbeta; done
</pre>
</pre>


Line 129: Line 205:
<pre>
<pre>


for table in $(cat manPen1.push); do sudo mypush manPen1 $table mysqlbeta; done
for table in $(cat manPen1.push); do sudo mypush manPen1 $table hgwbeta; done
</pre>
</pre>


'''Option 3: Use the bigPush.sh script, list tables in a file'''
List space separated tables in quotes.
<pre>
hgwdev > bigPush.csh $db myTableList
</pre>


'''Option 4: Use the bigPush.sh script, list tables in quotes'''
'''Option 4: Use the bigPush.sh script, list tables in quotes'''
Line 150: Line 219:
You can count the number of tables in a database if you want to compare the number of tables listed in your push file to the number of tables that have been pushed to beta.  
You can count the number of tables in a database if you want to compare the number of tables listed in your push file to the number of tables that have been pushed to beta.  


 
'''Option 5: Push all tables in the db, then remove the tables you don't need'''
<pre>
hgwdev> sudo mypush $db '*' hgwbeta
</pre>


FROM HIVE:
FROM HIVE:
Line 158: Line 230:
</pre>
</pre>


FROM MYSQLBETA
FROM HGWBETA
<pre>
<pre>
use manPen1
use manPen1
Line 175: Line 247:


<pre>
<pre>
for table in $(cat hg38.push); do sudo mypush hg38 $table mysqlbeta; done
for table in $(cat hg38.push1); do sudo mypush hg38 $table hgwbeta; done
</pre>
</pre>


<pre>
<pre>
for table in $(cat mm10.push); do sudo mypush mm10 $table mysqlbeta; done
for table in $(cat mm10.push1); do sudo mypush mm10 $table hgwbeta; done
</pre>
</pre>


Line 187: Line 259:
You can copy items from hgcentraltest to hgcentralbeta with the copyHgcentral script.  For the usage statement, run:
You can copy items from hgcentraltest to hgcentralbeta with the copyHgcentral script.  For the usage statement, run:


  hgwdev > copyHgcentral -h
  copyHgcentral -h


* The copyHgcentral script must be run in test mode first.   
* The copyHgcentral script must be run in test mode first.   
Line 193: Line 265:
* Once test mode has been run and reviewed, run execute mode to copy from hgcentraltest to hgcentralbeta.   
* Once test mode has been run and reviewed, run execute mode to copy from hgcentraltest to hgcentralbeta.   
* Note that test mode generates output files which must be manually deleted afterward.  Be sure to run copyHgcentral in hive or your home directory and not in a directory where temp files should not be.
* Note that test mode generates output files which must be manually deleted afterward.  Be sure to run copyHgcentral in hive or your home directory and not in a directory where temp files should not be.
* Note that copyHgcentral can be run for "all" (blatServers, dbDb, defaultDb, genomeClade):
NOTE
* Note, copyHgcentral can and should be run for "all" (blatServers, dbDb, defaultDb, genomeClade), as below and you can skip 4 next steps


<pre>
<pre>
hgwdev > copyHgcentral test $db all dev beta
copyHgcentral test $db all dev beta
copyHgcentral execute $db all dev beta
</pre>
</pre>


====<span style="color:dodgerblue">Beta: copyHgcentral test db blatServers dev beta ====
====<span style="color:dodgerblue">Beta: copyHgcentral test db blatServers dev beta ====


Generates files, run in hive:
Redundant if you did "all" above. Generates files, run in hive:


</span>
</span>
<pre>
<pre>
hgwdev > copyHgcentral test $db blatServers dev beta
copyHgcentral test $db blatServers dev beta
</pre>
</pre>
<pre>
<pre>
hgwdev > copyHgcentral execute $db blatServers dev beta
copyHgcentral execute $db blatServers dev beta
</pre>
</pre>


You can also check on mysqlbeta:
You can also check on hgsqlbeta:


<pre>
<pre>
Line 222: Line 296:
</span>
</span>


Generates files, run in hive:
Redundant if you did "all" above. Generates files, run in hive:


<pre>
<pre>
Line 231: Line 305:
</pre>
</pre>


You can also check on mysqlbeta:
You can also check on hgwbeta:


<pre>
<pre>
Line 242: Line 316:
</span>
</span>


Generates files, run in hive:
Redundant if you did "all" above. Generates files, run in hive:


<pre>
<pre>
Line 253: Line 327:
You can also check on hgsqlbeta
You can also check on hgsqlbeta
<pre>
<pre>
hgsql -h hgwbeta
use hgcentralbeta;
use hgcentralbeta;


Line 285: Line 361:
</pre>
</pre>


====<span style="color:dodgerblue">Beta: copyHgcentral: liftOverChain (manual move) ====
====<span style="color:dodgerblue">Beta: copyHgcentral: liftOverChain: manual move ====
</span>
</span>
<b>**If you completed liftOverChain+Nets in Track QA steps, run these commands to check hgcentralbeta:</b>
<pre>
hgsql -h hgwbeta -Ne "SELECT * FROM liftOverChain WHERE fromDb = '$db' OR toDb = '$db'" hgcentralbeta
</pre>


liftOverChain is not copied with the copyHgcentral script, it needs to be copied manually.
liftOverChain is not copied with the copyHgcentral script, it needs to be copied manually.


* Only copy lines from liftOverChain on hgcentraltest to hgcentralbeta if there are liftOver files listed in the pushQ and if the assemblies they go to/from exist on the RR.  
* Only copy lines from liftOverChain on hgcentraltest to hgcentralbeta if there are liftOver files listed in the pushQ and if the assemblies they go to/from exist on the RR.  
* Check for lines in liftOverChain that should be in the pushQ, but aren't (e.g., the liftOver from a previous assembly).  
* Check for lines in liftOverC'''Bold text'''hain that should be in the pushQ, but aren't (e.g., the liftOver from a previous assembly).  
* Add lines related to your assembly, any previous versions of your organism, and any other organisms that are associated with liftOver files and your assembly.  
* Add lines related to your assembly, any previous versions of your organism, and any other organisms that are associated with liftOver files and your assembly.  
* More details on the [http://genomewiki.ucsc.edu/genecats/index.php/Chains_and_Nets_QA Chain and Net QA wiki page].
* More details on the [http://genomewiki.ucsc.edu/genecats/index.php/Chains_and_Nets_QA Chain and Net QA wiki page].
Line 299: Line 380:
Check beta, load if not present and recheck:
Check beta, load if not present and recheck:
<pre>
<pre>
  hgsql -h mysqlbeta -Ne "SELECT * FROM liftOverChain WHERE fromDb = '$db' OR toDb = '$db'" hgcentralbeta  
  hgsql -h hgwbeta -Ne "SELECT * FROM liftOverChain WHERE fromDb = '$db' OR toDb = '$db'" hgcentralbeta  


  hgsql -h mysqlbeta -e "LOAD DATA LOCAL INFILE 'chain.dev' INTO TABLE liftOverChain" hgcentralbeta
  hgsql -h hgwbeta -e "LOAD DATA LOCAL INFILE 'chain.dev' INTO TABLE liftOverChain" hgcentralbeta
</pre>
 
'''TIP: An easy way to add liftOver rows!'''
 
#Run checkMetaData.csh $db
'''NOTE:''' If the assembly is not the first for that organism, the defaultDb will remain on hgcentraltestOnly
#This creates files in the dir in which you run it.
#Use the file "liftOverChain.$db.hgcentraltestOnly" as your LOAD file:
<pre>
hgsql -h hgwbeta -e "LOAD DATA LOCAL INFILE 'liftOverChain.$db.hgcentraltestOnly' INTO TABLE liftOverChain" hgcentralbeta
</pre>
</pre>
After loading, run checkMetaData.csh $db again to make sure there are no differences.


====<span style="color:dodgerblue">Beta: checkMetaData ====
====<span style="color:dodgerblue">Beta: checkMetaData ====
Line 311: Line 403:
* Run this script in a temporary folder or hive; it creates some comparison files that can be deleted after the check.
* Run this script in a temporary folder or hive; it creates some comparison files that can be deleted after the check.


====<span style="color:dodgerblue">Beta: Check that your assembly is listed in align.dbs====
====<span style="color:dodgerblue">Beta: Double check gbdb files====
</span>
 
The new assembly should already be listed in the files align.dbs and hgwdev.dbs in the source tree at ~/kent/src/hg/makeDb/genbank/etc/.
 
 
====<span style="color:dodgerblue">Beta: Double-check /gbdb files====
</span>
</span>


* Double-check that hgnfs1 (which is /gbdb on hgwbeta) has the files listed in Remine file list.  
* Double-check that hgnfs1 (which is /gbdb on hgwbeta) has the files listed in Redmine file list.  


<pre>
<pre>
$ /gbdb/manPen1
cd /gbdb/$db


ls -ld $(find .)
ls -ld $(find .)
Line 339: Line 425:


In order to check your assembly on beta, you will need to:
In order to check your assembly on beta, you will need to:
* Request a push of any listed supporting files in''' /gbdb from hgwdev to hgnfs1'''   
* Push any listed supporting files in''' /gbdb from hgwdev to hgwbeta'''  using the '''gbdbPush''' script
* Note that hgwbeta and the RR share the files on hgnfs1, so once these files are in place, there is not another push required when the track is released to the RR.  
* Using the script, paste the gbdb file paths from the redmine.$db.file.list one at a time and end the script using a period ( . ).
* NOTE THAT STAR CHARACTERS (*) will BREAK THE SCRIPT. PLEASE CHECK and avoid star chars.
<pre>
$ sudo gbdbPush
[sudo] password for user:
 
Enter the files to push to, one per line.  End with a '.'
 
### BEGIN FILE LIST ###
/gbdb/hg38/gdcCancer/gdcCancer.bb
.
### END FILE LIST ###
</pre>
 
* Note that  the gbdb files are sync'd to hgdownload on Sundays. If needed sooner, request a push.  
* Note that  the gbdb files are sync'd to hgdownload on Sundays. If needed sooner, request a push.  
* If there are images associated with any track description pages, be sure to run a <tt>make beta</tt> from within kent/src/hg/htdocs/ to get the images to beta.
* If there are images associated with any track description pages, be sure to run a <tt>make beta</tt> from within kent/src/hg/htdocs/ to get the images to beta.
* [https://groups.google.com/a/soe.ucsc.edu/forum/#!searchin/push-request/staging$20on$20beta%7Csort:date/push-request/Gt8cRK4eTuw/gPbM5I2SAgAJ Example request]
* They may not be listed in the file list, but you also need to push the /gbdb/$db/trackDb.ixx and /gbdb/$db/trackDb.ix files.
* They may not be listed in the file list, but you also need to push the /gbdb/$db/trackDb.ixx and /gbdb/$db/trackDb.ix files, as in the example above.  
* After push, you can see files: ssh qateam@hgwbeta ; cd /gbdb/$db
* After push, you can see files: ssh hgwbeta@qateam ; cd gbdb/$db
 
====<span style="color:dodgerblue">Beta: Admins/pushers completed gbdb push request====
</span>


====<span style="color:dodgerblue">Beta: Do a make beta in trackDb for your assembly====
====<span style="color:dodgerblue">Beta: Do a make beta in trackDb for your assembly====
Line 393: Line 488:
   make public -j 4 DBS="dm6 ce11 sacCer3 droEre1 droSec1 droSim1 droYak2 droAna2 dp3 droMoj2 droVir2 droGri1 droPer1"
   make public -j 4 DBS="dm6 ce11 sacCer3 droEre1 droSec1 droSim1 droYak2 droAna2 dp3 droMoj2 droVir2 droGri1 droPer1"


====<span style="color:dodgerblue">Beta: Do a make beta in trackDb for chain/net organisms====
====<span style="color:dodgerblue">Beta: Do a make beta in trackDb for chain net organisms====
If your assembly has alignments to other organisms, such as chain/net alignments to hg38 or to the previous assembly version of your organism, be sure to also do a 'make beta' for those assemblies.
If your assembly has alignments to other organisms, such as chain/net alignments to hg38 or to the previous assembly version of your organism, be sure to also do a 'make beta' for those assemblies (usually previous organism versions, hg38, and mm10)
 
<pre>
[http://genomewiki.ucsc.edu/genecats/index.php/Assembly_QA_Part_3_BETA_Steps#Beta:_Do_a_.27make_beta.27_in_trackDb_for_your_assembly See details in above section]
make beta DBS=$otherChainAssemblies
 
</pre>
 
----


====<span style="color:dodgerblue">Beta: Do a make public in trackDb for your assembly====
====<span style="color:dodgerblue">Beta: Turn on GenBank updates add:commit:push====
</span>
</span>


Make your track public by using the "make public" command on hgwdev while in the trackDb directory (src/hg/makeDb/trackDb):
In order for your assembly to receive Genbank updates on hgwbeta, add your assembly to hgwbeta.dbs alphabetically. If a previous assembly is there, put the previous assembly in the lower list alphabetically.
 
    [user@hgwdev trackDb]$ make public DBS=$db
 
Example to make beta on more than one db at a time:
 
  make beta DBS='$db1 $db2 $db3 $db4 etc'
 
Your track should now be visible on the [http://hgwbeta-public.cse.ucsc.edu/cgi-bin/hgTracks hgwbeta-public server].
 
If your track is not visible, you may want to check that your track has the correct release tag. Also see [[http://genomewiki.ucsc.edu/index.php/ThreeStateTrackDb Three State TrackDb]] for more information.
 
====<span style="color:dodgerblue">Beta: Do a make public in trackDb for chain/net organisms====
</span>
 
If your assembly has alignments to other organisms, such as chain/net alignments to hg38 or to the previous assembly version of your organism, be sure to also do a 'make public' for those assemblies.
 
[http://genomewiki.ucsc.edu/genecats/index.php/Assembly_QA_Part_3_BETA_Steps#Beta:_Do_a_.27make_public.27_in_trackDb_for_your_assembly See details in above section]
 


* Add the new assembly to ~/kent/src/hg/makeDb/genbank/etc/hgwbeta.dbs in alphabetical order. 
* Be sure to save, git add, git commit, and git push the file.
* This turns on GenBank updates for the assembly. Currently (June 2019), GenBank updates are done quarterly (every 3 months).
* Do not yet edit the rr.dbs, this comes later when the assembly is on the RR.


<pre>
<pre>
cat ~/kent/src/hg/makeDb/genbank/etc/align.dbs | grep manPen
cat ~/kent/src/hg/makeDb/genbank/etc/hgwbeta.dbs | grep $db
manPen1
</pre>
</pre>
If your assembly is missing from align.dbs , check with Brian Raney.
====<span style="color:dodgerblue">Beta: Turn on GenBank updates & add/commit/push====
</span>
* Once your assembly is listed in align.dbs, turn on GenBank updates on hgwbeta before 4:30 p.m.
* Add the new assembly to ~/kent/src/hg/makeDb/genbank/etc/hgwbeta.dbs in alphabetical order. 
* Be sure to save, git add, git commit, and git push the file.
* Do not yet edit the rr.dbs, this comes later when the assembly is on the RR.


====<span style="color:dodgerblue">Beta: GenBank updates: make libs & run make====
====<span style="color:dodgerblue">Beta: GenBank updates: make libs and run make====
</span>
</span>
After committing the change, make sure your libs are up to date:
After committing the change, make sure your libs are up to date:
Line 447: Line 517:
  cd ~/kent/src/hg/makeDb/genbank/  
  cd ~/kent/src/hg/makeDb/genbank/  
  git pull  
  git pull  
  make install-rr install-server
  make install-server


To see whether updates have run (at least a day after the *.dbs files were updated), check the update times of the table 'gbLoaded':
To see whether updates have run (at least the day after the *.dbs files were updated), check the update times of the table 'gbLoaded':


====<span style="color:dodgerblue">Beta: GenBank updates: check Genbank update times====
====<span style="color:dodgerblue">Beta: GenBank updates: check Genbank update times====
Line 455: Line 525:
  hgwdev > updateTimes.csh $db gbLoaded verbose
  hgwdev > updateTimes.csh $db gbLoaded verbose


For example, you'll see updates for dev and beta (but not yet for the rr/euro/asia):
For example, you'll see last update time for that particular SQL table on dev (but not yet for the beta/rr/euro/asia):


<pre>
<pre>
Line 470: Line 540:
</pre>
</pre>


The update times will be out of sync between machines, but not by more than 24 hours or so if updates are running. The gbLoaded table will be updated regardless of whether changes to other GenBank tables were picked up. More genbank update instructions are available at [[Genbank updates]].
The update times may be out of sync between machines, but not by more than 24 hours or so if updates are running. The gbLoaded table should have been pushed with the tables to beta. In the future, this update time may not change for a while,  this is because we '''now pull GenBank updates quarterly vs. nightly'''. Make sure you still add you $db to the hgwdev.dbs and hgwbeta.dbs files. More genbank update instructions are available at [[Genbank updates]].


The etc-update-server part of the make will cause the downloads mentioned below in the "Verify downloads" section to be created.
The etc-update-server part of the make will cause the downloads mentioned below in the "Verify downloads" section to be created.
Line 493: Line 563:
Check table update times:
Check table update times:
<pre>
<pre>
hgwdev > cd ~/kent/src/hg/makeDb/schema  
cd ~/kent/src/hg/makeDb/schema  
joinerCheck -database=$db -times all.joiner
</pre>


hgwdev > joinerCheck -database=$db -times all.joiner
You should get no output if it ran successfully.
</pre>


====<span style="color:dodgerblue">Beta: joinerCheck tableCoverage====
====<span style="color:dodgerblue">Beta: joinerCheck tableCoverage====
Line 506: Line 577:
If not all of the tables are listed, email the developer asking him to add those tables to the tablesIgnored $db. According to Hiram it is probably ok for us to edit all.joiner ourselves.
If not all of the tables are listed, email the developer asking him to add those tables to the tablesIgnored $db. According to Hiram it is probably ok for us to edit all.joiner ourselves.


====<span style="color:dodgerblue">Beta: Check release tags: compare dev and beta tracks side-by-side====
====<span style="color:dodgerblue">Beta: Check release tags: compare dev and beta tracks side by side====
</span>
</span>


Compare your tracks by bringing up a dev and a beta browser window side-by-side. If some beta tracks can't be seen, you may need to edit the release tag.
Compare your tracks by bringing up a dev and a beta browser window side-by-side. If some beta tracks can't be seen, you may need to edit the release tag.
See this page for more information:
http://genomewiki.ucsc.edu/index.php/ThreeStateTrackDb


Where are your release tags? You can do a recursive search, like this:
:See this page for more information:
:http://genomewiki.ucsc.edu/index.php/ThreeStateTrackDb
 
Where are your release tags? They're in your .ra files, you can do a recursive search, like this:


<pre>
<pre>
Line 519: Line 591:
</pre>
</pre>


====<span style="color:dodgerblue">Beta: Remove release tag for big*/vcf track types====
</span>


Once you verify that the track looks good on hgwbeta, remove the release tag from trackDb.ra.
====<span style="color:dodgerblue">Beta: Check Tools: LiftOver ====
 
====<span style="color:dodgerblue">Beta: Check Tools > LiftOver ====
</span>
</span>


Check LiftOver on beta and beta public.
Check LiftOver on beta and beta public.


====<span style="color:dodgerblue">Beta: Check Tools > BLAT (dna)====
====<span style="color:dodgerblue">Beta: Check Tools: BLAT: dna====
</span>
</span>
Copy some dna and check BLAT for a DNA search on beta and beta public.
Copy some dna and check BLAT for a DNA search on beta and beta public.


====<span style="color:dodgerblue">Beta: Check Tools > BLAT (protein) ====
====<span style="color:dodgerblue">Beta: Check Tools: BLAT: protein ====
</span>
</span>


Copy some protein and check BLAT for a protein search on beta and beta public.
Copy some protein and check BLAT for a protein search on beta and beta public.


====<span style="color:dodgerblue">Beta: Check Tools > PCR ====
====<span style="color:dodgerblue">Beta: Check Tools: PCR ====
</span>
</span>


Check PCR on beta and beta public.
Check PCR on beta and beta public.
See [http://genomewiki.ucsc.edu/genecats/index.php/Assembly_QA_Part_1_DEV_Steps#Dev:_isPCR_test how to test PCR details from DEV steps]


====<span style="color:dodgerblue">Beta: Review chain/net track QA wiki ====
====<span style="color:dodgerblue">Beta: Review chain net track QA wiki ====
</span>
</span>
* Review the Chain/Net QA wiki page, see the section [http://genomewiki.ucsc.edu/genecats/index.php/Chains_and_Nets_QA#QAing_on_hgwbeta QAing on hgwbeta]
* Review the Chain/Net QA wiki page, see the section [http://genomewiki.ucsc.edu/genecats/index.php/Chains_and_Nets_QA#QAing_on_hgwbeta QAing on hgwbeta]
Line 550: Line 619:
Your assembly may have chain/nets for a composite chain/net track. For example, you may have a 'turkey' assembly that belongs in the "Vertabrate  Chain/Net" composite track on hg38. If your chain/nets appear next to the composite track, rather than within the composite track, you may need to create separate chainNetMelGal1.ra and chainNetMelGal5.ra files in trackDb/human. Once you have those files created then you will need to add the appropriate includes to the trackDb/human/hg38/trackDb.ra and trackDb/human/hg19/trackDb.chainNet.ra files.
Your assembly may have chain/nets for a composite chain/net track. For example, you may have a 'turkey' assembly that belongs in the "Vertabrate  Chain/Net" composite track on hg38. If your chain/nets appear next to the composite track, rather than within the composite track, you may need to create separate chainNetMelGal1.ra and chainNetMelGal5.ra files in trackDb/human. Once you have those files created then you will need to add the appropriate includes to the trackDb/human/hg38/trackDb.ra and trackDb/human/hg19/trackDb.chainNet.ra files.


====<span style="color:dodgerblue">Beta: Do chain/net quick qa or full qa ====
====<span style="color:dodgerblue">Beta: Do chain net quick qa or full qa ====
</span>
</span>
DO NOT RUN these three commands since blastz has been replaced by lastz!!! Instead run findScores.pl, as shown at bottom


Details in Chain/Net QA wiki page, section [http://genomewiki.ucsc.edu/genecats/index.php/Chains_and_Nets_QA#QAing_on_hgwbeta QAing on hgwbeta]
Details in Chain/Net QA wiki page, section [http://genomewiki.ucsc.edu/genecats/index.php/Chains_and_Nets_QA#QAing_on_hgwbeta QAing on hgwbeta]
Line 566: Line 637:
</pre>
</pre>


The following getMatrixLines.csh script is broken:
<pre>
<pre>
getMatrixLines.csh [database] [other database]
getMatrixLines.csh [database] [other database]
</pre>
</pre>


====<span style="color:dodgerblue">Beta: Re-push any tables? check updateTimes====
You can see the same information with:
<pre>
$ findScores.pl susScr11 hg38
looking in file:
/hive/data/genomes/hg38/bed/lastz.susScr11/axtChain/run/chain.csh
chainMinScore 3000
chainLinearGap medium
default values:
matrix 16 91,-114,-31,-123,-114,100,-125,-31,-31,-125,100,-114,-123,-31,-114,91
</pre>
 
====<span style="color:dodgerblue">Beta: Re push any tables if needed after checking updateTimes====
</span>
</span>


* Make sure no tables need to be repushed from hgwdev to hgwbeta.
* Make sure no tables need to be repushed from hgwdev to hgwbeta.
* Note that if at any point you need to re-push some genbank tables, you must push ALL genbank tables together.
* Note that if at any point you need to re-push some genbank tables, you must push ALL genbank tables together. <-- (Don't need to run this step anymore)
* Remember that Dev should have several tables that beta doesn't (e.g., author, cds, cell, chainRBest*, chainSyn*)


You can use  
You can use  
Line 581: Line 665:
</pre>
</pre>


to compare table update times between hgwdev and hgwbeta. Everything but hgFindSpec, history, tableDescriptions, trackDb and the genbank tables should have the same update times.  
Or you can check tables in a list by using a loop like this:
<pre>
for table in $(cat hg38.push); do updateTimes.csh hg38 $table verboseNoMirror >> hg38.updateTimes.devVSbeta ; done
</pre>


To see all of the tables in the assembly that are related to genbank do this:
to compare table update times between hgwdev and hgwbeta. Everything but hgFindSpec, history, tableDescriptions, and trackDb tables should have the same update times.
 
To see all of the tables in the assembly that are related to genbank do this (Don't need to run this step anymore):
  hgwdev > hgsql -Ne 'show tables' $db | egrep -f /cluster/data/genbank/etc/genbank.tbls
  hgwdev > hgsql -Ne 'show tables' $db | egrep -f /cluster/data/genbank/etc/genbank.tbls


====<span style="color:dodgerblue">Beta: Do a make public in trackDb for your assembly====
</span>


Make your track public by using the "make public" command on hgwdev while in the trackDb directory (src/hg/makeDb/trackDb):


  make public DBS=$db


🔵  Done with BETA steps? Go to [[Assembly QA Part 4 RR Steps | Assembly QA Part 4: RR Steps]]
Example to make beta on more than one db at a time:
 
 
 
 
 


  make beta DBS='$db1 $db2 $db3 $db4 etc'


Your track should now be visible on the [http://hgwbeta-public.cse.ucsc.edu/cgi-bin/hgTracks hgwbeta-public server].


If your track is not visible, you may want to check that your track has the correct release tag. Also see [[http://genomewiki.ucsc.edu/index.php/ThreeStateTrackDb Three State TrackDb]] for more information.


====<span style="color:dodgerblue">Beta: Do a make public in trackDb for chain net organisms====
</span>


If your assembly has alignments to other organisms, such as chain/net alignments to hg38 or to the previous assembly version of your organism, be sure to also do a 'make public' for those assemblies.


====<span style="color:dodgerblue">Beta: Remove release tag for big vcf track types====
</span>


Once you verify that the track looks good on hgwbeta, remove the release tag from trackDb.ra.


:More information about editing release tags can be found here:
:http://genomewiki.ucsc.edu/index.php/ThreeStateTrackDb#Updated_trackDb_release_process




🔵  Done with BETA steps? Go to [[Assembly_QA_Part_4_RR_Steps | Assembly QA Part 4: RR Steps]]





Latest revision as of 21:41, 12 April 2022

These steps were revised in 2017, but you can also see the old steps: Releasing an assembly (old steps)

Navigation Menu

Home Page
Assembly QA Part 1: DEV Steps
Assembly QA Part 2: Track Steps
Assembly QA Part 3: BETA Steps
Assembly QA Part 4: RR Steps


Beta: Check for Ensembl tracks

Before pushing Ensembl tracks to beta, review the Ensembl wiki page and the Ensembl_QA script.

Beta: Copy redmine table list to hive

Should have been done in Dev steps: Alignment files are to valid assemblies, if not copy file and table list to assembly hive directory as below.


In Redmine for your assembly, look at the field, "Table List." The engineer should have provided a path to redmine.$db.table.list E.g., /hive/data/genomes/manPen1/pushQ/redmine.manPen1.table.list

Your file contents should be something like this:

head redmine.manPen1.table.list
hg38.chainManPen1
hg38.chainManPen1Link
hg38.netManPen1
manPen1.augustusGene

Copy the file list to your assembly directory in hive.

From hive, copy the file list to your assembly dir:

cp /hive/data/genomes/$db/pushQ/redmine.$db.table.list .

Make a note of which assemblies are listed in this file list by looking at the unique databases/assemblies in column 1. For example:

cut -d'.' -f1 redmine.$db.table.list | sort -u

hg38
manPen1
mm10

Beta: Remove trackDb and hgFindSpec tables from your HIVE table list

Remove tables from the list that start with trackDb or hgFindSpec:

sed -i.bak '/trackDb/d' cleanTableList

sed -i.bak '/hgFindSpec/d' cleanTableList

You'll be using this cleanTableList when doing your push.

Another way to do this:

grep -Ev "(trackDb|hgFindSpec)" redmine.$db.table.list > tableList.hgFindSpecANDtrackDb.removed

Beta: Special steps for extFile tables

  • If this is a new assembly and this is the first Conservation track, the extFile table probably doesn't exist on beta and likely only has a single entry on dev. If this is the case, it is ok to just push the entire table from dev to beta.
  • Include extFile table in push request when you are ready to push to the RR.
  • Follow the steps above to remove seq or extFile tables from cleanTableList.
  • NOTE: Do not push seq or extFile tables from dev to beta.
  • You must use the copyExtSeqRows.csh script to move only the rows needed.
  • There may be more than one set of entries in the extFile table. Make sure you only push the set that pertains to the actual files you are pushing to hgnfs1 (e.g. /gbdb/ponAbe2/multiz8way/anno/maf/*)
  • These are the lines that will need to be added to beta:

mysql> select path from extFile where path like "%anno/maf%";

Beta: Make table PUSH lists for each assembly

In the example above, the assembly manPen1 has tables in 3 different databases (manPen1, hg38, and mm10). It will be helpful to create a file, containing only table names, for each unique database listed in the Table List. These files can then be used for pushing tables in a loop.

Using the example above, make 3 files with a list of tables to push for each assembly,

  • manPen1 with tables to push
db="manPen1"
cat redmine.$db.table.list| grep "mm10." | cut -d'.' -f2 > mm10.push
  • hg38 with tables to push
cat redmine.$db.table.list| grep "hg38." | cut -d'.' -f2 > hg38.push
  • mm10 with tables to push: NOTE: This will need additional cleaning in the next step.
cat redmine.$db.table.list| grep "$db." | cut -d'.' -f2 > $db.push 

Since the above commands are specific to this example, you could have different patterns in your file list. Be sure to do comparison wc-l counts to be sure you didn't miss any tables.

Beta: Remove certain tables from your assembly push list

Here is a list of tables NOT to push. see notes on this Gspreadsheet

  • You may need to remove "extFile" on this exclude list depending on the outcome of the previous steps.
author
cds
cell
description
development
extFile
gbCdnaInfo
gbExtFile
gbMiscDiff
gbSeq
gbStatus
gbWarn
geneName
genscanSubopt
hgFindSpec_public or hgFindSpec_username
imageClone
keyword
library
mrnaClone
organism
productName
refLink
refSeqStatus
refSeqSummary
BJRgbStatus
sex
source
tissue
trackDb_public or trackDb_username

Access this file hive, or if it's removed, you can put this list in a file, "tablesNotToPush" and use grep to remove them from your assembly.push table (e.g., manPen1.push ) that you made in the previous step.

grep -vf /hive/data/staging/tablesNotToPush $db.push > $db.push1
sdiff $db.push $db.push1

Do this for all three assembly push files and if your sdiff output shows the two files are the same, then continue with one more check.

Beta: Compare assembly.push list with dev mySQL table list

It's worthwhile to compare the assembly.push list with the tables that are on dev for your assembly. For example, there may be tables added in dev since Hiram's redmine.$db.table.list was created that should not be pushed yet. If you find any discrepancies between the tables actually on dev, and the tables in your push list, ask the developer about it.

1. Get the tables in your dev assembly db.

hgsql -Ne "show tables" $db > devTablesAll

2. Remove the tables that you know won't be pushed:

grep -vf /hive/data/staging/tablesNotToPush devTablesAll > devMySqlTableList

This command will show a side-by side difference between the tables on dev and the tables you will push.

sdiff -s devMySqlTableList $db.push

Examine which tables are missing, ask if you have questions. Common differences for the sql table list include chainRBestHg38%, chainSyn%, gapOverlap, hgFindSpec_%, netRBest%, netSyn%, tandemDups, trackDb_%. The important this is to not miss any super interesting tables in the push list.

Beta: Push all tables for your assembly to beta


OPTION 3: Use the bigPush.sh script, push tables in a file These commands also push the two other commonly chained assemblies (the next step).

bigPush.sh $db $db.push
bigPush.sh mm10 mm10.push
bigPush.sh hg38 hg38.push

Push the clean table lists that you created in the steps above from hgwdev to hgwbeta:

Option 1: Push 1 table at a time

This command will push one table from dev to beta for your database/assembly:

 sudo mypush $db $table hgwbeta

Option 2: Push tables in a loop

for table in $(cat yourTableList); do sudo mypush $db $table hgwbeta; done

For example:


for table in $(cat manPen1.push); do sudo mypush manPen1 $table hgwbeta; done


Option 4: Use the bigPush.sh script, list tables in quotes

List space separated tables in quotes.

hgwdev > bigPush.csh $db "table1 table2 table3 table4" 

You can count the number of tables in a database if you want to compare the number of tables listed in your push file to the number of tables that have been pushed to beta.

Option 5: Push all tables in the db, then remove the tables you don't need

hgwdev> sudo mypush $db '*' hgwbeta

FROM HIVE:

wc -l manPen1.push
55 manPen1.push

FROM HGWBETA

use manPen1
USE databasename; SHOW TABLES; SELECT FOUND_ROWS();

+--------------+
| FOUND_ROWS() |
+--------------+
|           55 |
+--------------+

Beta: Push all tables to other assemblies to beta

Repeat the push steps for other assemblies listed in your table list (e.g., chain/net tables to other assemblies). For example:

for table in $(cat hg38.push1); do sudo mypush hg38 $table hgwbeta; done
for table in $(cat mm10.push1); do sudo mypush mm10 $table hgwbeta; done

Beta: Review copyHgcentral steps

You can copy items from hgcentraltest to hgcentralbeta with the copyHgcentral script. For the usage statement, run:

copyHgcentral -h
  • The copyHgcentral script must be run in test mode first.
  • Test mode will show you the state of hgcentraltest, hgcentralbeta and hgcentral.
  • Once test mode has been run and reviewed, run execute mode to copy from hgcentraltest to hgcentralbeta.
  • Note that test mode generates output files which must be manually deleted afterward. Be sure to run copyHgcentral in hive or your home directory and not in a directory where temp files should not be.

NOTE

  • Note, copyHgcentral can and should be run for "all" (blatServers, dbDb, defaultDb, genomeClade), as below and you can skip 4 next steps
 copyHgcentral test $db all dev beta
 copyHgcentral execute $db all dev beta

Beta: copyHgcentral test db blatServers dev beta

Redundant if you did "all" above. Generates files, run in hive:

copyHgcentral test $db blatServers dev beta
copyHgcentral execute $db blatServers dev beta

You can also check on hgsqlbeta:

use hgcentralbeta;

select * from blatServers where db='manPen1';

Beta: copyHgcentral test db dbDb dev beta

Redundant if you did "all" above. Generates files, run in hive:

hgwdev > copyHgcentral test $db dbDb dev beta
hgwdev > copyHgcentral execute $db dbDb dev beta

You can also check on hgwbeta:

use hgcentralbeta;

select * from dbDb where name='manPen1' \G;

Beta: copyHgcentral test db defaultDb dev beta

Redundant if you did "all" above. Generates files, run in hive:

hgwdev > copyHgcentral test $db defaultDb dev beta
hgwdev > copyHgcentral execute $db defaultDb dev beta

You can also check on hgsqlbeta

hgsql -h hgwbeta

use hgcentralbeta;

select * from defaultDb where name="manPen1"limit 1;

Beta: copyHgcentral test db genomeClade dev beta

NOTE: This table probably will not need to be updated. It contains records like this:

mysql> select * from genomeClade order by rand() limit 5;
+-----------------+------------+----------+
| genome          | clade      | priority |
+-----------------+------------+----------+
| GRCh38.p2       | haplotypes |      134 |
| C. japonica     | worm       |       70 |
| Atlantic cod    | vertebrate |      125 |
| D. melanogaster | insect     |       10 |
| D. persimilis   | insect     |       55 |
+-----------------+------------+----------+

Generates files, run in hive:

hgwdev > copyHgcentral test $db genomeClade dev beta
hgwdev > copyHgcentral execute $db genomeClade dev beta

Beta: copyHgcentral: liftOverChain: manual move

**If you completed liftOverChain+Nets in Track QA steps, run these commands to check hgcentralbeta:

hgsql -h hgwbeta -Ne "SELECT * FROM liftOverChain WHERE fromDb = '$db' OR toDb = '$db'" hgcentralbeta

liftOverChain is not copied with the copyHgcentral script, it needs to be copied manually.

  • Only copy lines from liftOverChain on hgcentraltest to hgcentralbeta if there are liftOver files listed in the pushQ and if the assemblies they go to/from exist on the RR.
  • Check for lines in liftOverCBold texthain that should be in the pushQ, but aren't (e.g., the liftOver from a previous assembly).
  • Add lines related to your assembly, any previous versions of your organism, and any other organisms that are associated with liftOver files and your assembly.
  • More details on the Chain and Net QA wiki page.
 hgsql -Ne "SELECT * FROM liftOverChain WHERE fromDb = '$db' OR toDb = '$db'" hgcentraltest > chain.dev 

Check beta, load if not present and recheck:

 hgsql -h hgwbeta -Ne "SELECT * FROM liftOverChain WHERE fromDb = '$db' OR toDb = '$db'" hgcentralbeta 

 hgsql -h hgwbeta -e "LOAD DATA LOCAL INFILE 'chain.dev' INTO TABLE liftOverChain" hgcentralbeta

TIP: An easy way to add liftOver rows!

  1. Run checkMetaData.csh $db

NOTE: If the assembly is not the first for that organism, the defaultDb will remain on hgcentraltestOnly

  1. This creates files in the dir in which you run it.
  2. Use the file "liftOverChain.$db.hgcentraltestOnly" as your LOAD file:
hgsql -h hgwbeta -e "LOAD DATA LOCAL INFILE 'liftOverChain.$db.hgcentraltestOnly' INTO TABLE liftOverChain" hgcentralbeta

After loading, run checkMetaData.csh $db again to make sure there are no differences.

Beta: checkMetaData

After completing copyHgcentral steps, run checkMetaData.csh $db

  • This checks that all of the metadata is the same on hgwdev and on hgwbeta.
  • Run this script in a temporary folder or hive; it creates some comparison files that can be deleted after the check.

Beta: Double check gbdb files

  • Double-check that hgnfs1 (which is /gbdb on hgwbeta) has the files listed in Redmine file list.
cd /gbdb/$db

ls -ld $(find .)
  • Remove any unnecessary files.
  • If any files were updated on hgwdev in the course of QAing tracks, make sure the correct version is on hgnfs1.
  • To see the timestamps of files with symlinks on hgwdev, use the options "-lL" with ls.
  • The large "L" shows information for a file that a link references, rather than for the link itself.
  • You may notice that hgwdev has TRIX .ix and .ixx files, that is OK. These live on beta in a different location, /data/trix, more information at the at the TrackDb page.

Note: files on hgwdev are double the size as beta, this is not a concern.

Beta: Push gbdb files

In order to check your assembly on beta, you will need to:

  • Push any listed supporting files in /gbdb from hgwdev to hgwbeta using the gbdbPush script
  • Using the script, paste the gbdb file paths from the redmine.$db.file.list one at a time and end the script using a period ( . ).
  • NOTE THAT STAR CHARACTERS (*) will BREAK THE SCRIPT. PLEASE CHECK and avoid star chars.
$ sudo gbdbPush 
[sudo] password for user: 

Enter the files to push to, one per line.  End with a '.'

### BEGIN FILE LIST ###
/gbdb/hg38/gdcCancer/gdcCancer.bb
.
### END FILE LIST ###
  • Note that the gbdb files are sync'd to hgdownload on Sundays. If needed sooner, request a push.
  • If there are images associated with any track description pages, be sure to run a make beta from within kent/src/hg/htdocs/ to get the images to beta.
  • They may not be listed in the file list, but you also need to push the /gbdb/$db/trackDb.ixx and /gbdb/$db/trackDb.ix files.
  • After push, you can see files: ssh qateam@hgwbeta ; cd /gbdb/$db

Beta: Do a make beta in trackDb for your assembly

Do make beta on hgwdev in kent/src/hg/makeDb/trackDb like so:

 make beta DBS=$db

Example to make beta on more than one db at a time:

 make beta DBS='$db1 $db2 $db3 $db4 etc'




Running multiple dbs in parallel to save time

Multiple assemblies can be run in parallel by using the make -j option (as of 2/10/17, thanks to Mark Diekhans). Updating all dev dbs used to take about 50 minutes, and now it can take about 5 minutes (at 16 in parallel). While Mark has safely run 16 dbs at a time on dev, it is recommended to only run 8 or less at a time on beta or the RR. Use make -j # beta and make -j # public, where the number (make -j 16 alpha) represents how many parallel processes (16) are running.

For example, if you do:
  make -j 8 alpha
it updates everything, 8 at a time. If you do:
  make -j 2 DBS="hg19 hg38 mm10 felCat5"
it updates those 4 databases, 2 at a time .
Note: the 'make in parallel' process creates and removes temporary files:
The tmp dirs are found with:
 kent/src/inc/portable.h:
   char *getTempDir(void);
   /* get temporary directory to use for programs.  This first checks TMPDIR environment
    * variable, then /data/tmp, /scratch/tmp, /var/tmp, /tmp.  Return is static and
    * only set of first call */

Examples:

 make beta -j 4 DBS="dm6 ce11 sacCer3 droEre1 droSec1 droSim1 droYak2 droAna2 dp3 droMoj2 droVir2 droGri1 droPer1"
 make public -j 4 DBS="dm6 ce11 sacCer3 droEre1 droSec1 droSim1 droYak2 droAna2 dp3 droMoj2 droVir2 droGri1 droPer1"

Beta: Do a make beta in trackDb for chain net organisms

If your assembly has alignments to other organisms, such as chain/net alignments to hg38 or to the previous assembly version of your organism, be sure to also do a 'make beta' for those assemblies (usually previous organism versions, hg38, and mm10)

make beta DBS=$otherChainAssemblies

Beta: Turn on GenBank updates add:commit:push

In order for your assembly to receive Genbank updates on hgwbeta, add your assembly to hgwbeta.dbs alphabetically. If a previous assembly is there, put the previous assembly in the lower list alphabetically.

  • Add the new assembly to ~/kent/src/hg/makeDb/genbank/etc/hgwbeta.dbs in alphabetical order.
  • Be sure to save, git add, git commit, and git push the file.
  • This turns on GenBank updates for the assembly. Currently (June 2019), GenBank updates are done quarterly (every 3 months).
  • Do not yet edit the rr.dbs, this comes later when the assembly is on the RR.
cat ~/kent/src/hg/makeDb/genbank/etc/hgwbeta.dbs | grep $db

Beta: GenBank updates: make libs and run make

After committing the change, make sure your libs are up to date:

cd ~/kent/src ; make libs

then go ahead and run the make:

cd ~/kent/src/hg/makeDb/genbank/ 
git pull 
make install-server

To see whether updates have run (at least the day after the *.dbs files were updated), check the update times of the table 'gbLoaded':

Beta: GenBank updates: check Genbank update times

hgwdev > updateTimes.csh $db gbLoaded verbose

For example, you'll see last update time for that particular SQL table on dev (but not yet for the beta/rr/euro/asia):

updateTimes.csh manPen1 gbLoaded verbose

gbLoaded
=============
dev  2017-04-18 11:40:07
beta 2017-04-18 11:40:07

rr
euro
asia

The update times may be out of sync between machines, but not by more than 24 hours or so if updates are running. The gbLoaded table should have been pushed with the tables to beta. In the future, this update time may not change for a while, this is because we now pull GenBank updates quarterly vs. nightly. Make sure you still add you $db to the hgwdev.dbs and hgwbeta.dbs files. More genbank update instructions are available at Genbank updates.

The etc-update-server part of the make will cause the downloads mentioned below in the "Verify downloads" section to be created.

Beta: joinerCheck common keys

Check that common keys between tables are in sync:

 hgwdev > cd ~/kent/src/hg/makeDb/schema 

 hgwdev > joinerCheck -database=$db -keys all.joiner

If there are errors related to genbank identifiers, it is likely because of the genbank load process, and not an issue with your database. Run joinerCheck once the tables are on beta to confirm:

 hgwdev > HGDB_CONF=~/.hg.conf.beta joinerCheck -keys -identifier=$identifier all.joiner

Beta: joinerCheck table times

Check table update times:

cd ~/kent/src/hg/makeDb/schema 
joinerCheck -database=$db -times all.joiner

You should get no output if it ran successfully.

Beta: joinerCheck tableCoverage

Check that all tables in this database are mentioned/referenced in all.joiner

 hgwdev > joinerCheck -database=$db -tableCoverage all.joiner 

If not all of the tables are listed, email the developer asking him to add those tables to the tablesIgnored $db. According to Hiram it is probably ok for us to edit all.joiner ourselves.

Beta: Check release tags: compare dev and beta tracks side by side

Compare your tracks by bringing up a dev and a beta browser window side-by-side. If some beta tracks can't be seen, you may need to edit the release tag.

See this page for more information:
http://genomewiki.ucsc.edu/index.php/ThreeStateTrackDb

Where are your release tags? They're in your .ra files, you can do a recursive search, like this:

find . -name '*ra' | xargs grep "manPen1"


Beta: Check Tools: LiftOver

Check LiftOver on beta and beta public.

Beta: Check Tools: BLAT: dna

Copy some dna and check BLAT for a DNA search on beta and beta public.

Beta: Check Tools: BLAT: protein

Copy some protein and check BLAT for a protein search on beta and beta public.

Beta: Check Tools: PCR

Check PCR on beta and beta public. See how to test PCR details from DEV steps

Beta: Review chain net track QA wiki

Your assembly may have chain/nets for a composite chain/net track. For example, you may have a 'turkey' assembly that belongs in the "Vertabrate Chain/Net" composite track on hg38. If your chain/nets appear next to the composite track, rather than within the composite track, you may need to create separate chainNetMelGal1.ra and chainNetMelGal5.ra files in trackDb/human. Once you have those files created then you will need to add the appropriate includes to the trackDb/human/hg38/trackDb.ra and trackDb/human/hg19/trackDb.chainNet.ra files.

Beta: Do chain net quick qa or full qa

DO NOT RUN these three commands since blastz has been replaced by lastz!!! Instead run findScores.pl, as shown at bottom


Details in Chain/Net QA wiki page, section QAing on hgwbeta

Full QA, run:

chainNetTrio.csh  [database] [other database]

Quick QA, run

getChainLines.csh [database] [other database]

The following getMatrixLines.csh script is broken:

getMatrixLines.csh [database] [other database]

You can see the same information with:

$ findScores.pl susScr11 hg38
looking in file:
/hive/data/genomes/hg38/bed/lastz.susScr11/axtChain/run/chain.csh
chainMinScore 3000
chainLinearGap medium
default values:
matrix 16 91,-114,-31,-123,-114,100,-125,-31,-31,-125,100,-114,-123,-31,-114,91

Beta: Re push any tables if needed after checking updateTimes

  • Make sure no tables need to be repushed from hgwdev to hgwbeta.
  • Note that if at any point you need to re-push some genbank tables, you must push ALL genbank tables together. <-- (Don't need to run this step anymore)
  • Remember that Dev should have several tables that beta doesn't (e.g., author, cds, cell, chainRBest*, chainSyn*)

You can use

 
hgwdev > updateTimesDb.sh -d $db

Or you can check tables in a list by using a loop like this:

for table in $(cat hg38.push); do updateTimes.csh hg38 $table verboseNoMirror >> hg38.updateTimes.devVSbeta ; done

to compare table update times between hgwdev and hgwbeta. Everything but hgFindSpec, history, tableDescriptions, and trackDb tables should have the same update times.

To see all of the tables in the assembly that are related to genbank do this (Don't need to run this step anymore):

hgwdev > hgsql -Ne 'show tables' $db | egrep -f /cluster/data/genbank/etc/genbank.tbls

Beta: Do a make public in trackDb for your assembly

Make your track public by using the "make public" command on hgwdev while in the trackDb directory (src/hg/makeDb/trackDb):

  make public DBS=$db

Example to make beta on more than one db at a time:

 make beta DBS='$db1 $db2 $db3 $db4 etc'

Your track should now be visible on the hgwbeta-public server.

If your track is not visible, you may want to check that your track has the correct release tag. Also see [Three State TrackDb] for more information.

Beta: Do a make public in trackDb for chain net organisms

If your assembly has alignments to other organisms, such as chain/net alignments to hg38 or to the previous assembly version of your organism, be sure to also do a 'make public' for those assemblies.

Beta: Remove release tag for big vcf track types

Once you verify that the track looks good on hgwbeta, remove the release tag from trackDb.ra.

More information about editing release tags can be found here:
http://genomewiki.ucsc.edu/index.php/ThreeStateTrackDb#Updated_trackDb_release_process


🔵 Done with BETA steps? Go to Assembly QA Part 4: RR Steps