Assembly QA Part 3 BETA Steps

This page is currently a draft in progress. For now, use Releasing an assembly instead.

Navigation Menu Home Page Assembly QA Part 1: DEV Steps Assembly QA Part 2: Track Steps Assembly QA Part 3: BETA Steps Assembly QA Part 4: RR Steps Assembly QA Part 5: Post Release Steps

NOTE TO SELF ADD LINKS TO ALL CHAIN-NET STEPS http://genomewiki.ucsc.edu/genecats/index.php/Chains_and_Nets_QA

Tracks: Populate spreadsheet steps

• We need to create a checklist for your beta steps.
• You can add a new tab to track beta steps, or you can pick up where you left off on the same tab as your "dev" steps.

To populate the "wiki link" for each step, add this formula to cell A2 (or the row after your last "dev" step) in your new "track checklist" spreadsheet and drag the formula down:

A2 (or the row after your last "dev" step)

=HYPERLINK("http://genomewiki.ucsc.edu/genecats/index.php/Assembly_QA_Part_3_BETA_Steps#"&SUBSTITUTE(B2," ", "_"),"link")

IMPORTANT: Drag the formula for "A" down the spreadsheet to populate the other rows.

To populate the "track checklist steps," add this formula to cell B2 ((or the row after your last "dev" step) in your new "track checklist" spreadsheet. Do NOT drag the formula down.

B2

=IMPORTXML("http://genomewiki.ucsc.edu/genecats/index.php/Assembly_QA_Part_3_BETA_Steps#", "/html/body/div/div/div/div/div/h4/span/span")

This formula will populate all the rows below it with the wiki section titles. You do no need to drag this formula down.

Tracks: Push to hgwbeta

Note for Ensembl gene tracks:

• Before pushing to beta, use the Ensembl_QA script.

Special case for 'seq' and 'extFile' tables:

• Do not push seq or extFile tables from dev to beta. You must use the copyExtSeqRows.csh script to move only the rows needed. More information can be found here.

Notes on existing tracks:

• If this is an update to an to an existing track, you may want to hold off on this step so that you can compare old and new tracks on hgwdev and hgwbeta.
• Open the track on hgwbeta before staging it to make sure that the update won't cause a cart clash for users currently looking at the track (as evidenced by a completely blank screen, for instance). If you need to do a cartReset to get the track to show up correctly, something is wrong.

Push all tables (EXCEPT seq and extFile tables, see note below) from hgwdev to hgwbeta:

 sudo mypush $db $table mysqlbeta

Next, make beta on hgwdev in kent/src/hg/makeDb/trackDb like so:

 make beta DBS=$db

Example to make beta on more than one db at a time:

 make beta DBS='$db1 $db2 $db3 $db4 etc'

---

Running multiple dbs in parallel to save time

Multiple assemblies can be run in parallel by using the make -j option (as of 2/10/17, thanks to Mark Diekhans). Updating all dev dbs used to take about 50 minutes, and now it can take about 5 minutes (at 16 in parallel). While Mark has safely run 16 dbs at a time on dev, it is recommended to only run 8 or less at a time on beta or the RR. Use make -j # beta and make -j # public, where the number (make -j 16 alpha) represents how many parallel processes (16) are running.

For example, if you do:

  make -j 8 alpha

it updates everything, 8 at a time. If you do:

  make -j 2 DBS="hg19 hg38 mm10 felCat5"

it updates those 4 databases, 2 at a time .

Note: the 'make in parallel' process creates and removes temporary files:

The tmp dirs are found with:

 kent/src/inc/portable.h:
   char *getTempDir(void);
   /* get temporary directory to use for programs.  This first checks TMPDIR environment
    * variable, then /data/tmp, /scratch/tmp, /var/tmp, /tmp.  Return is static and
    * only set of first call */

Examples:

 make beta -j 4 DBS="dm6 ce11 sacCer3 droEre1 droSec1 droSim1 droYak2 droAna2 dp3 droMoj2 droVir2 droGri1 droPer1"
 make public -j 4 DBS="dm6 ce11 sacCer3 droEre1 droSec1 droSim1 droYak2 droAna2 dp3 droMoj2 droVir2 droGri1 droPer1"

---

Request a push of any listed supporting files in /gbdb from hgwdev to hgnfs1 and check on hgwbeta. Note that hgwbeta and the RR share the files on hgnfs1, so once these files are in place, there is not another push required when the track is released to the RR. Be sure to send a push request to have the gbdb files pushed to hgdownload in advance of the usual Sunday sync, if this is necessary for your track.

If there are images associated with any track description pages, be sure to run a make beta from within kent/src/hg/htdocs/ to get the images to beta.

Tracks: Make Public

Make your track public by using the "make public" command on hgwdev while in the trackDb directory (src/hg/makeDb/trackDb):

   [user@hgwdev trackDb]$ make public DBS=$db

Your track should now be visible on the hgwbeta-public server.

If your track is not visible, you may want to check that your track has the correct release tag. Also see [Three State TrackDb] for more information.

Tracks: Remove release tag for big*/vcf track types

Once you verify that the track looks good on hgwbeta, remove the release tag from trackDb.ra.

Tracks: Remove release tag for big*/vcf track types

Once you verify that the track looks good on hgwbeta, remove the release tag from trackDb.ra.

🔵 Done with BETA steps? Go to Assembly QA Part 4: RR Steps