Threestatemetadb

This is a proposal for instituting a metaDb release process developed by Brian with all real programming to implement the metaDb table done by Tim. Please see the page ThreeStateTrackDb page for some description of how that three state process works. ThreeStateTrackDb

This address the following issues:

• track metadata is stored in the trackDb table, ignoring the fact that we have metadata for downloads that aren't tables.

Updated trackDb release process

The three state metaDb release process is:

• alpha state: make alpha run on hgwdev loads all tracks with release alpha into trackDb and hgFindSpec regardless of the table existing.
• beta state: make beta run on hgwbeta loads all tracks with release beta into trackDb and hgFindSpec only if the table exists.
• public state: make public run on hgwbeta loads all tracks with release public into trackDb_public and hgFindSpec_public only of the table exists.

Having no release tag is equivalent to having all 3 release tags present.

QA can look at trackDb_public on hgwbeta-public.cse.ucsc.edu as a final check before pushing it to the RR. Then trackDb_public and hgFindSpec_public will be pushed from mysqlbeta to mysqlrr, renaming them to trackDb and hgFindSpec.

Tracks that don't already exist on the RR function as they do now:

• Developer creates table(s) and adds a corresponding stanza to trackDb.ra, with no release tags
• QA pushes table to mysqlbeta, does a make beta, and QAs track
• When track is ready, QA does a make public, then pushes the tables and trackDb_public to mysqlrr, renaming them to trackDb

If a developer subsequently wants to make changes to the track, it would work like this:

• the old stanza:

track someTrack
shortLabel Mediocre RNAs
visibility hide

• becomes two stanzas:

track someTrack
release alpha
shortLabel Great RNAs
visibility pack

track someTrack
release beta,public
shortLabel Mediocre RNAs
visibility hide

• nothing leaks out to the RR before it is ready. QA looks at the changes on hgwbeta by changing the first stanza to release alpha,beta and the second stanza to release public

• when it is deemed worthy, the trackDb.ra entry can be collapsed back to one stanza, with no release tags (although leaving release alpha,beta,public in there would have the exact same effect):

track someTrack
shortLabel Great RNAs
visibility pack

Managing large composite tracks

To address this issue, each large composite track will be moved to its own file with all of its contained track stanzas. To minimize the amount of editing required, the include directive will be modified to have a release attribute. Since includes are processed line-per-line, not as part of a stanza, an attribute is an easier approach than adding a release tag.

For example, if a developer created the file encGencode.ra, the following line could be added to trackDb.ra (or trackDb.wgEncode.ra, or whatever):

include encGencode.ra alpha

When QA gets it, this will become:

include encGencode.ra alpha,beta

And when it is released:

include encGencode.ra alpha,beta,public

Updating existing public tracks

When changes need to be made to an already-released composite track, the composite trackDb file is copied to a new file name and it is added to cvs. So two copies of the entire file will exist, and trackDb.ra will look like:

 include encGencode.new.ra alpha
 include encGencode.ra beta,public

When the track is in QA and staged on hgwbeta, this will become:

 include encGencode.new.ra alpha,beta
 include encGencode.ra public

To release the new version, QA will:

 cp encGencode.new.ra encGencode.ra
 cvs commit encGencode.ra

And change the include line back to:

   include encGencode.ra

The changes won't leak to the RR before QA approves them and removes the release restrictions on the include directive (which is equivalent to a release restriction of alpha,beta,public).

Updating a track that is mid-QA

Often, a developer needs to start more work on a track before it is through the QA process and released and make the changes visible on genome-test for data contributors. In this case, a third file can be created and committed to CVS:

 include encGencode.latestForContributor.ra alpha  #name is arbitrary
 include encGencode.new.ra beta
 include encGencode.ra public

When QA of the original update is complete that change released, the developer is free create a pushQ entry for it. QA can copy the latest version to encGencode.new.ra when ready to move it to beta. Once the latest changes have been cvs committed to encGencode.new.ra and there are no include directives for the arbitrarily-named encGencode.latestForContributor.ra file, the latter can be cvs removed.

Migration Plan

The goal of the migration plan is to minimize any disruption in tracks currently being developed.

Changes that can be made while the current release model continues to function:

• Check for any existing tables were tracks have overlapping tags. For instance, there are composite tracks where some tables have the beta tag and some have no tag. Correct any that are found.
• Add generic release tag changes to the library code and hgTrackDb program
• Modify all instances of release beta to be release beta,public.
• Modify trackDb/makefile where make strict requests the beta,public tags
• Setup hgwbeta-public
• Convert ENCODE composite tracks to one per include.
• edit any trackDb that is staged on hgwbeta but not yet released back to release beta (remove the public)

At this point, all mechanism are in place

• replace make strict with make beta and make public
• switch to new methodology