ThreeStateTrackDb: Difference between revisions

From genomewiki
Jump to navigationJump to search
No edit summary
 
(14 intermediate revisions by 6 users not shown)
Line 1: Line 1:
This is a proposal for modifying the trackDb release process developed by Brian, Brooke, and Markd.  It is intended as a straight-forward modification to the existing process to address some of the current problems; not as a major,comprehensive change.
This is a proposal for modifying the trackDb release process developed by Brian, Brooke, and Markd.  It is intended as a straight-forward modification to the existing process to address some of the current problems; not as a major, comprehensive change.


This address the following issues:
This addresses the following issues:
* Release tags currently do not prevent modifications to tracks already on the RR from leaking to the RR while in QA.  This is caused by the release tag not accurately modeling the process.  This is addressed by adding a third state: "public".
* Release tags currently do not prevent modifications to tracks already on the RR from leaking to the RR while in QA.  This is caused by the release tag not accurately modeling the process.  This is addressed by adding a third state: "public".
* Managing releases of updated versions of ENCODE composite tracks is complex, requiring editing a potentially large number of release tags in trackDb.ra stanza files.  Also, the release tag doesn't currently work on the composite track stanza.
* Managing releases of updated versions of ENCODE composite tracks is complex, requiring editing a potentially large number of release tags in trackDb.ra stanza files.  Also, the release tag doesn't currently work on the composite track stanza.
Line 49: Line 49:
  shortLabel Great RNAs
  shortLabel Great RNAs
  visibility pack
  visibility pack


=== Managing large composite tracks ===
=== Managing large composite tracks ===
Line 54: Line 55:
To address this issue, each large composite track will be moved to its own file with all of its contained track stanzas.  To minimize the amount of editing required, the include directive will be modified to have a release attribute.  Since includes are processed line-per-line, not as part of a stanza, an attribute is an easier approach than adding a release tag.
To address this issue, each large composite track will be moved to its own file with all of its contained track stanzas.  To minimize the amount of editing required, the include directive will be modified to have a release attribute.  Since includes are processed line-per-line, not as part of a stanza, an attribute is an easier approach than adding a release tag.


For example, if a developer created the file '''encGencode.ra''', the following line could be added to trackDb.ra (or trackDb.wgEncode.ra, or whatever):
For example, if a developer created the file '''wgEncodeGencode.ra''', the following line could be added to trackDb.ra (or trackDb.wgEncode.ra, or whatever):


  include encGencode.ra alpha
  include wgEncodeGencode.ra alpha


When QA gets it, this will become:
When QA gets it, this will become:


  include encGencode.ra alpha,beta
  include wgEncodeGencode.ra alpha,beta


And when it is released:
And when it is released:


  include encGencode.ra alpha,beta,public
  include wgEncodeGencode.ra
 
(no release restrictions is the same as having: alpha,beta,public)


====Updating existing public tracks====
====Updating existing public tracks====
When changes need to be made to an already-released composite track, the composite trackDb file is copied to a new file name and it is added to cvs.  So two copies of the entire file will exist, and trackDb.ra will look like:
When changes need to be made to an already-released composite track, the composite trackDb file is copied to a new file name and it is added to git.  So two copies of the entire file will exist (the *.new.ra having all the new changes/updates) and trackDb.ra will look like:


   include encGencode.new.ra alpha
   include wgEncodeGencode.new.ra alpha
   include encGencode.ra beta,public
   include wgEncodeGencode.ra beta,public


When the track is in QA and staged on hgwbeta, this will become:
When the track is in QA and staged on hgwbeta, this will become:


   include encGencode.new.ra alpha,beta
   include wgEncodeGencode.new.ra alpha,beta
   include encGencode.ra public
   include wgEncodeGencode.ra public


To release the new version, QA will:
To release the new version, QA will:


   cp encGencode.new.ra encGencode.ra
   cp wgEncodeGencode.new.ra wgEncodeGencode.ra
  cvs commit encGencode.ra


And change the include line back to:
commit the changes. And change the include line back to:


     include encGencode.ra
     include wgEncodeGencode.ra


The changes won't leak to the RR before QA approves them and removes the release restrictions on the include directive (which is equivalent to a release restriction of alpha,beta,public).
The changes won't leak to the RR before QA approves them and removes the release restrictions on the include directive (which is equivalent to a release restriction of alpha,beta,public).


==== Updating a track that is mid-QA====
==== Updating a track that is mid-QA====
Often, a developer needs to start more work on a track before it is through the QA process and released and make the changes visible on genome-test for data contributors.  In this case, a third file can be created and committed to CVS:
Often, a developer needs to start more work on a track before it is through the QA process and released and make the changes visible on genome-test for data contributors.  In this case, a third file can be created and committed to git:
 
  include wgEncodeGencode.latestForContributor.ra alpha  #name is arbitrary
  include wgEncodeGencode.new.ra beta
  include wgEncodeGencode.ra public
 
When QA of the original update is complete that change released, the developer is free create a pushQ entry for it.  QA can copy the latest version to wgEncodeGencode.new.ra when ready to move it to beta.  Once the latest changes have been git committed to wgEncodeGencode.new.ra and there are no include directives for the arbitrarily-named wgEncodeGencode.latestForContributor.ra file, the latter can be git removed.
 
====Existing super-tracks MUST use release tags====
 
IT SEEMS LIKE THIS IS PROBABLY FIXED in 2011
AND YOU CAN IGNORE THIS SECTION.


  include encGencode.latestForContributor.ra alpha  #name is arbitrary
When you add new tracks to previously released super-tracks, the old paradigm of needing a table + trackDb entry in order for your track to be anything but completely invisible on the RR *does not apply*. So, if you are adding a new track to a super-track that is already on the RR, even though the tables are not yet on the RR, you must use release tags.
  include encGencode.new.ra beta
  include encGencode.ra public


When QA of the original update is complete that change released, the developer is free create a pushQ entry for it.  QA can copy the latest version to encGencode.new.ra when ready to move it to beta. Once the latest changes have been cvs committed to encGencode.new.ra and there are no include directives for the arbitrarily-named encGencode.latestForContributor.ra file, the latter can be cvs removed.
Currently, if you do not use release tags and add a new track to a super-track, users will be able to see your new track listed on the super-track control page. However, when the user tries to turn it on, they won't see anything. When the user tries to go to the hgTrackUi page, they will get an error.


=== Migration Plan ===
This issue is documented in redmine: http://redmine.soe.ucsc.edu/issues/2046.


The goal of the migration plan is to minimize any disruption in tracks currently being developed.
=== hgFindSpec ===
As of May, 2012, release tags apply to search stanzas in trackDb.ra as well as to track stanzas. Galt's comments on the topic:


Changes that can be made while the current release model continues to function:
"hgFindSpec now takes -release=<tag>, where tag is alpha, beta, or public.  If no release is specified, alpha is the default.  It works on both include and stanza levels.  It supports the standard root/org/db levels.  It does not have the fancy superTrack, subTrack, views, etc. issues that trackDb has.
searchName must be unique for the given release tag.  It will errAbort with a message if it finds a duplicate.  loadTracks now sends the -release option to hgFindSpec."


* Check for any existing tables were tracks have overlapping tags.  For instance, there are composite tracks where some tables have the beta tag and some have no tag.  Correct any that are found.
* Add generic release tag changes to the library code and hgTrackDb program
* Modify all instances of '''release beta''' to be '''release beta,public'''.
* Modify trackDb/makefile where '''make strict''' requests the beta,public tags
* Setup hgwbeta-public
* Convert ENCODE composite tracks to one per include.
* edit any trackDb that is staged on hgwbeta but not yet released back to '''release beta''' (remove the public)


At this point, all mechanism are in place
----
* replace '''make strict''' with '''make beta''' and '''make public'''
Some keywords for easier searching: 3-state, three, three-state, three state.  See related page [[http://genomewiki.ucsc.edu/genecats/index.php/Push-Request_Etiquette Push-Request_Etiquette].
* switch to new methodology
[[Category:Browser QA]]

Latest revision as of 21:10, 15 October 2020

This is a proposal for modifying the trackDb release process developed by Brian, Brooke, and Markd. It is intended as a straight-forward modification to the existing process to address some of the current problems; not as a major, comprehensive change.

This addresses the following issues:

  • Release tags currently do not prevent modifications to tracks already on the RR from leaking to the RR while in QA. This is caused by the release tag not accurately modeling the process. This is addressed by adding a third state: "public".
  • Managing releases of updated versions of ENCODE composite tracks is complex, requiring editing a potentially large number of release tags in trackDb.ra stanza files. Also, the release tag doesn't currently work on the composite track stanza.

Updated trackDb release process

The three state trackDb (and hgFindSpec) release process is:

  • alpha state: make alpha run on hgwdev loads all tracks with release alpha into trackDb and hgFindSpec regardless of the table existing.
  • beta state: make beta run on hgwbeta loads all tracks with release beta into trackDb and hgFindSpec only if the table exists.
  • public state: make public run on hgwbeta loads all tracks with release public into trackDb_public and hgFindSpec_public only if the table exists.

Having no release tag is equivalent to having all 3 release tags present.

QA can look at trackDb_public on hgwbeta-public.cse.ucsc.edu as a final check before pushing it to the RR. Then trackDb_public and hgFindSpec_public will be pushed from mysqlbeta to mysqlrr, renaming them to trackDb and hgFindSpec.

Tracks that don't already exist on the RR function as they do now:

  • Developer creates table(s) and adds a corresponding stanza to trackDb.ra, with no release tags
  • QA pushes table to mysqlbeta, does a make beta, and QAs track
  • When track is ready, QA does a make public, then pushes the tables and trackDb_public to mysqlrr, renaming them to trackDb

If a developer subsequently wants to make changes to the track, it would work like this:

  • the old stanza:
track someTrack
shortLabel Mediocre RNAs
visibility hide
  • becomes two stanzas:
track someTrack
release alpha
shortLabel Great RNAs
visibility pack
track someTrack
release beta,public
shortLabel Mediocre RNAs
visibility hide
  • nothing leaks out to the RR before it is ready. QA looks at the changes on hgwbeta by changing the first stanza to release alpha,beta and the second stanza to release public
  • when it is deemed worthy, the trackDb.ra entry can be collapsed back to one stanza, with no release tags (although leaving release alpha,beta,public in there would have the exact same effect):
track someTrack
shortLabel Great RNAs
visibility pack


Managing large composite tracks

To address this issue, each large composite track will be moved to its own file with all of its contained track stanzas. To minimize the amount of editing required, the include directive will be modified to have a release attribute. Since includes are processed line-per-line, not as part of a stanza, an attribute is an easier approach than adding a release tag.

For example, if a developer created the file wgEncodeGencode.ra, the following line could be added to trackDb.ra (or trackDb.wgEncode.ra, or whatever):

include wgEncodeGencode.ra alpha

When QA gets it, this will become:

include wgEncodeGencode.ra alpha,beta

And when it is released:

include wgEncodeGencode.ra

(no release restrictions is the same as having: alpha,beta,public)

Updating existing public tracks

When changes need to be made to an already-released composite track, the composite trackDb file is copied to a new file name and it is added to git. So two copies of the entire file will exist (the *.new.ra having all the new changes/updates) and trackDb.ra will look like:

 include wgEncodeGencode.new.ra alpha
 include wgEncodeGencode.ra beta,public

When the track is in QA and staged on hgwbeta, this will become:

 include wgEncodeGencode.new.ra alpha,beta
 include wgEncodeGencode.ra public

To release the new version, QA will:

 cp wgEncodeGencode.new.ra wgEncodeGencode.ra

commit the changes. And change the include line back to:

   include wgEncodeGencode.ra

The changes won't leak to the RR before QA approves them and removes the release restrictions on the include directive (which is equivalent to a release restriction of alpha,beta,public).

Updating a track that is mid-QA

Often, a developer needs to start more work on a track before it is through the QA process and released and make the changes visible on genome-test for data contributors. In this case, a third file can be created and committed to git:

 include wgEncodeGencode.latestForContributor.ra alpha  #name is arbitrary
 include wgEncodeGencode.new.ra beta
 include wgEncodeGencode.ra public

When QA of the original update is complete that change released, the developer is free create a pushQ entry for it. QA can copy the latest version to wgEncodeGencode.new.ra when ready to move it to beta. Once the latest changes have been git committed to wgEncodeGencode.new.ra and there are no include directives for the arbitrarily-named wgEncodeGencode.latestForContributor.ra file, the latter can be git removed.

Existing super-tracks MUST use release tags

IT SEEMS LIKE THIS IS PROBABLY FIXED in 2011 AND YOU CAN IGNORE THIS SECTION.

When you add new tracks to previously released super-tracks, the old paradigm of needing a table + trackDb entry in order for your track to be anything but completely invisible on the RR *does not apply*. So, if you are adding a new track to a super-track that is already on the RR, even though the tables are not yet on the RR, you must use release tags.

Currently, if you do not use release tags and add a new track to a super-track, users will be able to see your new track listed on the super-track control page. However, when the user tries to turn it on, they won't see anything. When the user tries to go to the hgTrackUi page, they will get an error.

This issue is documented in redmine: http://redmine.soe.ucsc.edu/issues/2046.

hgFindSpec

As of May, 2012, release tags apply to search stanzas in trackDb.ra as well as to track stanzas. Galt's comments on the topic:

"hgFindSpec now takes -release=<tag>, where tag is alpha, beta, or public. If no release is specified, alpha is the default. It works on both include and stanza levels. It supports the standard root/org/db levels. It does not have the fancy superTrack, subTrack, views, etc. issues that trackDb has. searchName must be unique for the given release tag. It will errAbort with a message if it finds a duplicate. loadTracks now sends the -release option to hgFindSpec."



Some keywords for easier searching: 3-state, three, three-state, three state. See related page [Push-Request_Etiquette.