ThreeStateTrackDb: Difference between revisions

From genomewiki
Jump to navigationJump to search
No edit summary
No edit summary
 
(32 intermediate revisions by 8 users not shown)
Line 1: Line 1:
This is a proposal for modifying the trackDb release process developed by Brian, Brooke, and Markd.  It is intended as a straight-forward modification to the existing process to address some of the current problems; not as a major,
This is a proposal for modifying the trackDb release process developed by Brian, Brooke, and Markd.  It is intended as a straight-forward modification to the existing process to address some of the current problems; not as a major, comprehensive change.
comprehensive change.


==Issue #1==
This addresses the following issues:
Release tags currently do not prevent modifications to tracks already on the RR from leaking to the RR while in QA.  This is caused by the release tag not accurately modeling the process.  This is addressed by adding a third state: "public".
* Release tags currently do not prevent modifications to tracks already on the RR from leaking to the RR while in QA.  This is caused by the release tag not accurately modeling the process.  This is addressed by adding a third state: "public".
* Managing releases of updated versions of ENCODE composite tracks is complex, requiring editing a potentially large number of release tags in trackDb.ra stanza files.  Also, the release tag doesn't currently work on the composite track stanza.


The three states would function like so:
=== Updated trackDb release process ===
The three state trackDb (and hgFindSpec) release process is:


alpha state:
* alpha state: '''make alpha''' run on hgwdev loads all tracks with '''release alpha''' into trackDb and hgFindSpec regardless of the table existing.
'make alpha' run on hgwdev loads all tracks with 'release alpha' into trackDb regardless of the table existing.
* beta state: '''make beta''' run on hgwbeta loads all tracks with '''release beta''' into trackDb and hgFindSpec only if the table exists.
* public state: '''make public''' run on hgwbeta loads all tracks with '''release public''' into trackDb_public and hgFindSpec_public only if the table exists.


beta state:
'''Having no release tag is equivalent to having all 3 release tags present.'''
'make beta' run on hgwbeta loads all tracks with 'release beta' into trackDb only of the table exists.


public state:
QA can look at trackDb_public on hgwbeta-public.cse.ucsc.edu as a final check before pushing it to the RR.  Then trackDb_public and hgFindSpec_public will be pushed from mysqlbeta to mysqlrr, renaming them to trackDb and hgFindSpec.
'make public' run on hgwbeta loads all tracks with 'release public' into trackDb_public only of the table exists.


Having no release tag is equivalent to having all 3 release tags present.
Tracks that don't already exist on the RR function as they do now:
 
QA can look at trackDb_public on hgwbeta-public.cse.ucsc.edu as a final check before pushing it to the RR.  Then trackDb_public will be pushed from mysqlbeta to mysqlrr, where it will be renamed trackDb.
 
 
A normal, "old school" track will function in much the same way as it functions now:


* Developer creates table(s) and adds a corresponding stanza to trackDb.ra, with no release tags
* Developer creates table(s) and adds a corresponding stanza to trackDb.ra, with no release tags
* QA pushes table to mysqlbeta, does a 'make beta', and QAs track
* QA pushes table to mysqlbeta, does a '''make beta''', and QAs track
* When track is ready, QA does a 'make public', then pushes table and trackDb_public* to mysqlrr (and trackDb_public is renamed trackDb along the way)
* When track is ready, QA does a '''make public''', then pushes the  tables and trackDb_public to mysqlrr, renaming them to trackDb


If a developer subsequently wants to make changes to the track, it would work like this:
If a developer subsequently wants to make changes to the track, it would work like this:
Line 47: Line 42:
  visibility hide
  visibility hide


* nothing leaks out to the RR before it is ready.  QA looks at the changes on hgwbeta by changing the first stanza to 'release alpha,beta' and the second stanza to 'release public'
* nothing leaks out to the RR before it is ready.  QA looks at the changes on hgwbeta by changing the first stanza to '''release alpha,beta''' and the second stanza to '''release public'''


* when it is deemed worthy, QA collapses the trackDb.ra entry back to one stanza, with no release tags (although leaving 'release alpha,beta,public' in there would have the exact same effect):
* when it is deemed worthy, the trackDb.ra entry can be collapsed back to one stanza, with no release tags (although leaving '''release alpha,beta,public''' in there would have the exact same effect):


  track someTrack
  track someTrack
Line 56: Line 51:




==Issue #2==
=== Managing large composite tracks ===
Managing releases of updated versions of ENCODE composite tracks is complex, requiring editing a potentially large number of release tags in trackDb.ra stanza files.  Also, the release tag doesn't currently work on the composite track stanza.


To address this issue, each large composite track will be moved to its own file, named in the form: trackDb.compositeName.version.ra, and include all of its contained track stanzas.  To minimize the amount of editing required, the include directive will be modified to have a release attribute.  Since includes are processed line-per-line, not as part of a stanza, an attribute is an easier approach than adding a release tag.
To address this issue, each large composite track will be moved to its own file with all of its contained track stanzas.  To minimize the amount of editing required, the include directive will be modified to have a release attribute.  Since includes are processed line-per-line, not as part of a stanza, an attribute is an easier approach than adding a release tag.


For example, if a developer created the file trackDb.encGencode.1.ra, the following line could be added to trackDb.ra (or trackDb.wgEncode.ra, or whatever):
For example, if a developer created the file '''wgEncodeGencode.ra''', the following line could be added to trackDb.ra (or trackDb.wgEncode.ra, or whatever):


  include trackDb.encGencode.1.ra alpha
  include wgEncodeGencode.ra alpha


When QA gets it, this will become:
When QA gets it, this will become:


  include trackDb.encGencode.1.ra alpha,beta
  include wgEncodeGencode.ra alpha,beta


And when it is released:
And when it is released:


  include trackDb.encGencode.1.ra alpha,beta,public
  include wgEncodeGencode.ra
  (NOTE TO MARKD/BRANEY: should there be no attribute instead??)
 
(no release restrictions is the same as having: alpha,beta,public)
 
====Updating existing public tracks====
When changes need to be made to an already-released composite track, the composite trackDb file is copied to a new file name and it is added to git.  So two copies of the entire file will exist (the *.new.ra having all the new changes/updates) and trackDb.ra will look like:
 
  include wgEncodeGencode.new.ra alpha
  include wgEncodeGencode.ra beta,public
 
When the track is in QA and staged on hgwbeta, this will become:
 
  include wgEncodeGencode.new.ra alpha,beta
  include wgEncodeGencode.ra public
 
To release the new version, QA will:
 
  cp wgEncodeGencode.new.ra wgEncodeGencode.ra
 
commit the changes. And change the include line back to:
 
    include wgEncodeGencode.ra
 
The changes won't leak to the RR before QA approves them and removes the release restrictions on the include directive (which is equivalent to a release restriction of alpha,beta,public).
 
==== Updating a track that is mid-QA====
Often, a developer needs to start more work on a track before it is through the QA process and released and make the changes visible on genome-test for data contributors. In this case, a third file can be created and committed to git:
 
  include wgEncodeGencode.latestForContributor.ra alpha  #name is arbitrary
  include wgEncodeGencode.new.ra beta
  include wgEncodeGencode.ra public
 
When QA of the original update is complete that change released, the developer is free create a pushQ entry for it.  QA can copy the latest version to wgEncodeGencode.new.ra when ready to move it to beta.  Once the latest changes have been git committed to wgEncodeGencode.new.ra and there are no include directives for the arbitrarily-named wgEncodeGencode.latestForContributor.ra file, the latter can be git removed.
 
====Existing super-tracks MUST use release tags====
 
IT SEEMS LIKE THIS IS PROBABLY FIXED in 2011
AND YOU CAN IGNORE THIS SECTION.
 
When you add new tracks to previously released super-tracks, the old paradigm of needing a table + trackDb entry in order for your track to be anything but completely invisible on the RR *does not apply*. So, if you are adding a new track to a super-track that is already on the RR, even though the tables are not yet on the RR, you must use release tags.
 
Currently, if you do not use release tags and add a new track to a super-track, users will be able to see your new track listed on the super-track control page. However, when the user tries to turn it on, they won't see anything. When the user tries to go to the hgTrackUi page, they will get an error.
 
This issue is documented in redmine: http://redmine.soe.ucsc.edu/issues/2046.


When changes need to be made to an already-released composite track, the file is copied, the version number in the name is incremented, and it is added to cvsSo two copies of the entire file will exist, and trackDb.ra will look like:
=== hgFindSpec ===
As of May, 2012, release tags apply to search stanzas in trackDb.ra as well as to track stanzasGalt's comments on the topic:


  include trackDb.encGencode.2.ra alpha
"hgFindSpec now takes -release=<tag>, where tag is alpha, beta, or public.  If no release is specified, alpha is the default.  It works on both include and stanza levels.  It supports the standard root/org/db levels. It does not have the fancy superTrack, subTrack, views, etc. issues that trackDb has.
  include trackDb.encGencode.1.ra beta,public
searchName must be unique for the given release tag. It will errAbort with a message if it finds a duplicate. loadTracks now sends the -release option to hgFindSpec."


The changes won't leak to the RR before QA approves them and promotes the version 2 file to alpha,beta,public.  Once an older version of a composite track is no longer on any systems, the corresponding .ra file can be cvs removed.


Note that the version numbers here are just sequential, and they do not necessarily correspond to the ENCODE "V2" track releases.
----
Some keywords for easier searching: 3-state, three, three-state, three state.  See related page [[http://genomewiki.ucsc.edu/genecats/index.php/Push-Request_Etiquette Push-Request_Etiquette].
[[Category:Browser QA]]

Latest revision as of 21:10, 15 October 2020

This is a proposal for modifying the trackDb release process developed by Brian, Brooke, and Markd. It is intended as a straight-forward modification to the existing process to address some of the current problems; not as a major, comprehensive change.

This addresses the following issues:

  • Release tags currently do not prevent modifications to tracks already on the RR from leaking to the RR while in QA. This is caused by the release tag not accurately modeling the process. This is addressed by adding a third state: "public".
  • Managing releases of updated versions of ENCODE composite tracks is complex, requiring editing a potentially large number of release tags in trackDb.ra stanza files. Also, the release tag doesn't currently work on the composite track stanza.

Updated trackDb release process

The three state trackDb (and hgFindSpec) release process is:

  • alpha state: make alpha run on hgwdev loads all tracks with release alpha into trackDb and hgFindSpec regardless of the table existing.
  • beta state: make beta run on hgwbeta loads all tracks with release beta into trackDb and hgFindSpec only if the table exists.
  • public state: make public run on hgwbeta loads all tracks with release public into trackDb_public and hgFindSpec_public only if the table exists.

Having no release tag is equivalent to having all 3 release tags present.

QA can look at trackDb_public on hgwbeta-public.cse.ucsc.edu as a final check before pushing it to the RR. Then trackDb_public and hgFindSpec_public will be pushed from mysqlbeta to mysqlrr, renaming them to trackDb and hgFindSpec.

Tracks that don't already exist on the RR function as they do now:

  • Developer creates table(s) and adds a corresponding stanza to trackDb.ra, with no release tags
  • QA pushes table to mysqlbeta, does a make beta, and QAs track
  • When track is ready, QA does a make public, then pushes the tables and trackDb_public to mysqlrr, renaming them to trackDb

If a developer subsequently wants to make changes to the track, it would work like this:

  • the old stanza:
track someTrack
shortLabel Mediocre RNAs
visibility hide
  • becomes two stanzas:
track someTrack
release alpha
shortLabel Great RNAs
visibility pack
track someTrack
release beta,public
shortLabel Mediocre RNAs
visibility hide
  • nothing leaks out to the RR before it is ready. QA looks at the changes on hgwbeta by changing the first stanza to release alpha,beta and the second stanza to release public
  • when it is deemed worthy, the trackDb.ra entry can be collapsed back to one stanza, with no release tags (although leaving release alpha,beta,public in there would have the exact same effect):
track someTrack
shortLabel Great RNAs
visibility pack


Managing large composite tracks

To address this issue, each large composite track will be moved to its own file with all of its contained track stanzas. To minimize the amount of editing required, the include directive will be modified to have a release attribute. Since includes are processed line-per-line, not as part of a stanza, an attribute is an easier approach than adding a release tag.

For example, if a developer created the file wgEncodeGencode.ra, the following line could be added to trackDb.ra (or trackDb.wgEncode.ra, or whatever):

include wgEncodeGencode.ra alpha

When QA gets it, this will become:

include wgEncodeGencode.ra alpha,beta

And when it is released:

include wgEncodeGencode.ra

(no release restrictions is the same as having: alpha,beta,public)

Updating existing public tracks

When changes need to be made to an already-released composite track, the composite trackDb file is copied to a new file name and it is added to git. So two copies of the entire file will exist (the *.new.ra having all the new changes/updates) and trackDb.ra will look like:

 include wgEncodeGencode.new.ra alpha
 include wgEncodeGencode.ra beta,public

When the track is in QA and staged on hgwbeta, this will become:

 include wgEncodeGencode.new.ra alpha,beta
 include wgEncodeGencode.ra public

To release the new version, QA will:

 cp wgEncodeGencode.new.ra wgEncodeGencode.ra

commit the changes. And change the include line back to:

   include wgEncodeGencode.ra

The changes won't leak to the RR before QA approves them and removes the release restrictions on the include directive (which is equivalent to a release restriction of alpha,beta,public).

Updating a track that is mid-QA

Often, a developer needs to start more work on a track before it is through the QA process and released and make the changes visible on genome-test for data contributors. In this case, a third file can be created and committed to git:

 include wgEncodeGencode.latestForContributor.ra alpha  #name is arbitrary
 include wgEncodeGencode.new.ra beta
 include wgEncodeGencode.ra public

When QA of the original update is complete that change released, the developer is free create a pushQ entry for it. QA can copy the latest version to wgEncodeGencode.new.ra when ready to move it to beta. Once the latest changes have been git committed to wgEncodeGencode.new.ra and there are no include directives for the arbitrarily-named wgEncodeGencode.latestForContributor.ra file, the latter can be git removed.

Existing super-tracks MUST use release tags

IT SEEMS LIKE THIS IS PROBABLY FIXED in 2011 AND YOU CAN IGNORE THIS SECTION.

When you add new tracks to previously released super-tracks, the old paradigm of needing a table + trackDb entry in order for your track to be anything but completely invisible on the RR *does not apply*. So, if you are adding a new track to a super-track that is already on the RR, even though the tables are not yet on the RR, you must use release tags.

Currently, if you do not use release tags and add a new track to a super-track, users will be able to see your new track listed on the super-track control page. However, when the user tries to turn it on, they won't see anything. When the user tries to go to the hgTrackUi page, they will get an error.

This issue is documented in redmine: http://redmine.soe.ucsc.edu/issues/2046.

hgFindSpec

As of May, 2012, release tags apply to search stanzas in trackDb.ra as well as to track stanzas. Galt's comments on the topic:

"hgFindSpec now takes -release=<tag>, where tag is alpha, beta, or public. If no release is specified, alpha is the default. It works on both include and stanza levels. It supports the standard root/org/db levels. It does not have the fancy superTrack, subTrack, views, etc. issues that trackDb has. searchName must be unique for the given release tag. It will errAbort with a message if it finds a duplicate. loadTracks now sends the -release option to hgFindSpec."



Some keywords for easier searching: 3-state, three, three-state, three state. See related page [Push-Request_Etiquette.