ENCODE QA

From genomewiki
Revision as of 23:56, 22 February 2010 by Katrina (talk | contribs) (→‎Other things to check by hand: added releaseN requirement to have a release notes section)
Jump to navigationJump to search

Getting started

If this is a subsequent release, see #Subsequent Release of Data (e.g. Release 2) first.

  1. Make a list of all tables (new & updated that need to be pushed to beta)
  2. Do bigPush.csh using list created above
  3. Remove "release alpha" lines from subtracks in trackDb.wgEncode.ra that are for this release:
    • note: some new files may be for subsequent release...leave release alphas for them!
    1. cvsup
    2. run removeAlphas script (> to a file)
    3. diff between file & trackDb.wgEncode.ra (diff file1 file2)
      • double check # of release alphas matches # of tables (diff file1 file2 | grep release | wc --lines)
    4. copy file over trackDb.wgEncode.ra
    5. cvs diff to check new copy against repository copy
    6. commit change
    7. cvs diff again to make sure they are the same
    8. remove file
    9. make strict DBS=<db>
  4. Push any new wib files if applicable

run qaEncodeTracks.csh

You will need to dump the list of tables from the pushQ to a file (i.e. tableList in the usage statement). Then run qaEncodeTracks.csh, which does:

  • countPerChrom
  • check for entry in tableDescriptions table
  • check that shortLabel does not exceed 16 characters
  • check that longLabel does not exceed 80 characters
  • check that there are no underscores in the table names
  • check for indices on the tables
  • check that positional tables are sorted
  • checkOffEnd

Other things to check by hand

  1. Run genePredCheck/pslCheck if applicable. (i.e. if your track is a gene prediction track)
  2. To remove "release alpha" from trackDb use this script: removeAlphas
  3. make sure there is a link to the help doc (in the config section: "Select views (help)")
  4. check metadata field of trackDb entry
  5. read description page
    • is it detailed enough, especially Methods
    • are the citations in correct format
    • does the "Display Conventions and Configuration" section cover all track types
    • test all hyper-links
    • releaseN tracks should contain a section called "Release Notes" which should state the release# and provide a description of the changes in that particular release.
  6. Release log: Check to be sure the release log entry is right. Usually it should just be the shortlabel, but if there is something weird about the data that needs to be noted, make sure it fits in nicely with the current release log entries. (url should be of the format: ../../cgi-bin/hgTrackUi?db=hg18&g=wgEncodeAffyRnaChip )
  7. configuration section (does it work)
  8. multi-view config: matrix etc.
  9. check "reset to defaults" button (does it work)
  10. Make sure there's a link to the ENCODE Data Release Policy (at the bottom of the description page).
  11. Make sure the tracks in the Tier1 and Tier2 Cell Lines are properly colored (no black, and all tracks from one Cell Line have the same color).

Files

First, a note about finding the files. One of the most time-consuming things we do is track down items that should have been placed in the "Files" section of the pushQ entries but weren't. It takes us a long time to (a) figure out what's missing, and (b) find it. If developers can ensure that both the /gbdb and /goldenPath files are there, it would be a huge help!

Download File specifics:

  1. Check the index page (which created from two files, preamble.html & index.html) in the downloads directory (e.g. this page ):
    • preamble.html - the top description part of the index page (not the list of files)
      • new track: an introductory paragraph with a brief description, and link to the trackUi.
      • releaseN: should include the release number at end of description (e.g. "This is Release 2 of this track. Release notes are included in the track description.")
        • Also, make sure there is a preamble.html file in the /releaseN directory
      • If you need to edit the preamble.html (not the list of files), follow these steps:
        1. Edit the preamble.html file (note that it is not in CVS) in the downloads directory on hgwdev
        2. Regenerate index.html by running the script: encodeDownloadsPage.pl index.html
        3. Look at results on genome-test. If necessary, go back to step 1.
    • index.html - the list of files on the index page (not the description on top)
      • should contain the name of each file being released (and only the name of those files in this release)
        • new track: use the PushQ file list (shouldn't be files with V2 or V3, etc.) and your best judgement to determine that all the appropriate files (and only those files are listed).
        • releaseN: run the encodeDownloadsPage.pl script directly in the /releaseN directory (hgwdev: /data/apache/htdocs/goldenPath/<db>/<trackName>/releaseN/) to generate a new index.html page that you know contains all the files in that directory.
      • make sure this file is executable (because of the way the links are created)
  2. Downloads directory also needs to contain the following:
    • files.txt - plaintext version of index.html; lists files with metadata
    • md5sum.txt - checksum of all files in download directory
  3. When you are ready to release make sure your track is listed on the downloads page (http://genome.ucsc.edu/ENCODE/downloads.html ) - if it isn't listed, add a line for your track to this page and push the following from hgwdev -> hgwbeta, RR:
 /usr/local/apache/htdocs/ENCODE/downloads.html


Pushing Files

Pushing the three main types of files involved in ENCODE tracks.

  • gbdb Files

Files of this form get pushed hgwdev -> hgnfs1 

 /gbdb/hg18/wib/wgEncode*.wib
  • Other Files

Files of this form get pushed hgwdev -> hgwbeta & RR (they are quite often accidentally omitted from the pushQ entry -- you will need these types of protocol PDF files, if this is the first subtrack released for this cell line from this lab) 

 /usr/local/apache/htdocs/ENCODE/protocols/cell/*.pdf
  • Download Files

Files of this form get pushed hgwdev -> hgdownload 

 /usr/local/apache/htdocs/goldenPath/hg18/encodeDCC/wgEncode*/index.html
 /usr/local/apache/htdocs/goldenPath/hg18/encodeDCC/wgEncode*/wgEncode*.[bed/wig].gz

When pushing download files for a subsequent release track (e.g. release 2), push files 

 from hgwdev: /usr/local/apache/htdocs/goldenPath/<db>/encodeDCC/<trackName>/releaseN/
 to hgdownload: /usr/local/apache/htdocs/goldenPath/<db>/encodeDCC/<trackName>/
 (Note: no releaseN directory on hgdownload)
  • After Pushing Files:

Once the files have been pushed you can check to see if the push was successful using this script: checkPushedFiles.csh

validateFiles

  • No longer run, here are Tim's comments about QA running validateFiles: "To get these things through the pipeline, we run them through validateFiles, so I think your running them through again is one time too many. But if you are going to, then each lab and each file type may have negotiated limits (which may change between submissions). These limits are found in the relevant submission directory DAF files."
  • Old validateFiles process:

Test a smattering of different file types using this tool: validateFiles (type the program name without arguments to see the usage statement). If there are no errors, there will be no output. For example, for files of type tagAlign, invoke the tool like this: 

validateFiles -type=tagAlign -genome=/gbdb/hg18/hg18.2bit /usr/local/apache/htdocs/goldenPath/hg18/encodeDCC/wgEncodeHudsonalphaChipSeq/wgEncodeHudsonalphaChipSeqAlignmentsRep1Gm12878Control.tagAlign.gz

For tagAligns there are several relevant validateFiles options: 

  mismatches  - frequently 2 but negotiated for each lab.  Set this to 5 to be tolerant
  matchFirst - negotiated.  You should set this to 25 and even then you may need to adjust it
  nMatch - negotiated, but you should always have this parameter set.

If you want to be exact, then the metadata as seen on the downloads page tells which submission directory the file belongs to, and the most recent *.DAF (or *.daf) will have a validationSettings line in it which will include the settings that belong to each file type. Example: 

  /hive/groups/encode/dcc/pipeline/encpipeline_prod/773/UtaChIPseqBOonlyV1.DAF

has the line: 

  validationSettings allowReloads;validateFiles.tagAlign:mmCheckOnInN=100,mismatches=3,nMatch,matchFirst=25

This means that the tag aligns were validated with -mismatches=3 -nMatch -matchFirst=25

Testing in the Browser

  1. test one point from table to view in GB
  2. zoom into base level (at different visibilities)
  3. zoom way out 1million bps (at different visibilities)
  4. searching: should items be searchable
  5. default visibility: should this track be on by default?
  6. Check URL in Release Log (should go to track description page)
  7. Check for lab contact (sanitize email addresses using encodeEmail.pl script)

Performance Tests

  1. Does the first 'Signal' subtrack pass the chr1 test (chr1 loads in less than one minute)
  2. Do all views for one experiment pass chr1 test (e.g. Pol2 in GM12878 cells)?
  3. A user-oriented test would be to test the performance in a gene-size region of the track with just the default-on subtracks (for the Yale track, and many other ENCODE tracks, default-on subtracks will be all experiments in the GM12878 cell lines, Signal view only -- this should be the configuration you see after a cart reset, then turning the overall track vis to full).
  4. Note that ENCODE tracks can have any number of subtracks, and will continue to grow with time. We should definitely assure that useful subsets can be displayed in user-friendly time.

Subsequent Release of Data (e.g. Release 2)

Periodically, the existing ENCODE tracks will be augmented with new data as labs complete experiments on new cell lines, etc. The new data will come in various formats: some will replace existing data, some will be brand new, some old data will be eliminated, etc. The data wrangler will create a text document, check it into CVS, and place it here: kent/src/hg/makeDb/doc/encodeDccHg18/*.txt

This document should contain complete lists of each table and file and what its disposition is. The tables and files will fall into categories similar to this:

  • Untouched - are on public browser and shall remain
  • Eliminated - are currently on RR but will no longer be needed and should not be referenced by the public site
  • Replaced - are on public browser but will be replaced by something with another name (nothing should be replace by something with the same name)
  • Versioned (replacing) - are only currently on test but will need to be pushed to RR as replacements for existing datasets (with a different name)
  • New - are only currently on test but will need to be pushed to the RR.

Invariably, this document will not match reality. It may be the case that some of the tables/files do not exist at all, the names are incorrect, they are not present on the machine as listed in the file, they do not match the list that is in the pushQ. The first challenge in QAing a subsequent ENCODE release is to determine if/how the file diverges from reality. To do this, compare the file to the "snapshot" of what is included in the release (and what you should QA), which can be found in the "release2" list in the downloads directory (hgwdev: /data/apache/htdocs/goldenPath/<db>/<trackName>/release<x>/). If the file differs from the downloads directory, then send that information to the data wrangler and pop the track into the B-queue while they sort it out. Otherwise, QA spends far too much time figuring out exactly what they are expected to QA.

Once the list is finalized, proceed with the QA work as outlined above. Note the additional steps in the #Files section for how to handle the /releaseN directory.

Releasing to RR

  1. Check the trackDb diffs to see if there is any reason that <db>.trackDb can't go out today
    • If you determine that it's OK to push <db>.trackDb, then write to browser-qa "hands off <db>.trackDb"
  2. Check release log field in PushQ...needs to start with ENCODE
  3. Push track tables from mysqlbeta -> mysqlrr (not trackDb yet)
  4. Drop tables from hgwbeta that need to be removed (being replaced by V2 tables)
  5. Clean up trackDb (remove blocks for the dropped tables - currently marked release beta)
  6. Make strict for <db> on hgwbeta and check to make sure tracks look like you expect them to (all there and accounted for)
  7. Drop tables being removed from the RR
  8. Push trackDb, tableDescriptions and, if items are searchable (not usually for Encode), hgFindSpec
  9. Push download files and index.html (from hgwdev to hgdownload)
    • If this is a releaseN, even though there is a releaseN directory on hgwdev, do not create one on hgdownload.
    • see the Download Files section of #Pushing Files for specifics
  10. Drop download files that need to be dropped (from hgdownload)
  11. Drop .wib files that need to be dropped (from hgnfs1)
  12. Check the ENCODE/downloads.html page to see if your track is listed. If not (mostly for first releases), edit and push ENCODE/downloads.html from hgwdev -> hgwbeta & RR (a special Encode download release log)
  13. Push cv.ra file (only if there is a matrix)
  14. Cc the data wrangler for this track on all your pushes Cc encode@soe.ucsc.edu on your final push
Retrieved from "https://genomewiki.ucsc.edu/index.php?title=ENCODE_QA&oldid=9498"