ENCODE QA

From Genecats
Jump to navigationJump to search

Getting Started

Note: see the Old ENCODE QA page for the pre-bootcamp ENCODE QA process

Claim track (redmine)

  1. Look at the Redmine PushQ query in the Redmine ENCODE project.
    • In general, select the track with the highest priority. If two tracks have the same priority, take the one with the soonest due date. Also take into account how many other tracks that wrangler currently has in "Reviewing" status (under active QA).
  2. Change Assignee to yourself
  3. Change Redmine status from "Approved" to "Reviewing"
  4. Add yourself (and the wrangler) as a watcher to the redmine ticket (so if you assign it back to the developer you will get updates)

Note: QA no longer uses the '% Done' field in the Redmine Issue to estimate QA progress. This is now just for wrangler use.

Run encodeQaInit

This script makes preparations for QA (run the script without any arguments to see the usage):

The script:

  • runs qaEncodeTracks script
  • performs a verification of the notes file
  • sets the release status of all the subIds for the current release to "Reviewing" (no longer need to ask the wrangler to do this)

The script creates:

  • directory in /hive/groups/encode/encodeQa/hg19/ for that track and release (the subsequent items are in this directory)
    e.g. /hive/groups/encode/encodeQa/hg19/wgEncodeRikenCage/release2
  • allTables - list of all tables for this release
  • beta.mdb.ra
  • checkPushFilesList - lists files that should be pushed to hgdownload; use to verify all files were successfully pushed to hgdownload, see Verify all downloads pushed section
  • claimMail - email for Kate about claiming track
  • downloads (sym link) - links to downloads directory of the track
  • htmlDownloadSnippet - html for adding track to the appropriate ENCODE downloads page (human or mouse)
  • fullFilesListNoRevoked - list of all files downloadable through hgFileUi
  • methods - created from this ENCODE QA wiki page index to be used as a checklist and notes file for QA
  • newTables - list of new tables for this release
  • notes.file (sym link) - links to the notes file for a track describing what this release consists of (created by encodeMkChangeNotes)
  • pushFilesMail - email to send for pushing the download files (releasing)
  • pushGbdbsMail - email to send for pushing the gbdb files (staging)
  • pushTableMail - email to send for pushing the tables and trackDb & friends (releasing)
  • release.sql - sql commands for to create the releaseLog (after releasing the track)
  • script.output - output from qaEncodeTracks script
  • subIds - list of all the subIds involved in this release

Re-running encodeQaInit:

If the release changes in a way that the notes.file changes (tables/file/gbdbs are added, removed, etc), then the push*Mail, tableList, etc will no longer be accurate. In some cases, it might be best to re-run encodeQaInit so that these files don't have to fixed by hand.

Details:

  • methods will not be replaced if it exists already
  • release.sql will not be replaced, but it will be updated:
    • lastdate = last time encodeQaInit was run (updates every time encodeQaInit is run)
    • initdate = date encodeQaInit was first run (not ever overwritten)
  • all other files will be replaced
  • if you want to be able to refer back and see what tables and gbdbs you pushed, before running the script, rename newTables (e.g. tableListOld), or save the email files

Create a symbolic link

  1. From your home directory enter:
ln -s /hive/groups/encode/encodeQa/*database*/*wgEncodeTrackName*/*release* trackName

Example: ln -s /hive/groups/encode/encodeQa/hg19/wgEncodeUwDnase/release5 UwDnase

  1. Symbolic links allow you quickly jump from your home directory. For example by typing: cd UwDnase
  2. To delete a symbolic link, if you tab complete the name, be sure to delete the final "/" character. For example, "rm UwDnase" will work, while "rm UwDnase/" will not.

email Cricket (claimMail)

and cc Kate. From the command line encodeQa directory: 

mail -s "claiming trackName (Release #)" -c yourDevLogin -c kate cricket < claimMail

(good habit to read the email before sending)

Review the notes file

  1. Review the #Notes file to familiarize yourself with the components of the release. To find the notes file:
    • use the notes.file sym link in the encodeQa directory
    • or go to /kent/src/hg/makeDb/doc/encodeDcc/__<db>__/
  2. If this is a subsequent release, see #Subsequent Release of Data (e.g. Release 2) first.
  3. Compare the notes file to the hgTrackUi (to make sure it reflects the notes file) on dev (or on beta if dev already has the next release staged).
  4. If a Release N, compare the hgTrackUi on dev to the previous release's hgTrackUi on the RR to help verify notes file & new hgTrackUi is correct (e.g. make sure things aren't missing from the new release in comparison to the previous release that aren't accounted for in the notes file).

Pre-QA

Some tracks may have already gone through some preliminary QA, see Pre-QA for more information.

Check script.output

Output from qaEncodeTracks.csh which is run by encodeQaInit and does:

  • countPerChrom
  • check for entry in tableDescriptions table
  • check that shortLabel does not exceed 17 characters
  • check that longLabel does not exceed 80 characters
  • check that there are no underscores in the table names
  • check for indices on the tables
  • check that positional tables are sorted
  • checkTableCoords (checks for any illegal coordinates)

Also, run genePredCheck/pslCheck if applicable. (i.e. if your track is a gene prediction track)

Staging on hgwbeta

Push /gbdb files (pushGbdbsMail)

Push new and, if applicable, updated /gbdb files (e.g. .wib, .bb, etc.) from hgwdev -> hgnfs1.

  • Review the pushGbdbsMail, then from the command line in the encodeQa directory:
mail -s "push files to hgnfs1 for trackName (Release #)" -c yourDevLogin push-request < pushGbdbsMail

Open track on beta (if subsequent release)

Open the track on hgwbeta in hgTracks before staging it.
This way, when you check the track on beta (in the last staging step) you'll be able to tell if the update will cause a cart clash for users who happen to be using it when you release it to the RR (as evidenced by a completely blank screen).

Push tables to mysqlbeta (bigPush.csh)

Use bigPush.csh using the newTables file created by encodeQaInit. For example: bigPush.csh mm9 newTables

Run encodeQaPrepareRelease (trackDb release tags and metaDb)

  1. From hgwbeta run encodeQaPrepareRelease with "beta" for the stage (tip: run it from the encodeQa directory created by encodeQaInit, and the summary file of metadata changes will be saved there). Running encodeQaPrepareRelease:
    1. updates the trackDb release tags for the track's include statements appropriately:
      • In /cluster/home/$usr/trackDb/$species/$db/trackDb.wgEncode.ra, finds the include statement for your track's .ra file and changes 'alpha' tag to an 'alpha,beta' tag and, if applicable (releaseN), change 'beta,public' to 'public')
      • see the Three State TrackDb page for more info on release tags and our three-state trackDb)
    2. prepares the metadata:
      • from /cluster/home/$usr/trackDb/$species/$db/metaDb:
      • copies metDb .ra file from ~metaDb/alpha -> metaDb/beta
      • adds .ra file name to the makefile in ~metaDb/beta
    3. creates a trackName.beta.metaDb.diff file which is a summary of the metadata changes which you can review.
  2. Do a git status to see what files changed; review changes
  3. Check-in the changes
    1. Add commit message such as: git commit -m "Staging Caltech RNA-Seq (Release 1) on beta (redmine #7777)"
  4. On hgwbeta, make beta DBS=__ from /cluster/home/$usr/trackDb/

Check track on beta

Check that the track looks good on beta.
If this is a subsequent release, you already had the track open on beta from #Open track on beta (if subsequent release). Refresh the page to see the changes.
If you get a blank screen:

  1. Don't reset your cart (at least not until you've completed these steps!)
  2. Notify the track wrangler that there is likely a problem with conflicting cart variables when the new data is used with an old cart.
  3. Dump the cart variables (manipulate the URL to: http://hgwbeta.cse.ucsc.edu/cgi-bin/cartDump then hit enter) and save them in a file for people to look at.

hgTrackUi

Functionality (track controls)

Display Modes

  • If in a super-track, by default, composite "Maximum display mode" should be set to dense. Super-track should be set to hide.
  • If not in a super-track, by default composite "Maximum display mode" should be set to hide.
  • If multiple views, Kate wants these settings by default:
    • Peaks -> pack
    • Alignments -> hide
    • All else -> full
  • NOTE: Check with the wrangler in case there are custom visibility settings
  • Changing display mode of views should affect the subtrack list & hgTracks

Config Settings of Views

  • settings function correctly
  • settings of different views are independent
  • Signals, by default, should have the following settings (unless lab has requested otherwise or other good reason):
    • Data view scaling: use vertical viewing range (rather than auto-scale)
      • (Pre-QA skip) in dense, default fixed range should result in meaningful banding at full chromosome (not all gray)
    • Windowing function: mean + whiskers

Matrix

  • By default, matrix boxes should be fully checked or fully unchecked (not grayed), if not, this is trackDb setting issue that the wrangler should fix.
  • Matrix headers:
    • For human, Tier 1 and Tier 2 cell lines:
      • should be listed first (Tier 1 in alphabetical order followed by Tier 2 in alphabetical order)
      • should be labeled as Tier 1 or 2. The tier should follow the cell line name, in parentheses and bolded. No hyperlink, no italics, e.g. cellLineA (Tier 1)
    • matrix headers are links to a working hgEncodeVocab page for the item (cell line, factor, etc)
    • +/- buttons function correctly
    • selections in matrix result in appropriate selection changes in subtrack list

Subtrack list

  • adjusts according to matrix & view (hide -> non-hide) selections
  • 'only selected/visible' and 'all' radio selections function
  • sorting functions (clicking on column headings)
  • schema links work and has a "description" column
    • if the table is very large, there may not be an "info" column
    • if the table is merely a reference to a big* filename, there will not be a "description" column

MetaData

  • make sure metaData is present by clicking on the down arrow (v)
  • check a few to make sure they have somewhat consistent fields
  • spot check a few fields to make sure they make sense
  • make sure subtracks have dccAccession numbers (aka UCSC Accession), if none, may not be ready for QA; ask wrangler.
    • expId & dccAccession should correspond, dccAccession = wgEncodeE<H or M><expId> (the E=experiment, H=human, M=mouse)
    • these #numbers should be the same among subtracks of the same "experiment," even across assemblies of an organism (e.g. same number on hg18 & hg19)
      • NOTE: expID should only be displayed on hgwdev.
    • can check to make sure a composite (or single track) has all its expIDs and dccAccessions using experimentify option on mdbPrint:
mdbPrint <db> -composite=<compositeName> -experimentify
mdbPrint <db> -obj=<trackName> -experimentify

Links

  • check that all links work, and (PRe-QA skip) where applicable, are relative

Content (.html description page)

Labels

  • Check labels adhere to Kate's instructions

Sections

  • Make sure all sections are present, in order, and have the correct headings (the list below has the correct headings and is in the correct order)
  • Check grammar, spelling, readability, completeness, correctness
  • style should be consistent with the rest of the site
    • Description should be in a passive 3rd person voice
    • references to "data" are plural
    • value and units have space between them (e.g. 50 bp rather than 50bp)
  • links should be hyper linked text rather than just plain URLs
  • Latin Terminology
    • Latin or foreign words or phrases should not be italicized
    • Genus/species names should be italicized
Description
  • Brief overall summary of track.
Display Conventions and Configuration
  • Contains info about each view in track
  • No description for views only available in downloads
  • link to multi-view instructions if there are multiple viewing options.
  • Tracks with Bam alignments (in metadata, fileName will end with ".bam") should have a link to the Sam Format Specification and should explain any non-standard tags, those starting with X, Y or Z or that are not listed in the tag section
Methods
  • Make sure it is detailed enough.
Verification
  • optional
Release Notes
  • Optional for first release
  • Required for subsequent releases
  • Should start with "Release # (Month Year) of this track...."
  • Provides a description of the changes of this particular release.
Credits
  • Must have contact person
  • Name is a hyperlink to email
  • Email must be sanitized (using encodeEmail.pl script). To check go into your track's .html file and make sure the 'mailto' address is encoded.
References
  • Correct format, see CBSE citation format
  • Alphabetical order
  • Make sure URL to references don't contain the rel='nofollow' attribute.
Publications

This is an optional listing publications that reference or use ENCODE data from this track. Information for this section is provided to us by NHGRI.

Data Release Policy
  • Standard language, Supertrack:
Data users may freely use ENCODE data, but may not, without prior consent, submit publications that use an unpublished ENCODE dataset until nine months following the release of the dataset. This date is listed in the Restricted Until column on the track configuration page and the download page. The full data release policy for ENCODE is available here.
  • Standard language, Track:
Data users may freely use ENCODE data, but may not, without prior consent, submit publications that use an unpublished ENCODE dataset until nine months following the release of the dataset. This date is listed in the Restricted Until column, above. The full data release policy for ENCODE is available here.

Links

  • Check standard links are present, and, where applicable, are relative:
    • ENCODE Data policy (in Data Release Policy section)
    • help for multi-view (Next to "Select views" in track Control Section in Display Conventions and Configuration section)
    • contact email (see #Credits for more info)
  • If there is supplemental data, make sure there is a link and that it points to hgdownload and not hgdownload-test

hgc details

Check the following for each view:

Accuracy of details

  • details that are displayed correspond with the record in the table

Makes sense

  • table values seem correct

Useful

  • you understand what is being displayed
  • internal, non-functioning fields are not displayed (e.g. if all values in a field have "-1" as a placeholder, we shouldn't display that field)

Complete

  • all useful information from is present (there's nothing important that is missing)

Clear

  • details are presented and labeled clearly
  • layout is user friendly

Links

  • check that all links are relevant, work, and where applicable, are relative
  • standard links: downloads, metadata, schema

hgTracks

Display

Views (zoomed in/out)

  • check the display of all Views in all display modes when zoomed in to the base pair level & zoomed out to 1 million bp

Table coordinates + features

  • an items' cooridnates and other display features (exons, etc) display as expected/correctly based on table
    • a line from the table for comparing against the display can be obtained from schema or mysql db for regular tables
    • for bam files, the schema will only give you a filePath. You will need to use SamTools to obtain a point to test.
      1. add /hive/data/outside/samtools/svn_${MACHTYPE}/samtools to $PATH in your .bashrc
      2. run samtools on the command line using the fileName found in the schema (see following example). The output will give you the start position and then it gives you read length (in the CIGAR string); if the CIGAR string is simple, e.g. 76M, add the read length, 76, to the start position to get the end position. If the CIGAR string is complicated, e.g. 43S17M494510N16M, just use the actual sequence to determine the length (paste sequence into Word & get word count) and add this to the start position to get the end position. This will give the point needed to put into the browser for testing purposes.
 samtools view -x filePath chrx:xxxx-xxxxxx | head
 samtools view -x /gbdb/hg19/bbi/wgEncodeCshlLongRnaSeqHuvecCellPapAlnRep1.bam chr1:2000000-3000000 | head
  • for big* files, you can't get individual record, but use bigWigInfo or bigBedInfo to get general stats, be sure bigWigs are version 4.

Searchable

  • Are items searchable; should they be? Most likely not for ENCODE. (position/search box at the top of the browser image)
  • Do a quick search of a subtrack in track search (button found at the bottom of the browser image) to make sure that it is interacting correctly.

Colors

Human
  • Human Tier 1 cell lines should be displayed in a unique color (other than black)
  • Human 'original' Tier 2 cell lines (HeLa-S3, HepG2, and HUVEC) should be displayed in a unique color (other than black), but newly promoted Tier 2 cell lines should be displayed in black until a unique color for each is determined by the consortium.
  • it is OK if other tracks are in color, but not necessary
Mouse
  • For mouse, tissues/cell lines are grouped based on organ systems, and each has a unique color:
    1. Skeletal: Dark Purple 102,50,200
    2. Muscular: Brown (139,69,19)
    3. Circulatory: Red (153,38,0)
    4. Nervous: Grey (105,105,105)
    5. Respiratory; Black (empty)
    6. Digestive: Orange (230,159,0)
    7. Excretory: Purple (204,121,167)
    8. Endocrine: Pink (189,0,157)
    9. Reproductive: Green (0,158,115)
    10. Lymphatic/immune System: Blue (86,180,233)
    11. Stem Cell: Dark Blue (65,105,225)
  • Visualize the above colors in the browser Mouse color Session
  • If a cell line is not colored that should be, the appropriate color for the cell lines is listed as a 'color' value in the cv.ra.

Defaults (composite/subtracks)

  • should this composite track be on by default? (For ENCODE, usually no)
  • check the which subtracks are set as default selection, make sure:
    • there aren't too many
    • important cell lines are on by default
  • default Tier 1 and Tier 2 subtracks should display first

Compare to hg18

  • If track is the first release in hg19, compare a point on the hg19 browser of the track to the equivalent position in hg18.
    1. use "convert" from hg19 position to see the equivalent position in hg18.
    2. go back to your region in hg19, open new window and paste in hg18 equivalent position and compare hg19 to hg18.
  • Note: Comparisons to hg18 should be very cursory. Any differences should be noted in the redmine ticket, but not necessarily investigated unless a user also brings up an issue. The thinking behind this is that when there are differences, it is most likely an error with hg18, not hg19 and we are unnecessarily holding up hg19)

Performance

Chrom 1 Test (signal & experiment)

When position is set to all of chromosome 1, data of interest (in full mode) loads in less than 1 minute:

  • signal: check time of loading first signal subtrack
  • experiment: check time of loading all views for one experiment (e.g. Pol2 in GM12878 cells)?

Defaults at Gene Sized Region Test

Set position to a gene-size region with your track's default subtracks on and the default browser tracks on (easiest to reset cart, turn on track)

  • should display quickly and not be "too much" data

Data make sense

Compare subtracks within views

  • Do all the subtracks within a view somewhat correlate?

Compare subtracks of related views

  • For example:
    • Does the All Signal Raw Signal subtrack of an experiment really seem to comprise of the data in the Plus Raw Signal & Minus Raw Signal?
    • Do Peaks really represent the high Signal areas of the Signal View subtracks?

Do the data make sense Biologically?

  • Turn on other tracks to compare.
    • compare to the gene tracks
    • compare to subtracks of similar tracks
    • For example:
      • RNA-seq data should correlate with the exons in a genes track
      • TFBS tracks should correlate with the beginning of gene transcripts

Files

hgFileUi

  • 'Downloads' links on hgTrackUi should now go to hgFileUi
    • if not, ask wrangler to add "fileSortOrder" information to trackDb entry

file count (notes.file)

  • Check # of files displayed is correct (use notes.file). Pre-QA skip.
  • Note: If you are not seeing the correct number of files on beta, first try clearing the cache
    • Add the following to the end of the URL in your address bar: &clearCache=true

download button

  • Make sure download button prompts a download (and doesn't take you to an error page)

useful columns w/ good titles

  • Columns are useful
  • Column titles are correct and make sense (e.g. dccAccession title is "UCSC Accession")

sort columns

  • Check the sorting of columns. Clicking on the title of the column should sort the table on that column.

file filter

  • Check the filtering of files

links

  • Check that the "Track Settings" link takes you back to the track's hgTrackUi page
  • Check that the navigation, file filter title links, and other links work
  • Make sure files.txt & md5Sum.txt links are present and function
  • Make sure the download server link goes to the download directory for that track see #download server for more info.

download server

  • linked from hgFileUi with the *download server* link
  • have wrangler remove index.html or preamble.html files from the current release directories if they exist (it is OK in older directories, e.g. release1 if this is a release3).

README

  • README should be displayed automatically followed by the list of files in the directory
  • contains a URL to the track's hgFileUi (you can double check by copying link, pasting in a browser, and changing hgdownload to hgdownload-test).
  • there may be more files/directories in here than seen in hgFileUi. This is OK. Because we are not dropping obsolete files, they will be present in this directory. Also, on hgdownload-test there will also be releaseN directories. These are part of the process of preparing a track and are OK. These, however, *won't* be pushed to the hgdownload upon release of the track.

Release to RR

Note: Cc the data wrangler for this track on all your pushes and Cc encode-staff@soe.ucsc.edu on your final push.

Push downloads (pushFilesMail)

  • Push download files, index.html, files.txt and md5sum.txt (from hgwdev to hgdownload):
mail -s "push download files for trackName (Release #)" -c wranglerDevLogin push-request < pushFilesMail
  • If track has supplemental data (files linked from the description pages in a /supplemental directory), make sure the pushFilesMail includes them (if they're in the notes.file, they should be in the pushFilesMail).
  • Notice that even though there is a releaseN directory on hgwdev, it is not pushed to a releaseN directory on hgdownload (see #Download files for specifics)
  • Note, this push can take hours, so you may want to start the day before you want to release.

Verify all downloads pushed (encodeQaCheckHgdownloadFiles)

Run encodeQaCheckHgdownloadFiles to make sure all files got pushed to hgdownload

  • use 'checkPushFilesList' as the list of files (created by encodeQaInit and in the encodeQa directory for the track/release)

Run encodeQaPrepareRelease (trackDb release tags & metaDb)

  1. Check metaDb/alpha vs. beta to make sure you have most updated metadata (diff beta/<trackName.ra> alpha/<trackName>.ra) or try the qaRaDiff script (soon to be renamed raDiff).
    • if diffs are due to next release, don't copy to beta, if diffs are for current release, copy to beta & double check in hgTrackUi, etc.
  2. Run encodeQaPrepareRelease with "public" as the stage, which:
    1. updates the release tags in trackDb.wgEncode.ra (of the $db directory): removes any release tags from the current <trackName>.releaseN.ra include line (or can be <trackName>.ra if 1st release)
    2. updates the metaDb, from /cluster/home/$usr/trackDb/$species/$db/metaDb:
      1. copies metaDb <trackName>.ra file from ~metaDb/beta -> metaDb/public
      2. adds <trackName>.ra file name to the make file in ~metaDb/public (if a first release)
  3. Review changes
  4. Check-in changes
  5. Announce intent to make public on db
  6. On hgwbeta, make public DBS=$db from /cluster/home/$usr/trackDb/

Check track on beta-public

Check track on hgwbeta-public

  • hgwbeta-public uses mysqlbeta for the tables, but uses the CGIs that are on the RR.

Push tables + trackDb&friends (pushTableMail)

  • Push the tables from mysqlbeta -> mysqlrr and trackDb & friends:
mail -s "push tables and trackDb & friends for trackName (Release #)" -c encode-staff push-request < pushTableMail
  • no longer have to push tableDescriptions because it gets pushed out once a week by a designated QA'er

Check track on RR

Once all pushes complete, check track on RR

Set subIds to Released

Set the ENCODE status of the subIds (listed in the 'subIds' file of the enocode directory created by encodeQaInit) for this release to "Released":

  • from the bash shell command line on hgwdev, enter this 'for loop':
for i in $(cat filenamewithpath); do encodeStatus.pl $i released; done

or, if you need to change just a few subIds, you can enter the individual subIds with spaces in between them instead of doing the cat fileName: 

for i in subIdsWithSpacesSeparating; do encodeStatus.pl $i released; done

For example: 

for i in 2007 3113; do encodeStatus.pl $i released; done

Close Redmine ticket

  • If there aren't any lingering issues, close the ticket by setting the status to "Released".

Run encodeQaSqlRelease

On hgwbeta, encodeQaSqlRelease creates a pushQ entry directly in the L queue of the Main pushQ so the ENCODE track will have an entry in the release log.

  • The Release Log field of the entry created:
    • should be the long label (or short if too long) and, if releaseN, release number in parentheses
    • should contain ENCODE (or it it won't show up on ENCODE downloads page)
  • The Release Log URL:
    • should be a relative link with the db specified (e.g. ../../cgi-bin/hgTrackUi?db=hg19&g=wgEncodeRikenCage)

ENCODE downloads (htmlDownloadSnippet)

Check ENCODE downloads page (human | mouse), if your track isn't there:

  1. add it by copying the text in the htmlDownloadSnippet file and adding it to /kent/src/hg/htdocs/ENCODE/downloads.html under the appropriate group in alphabetical order.
  2. if necessary, also add its super-track:
    • super-track title should be a non-underlined link to the super-track hgTrackUi, for example:
      <A style="text-decoration:none" HREF="http://genome.ucsc.edu/cgi-bin/hgTrackUi?db=hg19&g=wgEncodeRnaSeqSuper" TARGET=_blank>RNA-seq</A>
  3. push the following from hgwbeta -> RR:
/usr/local/apache/htdocs-hgdownload/ENCODE/downloads.html

Other info

Subsequent Release of Data (e.g. Release 2)

Periodically, released ENCODE tracks will be augmented with new data as labs complete experiments on new cell lines, etc. The new data will come in various formats: some will replace existing data, some will be brand new, some old data will be eliminated, etc.

Notes file

The data wrangler will create a notes file using the encodeMkChangeNotes script, check it into git, and place it here: kent/src/hg/makeDb/doc/encodeDcc$db/*.txt

This document should contain complete lists of each table and file and what its disposition is. The tables and files will fall into categories similar to this:

  • A) Untouched - are on public browser and should remain
  • B) Deprecated - are currently on RR but will no longer be needed and should not be referenced by the public site.
NOTE: NO FILES SHOULD BE REMOVED from the downloads directory on hgdownloads (RR).
This list is provided for completeness.
  • C) New - are only currently on test but will need to be pushed to the RR.
  • D) Additional items of note

This document may not match reality. It may be the case that some of the tables/files do not exist at all, the names are incorrect, they are not present on the machine as listed in the file, they do not match the list that is in the pushQ. The first challenge in QAing a subsequent ENCODE release is to determine if/how the notes file diverges from reality. To do this, compare the file to the "snapshot" of what is included in the release (and what you should QA), which can be found in the "release2" list in the downloads directory (hgwdev: /data/apache/htdocs/goldenPath/<db>/<trackName>/release<x>/). If the file differs from the downloads directory, then send that information to the data wrangler and pop the track into the B-queue while they sort it out. Otherwise, QA spends far too much time figuring out exactly what they are expected to QA.

Once the list is finalized, proceed with the QA work as outlined above. Note the additional steps in the #Files section for how to handle the /releaseN directory.

MetaDb changes

There also may be some metadata changes to fix errors or add information such as the GEO Series and GEO Sample.

  • Be sure to QA these metadata changes
  • Check for related redmine issues (addition of GEO accessions should definitely have a related issue for the addition of this info)
  • You can also do a diff to check for any other metadata changes.From kent/src/hg/makeDb/trackDb/<org>/<db>/metaDb, do the following diff:

diff beta/wgEncodeYourTrack alpha/wgEncodeYourTrack

Pushing Files

Pushing the three main types of files involved in ENCODE tracks.

gbdb files

Files of this form get pushed hgwdev -> hgnfs1 

/gbdb/hg18/wib/wgEncode*.wib

Download files

Download files for an original release get pushed hgdownload-test on hgwdev -> hgdownload (list the entire file path as usual) 

/usr/local/apache/htdocs-hgdownload/goldenPath/hg18/encodeDCC/wgEncode*/index.html
/usr/local/apache/htdocs-hgdownload/goldenPath/hg18/encodeDCC/wgEncode*/wgEncode*.[bed/wig].gz

When pushing download files for a subsequent release track (e.g. release 2), push files as follows (but in your request, list the from/to paths at the top followed by a list of the file names without the full path) 

from hgwdev: /usr/local/apache/htdocs-hgdownload/goldenPath/<db>/encodeDCC/<trackName>/releaseN/
to hgdownload: /usr/local/apache/htdocs/goldenPath/<db>/encodeDCC/<trackName>/
(Note: no releaseN directory on hgdownload)
  • Once the files have been pushed you can check to see if the push was successful using this script: checkPushedFiles.csh

Other files

Files of this form get pushed hgwbeta -> RR. Because they used to be omitted from the pushQ entry often, the directories containing these files are now pushed weekly byKatrina on Fridays. So QAers no longer have to worry about pushing these. They are not in source control so go out way ahead of the track usually. 

/usr/local/apache/htdocs/ENCODE/protocols/cell/*.pdf

Relative Links

In html on our site, you can create relative links (on dev, the link goes to the page on dev, on beta, it goes to beta, etc.) by using part of the path based on the your file's location in the source tree relative to the location of the file or cgi you are linking to.
For example from ~trackDb/human/hg19, here is how you point to:

  • a golden path file:
<A HREF="../../goldenPath/help/multiView.html" TARGET=_BLANK>here</A>.
  • cgi:
<A TARGET=_BLANK HREF="/cgi-bin/hgEncodeVocab?type=cellType">cell lines</A>
  • ENCODE protocols:
<A HREF="../../ENCODE/protocols/cell">ENCODE cell culture protocols </A>.
  • ENCODE portal:
<A TARGET=_BLANK HREF="/ENCODE/index.html">Encyclopedia of DNA Elements (ENCODE) Project</A>
  • ENCODE data release policy:
<A TARGET=_BLANK HREF="../ENCODE/terms.html" TARGET=_BLANK>here</A>

Old info

File Validation

  • No longer run, here are Tim's comments about QA running validateFiles: "To get these things through the pipeline, we run them through validateFiles, so I think your running them through again is one time too many. But if you are going to, then each lab and each file type may have negotiated limits (which may change between submissions). These limits are found in the relevant submission directory DAF files."
  • Old validateFiles process:

Test a smattering of different file types using this tool: validateFiles (type the program name without arguments to see the usage statement). If there are no errors, there will be no output. For example, for files of type tagAlign, invoke the tool like this:

validateFiles -type=tagAlign -genome=/gbdb/hg18/hg18.2bit /usr/local/apache/htdocs/goldenPath/hg18/encodeDCC/wgEncodeHudsonalphaChipSeq/wgEncodeHudsonalphaChipSeqAlignmentsRep1Gm12878Control.tagAlign.gz

For tagAligns there are several relevant validateFiles options: 

 mismatches  - frequently 2 but negotiated for each lab.  Set this to 5 to be tolerant
 matchFirst - negotiated.  You should set this to 25 and even then you may need to adjust it
 nMatch - negotiated, but you should always have this parameter set.

If you want to be exact, then the metadata as seen on the downloads page tells which submission directory the file belongs to, and the most recent *.DAF (or *.daf) will have a validationSettings line in it which will include the settings that belong to each file type. Example: 

 /hive/groups/encode/dcc/pipeline/encpipeline_prod/773/UtaChIPseqBOonlyV1.DAF

has the line: 

 validationSettings allowReloads;validateFiles.tagAlign:mmCheckOnInN=100,mismatches=3,nMatch,matchFirst=25

This means that the tag aligns were validated with -mismatches=3 -nMatch -matchFirst=25

Retrieved from "https://genomewiki.ucsc.edu/genecats/index.php?title=ENCODE_QA&oldid=1115"