Chains and Nets QA: Difference between revisions

From genomewiki
Jump to navigationJump to search
No edit summary
(Same information, just restructured it to make sense. Also pulled some information from referenced webpages.)
Line 1: Line 1:
[[Category:Browser QA]]
[[Category:Browser QA]]


Nets/Chains:
There are two components of chains/nets [http://genomewiki.ucsc.edu/index.php/Chains_Nets]: the actual chains/nets themselves and the liftOver directories/files. The liftOver [http://genome.ucsc.edu/goldenPath/help/hgTracksHelp.html#Convert] files make it possible to convert coordinates from one assembly to another. The liftOver files also have several associated files that will need to be pushed to hgdownloads at the end of the QA process. Chains and Nets are typically QAed in reciprocal pairs as similar tests are being done for both database-chain/net sets; however, this wiki only documents QAing one track.
The primary thing is to run this script:
chainNetTrio.csh  [database] [other database]


ex: chainNetTrio.csh tetNig2 GasAcu1
==Prior to QAing on hgwbeta: Pushing from hgwdev==
1) Use sudo mypush to push chain/net tables from hgwdev to hgwbeta. This will include 3 tables: a chain, net and chain link table. Syntax for this command is: sudo mypush [from database] [tableName] mysqlbeta. See the example below:


I recommend making a separate directory as the script generates a bunch
*hgwdev: sudo mypush danRer6 chainMm9 mysqlbeta
of files. Ann said she changed the script to tell you which files to
*hgwdev: sudo mypush danRer6 netMm9 mysqlbeta
look at, but in case it doesn't for whatever reason, you want to look at
*hgwdev: sudo mypush danRer6 chainMm9Link mysqlbeta
*.chain.* , *.chain2.* , *.net.*


Just read through making sure all the checks went okay and doing the few
2) Make strict on beta so that trackDb is updated to include the tables just pushed. Syntax for this command is: make strict DBS=[from database].
manual checks


If you're QAing, for example, chainNet from tetNig2 to hg19, you will also
*ex: hgwbeta: make strict DBS=danRer6
see, in the file section of the pushQ entry, some files listed:


/usr/local/apache/htdocs/goldenPath/tetNig2/vsHg19/*.txt
3) Email a push request to copy /gbdb/[from database]/liftOver/[from database]To[to database].over.chain.gz from hgwdev to hgnfs1. Note that the liftover file will now be available on both beta and RR as it is on hgnfs1. However, because the hgcentral database on the RR doesn't have an entry pointing to this file, it will only be visible on beta.
/usr/local/apache/htdocs/goldenPath/tetNig2/vsHg19/*.gz
/usr/local/apache/htdocs/goldenPath/tetNig2/vsHg19/axtNet/*
/usr/local/apache/htdocs/goldenPath/tetNig2/liftOver/tetNig2ToHg19.over.chain.gz
/gbdb/tetNig2/liftOver/tetNig2ToHg19.over.chain.gz


*ex: email: Please push /gbdb/danRer6/liftOver/danRer6ToMm9.over.chain.gz from hgwdev to hgnfs1.


You can read about liftOver here (if you don't already know about it).
4) Add a line from hgcentraltest database to hgcentral beta so that file pushed in step 3 is visible on beta (this is similar to the same way that tables are not visible until trackDb is updated). To do this you first find the line on hgcentraltest.liftOverChain, copy it and then insert it into hgcentralbeta.liftOverChain. Note that it is very easy to make a mistake that can bring down hgcentral so make sure that your syntax and mysql statements are correct. See the example below:
[http://genome.ucsc.edu/goldenPath/help/hgTracksHelp.html#Convert]


And you can test your liftOver chains in two CGIs: hgLiftOver and hgConvert.
*hgwdev: hgsql hgcentraltest
[http://hgwdev.cse.ucsc.edu/cgi-bin/hgLiftOver]
*mysql> select * from liftOverChain where fromDb=“danRer6” and toDb=“mm9”;
[http://hgwdev.cse.ucsc.edu/cgi-bin/hgConvert]
*(copy the output into a text editor for later use)
*(open another window on hgwbeta) hgwbeta: hgsql hgcentralbeta
*(put the values copied from hgcentral test into this statement. please note that the formatting must be followed exactly)
mysql> insert into liftOverChain values ("danRer6","mm9","/gbdb/danRer6/liftOver/danRer6ToMm9.over.chain.gz","0.1","0","0","Y","1","N");
*(checking your work) mysql> select * from liftOverChain where fromDb=“danRer6” and toDb=“mm9”;


To get to the hgConvert CGI, start from your assembly (that your chains/nets
==QAing on hgwbeta==
are on -- tetNig2 in this example) and press the "Convert" link in the top blue
To QA a chain/net file first run this script: chainNetTrio.csh  [database] [other database].  
navigation bar.


To get the to hgLiftOver CGI, start from the home page and cick on
*ex: chainNetTrio.csh danRer6 mm9
"utilities" link, then click on "Batch Coordinate Conversion (liftOver)".


Not that you will need to stage these on beta, but just FYI, files that start
This script runs several other scripts and outputs the results into several files. It is recommended that you create a separate directory to run this script in so you don't end up with the random output files in another directory. Read through the results in these three output files: *.chain.* , *.chain2.* , *.net.*. Mainly check that all the scripts tested ok and do a few manual checks that requested in these files.  
with /usr/local/apache/htdocs/goldenPath get pushed from hgwdev to hgwbeta and  
the RR machines.  Files that start with /gbdb/ get pushed from hgwdev to hgnfs1.  


To push the files gbdb to hgwbeta:
There are two CGIs that use the liftOver data: hgConvert [http://hgwdev.cse.ucsc.edu/cgi-bin/hgConvert] and hgLiftOver [http://hgwdev.cse.ucsc.edu/cgi-bin/hgLiftOver]. To test these CGIs first select a top level short net that has very few other chains that match to the same area. To test hgConvert, first go to the genome browser of your "fromDb" database and center the window on the pre-selected net. Click on "Convert" link in the top blue bar. Configure the pull down menu to point to the "toDb" being tested and click "submit". The results should be the same as chain/net track details for that chain.
cd  /gbdb
cd “assembly”
cd liftover


Check on hgwbeta to see if that directory is there.
To test hgLiftOver first click on the "Utilities" link in the side blue bar on the home page and then click the first link to "Batch Conversion". Configure the pull down menus to match the toDB and fromDb being tested, enter the pre-selected test coordinates and click "submit". If the search fails, lower the minMatch (as low as 0.01) until some result for some region is obtained successfully. Check to see if the results match those from hgConvert and the chain/net track. Results in hgConvert may not match hgLiftOver but will still match the net and chains exactly. If results in hgLiftOver are NOT in hgConvert the developer should be contacted.


If it isn’t you need to send a push-request with the following info:
==After QAing on hgwbeta: Pushing to the RR==
Please push file path/filename
from hgwdev to hgnfs1


Reason: staging liftOver file on hgwbeta.
1) Email a push request to copy the three chain/net (chain, net and chain link) tables from mysqlbeta to mysqlrr.  
Additionally, the hgcentral database contains a line that needs to be edited
in order to enable the liftOvers to work on beta and on the RR. To do this:


Hgdev:
*ex: email: Please push the following tables from the danRer6 database: chainMm9, netMm9, chainMm9Link from mysqlbeta to mysqlrr.
    1) hgsql hgcentraltest;
    2) show tables;
    3) desc liftOverChain;
    4) select * from liftOverChain where fromDb like “assembly” and toDb like “assembly%”;
    5) check on hgcentralbeta using the same sql statements above.
    6) Find the line on dev and copy without the pipes at the front and end.
    7) Paste in a text editor.
      a. In the text editor surround the newly pasted text in ( ). 
      b. Add the statement: insert into liftOverChain values);
      c. Revmove spaces and pipes. Place all text into quotes.
      d. Statement should look like this: insert into liftOverChain values (“ “,” “);
    8) In hgwbeta:
      a. Paste this statement in the terminal (must in database) place your hands under your legs and make sure it is correct,   press enter
    9) Do this statement again to make sure that it has been inserted: select * from liftOverChain where fromDb like “assembly” and toDb like “assembly%”;
  10) Look at covert and liftover dropdowns on beta to confirm push.  
  11) Repeat when ready to release to RR (hgcentral)


2) Make strict on beta so that trackDb is completely up-to-date with anyone else's changes.


Here is some old documentation: [http://hgwdev.cse.ucsc.edu/qa/test-protocols/tracks/liftover.html]
*ex: hgwbeta: make strict DBS=danRer6


Testing Liftovers:
3) Email a push request to copy the updated trackDb tables from mysqlbeta to mysqlrr.  
  1) Pick a top level short net and zoom in.
  2) Paste in browser search position in hgliftover
  3) Click submit
  4) Compare output back to the net in the browser. It should be the same.


Testing Convert:
*ex: email: Please push the following table from the danRer6 database: trackDb from mysqlbeta to mysqlrr.
  1) Convert with the same net chosen
  2) Check to make sure that it matches the net.  


Moving files to RR:
Note: for steps 1-3 you may have to do some sneaky/selective pushing that is different from the steps above if there is already a chain/net on the RR for an older assembly.
  1) Push tables from hgwbeta to RR with push request.
  2) Push trackDb from mysqlbeta to mysqlrr.
  3) Test


Add line to hgcentral see above.
4) Test the chain/Net track on the RR to make sure it is working properly.


Push file sections of pushQ (liftOver file and vs directory)
5) Add line to hgcentral.liftOverChain from hgcentralbeta.liftOverChain. Use the same steps listed in step 4 under "Prior to QAing on hgwbeta". Note that you do not need to push the liftOver file as the RR and hgwbeta both have access to hgnfs1.


6) Test that hgLiftOver and hgConvert to make sure they are working as expected.


Moving files to hgdownload:
7) Email a push request to copy this file: /usr/local/apache/htdocs/goldenPath/[from database]/liftOver/[from database]To[to database].over.chain.gz and this directory: /usr/local/apache/htdocs/goldenPath/[from database]/vs[to database]/* from hgwdev to hgdownloads so that the public can download these files.
  1) Compare http://hgwdev.cse.ucsc.edu/downloads.html with http://hgdownload.cse.ucsc.edu/downloads.html
 
  2) Check for liftOver files and if yours is missing on hgdownloads you will need to ask for a push of the corresponding .over.chain.gz to be moved to the download server.
*ex: email: Please push /usr/local/apache/htdocs/goldenPath/danRer6/liftOver/danRer6ToMm9.over.chain.gz and /usr/local/apache/htdocs/goldenPath/danRer6/vsMm9/* from hgwdev to hgdownloads.
  3) You will then need to add the link to to the downloads page:
 
      a. To get to downloads on hgwdev:
8) To make a link to these new download files change the downloads.html file.
        From your home directory
 
        1. cd browser
*hgwdev (in the browser tree):vi downloads.html
        2. cvsup
* (find a link under "Pair-wise alignments" to a similar database. copy it, paste it and change it so that it points to the new vs directory)
        3. vi downloads.htm
* cvs commit downloads.htm
        4. Go to the line that has the previous version of the chains and hit yy p (this copies and pastes the text below)  
 
        5. Correct the links and names and save.
9) Make alpha in browser tree so that the changes in downloads.html are now in htdocs.
        6. cvs commit
 
        7. make alpha in browser tree.
*ex: hgwdev (inside browser tree): make alpha
        8. and push-request this file to be move to RR
 
10) Email a push request to copy this file: /usr/local/apache/htdocs/downloads.html from hgwdev to hgdownloads, thus allowing the public to access the download files.
 
*ex: email: Please push usr/local/apache/htdocs/downloads.html from hgwdev to hgdownloads
 
The end!

Revision as of 01:23, 5 March 2010


There are two components of chains/nets [1]: the actual chains/nets themselves and the liftOver directories/files. The liftOver [2] files make it possible to convert coordinates from one assembly to another. The liftOver files also have several associated files that will need to be pushed to hgdownloads at the end of the QA process. Chains and Nets are typically QAed in reciprocal pairs as similar tests are being done for both database-chain/net sets; however, this wiki only documents QAing one track.

Prior to QAing on hgwbeta: Pushing from hgwdev

1) Use sudo mypush to push chain/net tables from hgwdev to hgwbeta. This will include 3 tables: a chain, net and chain link table. Syntax for this command is: sudo mypush [from database] [tableName] mysqlbeta. See the example below:

  • hgwdev: sudo mypush danRer6 chainMm9 mysqlbeta
  • hgwdev: sudo mypush danRer6 netMm9 mysqlbeta
  • hgwdev: sudo mypush danRer6 chainMm9Link mysqlbeta

2) Make strict on beta so that trackDb is updated to include the tables just pushed. Syntax for this command is: make strict DBS=[from database].

  • ex: hgwbeta: make strict DBS=danRer6

3) Email a push request to copy /gbdb/[from database]/liftOver/[from database]To[to database].over.chain.gz from hgwdev to hgnfs1. Note that the liftover file will now be available on both beta and RR as it is on hgnfs1. However, because the hgcentral database on the RR doesn't have an entry pointing to this file, it will only be visible on beta.

  • ex: email: Please push /gbdb/danRer6/liftOver/danRer6ToMm9.over.chain.gz from hgwdev to hgnfs1.

4) Add a line from hgcentraltest database to hgcentral beta so that file pushed in step 3 is visible on beta (this is similar to the same way that tables are not visible until trackDb is updated). To do this you first find the line on hgcentraltest.liftOverChain, copy it and then insert it into hgcentralbeta.liftOverChain. Note that it is very easy to make a mistake that can bring down hgcentral so make sure that your syntax and mysql statements are correct. See the example below:

  • hgwdev: hgsql hgcentraltest
  • mysql> select * from liftOverChain where fromDb=“danRer6” and toDb=“mm9”;
  • (copy the output into a text editor for later use)
  • (open another window on hgwbeta) hgwbeta: hgsql hgcentralbeta
  • (put the values copied from hgcentral test into this statement. please note that the formatting must be followed exactly)

mysql> insert into liftOverChain values ("danRer6","mm9","/gbdb/danRer6/liftOver/danRer6ToMm9.over.chain.gz","0.1","0","0","Y","1","N");

  • (checking your work) mysql> select * from liftOverChain where fromDb=“danRer6” and toDb=“mm9”;

QAing on hgwbeta

To QA a chain/net file first run this script: chainNetTrio.csh [database] [other database].

  • ex: chainNetTrio.csh danRer6 mm9

This script runs several other scripts and outputs the results into several files. It is recommended that you create a separate directory to run this script in so you don't end up with the random output files in another directory. Read through the results in these three output files: *.chain.* , *.chain2.* , *.net.*. Mainly check that all the scripts tested ok and do a few manual checks that requested in these files.

There are two CGIs that use the liftOver data: hgConvert [3] and hgLiftOver [4]. To test these CGIs first select a top level short net that has very few other chains that match to the same area. To test hgConvert, first go to the genome browser of your "fromDb" database and center the window on the pre-selected net. Click on "Convert" link in the top blue bar. Configure the pull down menu to point to the "toDb" being tested and click "submit". The results should be the same as chain/net track details for that chain.

To test hgLiftOver first click on the "Utilities" link in the side blue bar on the home page and then click the first link to "Batch Conversion". Configure the pull down menus to match the toDB and fromDb being tested, enter the pre-selected test coordinates and click "submit". If the search fails, lower the minMatch (as low as 0.01) until some result for some region is obtained successfully. Check to see if the results match those from hgConvert and the chain/net track. Results in hgConvert may not match hgLiftOver but will still match the net and chains exactly. If results in hgLiftOver are NOT in hgConvert the developer should be contacted.

After QAing on hgwbeta: Pushing to the RR

1) Email a push request to copy the three chain/net (chain, net and chain link) tables from mysqlbeta to mysqlrr.

  • ex: email: Please push the following tables from the danRer6 database: chainMm9, netMm9, chainMm9Link from mysqlbeta to mysqlrr.

2) Make strict on beta so that trackDb is completely up-to-date with anyone else's changes.

  • ex: hgwbeta: make strict DBS=danRer6

3) Email a push request to copy the updated trackDb tables from mysqlbeta to mysqlrr.

  • ex: email: Please push the following table from the danRer6 database: trackDb from mysqlbeta to mysqlrr.

Note: for steps 1-3 you may have to do some sneaky/selective pushing that is different from the steps above if there is already a chain/net on the RR for an older assembly.

4) Test the chain/Net track on the RR to make sure it is working properly.

5) Add line to hgcentral.liftOverChain from hgcentralbeta.liftOverChain. Use the same steps listed in step 4 under "Prior to QAing on hgwbeta". Note that you do not need to push the liftOver file as the RR and hgwbeta both have access to hgnfs1.

6) Test that hgLiftOver and hgConvert to make sure they are working as expected.

7) Email a push request to copy this file: /usr/local/apache/htdocs/goldenPath/[from database]/liftOver/[from database]To[to database].over.chain.gz and this directory: /usr/local/apache/htdocs/goldenPath/[from database]/vs[to database]/* from hgwdev to hgdownloads so that the public can download these files.

  • ex: email: Please push /usr/local/apache/htdocs/goldenPath/danRer6/liftOver/danRer6ToMm9.over.chain.gz and /usr/local/apache/htdocs/goldenPath/danRer6/vsMm9/* from hgwdev to hgdownloads.

8) To make a link to these new download files change the downloads.html file.

  • hgwdev (in the browser tree):vi downloads.html
  • (find a link under "Pair-wise alignments" to a similar database. copy it, paste it and change it so that it points to the new vs directory)
  • cvs commit downloads.htm

9) Make alpha in browser tree so that the changes in downloads.html are now in htdocs.

  • ex: hgwdev (inside browser tree): make alpha

10) Email a push request to copy this file: /usr/local/apache/htdocs/downloads.html from hgwdev to hgdownloads, thus allowing the public to access the download files.

  • ex: email: Please push usr/local/apache/htdocs/downloads.html from hgwdev to hgdownloads

The end!

Retrieved from "https://genomewiki.ucsc.edu/index.php?title=Chains_and_Nets_QA&oldid=9583"