Requests for liftOver files: Difference between revisions

From Genecats
Jump to navigationJump to search
 
(7 intermediate revisions by 2 users not shown)
Line 1: Line 1:
[[Category:Browser QA tracks]]
[[Category:Browser QA tracks]]
==QA Procedure for generating liftOver files==


==Overview==
==Overview==
Line 11: Line 14:


Here is a quick review of the process, covered below on this page:
Here is a quick review of the process, covered below on this page:
====push the files in /gbdb/ on hgwdev to hgnfs1====
====push the files in /gbdb/ on hgwdev to beta / RR/world====
These files have the same name and live in a similar place as hgdownload-test (/$db/liftOver/$db1To$db2.over.chain.gz/), but are what is used by the CGIs to allow lifting.
These gbdb files have the same name and live in a similar place as hgdownload-test (/$db/liftOver/$db1To$db2.over.chain.gz/), but are what is used by the CGIs to allow lifting.
To get the files to hgwbeta gbdb for the push use the '''sudo gbdbPush''' script.


====edit hgcentral on beta and the RR once gbdb goes through====
====edit hgcentral on beta and the RR once gbdb goes through====
Line 20: Line 24:
To make the files available for download, a push of the files to hgdownload is also needed.
To make the files available for download, a push of the files to hgdownload is also needed.


==Push the files in /gbdb/ on hgwdev to hgnfs1==
==Push the files in /gbdb/ on hgwbeta to RR==


Here is an example push email where mm10 <---> danRer10 liftOver files and rn6 <---> danRer10 liftOver files and hg19 <---> danRer10 files are being released.
Here is an example push email where mm10 <---> danRer10 liftOver files and rn6 <---> danRer10 liftOver files and hg19 <---> danRer10 files are being released.


''First use ''''gbdbPush'''' '' to get the files to hgwbeta /gbdb location.
Then request a push from beta to the RR.
<pre>
<pre>
Hi Pushers,
Hi Pushers,
Line 36: Line 43:
/gbdb/danRer10/liftOver/danRer10ToHg19.over.chain.gz
/gbdb/danRer10/liftOver/danRer10ToHg19.over.chain.gz


       from hgwdev --> hgnfs1/genome-euro/-asia
       from hgwbeta --> RR/genome-euro/-asia
 
Note that items that are symlinked on hgwdev should become real files on hgnfs1.


Reason: Releasing LiftOvers as part of MLQ ######
Reason: Releasing LiftOvers as part of MLQ ######
Line 44: Line 49:
Thanks,
Thanks,
</pre>
</pre>
This push will make the files available on ''both hgwbeta and the RR'' as hgnfs1 is shared by both of them.


==Edit hgcentral on beta and the RR once gbdb push goes through==
==Edit hgcentral on beta and the RR once gbdb push goes through==
Line 57: Line 60:


===Check that the lines do not exist on Beta===
===Check that the lines do not exist on Beta===
  hgsql  -h mysqlbeta -Ne "select * from liftOverChain where fromDb='danRer6' and toDb='mm9';" hgcentralbeta
  hgsql  -h hgwbeta -Ne "select * from liftOverChain where fromDb='danRer6' and toDb='mm9';" hgcentralbeta


This will query beta and '''should come back empty''' indicating there are no lines like the ones.  We will next populate the lines using what we took from dev.
This will query beta and '''should come back empty''' indicating there are no lines like the ones.  We will next populate the lines using what we took from dev.


===Provided lines don't exist, load lines from dev on Beta===
===Provided lines don't exist, load lines from dev on Beta===
  hgsql -h mysqlbeta -e "load data local infile 'liftOver.dev' into table liftOverChain" hgcentralbeta
  hgsql -h hgwbeta -e "load data local infile 'liftOver.dev' into table liftOverChain" hgcentralbeta


===Check behavior on Beta===
===Check behavior on Beta===
At this point you should be able to go to hgLiftover and hgConvert and see things in operation on hgwbeta, where in this example danRer6 will have an option to lift coordinates to mm9
At this point you should be able to go to hgLiftover and hgConvert and see things in operation on hgwbeta, where in this example danRer6 will have an option to lift coordinates to mm9


===Check that lines do not exit on RR (before loading)===
===Check that lines do not exist on RR (before loading)===
  hgsql  -h genome-centdb -Ne "select * from liftOverChain where fromDb='danRer6' and toDb='mm9';" hgcentral
  hgsql  -h genome-centdb -Ne "select * from liftOverChain where fromDb='danRer6' and toDb='mm9';" hgcentral
   
   
This will query beta and '''should come back empty''' indicating there are no lines like the ones.  We will next populate the lines using what we took from dev.
This will query beta and '''should come back empty''' indicating there are no lines like the ones.  We will next populate the lines using what we took from dev.


===Provided lines don't exit, load lines on the RR===
===Provided lines don't exist, load lines on the RR===
  hgsql -h genome-centdb -e "load data local infile 'liftOver.dev' into table liftOverChain;" hgcentral
  hgsql -h genome-centdb -e "load data local infile 'liftOver.dev' into table liftOverChain;" hgcentral


Line 85: Line 88:
  hgsql -Ne "select * from liftOverChain where fromDb='mm9' and toDb='danRer6';" hgcentraltest > liftOver.dev
  hgsql -Ne "select * from liftOverChain where fromDb='mm9' and toDb='danRer6';" hgcentraltest > liftOver.dev
Check it doesn't exist on beta
Check it doesn't exist on beta
  hgsql  -h mysqlbeta -Ne "select * from liftOverChain where fromDb='mm9' and toDb='danRer6';" hgcentralbeta
  hgsql  -h hgwbeta -Ne "select * from liftOverChain where fromDb='mm9' and toDb='danRer6';" hgcentralbeta
Load on Beta (provided it doesn't exist)
Load on Beta (provided it doesn't exist) **NOTE BELOW**
  hgsql -h mysqlbeta -e "load data local infile 'liftOver.dev' into table liftOverChain" hgcentralbeta
  hgsql -h hgwbeta -e "load data local infile 'liftOver.dev' into table liftOverChain" hgcentralbeta
Check it doesn't exist on RR
Check it doesn't exist on RR
  hgsql  -h genome-centdb -Ne "select * from liftOverChain where fromDb='mm9' and toDb='danRer6';" hgcentral
  hgsql  -h genome-centdb -Ne "select * from liftOverChain where fromDb='mm9' and toDb='danRer6';" hgcentral
Load on RR (provided it doesn't exist):
Load on RR (provided it doesn't exist): **NOTE BELOW**
  hgsql -h genome-centdb -e "load data local infile 'liftOver.dev' into table liftOverChain;" hgcentral
  hgsql -h genome-centdb -e "load data local infile 'liftOver.dev' into table liftOverChain;" hgcentral
::NOTE:If you happen to load things twice on hgcentralbeta or hgcentral you can delete the rows you added and add them back
:::hgsql -h hgwbeta -Ne "delete from liftOverChain where fromDb='felCat3' and toDb='felCat8';" hgcentralbeta


==Push the files from hgdownload-test to hgdownload==
==Push the files from hgdownload-test to hgdownload==
Line 116: Line 122:
Note that it is probably good to include the md5sum.txt, but that the md5sum.txt may have additional files only seen on dev, this is alright. If users email about these additional files, we can always point them to genome-preview and hgdownload-test, or move on releasing the liftOver files if browsers exist on the RR.
Note that it is probably good to include the md5sum.txt, but that the md5sum.txt may have additional files only seen on dev, this is alright. If users email about these additional files, we can always point them to genome-preview and hgdownload-test, or move on releasing the liftOver files if browsers exist on the RR.


==Note about Pairwise Files==
==Note about *Not Required* Pairwise Files==
When doing a release for a mailing list request, there isn't a mandate to release the related Pairwise files, unless asked.  If you do push those files to hgdownload an added step will be needed to update downloads.html where there are '''Pairwise Alignments''' sections under each species and lines like '''Human/Chicken (galGal3)'''  that point to a [http://hgdownload.soe.ucsc.edu/goldenPath/hg19/vsGalGal3/ hg19/vsGalGal3] directory that needs pushing (and the '''link needs creation on downloads.html''').  The gbdb steps would remain unchanged.  
When doing a release for a mailing list request, there isn't a mandate to release the related Pairwise files, unless asked.  If you do push those files to hgdownload an added step will be needed to update downloads.html where there are '''Pairwise Alignments''' sections under each species and lines like '''Human/Chicken (galGal3)'''  that point to a [http://hgdownload.soe.ucsc.edu/goldenPath/hg19/vsGalGal3/ hg19/vsGalGal3] directory that needs pushing (and the '''link needs creation on downloads.html''').  The gbdb steps would remain unchanged.  



Latest revision as of 16:56, 7 October 2021


QA Procedure for generating liftOver files

Overview

The MLM FAQ on LiftOver files shares some guidelines:

liftOver files for assemblies with no browser

We do not provide liftOver files for assemblies for which we do not have at least a minimal browser. If a user requests such a file, please point them to this wiki page and wish them luck!

liftOver file requests for assemblies with a browser

When a specific liftOver file is requested from a user for an existing browser, we should definitely consider making it. It should go through the usual process (make a redmine, assign to an engineer, place in pushQ, QA and release). QA can do a minimal QA job on it (does it work? does it show up?) before pushing it to hgdownload and enabling it on the RR. There is no need to create companion net/chain tracks (unless the user specifically asked for that). The MLM who received the question is not responsible for QAing the liftOver file (it should just go into the pushQ like all data). After it is released, whoever QAd it should contact the user to let them know that it's available.

Here is a quick review of the process, covered below on this page:

push the files in /gbdb/ on hgwdev to beta / RR/world

These gbdb files have the same name and live in a similar place as hgdownload-test (/$db/liftOver/$db1To$db2.over.chain.gz/), but are what is used by the CGIs to allow lifting. To get the files to hgwbeta gbdb for the push use the sudo gbdbPush script.

edit hgcentral on beta and the RR once gbdb goes through

The lines on dev in hgcentraltest that allow the /gbdb/ files to load in the CGIs need to added to hgcentralbeta and hgcentral. This is probably the most detailed part as it requires careful manipulation of hgcentral on the RR.

push the files from hgdownload-test to hgdownload

To make the files available for download, a push of the files to hgdownload is also needed.

Push the files in /gbdb/ on hgwbeta to RR

Here is an example push email where mm10 <---> danRer10 liftOver files and rn6 <---> danRer10 liftOver files and hg19 <---> danRer10 files are being released.

First use 'gbdbPush' to get the files to hgwbeta /gbdb location.

Then request a push from beta to the RR.

Hi Pushers,

Please push:

/gbdb/mm10/liftOver/mm10ToDanRer10.over.chain.gz
/gbdb/danRer10/liftOver/danRer10ToMm10.over.chain.gz
/gbdb/rn6/liftOver/rn6ToDanRer10.over.chain.gz
/gbdb/danRer10/liftOver/danRer10ToRn6.over.chain.gz
/gbdb/hg19/liftOver/hg19ToDanRer10.over.chain.gz
/gbdb/danRer10/liftOver/danRer10ToHg19.over.chain.gz

      from hgwbeta --> RR/genome-euro/-asia

Reason: Releasing LiftOvers as part of MLQ ######

Thanks,

Edit hgcentral on beta and the RR once gbdb push goes through

Note that these examples must be edited to the appropriate databases and also be updated in both directions. In the below example danRer6 <---> mm9 is being released, but first in the danRer6 --> mm9 direction, it will need to be repeated a second time in the mm9 --> danRer6 direction.

Acquire the hgwdev entry

 hgsql -Ne "select * from liftOverChain where fromDb='danRer6' and toDb='mm9';" hgcentraltest > liftOver.dev

This will save the danRer6 --> mm9 lines to a file called liftOver.dev

Check that the lines do not exist on Beta

hgsql  -h hgwbeta  -Ne "select * from liftOverChain where fromDb='danRer6' and toDb='mm9';" hgcentralbeta

This will query beta and should come back empty indicating there are no lines like the ones. We will next populate the lines using what we took from dev.

Provided lines don't exist, load lines from dev on Beta

hgsql -h hgwbeta -e "load data local infile 'liftOver.dev' into table liftOverChain" hgcentralbeta

Check behavior on Beta

At this point you should be able to go to hgLiftover and hgConvert and see things in operation on hgwbeta, where in this example danRer6 will have an option to lift coordinates to mm9

Check that lines do not exist on RR (before loading)

hgsql  -h genome-centdb -Ne "select * from liftOverChain where fromDb='danRer6' and toDb='mm9';" hgcentral

This will query beta and should come back empty indicating there are no lines like the ones. We will next populate the lines using what we took from dev.

Provided lines don't exist, load lines on the RR

hgsql -h genome-centdb -e "load data local infile 'liftOver.dev' into table liftOverChain;" hgcentral

Check behavior on the RR

At this point you should be able to go to hgLiftover and hgConvert and see things in operation on hgwbeta, where in this example danRer6 will have an option to lift coordinates to mm9.

Repeat the process in the reverse direction

Now reverse the names and repeat the process. I suggest removing liftOver.dev before starting.

For reference here are the steps (note that everywhere danRer6 has been replaced by mm9 and vice-versa and the load steps are the same). Acquire:

hgsql -Ne "select * from liftOverChain where fromDb='mm9' and toDb='danRer6';" hgcentraltest > liftOver.dev

Check it doesn't exist on beta

hgsql  -h hgwbeta  -Ne "select * from liftOverChain where fromDb='mm9' and toDb='danRer6';" hgcentralbeta

Load on Beta (provided it doesn't exist) **NOTE BELOW**

hgsql -h hgwbeta -e "load data local infile 'liftOver.dev' into table liftOverChain" hgcentralbeta

Check it doesn't exist on RR

hgsql  -h genome-centdb -Ne "select * from liftOverChain where fromDb='mm9' and toDb='danRer6';" hgcentral

Load on RR (provided it doesn't exist): **NOTE BELOW**

hgsql -h genome-centdb -e "load data local infile 'liftOver.dev' into table liftOverChain;" hgcentral
NOTE:If you happen to load things twice on hgcentralbeta or hgcentral you can delete the rows you added and add them back
hgsql -h hgwbeta -Ne "delete from liftOverChain where fromDb='felCat3' and toDb='felCat8';" hgcentralbeta

Push the files from hgdownload-test to hgdownload

Send a push-request to release the files to hgdownload:

Hi Pushers,

Please push the following download file(s):

/usr/local/apache/htdocs-hgdownload/goldenPath/galGal4/liftOver/galGal4ToHg19.over.chain.gz
/usr/local/apache/htdocs-hgdownload/goldenPath/galGal4/liftOver/md5sum.txt
/usr/local/apache/htdocs-hgdownload/goldenPath/hg19/liftOver/hg19ToGalGal4.over.chain.gz
/usr/local/apache/htdocs-hgdownload/goldenPath/hg19/liftOver/md5sum.txt

   from hgwdev --> hgdownload
         (in the path, "htdocs-hgdownload" should become "htdocs")

Reason: Adding some user requested files to hgdownload, refs #####

Thanks,

Note that it is probably good to include the md5sum.txt, but that the md5sum.txt may have additional files only seen on dev, this is alright. If users email about these additional files, we can always point them to genome-preview and hgdownload-test, or move on releasing the liftOver files if browsers exist on the RR.

Note about *Not Required* Pairwise Files

When doing a release for a mailing list request, there isn't a mandate to release the related Pairwise files, unless asked. If you do push those files to hgdownload an added step will be needed to update downloads.html where there are Pairwise Alignments sections under each species and lines like Human/Chicken (galGal3) that point to a hg19/vsGalGal3 directory that needs pushing (and the link needs creation on downloads.html). The gbdb steps would remain unchanged.

If you were pushing these files, here is an example hgdownload push that inlcudes the vs$Db directories, in this example danRer10 <---> hg19. But note we don't need to do this unless requested.

/usr/local/apache/htdocs-hgdownload/goldenPath/danRer10/md5sum.txt
/usr/local/apache/htdocs-hgdownload/goldenPath/danRer10/liftOver/danRer10ToHg19.over.chain.gz
/usr/local/apache/htdocs-hgdownload/goldenPath/danRer10/liftOver/danRer10ToRn6.over.chain.gz
/usr/local/apache/htdocs-hgdownload/goldenPath/danRer10/liftOver/danRer10ToMm10.over.chain.gz
/usr/local/apache/htdocs-hgdownload/goldenPath/danRer10/vsHg19/*
/usr/local/apache/htdocs-hgdownload/goldenPath/danRer10/vsHg19/reciprocalBest/*
/usr/local/apache/htdocs-hgdownload/goldenPath/danRer10/vsHg19/reciprocalBest/axtRBestNet/*

/usr/local/apache/htdocs-hgdownload/goldenPath/hg19/md5sum.txt
/usr/local/apache/htdocs-hgdownload/goldenPath/hg19/liftOver/hg19ToDanRer10.over.chain.gz
/usr/local/apache/htdocs-hgdownload/goldenPath/hg19/vsDanRer10/*
/usr/local/apache/htdocs-hgdownload/goldenPath/hg19/vsDanRer10/axtNet/*
/usr/local/apache/htdocs-hgdownload/goldenPath/hg19/vsDanRer10/reciprocalBest/*
/usr/local/apache/htdocs-hgdownload/goldenPath/hg19/vsDanRer10/reciprocalBest/axtRBestNet/*'

And here is a reminder to edit the downloads.html page and push that as well:

Hi Pushers,

Please push the following file from hgwdev:
   /usr/local/apache/htdocs-hgdownload/downloads.html

   to same location on hgdownload *except*
   replace "htdocs-hgdownload" with "htdocs" in the directory path

Reason: Adding links to hg19 and danRer10 pairwise directories, refs #####