UCSC Genes Staging Process

The UCSC Gene set is created at UCSC for three vertebrate organisms: human, mouse, and rat. It is built once during the initial release of a new assembly, then updated sporadically after that. The process for QAing, staging, and releasing an update to the UCSC Gene track is complicated enough that it deserves to be documented.

Data Involved

• Databases
  • Assembly Database (e.g. hg18) -- usually about 70 tables (more details on this below)
  • UniProt Database (e.g. sp080707)
  • Proteome Database (e.g. proteins080707)

• Files
  • Index files to speed searching (e.g. /gbdb/hg18/knownGene.ix and /gbdb/hg18/knownGene.ixx)
  • Known Gene list for Google to index (e.g. /usr/local/apache/htdocs/knownGeneList/hg18/*)
  • hgdownload files (e.g. /goldenPath/proteinDB/proteins080707/database/README.txt)

• Tracks
  • In addition to the new (or updated) UCSC Genes track, there will also be a new (or updated) "Previous Version of UCSC Genes" track. This track is supported by one table; something along the lines of: knownGeneOld2.

Tables in the Assembly Database

There are many tables involved in the UCSC Gene set. For a complete list of all possible tables ever used to support any UCSC Genes set in any of the three organisms, see: /cluster/bin/scripts/kgTables. If there are tables supporting the UCSC Gene set in the assembly you are working with that are not on this list, please add them to the list. You might also consider checking the list of tables in the pushQ entry against the list of tables for the previous UCSC Gene set (sometimes developers forget to build all of the necessary tables).

For the Summer 2008 update to the UCSC Gene set on hg18, the tables are:

affyHumanExonGs, affyHumanExonGsMedian, affyHumanExonGsRatio, affyHumanExonGsRatioMedian, bioCycMapDesc, bioCycPathway, ccdsKgMap, ceBlastTab, cgapAlias, cgapBiocDesc, cgapBiocPathway, chromInfo, dmBlastTab, drBlastTab, foldUtr3, foldUtr5, gnfAtlas2Distance, gnfU95Distance, humanHprdP2P, humanVidalP2P, humanWankerP2P, keggMapDesc, keggPathway, kg3ToKg4, kgAlias, kgColor, kgProtAlias, kgProtMap2, kgSpAlias, kgTxInfo, kgXref, knownAlt, knownBlastTab, knownCanonical, knownGene, knownGeneMrna, knownGeneOld3, knownGenePep, knownIsoforms, knownToAllenBrain, knownToCdsSnp, knownToEnsembl, knownToGnf1h, knownToGnfAtlas2, knownToHInv, knownToHprd, knownToLocusLink, knownToPfam, knownToRefSeq, knownToSuper, knownToU133, knownToU133Plus2, knownToU95, knownToVisiGene, mmBlastTab, pbAnomLimit, pbResAvgStd, pbStamp, pepCCntDist, pepExonCntDist, pepHydroDist, pepIPCntDist, pepMolWtDist, pepMwAa, pepPi, pepPiDist, pepResDist, pfamDesc, rnBlastTab, scBlastTab, scopDesc, spMrna.

Details About UniProt and Proteome Databases

Each UCSC Gene set is related to one UniProt database and one Proteome Database. Each of these databases can support more than one UCSC Gene set (e.g. a single UniProt database might support the UCSC Genes on both hg18 and mm9).

These databases are given a name based on the date they were created. All UniProt databases are named using the following convention: spYYMMDD (e.g. sp080707). All Proteome databases are named using the following convention: proteinsYYMMDD (e.g. proteins080707).

To make this transparent to the users, a symbolic link is used; users see "uniProt" (but are actually using spYYMMDD). Once you push these two databases to hgwbeta, ask the cluster-admin to update the symbolic link in the /var/lib/mysql directory for uniProt and proteome to point to the newly-pushed databases. Likewise for the push from hgwbeta to the public website.

Additionally, as you set up the new databases on hgwbeta (then on the public website) you will need to edit hgcentralbeta.gdbPdb (then hgcentral.gdbPdb) to point to the correct databases:

mysql> select * from hgcentraltest.gdbPdb where genomeDb = 'hg18'\G

genomeDb: hg18
proteomeDb: proteins080707

Staging on hgwbeta

• Databases and Tables

As usual, the new databases and tables will be built on hgwdev. After QAing on hgwdev, the whole set should be staged on hgwbeta.

Create new databases on hgwbeta for the new uniProt and proteome databaes. Push all tables from dev to beta into these two new databases. Update the gdbPdb table and ask the cluster-admin to update the symlinks (see above for details).

Push all of the necessary supporting tables (usually about 70 tables) from the assembly database from dev to beta.

• Searching

Searching for UCSC IDs is supported by these files:

/gbdb/hg18/knownGene.ix
/gbdb/hg18/knownGene.ixx

However, if this is a UCSC Gene update, and you push those files from hgwdev to hgnfs1 at this point, the searching on the public website for the current UCSC Genes will break (because it will be looking for the new IDs). So, you will have to put up with broken searching on hgwbeta until you are ready to make the final push to the public website.

Releasing to the public website

• Searching

Push these files from hgwdev to hgnfs1:

/gbdb/hg18/knownGene.ix
/gbdb/hg18/knownGene.ixx

• Gene Lists for Google

We attempt to get Google to index our list of UCSC Genes. To that end, this is the set of files that needs to be made available:

/usr/local/apache/htdocs/knownGeneList/<db>/*

Push those directories and files from hgwdev to the RR machines. It shows up in the browser here: http://genome.ucsc.edu/knownGeneList/hg18/

And is linked to from here: http://genome.ucsc.edu/knownGeneLists.html

The above page will need to be edited if this is a new UCSC Gene set (not an update).

Post-Release

We typically like to announce the release of a new UCSC Gene set. Ask Donna to prepare an announcement for the website and genome-announce. Also see this page: Post-Release-Checklist.