|
|
| (2 intermediate revisions by the same user not shown) |
| Line 1: |
Line 1: |
| | == Etc. == |
|
| |
|
| This page contains detailed installation instructions for setting up [http://genome.ucsc.edu UCSC Genome Browser] comparative genomics track data and [[GBrowse]] [[GBrowse#Plugins|plugins]] to display them. It assumes that you already have a working installation of [[GBrowse]] and [http://www.mysql.com/ mysql]. | | [[AngieTest|This page]] (but not [[Main_Page|this one]]) is for ''quick'' '''tests''' |
|
| |
|
| | === etc. === |
| | *of |
| | **<span style="color: green">'''''formatting'''''</span> |
| | **#<nowiki>options [and] <stuff></nowiki>. |
|
| |
|
| == Prerequisites ==
| | code here |
| * A working installation of [[GBrowse]] (see also [[GBrowse_Install_HOWTO.txt]])
| |
| * A working installation of [http://www.mysql.com/ MySQL http://www.mysql.com/]
| |
| * Plenty of disk space, ideally local to your server.
| |
| | |
| Disk space requirements vary depending on how many assemblies, alignments and types of tracks you would like to install. Things to consider:
| |
| * Temporary disk space for MySQL dump files used to load the database tables.
| |
| * MySQL server disk space requirements for database tables.
| |
| * If loading a Conservation track, permanent disk space for data files indexed by the MySQL database tables.
| |
| | |
| Here are some example approximate sizes for various tracks and organisms (note that each genome assembly usually has many Chain and Net tracks, and one Conservation track):
| |
| {| cellspacing="0" border="1"
| |
| !Track Description
| |
| !MySQL dump files
| |
| !MySQL table Data_length
| |
| !External files
| |
| |-
| |
| |Worm-Worm Chain
| |
| |170M
| |
| |120M
| |
| |n/a
| |
| |-
| |
| |Worm-Worm Net
| |
| |25M
| |
| |20M
| |
| |n/a
| |
| |-
| |
| |Worm 5-way Conservation
| |
| |75M
| |
| |65M
| |
| |465M
| |
| |-
| |
| |Human-Mammal Chain
| |
| |3,000M
| |
| |2,000M
| |
| |n/a
| |
| |-
| |
| |Human-Mammal Net
| |
| |510M
| |
| |400M
| |
| |n/a
| |
| |-
| |
| |Human 28-way Conservation
| |
| |2,530M
| |
| |2,000M
| |
| |79,300M
| |
| |}
| |
| | |
| | |
| | |
| == UCSC naming conventions and other important subtleties ==
| |
| | |
| An understanding of UCSC conventions is not really a prerequisite, but will be quite helpful for understanding the installation process.
| |
| | |
| === UCSC database names ===
| |
| | |
| UCSC builds a separate MySQL database of annotations for each genome assembly. Each database contains at least one table for each "track" (set of annotations). The database name begins with 2 or 6 letters indicating the species, followed by a sequence number which usually starts with 1 on the first assembly that UCSC processes.
| |
| | |
| Jim Kent wrote the UCSC Genome Browser while working on the Human Genome Project, so all databases for the human genome start with "hg". For example, hg18 is the database of annotations for the 18th assembly of the human genome displayed in the UCSC Genome Browser; the assembly itself is better known as NCBI Build 36.
| |
| | |
| The next species added to the browser was mouse, and the first letters of its genus and species names were used: "mm" for Mus musculus. That convention was followed for rat (rn), ''C. elegans'' (ce), ''D. melanogaster'' (dm) and several other species added before mid-2003. Then, in order to avoid name clashes among the deluge of newly sequenced species, a new convention was established: when new species are added to the browser, we use the first three letters of the genus in lowercase, and first three letters of the species with the first letter capitalized (e.g. "bosTau" for ''Bos taurus'').
| |
| | |
| === Chromosome naming ===
| |
| | |
| Most genome assemblies use sequence names for chromosomes that match chromosome designations in the literature -- for example, 1, 2, 3 for human, and I, II, III for worm. UCSC prefixes those sequence names with "chr", to make it easier for its software to recognize chromosome sequence names. UCSC's annotations for human refer to chr1, chr2, chr3 etc., and for worm, chrI, chrII, chrIII etc.
| |
| | |
| === Naming of database tables for comparative genomics tracks ===
| |
| | |
| For assemblies that consist of a reasonably small number of sequences (e.g. a few dozen chromosomes or mapping groups, possibly with concatenated unplaced sequences in virtual "chromosomes"), a track's data may be split into per-sequence tables instead of kept in one main table. This applies to the Chain tracks but not Net tracks, since Chain tracks contain much more data. When split, table names for a track end with the same track name, prefixed by the sequence name and _. For example, if a track named foo is split, its database tables are named chr*_foo (or using the MySQL wildcard, chr%\_foo).
| |
| | |
| On the other hand, if an assembly consists of thousands of separate unplaced sequences (e.g. WGS scaffolds), then it would kill MySQL to make one table per sequence for each split track. Therefore, splitting does not apply to such assemblies, and a track named foo would have one table named foo.
| |
| | |
| Chain tracks use two different types of database table to efficiently store the usually quite large amounts of data. The main tables are named chain''OtherDb'', and auxiliary "chainLink" tables are named chain''OtherDb''Link. (Or, if split, chr*_chain''OtherDb'' and chr*_chain''OtherDb''Link.)
| |
| | |
| Chain and Net tracks contain pairwise alignments to a particular assembly of some other species' genome. The database table names for those tracks include the UCSC database name for that other species' assembly. For example, the chained alignments of UCSC's first ''D. ananassae'' build (the query) to UCSC's third ''D. melanogaster'' build (the target or reference) would be stored in the dm3 database, as the tables chr*_chainDroAna1 and chr*_chainDroAna1Link -- split because dm3 consists of a manageable number of chromosomes and concatenated unplaced sequences. The corresponding netted chains would be stored in netDroAna1.
| |
| | |
| The Conservation track combines two data types: per-base conservation scores and multi-species alignments. Both data types use not only database tables, but also external files, to store the data due to its massive size.
| |
| | |
| The scores are stored in UCSC's "wiggle" format, which utilizes both database tables and external files. The database table is usually called phastCons''N''way where ''N'' is the number of aligned species including the reference. The database references external files that contain binary compressed data.
| |
| | |
| The multi-species alignments are stored in external files in the MAF format; those files are indexed in the multiz''N''way database table. The maf table refers to numeric IDs of the files; those numeric IDs and the actual file paths are stored in the extFile table.
| |
| | |
| The Conservation track will probably need a C extension for decent performance... describe that too!
| |
| | |
| === Assembly and track meta-info ===
| |
| | |
| UCSC's names for databases and track tables are rather opaque by themselves, so we store information about them in separate tables. Assembly databases are described by the dbDb table in the central database, hgcentral. Each assembly database contains a trackDb table that describes all tracks in that database.
| |
| | |
| | |
| == Installation Overview ==
| |
| | |
| # Download MySQL table data from UCSC.
| |
| # Create mysql database and load tables.
| |
| #* If adding the Conservation track, also download external files and C extension for the plugin.
| |
| # Download the plugin and glyph modules (from where? UCSC? modENCODE? bundled with GBrowse?)
| |
| # Install and configure plugin(s).
| |
| | |
| | |
| == Installation Details ==
| |
| | |
| === Which UCSC database? ===
| |
| First, identify the UCSC database name that corresponds to the assembly version on which your GBrowse installation is built. For example, WormBase version WS170 corresponds to UCSC database ce4. This version correspondence is usually available on the assembly description page at http://genome.ucsc.edu/cgi-bin/hgGateway ; you might need to change the clade and genome in order to see your species of interest and its available assembly dates. If you still aren't sure about the correspondence, ask mailto:genome@soe.ucsc.edu for assistance.
| |
| | |
| === Which tables for my track(s) of interest? ===
| |
| | |
| * tracks and the db tables required for them:
| |
| {| cellspacing="0" border="1"
| |
| !Track type
| |
| !MySQL table(s)
| |
| |-
| |
| |all tracks
| |
| |trackDb, hgcentral.dbDb
| |
| |-
| |
| |Chain
| |
| |chain''OtherDb'', chain''OtherDb''Link<br>or if split:<br>chr*_chain''OtherDb'', chr*_chain''OtherDb''Link
| |
| |-
| |
| |Net
| |
| |net''OtherDb''
| |
| |-
| |
| |Conservation
| |
| |multiz''N''way, extFile, multiz''N''wayFrames, multiz''N''waySummary, phastCons''N''way*
| |
| |}
| |
| | |
| | |
| === Download mysql dump files from UCSC ===
| |
| | |
| All UCSC Genome Browser data can be downloaded from http://hgdownload.cse.ucsc.edu/ . The HTTP, FTP and rsync protocols are supported.
| |
| | |
| * example download of db.table for each protocol: http, ftp, rsync
| |
| * example download of hgcentral.dbDb
| |
| | |
| === Create and load the MySQL database ===
| |
| * create mysql database w/same name as UCSC identifier
| |
| mysql -u root -pPassword -e "create database $DB;"
| |
| | |
| * load tables into mysql
| |
| mysql $DB -u root -pPassword < $TABLE.sql
| |
| mysql $DB -u root -pPassword -e "load data local infile '$TABLE.txt' into table $TABLE"
| |
| | |
| | |
| === Conservation track only: download external files ===
| |
| The Conservation track combines multiple alignments and per-base scores for multiple species. This is a very large amount of data, and in order to access and display it quickly, the actual data are kept in external files, and the database tables simply index those files. So, having installed the multiz''N''way and phastCons''N''way database tables, you can use mysql to determine what files are needed from UCSC.
| |
| | |
| Determine the files needed for multiz''N''way and phastCons''N''way by running by running these mysql commands:
| |
| select distinct(path) from extFile,multizNway where extFile.id = multizNway.extFile;
| |
| select distinct(file) from phastConsNway;
| |
| | |
| The paths all start with "/gbdb/", and can be downloaded from hgdownload.cse.ucsc.edu using the /gbdb/... path as follows, depending on your chosen protocol:
| |
| http://hgdownload.cse.ucsc.edu/gbdb/...
| |
| ftp://hgdownload.cse.ucsc.edu/gbdb/...
| |
| rsync rsync://hgdownload.cse.ucsc.edu/gbdb/...
| |
| | |
| * Ideally, create a local /gbdb/ and install the files in the same paths as referenced by the sql tables. If that is not possible, replace the pathnames in the sql tables with your local paths to the corresponding files.
| |
| | |
| === Download, install and configure the UCSC track plugin(s) ===
| |
| | |
| * Obtain the GBrowse UCSC*.pm plugin files (where will these be distributed from?) and install then in GBrowse's Generic_Genome_Browser/conf/plugins/ directory.
| |
| | |
| * Obtain the GBrowse ucsc_*.pm glyph files and install them in GBrowse's Generic_Genome_Browser/lib/Bio/Graphics/Glyph/ directory.
| |
| | |
| * Conservation track only: download and compile C extension??
| |
| | |
| * Add the plugin names (UcscChain, UcscNet, and/or UcscConservation) to the GBrowse .conf plugins setting (see also [[GBrowse#Plugins]])
| |
| | |
| * Add plugin settings like the following:
| |
| | |
| [UcscChain:plugin]
| |
| db = ce4
| |
| user = ''mysqlReadOnlyUser''
| |
| pass = ''mysqlReadOnlyUsersPassword''
| |
| seq_prefix = chr
| |
| split_prefix = chr
| |
| default_enable = chainCb3 chainCaeRem2
| |
|
| |
|
| {| cellspacing="0" border="1" | | {| cellspacing="0" border="1" |
| Line 182: |
Line 15: |
| !description | | !description |
| |- | | |- |
| |db | | |<span style="font-family: Courier New;">db</span> |
| |'''Yes''' | | |'''Yes''' |
| |The name of the MySQL database where UCSC tables have been loaded. | | |The name of the MySQL database where UCSC tables have been loaded. |
| |- | | |- |
| |user | | |<span style="font-family: Courier New;">user</span> |
| |'''Yes'''
| |
| |The name of a MySQL user that has permissions to read db.
| |
| |-
| |
| |pass
| |
| |'''Yes''' | | |'''Yes''' |
| |The password for user. | | |The name of a MySQL user that has permission to read <span style="font-family: Courier New;">db</span>. |
| |-
| |
| |seq_prefix
| |
| |'''probably'''
| |
| |If the datasource has sequence names like I, II, III or 2L, 2R, 3L, set this to chr so that they can be translated into UCSC's names: chrI, chrII, chrIII or chr2L, chr2R, chr3L etc. This is usually the case.
| |
| |-
| |
| |split_prefix
| |
| |'''probably'''
| |
| |If all of the sequences in the datasource are chromosome names (not separate scaffolds, contigs etc), set this to chr if seq_prefix is chr.
| |
| |-
| |
| |default_enable
| |
| |'''should be'''
| |
| |A whitespace-separated list of chain or net tracks to display by default. E.g., chain''OtherDb1'' chain''OtherDb2'' in the UcscChain:plugin section, net''OtherDb1'' net''OtherDb2'' in the UcscNet:plugin section. If not specified, all tracks will be displayed by default which could overwhelm the display.
| |
| |-
| |
| |host
| |
| |prob. not
| |
| |The machine on which the MySQL server, if not the same machine on which GBrowse runs.
| |
| |-
| |
| |port
| |
| |prob. not
| |
| |The port number used by the MySQL server, if not the default.
| |
| |-
| |
| |central_db
| |
| |prob. not
| |
| |The database in which the dbDb table is stored, if not db.
| |
| |-
| |
| |central_user
| |
| |prob. not
| |
| |The MySQL user for central_db, if not user.
| |
| |-
| |
| |central_pass
| |
| |prob. not
| |
| |The password for central_user, if not pass.
| |
| |-
| |
| |central_host
| |
| |prob. not
| |
| |The machine on which the MySQL server that serves up central_db, if not host.
| |
| |-
| |
| |central_port
| |
| |prob. not
| |
| |The port number used by central_host, if not port.
| |
| |} | | |} |
|
| |
|
| === Test GBrowse === | | ==== Re: category ==== |
| * tail the GBrowse web server's error log file and start using GBrowse
| | Don't say <nowiki>[[Category:HOWTO]]</nowiki> because we don't have that category. |
| * below the image, in the Tracks section, Analysis subsection, set the checkboxes by the UCSC plugin track names and click Update Image.
| |
| * UCSC tracks should appear in the display. If not, there may be no data in the current region; try viewing a region that contains an exon or gene. Or the error log may contain a message that indicates what is missing.
| |
| | |
| | |
| == Performance tweaks ==
| |
| | |
| === rsync mysql table binary files ===
| |
| | |
| === Build C lib for Conservation plugin ===
| |
| Will this have to be part of the regular installation for Conservation???
| |
| | |
| | |
| == Bug Reports and Support Requests ==
| |
| Please send questions and/or bug reports to mailto:genome@soe.ucsc.edu.
| |
| | |
| | |
| == AUTHORS == | |
| Copyright 2007, Angie Hinrichs mailto:angie@soe.ucsc.edu. Redistribution is welcome provided that acknowledgement of authorship is maintained.
| |
| | |
| [[Category:HOWTO]] | |
