Learn about the Browser: Difference between revisions

From genomewiki
Jump to navigationJump to search
(changed cse to soe)
 
(70 intermediate revisions by 6 users not shown)
Line 1: Line 1:
This list is sorted by increasing technical complexity: the first steps require only a webbrowser, the last ones a Linux webserver.
This page links to various pages that will help you to learn more about the UCSC Genome Browser. It is sorted by increasing technical complexity: the first steps require only a webbrowser, the last ones administrator access to a Linux webserver.


== Use the browser website ==
== Use the browser website ==
* Bob Kuhn is giving workshops throughout the year at several US conferences, in Europe, Australia and Asia
** Read about how to [http://genome.ucsc.edu/training/ | host]
* Video tutorials and slides by [http://www.openhelix.com/downloads/ucsc/ucsc_home.shtml Openhelix]
* Video tutorials and slides by [http://www.openhelix.com/downloads/ucsc/ucsc_home.shtml Openhelix]
* An introduction on [http://www.nature.com/scitable/ebooks/guide-to-the-ucsc-genome-browser-16569863/contents Nature Education]
* Basic materials written by the gurus: The [http://genome.ucsc.edu/goldenPath/help/hgTracksHelp.html User's guide] and the [http://genome.ucsc.edu/FAQ/ FAQs].  
* Basic materials written by the gurus: The [http://genome.ucsc.edu/goldenPath/help/hgTracksHelp.html User's guide] and the [http://genome.ucsc.edu/FAQ/ FAQs].  
* A similar page to this one [http://genome-test.cse.ucsc.edu/eng/ "Learning about the browser"]
* View a [http://genome.ucsc.edu/cgi-bin/hgPublicSessions gallery] of browser sessions that highlight interesting data sets.
* Type in (manually) a couple of [http://genome.ucsc.edu/FAQ/FAQcustom custom tracks] in [http://genome.ucsc.edu/FAQ/FAQformat.html different formats]
* Type in (manually) a couple of [http://genome.ucsc.edu/FAQ/FAQcustom custom tracks] in [http://genome.ucsc.edu/FAQ/FAQformat.html different formats] (just try bed for a start, you mostly won't need the others)
** Try typing "chr1 1 1000 Hello_World!" as a custom track
** If you need to graph data, there are [[Selecting_a_graphing_track_data_format|different graphing formats]]
* Subscribe to the [http://www.soe.ucsc.edu/mailman/listinfo/genome mailing list] or [http://genome.ucsc.edu/contacts.html search through it]. Most everyday questions have already been asked by someone else so searching gives you an answer usually faster than asking on the mailing list.
* Subscribe to the [http://www.soe.ucsc.edu/mailman/listinfo/genome mailing list] or [http://genome.ucsc.edu/contacts.html search through it]. Most everyday questions have already been asked by someone else so searching gives you an answer usually faster than asking on the mailing list.


== Download the data of the genome browser (sequences and annotations) ==
== Download the data of the genome browser (sequences and annotations) ==
* Be aware that internal coordinates (not website) are [[Coordinate_Transforms|0-based]]!
* Be aware that internal coordinates (not those shown on the website) are [[Coordinate_Transforms|0-based]]! (only exception: wiggle)
* Unlike Gbrowse and Ensembl, UCSC is storing the data partially in SQL (coordiantes, outline of x-y-plots) and partially in flat text files (sequences, alignments, details of x-y-plots)
* Unlike Gbrowse and Ensembl, UCSC is storing the data partially in SQL (coordiantes, outline of x-y-plots) and partially in flat text files (sequences, alignments, details of x-y-plots)
* Table Browser: The easiest way to access data (you don't have to care whether data is stored in MySQL or in textfiles):
* Table Browser: The easiest way to access data (you don't have to care whether data is stored in MySQL or in textfiles):
** [http://genome.ucsc.edu/goldenPath/help/hgTablesHelp.html Table Browser], [http://genomewiki.ucsc.edu/images/1/1a/ISMB2008_UCSC.ppt Example session]
** [http://genome.ucsc.edu/goldenPath/help/hgTablesHelp.html Table Browser], [[Image:ISMB2008_UCSC.ppt|Example session]]
* SQL-stored data ([http://genome.ucsc.edu/FAQ/FAQdownloads FAQ]):
* SQL-stored data ([http://genome.ucsc.edu/FAQ/FAQdownloads FAQ]):
** [http://genome.ucsc.edu/FAQ/FAQdownloads#download29 Public mysql access]
** [http://genome.ucsc.edu/FAQ/FAQdownloads#download29 Public mysql access]
** [http://genomewiki.cse.ucsc.edu/index.php/Image:Kent_allJoiner.ppt The all.joiner file], describes relations between all database tables
** [[Image:Kent_allJoiner.ppt|The all.joiner file]], describes relations between all database tables
** For all genes-related tables, there is a [http://genome-test.cse.ucsc.edu/images/knownGeneSchema.gif graphical map] (sub-optimal layout :-)
** For all genes-related tables, there is a [http://genome-test.soe.ucsc.edu/images/knownGeneSchema.gif graphical map] (sub-optimal layout :-)
* Flat-file data: Download from the [http://genome.ucsc.edu/FAQ/FAQdownloads#download1 ftp server] (stored in /gbdb on browser servers)
* Flat-file data: Download from the [http://genome.ucsc.edu/FAQ/FAQdownloads#download1 ftp server] (stored in /gbdb on browser servers)
** [http://genome.ucsc.edu/FAQ/FAQformat.html Text file formats]
** [http://genome.ucsc.edu/FAQ/FAQformat.html Text file formats]


== Install a copy of the browser on your own machine (Unix or Mac, Windows optional) ==
== Install a copy of the browser on your own machine (Unix or Mac) ==
* The [http://genome-test.cse.ucsc.edu/eng/ old documentation website with developer documentation] 
* Create a mirror of the UCSC site:
* Create a mirror of the UCSC site:
** [http://genome.ucsc.edu/admin/mirror.html mirror a complete browser]
** [http://genome.ucsc.edu/admin/mirror.html Official FAQ: mirror a complete browser], the main reference for browser installation
** [[Browser_Installation|Almost the same document]], but on the wiki, some more info
** [[Browser_Installation|Inofficial FAQ: Browser Installation]], on the wiki, a lot more info
** [[Minimal Browser Installation|Mirror only selected genomes]]
** [[Minimal Browser Installation|Mirror only selected genomes]]
** Adapt your cgi-bin/[[Hg.conf]] file for your mirror
** Be prepared that you'll need to create at least and also download parts of the hg18. At least mouse genomes still depend on it.
** Make sure that you keep your own tracks separate, read this before loading any local data: [[Local_tracks_at_mirror_sites|Local TrackDB table]]
** [[Browser_Mirrors|Updating the data automatically from UCSC]]
** [[Browser_Mirrors|Updating the data automatically from UCSC]]
** [[Using_custom_track_database|Pro and cons of storing custom tracks in MySQL or as flat files]]
** [[Using_custom_track_database|Pro and cons of storing custom tracks in MySQL or as flat files]]
** [[Cookie_Session|How are cookies handled by the browser?]]
** [[Cookie_Session|How are cookies handled by the browser?]]
** It seems that you can also run a mirror on windows (see [https://lists.soe.ucsc.edu/pipermail/genome-mirror/2008-November/001059.html]) but your mileage may vary to get everything to compile. If you have one running, please update this wiki.
** [[Category:Mirror_Site_FAQ|All other documents in this category]]
** The [http://genome-test.soe.ucsc.edu/eng/ old documentation website with developer documentation]
**[[Running your own gfServer]] (=BLAT server), needed in cases where you don't want or cannot use the UCSC BLAT servers.


== Analyze UCSC genome data files on your own machine ==
== Compile the UCSC source tree and analyze genomes yourself ==
** What is the [[the source tree]] ?
* The '''source tree''' (aka Kent source) is the collection of all tools used at the UCSC browser group
** What is available in the [[Kent source utilities]] ?
* You could also call this an '''API''', as you can use these tools from your own programs
** Download the most important [http://hgdownload.cse.ucsc.edu/admin/exe/ compiled binary programs]
* General introduction to the [[the source tree]]  
** [http://genome.ucsc.edu/admin/jk-install.html Compile all tools (includes the browser webserver)] yourself
* What is documented to be available in the [[Kent source utilities]]  
** ... and what is [http://hgwdev.soe.ucsc.edu/~kent/src/unzipped/sources really available] in the source tree?
* Some important tools are already compiled (even for windows!) [http://hgdownload.soe.ucsc.edu/admin/exe/ from here]
* [http://genome.ucsc.edu/admin/jk-install.html Compile all tools (includes the browser webserver)] yourself
** The best starting point is the main zipfile: http://hgdownload.soe.ucsc.edu/admin/jksrc.zip
** You need to set some [[Build_Environment_Variables|environment variables]] before you start the compilation
** You need to set some [[Build_Environment_Variables|environment variables]] before you start the compilation
** Walkthroughs for...
** Walkthroughs for...
*** [http://bergman-lab.blogspot.com/2009/03/compiling-ucsc-source-tree-utilities-on.html MacOS X users]
*** [http://bergmanlab.smith.man.ac.uk/?p=32 MacOS X users]
*** [[Source tree compilation on Debian/Ubuntu|Debian and Ubuntu]]
*** [[Source tree compilation on Debian/Ubuntu|Debian and Ubuntu]]
** The most common problem on the mailing list are harmless warnings that trigger errors. To ignore these, which is usually safe, [https://lists.soe.ucsc.edu/pipermail/genome-mirror/2006-November/000251.html remove the -Wall option from the makefile]
*** [[CentOS Notes]]
** See also the textfile README.building.source [http://hgwdev.cse.ucsc.edu/~kent/src/unzipped/product/README.building.source README.building.source]
*** Windows: see [https://lists.soe.ucsc.edu/pipermail/genome-mirror/2008-November/001059.html] but your mileage may vary to get everything to compile.
** [http://genome.ucsc.edu/contacts.html searching through the archives] or the [[:Category:Technical FAQ| Technical-FAQ-Category of the wiki]] when you have problems
 
** The most common problem on the mailing list are harmless warnings that trigger errors. "cc1: warnings being treated as errors". To ignore them, which is usually safe, [https://lists.soe.ucsc.edu/pipermail/genome-mirror/2006-November/000251.html remove the -Wall option from the makefile]
** See also the textfile README.building.source [http://hgwdev.soe.ucsc.edu/~kent/src/unzipped/product/README.building.source README.building.source]
** [http://genome.ucsc.edu/contacts.html Search through the archives] or the [[:Category:Technical FAQ| Technical-FAQ-Category of the wiki]] when you have problems
** Send an email to the mailing list if you cannot find the answer yourself
** Some more sophisticated genome pipelines
*** Create your own [[Whole_genome_alignment_howto|whole-genome alignment]]
*** Create your own [[LiftOver_Howto|liftOver file]]


== Modify your own copy of the browser ==
== Modify your own copy of the browser ==
* Add tracks to your own mirror:  
* You load track data into your mirror  
** with the hgLoadxxx utils (e.g. hgLoadBed)
** and show meta data (name, type, possible controls, etc) on the browser by editing a textfile called trackDb.ra
* TrackDB documentation:
** [http://www.soe.ucsc.edu/~sugnet/doc/trackHowto/browserTalk.pdf Charles Sugnet's presentation about TrackDB]  
** [http://www.soe.ucsc.edu/~sugnet/doc/trackHowto/browserTalk.pdf Charles Sugnet's presentation about TrackDB]  
** [[How_to_add_a_track_to_a_mirror]]
** [[How_to_add_a_track_to_a_mirror]]
** [http://hgwdev.cse.ucsc.edu/~kent/src/unzipped/product/README.trackDb The structure of the trackDB]
** [http://hgwdev.soe.ucsc.edu/~kent/src/unzipped/product/README.trackDb The structure of the trackDB]
** [http://hgwdev.cse.ucsc.edu/~kent/src/unzipped/hg/makeDb/trackDb/ The trackDB statements]
** [http://hgwdev.osoe.ucsc.edu/~kent/src/unzipped/hg/makeDb/trackDb/ Full TrackDB-Documentation: shows all possible trackDB statements]
** When you run into problems, search through the [http://genome.ucsc.edu/mirror.html mailing list] when you have problems and read the documentation in the directory [http://hgwdev.cse.ucsc.edu/~kent/src/unzipped/product/ kent/src/product]
*** Examples: [http://hgwdev.soe.ucsc.edu/~kent/src/unzipped/hg/makeDb/trackDb/human/hg19/trackDb.ra UCSC trackDb for hg19]
** [http://genome-test.cse.ucsc.edu/admin/ a similar place with hgSearchSpec docs and statistics (the system to search for ids)] can be found completely off-track
*** Examples for the composite statements in trackDb, from the [Encode trackDb http://hgwdev.soe.ucsc.edu/~kent/src/unzipped/hg/makeDb/trackDb/human/hg19/trackDb.wgEncode.ra]
* When you run into problems, search through the [http://genome.ucsc.edu/mirror.html mailing list] when you have problems and read the documentation in the directory [http://hgwdev.soe.ucsc.edu/~kent/src/unzipped/product/ kent/src/product]
** [http://genome-test.soe.ucsc.edu/~hiram/rgbItemExamples.html#RGB color codes]
** [http://genome-test.soe.ucsc.edu/admin/ a similar place with hgSearchSpec docs and statistics (the system to search for ids)] can be found completely off-track
* Create a browser for a [[Building a new genome database|completely new genome]]
* Create a browser for a [[Building a new genome database|completely new genome]]
 
** Slides of a class on [https://banana-slug.soe.ucsc.edu/_media/lecture_notes:genomebrowsersetup.pdf Creating a browser for new bacterial genome]
* The most difficult part: [[Writing_a_new_track_type|Adding a completely new track type and visualisation code to the browser]]


== Making Of: How the UCSC genome annotations are created ==
== Making Of: How the UCSC genome annotations are created ==
* How the UCSC folks created their tracks:
* How the UCSC folks created their tracks:
** The [http://hgwdev.cse.ucsc.edu/~kent/src/unzipped/hg/makeDb/doc/ UCSC's makeDb-files] are a log of all commands that are necessary to re-create all annotations for a particular genome.   
** The [http://hgwdev.soe.ucsc.edu/~kent/src/unzipped/hg/makeDb/doc/ UCSC's makeDb-files] are a log of all commands that are necessary to re-create all annotations for a particular genome.   
** Some explanations how to read the makeDb files: [http://hgwdev.cse.ucsc.edu/~kent/src/unzipped/hg/doc/bashVsCsh.txt bashVsCsh]  
** Some explanations how to read the makeDb files: [http://hgwdev.soe.ucsc.edu/~kent/src/unzipped/hg/doc/bashVsCsh.txt bashVsCsh]  
** The [http://hgwdev.soe.ucsc.edu/~kent/src/unzipped/hg/makeDb/trackDb/ trackDb-files] are the "track database" of the browser. They include all instructions on how annotations are displayed (e.g. the type of display, colors, settings, options, filenames, etc.)
** [[Implementation_Notes|Notes on the history of the internal tools: Autosql. Blastz. Chains and nets]]
** [[Implementation_Notes|Notes on the history of the internal tools: Autosql. Blastz. Chains and nets]]
* Whole-genome multiple alignments ([[Chains_Nets]])
* Whole-genome multiple alignments ([[Chains_Nets]])
Line 64: Line 92:
** Whole-genome alignment pipeline: [[Chains_Nets|Angies mental model]] and [[Whole genome alignment howto|Max's howto]]
** Whole-genome alignment pipeline: [[Chains_Nets|Angies mental model]] and [[Whole genome alignment howto|Max's howto]]
* [http://genome.ucsc.edu/FAQ/FAQlicense#license4 How to create a browser for a new genome from scratch]
* [http://genome.ucsc.edu/FAQ/FAQlicense#license4 How to create a browser for a new genome from scratch]
* [http://users.soe.ucsc.edu/~markd/genbank-update/ Genbank Updates]
* Cross-links to other databases
** Outlinks for genes are copied from uniProt. The whole uniProt update process is rather complicated, see [[UCSC_Genes_Staging_Process]]
== Developing with the UCSC API ==
* [https://lists.soe.ucsc.edu/pipermail/genome-mirror/2010-March/001677.html Debug the cgi-scripts with GDB]
* Understand the [http://biostar.stackexchange.com/questions/3669/is-there-such-a-thing-as-a-ucsc-api/3723#3723 binning scheme]


== Statistics, overviews ==
== Statistics, overviews ==
*[[Gene_Set_Summary_Statistics]]
*[[Gene_Set_Summary_Statistics]]
* [http://genome-test.cse.ucsc.edu/~hiram/WEBStats/ Web hit statistics of the UCSC browser] (does anyone know if Ensembl is publishing their webhits?)
* [http://genome-test.soe.ucsc.edu/~hiram/WEBStats/ Web hit statistics of the UCSC browser]  


[[Category:Technical FAQ]]
[[Category:Technical FAQ]]

Latest revision as of 07:30, 1 September 2018

This page links to various pages that will help you to learn more about the UCSC Genome Browser. It is sorted by increasing technical complexity: the first steps require only a webbrowser, the last ones administrator access to a Linux webserver.

Use the browser website

  • Bob Kuhn is giving workshops throughout the year at several US conferences, in Europe, Australia and Asia
  • Video tutorials and slides by Openhelix
  • An introduction on Nature Education
  • Basic materials written by the gurus: The User's guide and the FAQs.
  • View a gallery of browser sessions that highlight interesting data sets.
  • Type in (manually) a couple of custom tracks in different formats (just try bed for a start, you mostly won't need the others)
  • Subscribe to the mailing list or search through it. Most everyday questions have already been asked by someone else so searching gives you an answer usually faster than asking on the mailing list.

Download the data of the genome browser (sequences and annotations)

  • Be aware that internal coordinates (not those shown on the website) are 0-based! (only exception: wiggle)
  • Unlike Gbrowse and Ensembl, UCSC is storing the data partially in SQL (coordiantes, outline of x-y-plots) and partially in flat text files (sequences, alignments, details of x-y-plots)
  • Table Browser: The easiest way to access data (you don't have to care whether data is stored in MySQL or in textfiles):
  • SQL-stored data (FAQ):
  • Flat-file data: Download from the ftp server (stored in /gbdb on browser servers)

Install a copy of the browser on your own machine (Unix or Mac)

Compile the UCSC source tree and analyze genomes yourself

Modify your own copy of the browser

Making Of: How the UCSC genome annotations are created

Developing with the UCSC API

Statistics, overviews

Retrieved from "https://genomewiki.ucsc.edu/index.php?title=Learn_about_the_Browser&oldid=24959"