KNETFILE HOOKS: Difference between revisions
(Added samtools-0.1.{9,10,11}) |
(Adding tabix KNETFILE_HOOKS patch.) |
||
| Line 1: | Line 1: | ||
==Description== | ==Description== | ||
This is a description of patch files for the samtools C library | This is a description of patch files for the samtools C library and tabix C library (both available from [http://samtools.sourceforge.net/ samtools.sourceforge.net]) that add a layer of indirection to the external functions in knetfile.c, samtools'/tabix's network code used to fetch data from URLs of BAM or tabix-compressed .gz files <nowiki>(http://.../x.bam, ftp://.../x.bam, http://.../x.vcf.gz)</nowiki>. | ||
(samtools.sourceforge.net) that add a layer of indirection to | |||
the external functions in knetfile.c, samtools' network code | Samtools and tabix contain separate knetfile code; if you link against both libraries, then both (or neither) must be patched using their respective patch files. | ||
used to fetch data from URLs of BAM files <nowiki>(http://.../x.bam, | |||
ftp://.../x.bam)</nowiki>. | |||
The purpose of the new "hooks" into knetfile.c is to enable alternate | The purpose of the new "hooks" into knetfile.c is to enable alternate | ||
implementations of knetfile functionality to be substituted for the | implementations of knetfile functionality to be substituted for the | ||
samtools knetfile implementation. The UCSC Genome Browser source code | samtools (and/or tabix) knetfile implementation. The UCSC Genome Browser source code | ||
has network code that supports https and authentication in addition to | has network code that supports https and authentication in addition to | ||
basic http and ftp ([http://hgwdev.cse.ucsc.edu/~kent/src/unzipped/lib/net.c net.c]), and URL data caching code | basic http and ftp ([http://hgwdev.cse.ucsc.edu/~kent/src/unzipped/lib/net.c net.c]), and URL data caching code | ||
| Line 24: | Line 22: | ||
[[Getting_Started_With_Git]] | [[Getting_Started_With_Git]] | ||
==Patch files== | ==Patch files for samtools== | ||
[[Image:Knetfile_hooks.0.1.7.patch]] | [[Image:Knetfile_hooks.0.1.7.patch]] | ||
[[Image:Knetfile_hooks.0.1.8.patch]] | [[Image:Knetfile_hooks.0.1.8.patch]] | ||
| Line 31: | Line 29: | ||
[[Image:Knetfile_hooks.0.1.11.patch]] | [[Image:Knetfile_hooks.0.1.11.patch]] | ||
NOTE: the patch files include one other UCSC-specific change which you | ==Patch files for tabix== | ||
[[Image:Knetfile_hooks_tabix-0.2.3.patch]] | |||
NOTE: the samtools patch files include one other UCSC-specific change which you | |||
may or may not want to keep: they disable the check for empty BZGF record | may or may not want to keep: they disable the check for empty BZGF record | ||
at the end of the BAM file (bgzf_check_EOF). The check adds an extra | at the end of the BAM file (bgzf_check_EOF). The check adds an extra | ||
socket open delay (or for FTP, two extra socket opens). If you want to | socket open delay (or for FTP, two extra socket opens). If you want to | ||
keep the EOF check, edit the patch file to remove the bam.c section. | keep the EOF check, edit the patch file to remove the bam.c section. | ||
NOTE: the tabix patch files include another set of changes to allow linking against both samtools and tabix libraries: tabix's bgzf_* functions are renamed to ti_bgzf_* in order to avoid linking conflicts with samtools' bgzf_* functions, which are nearly identical but not completely. | |||
==Apply Patch== | ==Apply Patch== | ||
To apply the patch, start with a clean samtools build directory, and | To apply the patch, start with a clean samtools (or tabix) build directory, and | ||
run these commands: | run these commands: | ||
# samtools: | |||
patch -p1 < knetfile_hooks.0.1.11.patch | patch -p1 < knetfile_hooks.0.1.11.patch | ||
make | make | ||
(Replace 0.1.11 with the appropriate version number if necessary.) | (Replace 0.1.11 with the appropriate version number if necessary.) | ||
# tabix: | |||
patch -p1 < knetfile_hooks_tabix-0.2.3.patch | |||
==samtools-0.1.7== | ==samtools-0.1.7== | ||
| Line 76: | Line 83: | ||
[[Image:Knetfile_hooks.0.1.11.patch]] is nearly identical to 0.1.9 | [[Image:Knetfile_hooks.0.1.11.patch]] is nearly identical to 0.1.9 | ||
and 0.1.10 but for some line numbers. | and 0.1.10 but for some line numbers. | ||
==tabix-0.2.3== | |||
[[Image:Knetfile_hooks_tabix-0.2.3.patch]] is the first patch available for tabix. | |||
Revision as of 16:32, 16 March 2011
Description
This is a description of patch files for the samtools C library and tabix C library (both available from samtools.sourceforge.net) that add a layer of indirection to the external functions in knetfile.c, samtools'/tabix's network code used to fetch data from URLs of BAM or tabix-compressed .gz files (http://.../x.bam, ftp://.../x.bam, http://.../x.vcf.gz).
Samtools and tabix contain separate knetfile code; if you link against both libraries, then both (or neither) must be patched using their respective patch files.
The purpose of the new "hooks" into knetfile.c is to enable alternate implementations of knetfile functionality to be substituted for the samtools (and/or tabix) knetfile implementation. The UCSC Genome Browser source code has network code that supports https and authentication in addition to basic http and ftp (net.c), and URL data caching code (udc.c) that saves accessed portions of URLs on local disk, while checking for remote file updates. When samtools is rebuilt with this patch, the UCSC source code can be rebuilt with environment variable KNETFILE_HOOKS=1 (in addition to USE_BAM=1, see Build Environment Variables to take advantage of the new hooks in knetfile. Samtools calls knetfile functions as usual, but now those knetfile functions call functions from a UCSC wrapper on udc functionality (knetUdc.c).
UCSC source code
UCSC Genome Browser code is available via git Getting_Started_With_Git
Patch files for samtools
File:Knetfile hooks.0.1.7.patch File:Knetfile hooks.0.1.8.patch File:Knetfile hooks.0.1.9.patch File:Knetfile hooks.0.1.10.patch File:Knetfile hooks.0.1.11.patch
Patch files for tabix
File:Knetfile hooks tabix-0.2.3.patch
NOTE: the samtools patch files include one other UCSC-specific change which you may or may not want to keep: they disable the check for empty BZGF record at the end of the BAM file (bgzf_check_EOF). The check adds an extra socket open delay (or for FTP, two extra socket opens). If you want to keep the EOF check, edit the patch file to remove the bam.c section.
NOTE: the tabix patch files include another set of changes to allow linking against both samtools and tabix libraries: tabix's bgzf_* functions are renamed to ti_bgzf_* in order to avoid linking conflicts with samtools' bgzf_* functions, which are nearly identical but not completely.
Apply Patch
To apply the patch, start with a clean samtools (or tabix) build directory, and run these commands:
# samtools: patch -p1 < knetfile_hooks.0.1.11.patch make
(Replace 0.1.11 with the appropriate version number if necessary.)
# tabix: patch -p1 < knetfile_hooks_tabix-0.2.3.patch
samtools-0.1.7
File:Knetfile hooks.0.1.7.patch has been tested with samtools SVN versions 537 (~Feb '10) and 574 (latest as of May 11 2010). In SVN 574, some of the bam.c line numbers have changed by 2 since the patch file was created so patch prints out these messages:
patching file bam.c Hunk #1 succeeded at 72 with fuzz 2 (offset 1 line). Hunk #2 succeeded at 80 with fuzz 2.
-- but the patch is successful and the code builds correctly.
samtools-0.1.8
The knetfile code is fairly stable; however, bam_index has ongoing work which necessitated File:Knetfile hooks.0.1.8.patch.
samtools-0.1.9
A new subdirectory bcftools was added, and bcftools/main.c uses the macro knet_fileno() which is not supported by KNETFILE_HOOKS code. The "bcftools cat" command is implemented only when _USE_KNETFILE is defined and KENTFILE_HOOKS is *not* defined. File:Knetfile hooks.0.1.9.patch.
samtools-0.1.10
File:Knetfile hooks.0.1.10.patch is identical to 0.1.9 but is provided as a separate file for convenience.
samtools-0.1.11
File:Knetfile hooks.0.1.11.patch is nearly identical to 0.1.9 and 0.1.10 but for some line numbers.
tabix-0.2.3
File:Knetfile hooks tabix-0.2.3.patch is the first patch available for tabix.
