Emergency Backup BLAT Servers

Overview

In case certain blat servers go down, there is a a backup blat server ("blatx") that has ports pre-configured to host the "top 5" blatted assemblies (hg38, hg19, hg18, mm10, mm9). If the current blat server(s) hosting any of these 5 assemblies goes down, manually switch to the specified fail-over server in the list below by updating the hgcentral.blatServers table. A common solution is for the admins to restart the blat server that has gone down.

List of current backup "blatx" server ports

1. db: hg38
2. host: blatx
3. trans port: 17786 isTrans=1, canPcr=0
4. untrans port: 17787 isTrans=0, canPcr=1

1. db: hg19
2. host: blatx
3. trans port: 17778 isTrans=1, canPcr=0
4. untrans port: 17779 isTrans=0, canPcr=1

1. db: hg18
2. host: blatx
3. trans port: 17782 isTrans=1, canPcr=0
4. untrans port: 17783 isTrans=0, canPcr=1

1. db: mm10
2. host: blatx
3. trans port: 17780 isTrans=1, canPcr=0
4. untrans port: 17781 isTrans=0, canPcr=1

1. db: mm10
2. host: blatx
3. trans port: 17784 isTrans=1, canPcr=0
4. untrans port: 17785 isTrans=0, canPcr=1

"mm10 blat is down!" Example for QA to follow when moving a down-blat-server assembly to emergency blatx

How is mm10 blat currently configured?

hgsql -h genome-centdb -e "select * from blatServers where db='mm10'" hgcentral
+------+--------+-------+---------+--------+
| db   | host   | port  | isTrans | canPcr |
+------+--------+-------+---------+--------+
| mm10 | blat1d | 17779 |       0 |      1 |
| mm10 | blat1d | 17778 |       1 |      0 |
+------+--------+-------+---------+--------+

Update the blatServers table to point mm10 to the backup "blatx" server:

hgsql -h genome-centdb -e "update blatServers set host='blatx' where db='mm10'" hgcentral
hgsql -h genome-centdb -e "update blatServers set port='17780' where db='mm10' and isTrans='1'" hgcentral
hgsql -h genome-centdb -e "update blatServers set port='17781' where db='mm10' and isTrans='0'" hgcentral

Check the change:

hgsql -h genome-centdb -e "select * from blatServers where db='mm10'" hgcentral
+------+-------+-------+---------+--------+
| db   | host  | port  | isTrans | canPcr |
+------+-------+-------+---------+--------+
| mm10 | blatx | 17781 |       0 |      1 |
| mm10 | blatx | 17780 |       1 |      0 |
+------+-------+-------+---------+--------+

Mm10 blat should now be fixed (is on the emergency server), go to the RR and do a blat test to be sure.

When the failed blat server has been fixed, roll back to it. In this example, move mm10 from blatx back to blat1d:

hgsql -h genome-centdb -e "update blatServers set host='blat1d' where db='mm10'" hgcentral
hgsql -h genome-centdb -e "update blatServers set port='17778' where db='mm10' and isTrans='1'" hgcentral
hgsql -h genome-centdb -e "update blatServers set port='17779' where db='mm10' and isTrans='0'" hgcentral

mm10 blat should now be fixed on the original blat server, be sure to go to the RR and do another blat test.

Test with gfServer

A quick command-line test example:

gfClient blatx.soe.ucsc.edu 17845 /gbdb/strPur4/  test.fa dnaTestOut.psl

About gfServer logs

As of 1/25/18: Haifang says: I just upgraded gfServer on blatx and added two new options, -debugLog -seqLog.

>ssh qateam@blatx.soe.ucsc.edu

Chris Lee says:

debugLog and seqLog are options on gfServer, not gfClient. You can see the additional lines in the gfServer logs on the blatx machine:

[qateam@blatx.soe.ucsc.edu /home/qateam] tail -15 /scratch/strPur4/gfServer.log 
>query
gcccagctaaaaccaatcaccttcaagggataaaggttcatccaaatatg
taaaaaccgactgaagacagcttagttagaaacgtgtgtgggatcaacta
tggcgttgccgttgactagatctagatccctacttttttttgttctcaat
acggtatgtagcttggtatccgcctctgacaatgatcccagccctagctt
ctgatcatgatctggcacaaaatcaatgtaggtctagatctagatctatc
cgacttcgtaaacaacatagcctatttctttatcgttgaactatgagttt
atgactaaaacattcttcttcttcttcttctt
2018/01/25 09:58:52: debug: 1711519 101 clumps, 36593 hits
2018/01/25 09:59:18: info: gfServer version 36x2 on host blat4d, port 17845 connection from 132.249.245.79
2018/01/25 09:59:18: warn: Zero sized query
2018/01/25 10:00:12: info: gfServer version 36x2 on host blat4d, port 17845 connection from 132.249.245.79
2018/01/25 10:00:12: debug: 0ddf270562684f29pcr cctacattgattttgtgccagatcatgatcagaagctagggctgggatca gcccagctaaaaccaatcaccttcaagggataaaggttcatccaaat 4000
2018/01/25 10:00:12: debug: 1791616 PCR cctacattgattttgtgccagatcatgatcagaagctagggctgggatca gcccagctaaaaccaatcaccttcaagggataaaggttcatccaaat 63 clumps

Compared to a gfServer running without those options, like on blat1a:
[qateam@blat1a.cse.ucsc.edu /scratch/bosTau7] tail gfServer.log 
2018/01/25 07:24:13: info: gfServer version 34x12 on host blat1a, port 17845 connection from 128.114.119.132
2018/01/25 08:00:02: info: gfServer version 34x12 on host blat1a, port 17845 connection from 132.249.245.79
.... # more of the same