Emergency Backup BLAT Servers

Overview

In case certain blat servers go down, there is a a backup blat server ("blatx") that has ports pre-configured to host the "top 5" blatted assemblies (hg38, hg19, hg18, mm10, mm9). If the current blat server(s) hosting any of these 5 assemblies goes down, manually switch to the specified fail-over server in the list below by updating the hgcentral.blatServers table. A common solution is for the admins to restart the blat server that has gone down.

List of current backup "blatx" server ports

NOTE this information could be outdated, be sure to check things work.

hg38

host: blatx

trans port: 17786 isTrans=1, canPcr=0

untrans port: 17787 isTrans=0, canPcr=1

hg19

host: blatx

trans port: 17778 isTrans=1, canPcr=0

untrans port: 17779 isTrans=0, canPcr=1

hg18

host: blatx

trans port: 17782 isTrans=1, canPcr=0

untrans port: 17783 isTrans=0, canPcr=1

mm10

host: blatx

trans port: 17780 isTrans=1, canPcr=0

untrans port: 17781 isTrans=0, canPcr=1

mm9

host: blatx

trans port: 17784 isTrans=1, canPcr=0

untrans port: 17785 isTrans=0, canPcr=1

dm6

host: blatx

trans port: 17874 isTrans=1, canPcr=0

untrans port: 17875 isTrans=0, canPcr=1'

dm3

host: blatx

trans port: 17866 isTrans=1, canPcr=0

untrans port: 17867 isTrans=0, canPcr=1

rn6

host: blatx

trans port: 17868 isTrans=1, canPcr=0

untrans port: 17869 isTrans=0, canPcr=1'

danRer11

host: blatx

trans port: 17872 isTrans=1, canPcr=0

untrans port: 17873 isTrans=0, canPcr=1

danRer10

host: blatx

trans port: 17870 isTrans=1, canPcr=0

untrans port: 17871 isTrans=0, canPcr=1

danRer7

host: blatx

trans port: 17864 isTrans=1, canPcr=0

untrans port: 17865 isTrans=0, canPcr=1

"mm10 blat is down!" Example for QA to follow when moving a down-blat-server assembly to emergency blatx

How is mm10 blat currently configured?

hgsql -h genome-centdb -e "select * from blatServers where db='mm10'" hgcentral
+------+--------+-------+---------+--------+
| db   | host   | port  | isTrans | canPcr |
+------+--------+-------+---------+--------+
| mm10 | blat1d | 17779 |       0 |      1 |
| mm10 | blat1d | 17778 |       1 |      0 |
+------+--------+-------+---------+--------+

Update the blatServers table to point mm10 to the backup "blatx" server:

hgsql -h genome-centdb -e "update blatServers set host='blatx' where db='mm10'" hgcentral
hgsql -h genome-centdb -e "update blatServers set port='17780' where db='mm10' and isTrans='1'" hgcentral
hgsql -h genome-centdb -e "update blatServers set port='17781' where db='mm10' and isTrans='0'" hgcentral

Check the change:

hgsql -h genome-centdb -e "select * from blatServers where db='mm10'" hgcentral
+------+-------+-------+---------+--------+
| db   | host  | port  | isTrans | canPcr |
+------+-------+-------+---------+--------+
| mm10 | blatx | 17781 |       0 |      1 |
| mm10 | blatx | 17780 |       1 |      0 |
+------+-------+-------+---------+--------+

Mm10 blat should now be fixed (is on the emergency server), go to the RR and do a blat test to be sure.

When the failed blat server has been fixed, roll back to it. In this example, move mm10 from blatx back to blat1d:

hgsql -h genome-centdb -e "update blatServers set host='blat1d' where db='mm10'" hgcentral
hgsql -h genome-centdb -e "update blatServers set port='17778' where db='mm10' and isTrans='1'" hgcentral
hgsql -h genome-centdb -e "update blatServers set port='17779' where db='mm10' and isTrans='0'" hgcentral

mm10 blat should now be fixed on the original blat server, be sure to go to the RR and do another blat test.

Example return of hg38 to normal state

select * from blatServers where db='hg38';
update blatServers set host='blat1d', port=17902 where db='hg38' and isTrans=1;
update blatServers set host='blat1d', port=17903 where db='hg38' and isTrans=0;

By doing this directly (instead of with -e command) you can see a report like:

Query OK, 1 row affected (0.00 sec)

Rows matched: 1 Changed: 1 Warnings: 0

This is can comfort you to know you were only changing the row you meant to change. Note above ports may change.

Test with gfServer

A quick command-line test example:

gfClient blatx.soe.ucsc.edu 17845 /gbdb/strPur4/  test.fa dnaTestOut.psl

About gfServer logs

As of 1/25/18: Haifang says: I just upgraded gfServer on blatx and added two new options, -debugLog -seqLog. See more about revieing gfServer.log here.

>ssh qateam@blatx.soe.ucsc.edu

Chris Lee says:

debugLog and seqLog are options on gfServer, not gfClient. You can see the additional lines in the gfServer logs on the blatx machine:

[qateam@blatx.soe.ucsc.edu /home/qateam] tail -15 /scratch/strPur4/gfServer.log 
>query
gcccagctaaaaccaatcaccttcaagggataaaggttcatccaaatatg
taaaaaccgactgaagacagcttagttagaaacgtgtgtgggatcaacta
tggcgttgccgttgactagatctagatccctacttttttttgttctcaat
acggtatgtagcttggtatccgcctctgacaatgatcccagccctagctt
ctgatcatgatctggcacaaaatcaatgtaggtctagatctagatctatc
cgacttcgtaaacaacatagcctatttctttatcgttgaactatgagttt
atgactaaaacattcttcttcttcttcttctt
2018/01/25 09:58:52: debug: 1711519 101 clumps, 36593 hits
2018/01/25 09:59:18: info: gfServer version 36x2 on host blat4d, port 17845 connection from 132.249.245.79
2018/01/25 09:59:18: warn: Zero sized query
2018/01/25 10:00:12: info: gfServer version 36x2 on host blat4d, port 17845 connection from 132.249.245.79
2018/01/25 10:00:12: debug: 0ddf270562684f29pcr cctacattgattttgtgccagatcatgatcagaagctagggctgggatca gcccagctaaaaccaatcaccttcaagggataaaggttcatccaaat 4000
2018/01/25 10:00:12: debug: 1791616 PCR cctacattgattttgtgccagatcatgatcagaagctagggctgggatca gcccagctaaaaccaatcaccttcaagggataaaggttcatccaaat 63 clumps

Compared to a gfServer running without those options, like on blat1a:
[qateam@blat1a.soe.ucsc.edu /scratch/bosTau7] tail gfServer.log 
2018/01/25 07:24:13: info: gfServer version 34x12 on host blat1a, port 17845 connection from 128.114.119.132
2018/01/25 08:00:02: info: gfServer version 34x12 on host blat1a, port 17845 connection from 132.249.245.79
.... # more of the same

(Note: 128.114.198.32 is the new hgwdev, in the above example 132.249.245.79 was the old hgwdev address)

List of current original "blat1*" server ports

See the hgcentral backups: https://genecats.gi.ucsc.edu/qa/test-results/hgcentral/

You can navigate to the blatServers table and see the history for the specific tables.