Euronode: Difference between revisions

From Genecats
Jump to navigationJump to search
No edit summary
(Undo revision 4750 by Daniel (talk))
 
(10 intermediate revisions by 4 users not shown)
Line 5: Line 5:
We are moving to a system where we have geographically specific servers (nodes); we will try to direct users to the node which is appropriate for them. I call this geoMirroring to avoid confusion with our traditional mirrors.
We are moving to a system where we have geographically specific servers (nodes); we will try to direct users to the node which is appropriate for them. I call this geoMirroring to avoid confusion with our traditional mirrors.


The node for a given server is specified in the hg.conf file; e.g. the machines in california will have the following in their hg.conf file:
The node for a given server is specified in the hg.conf file; e.g. the machines in California will have the following in their hg.conf file:


  browser.node=1
  browser.node=1
Line 13: Line 13:
  browser.node=2
  browser.node=2


We may add other nodes in the future (e.g. in asia). If this configuration variable is not found, the redirection code is not executed (e.g. this will be the case in traditional mirrors).
We may add other nodes in the future (e.g. in asia). If this configuration variable is not found, the redirection code is not executed (this will be the case in traditional mirrors).


In the following discussion, <tt>hgcentral</tt> applies to RR, <tt>hgcentraltest</tt> applies to hgwdev. RR will include euronode.cse.ucsc.edu when we activate the euronode.
In the following discussion, <tt>hgcentral</tt> applies to RR (the relevant table for testing on hgwdev is <tt>hgcentraltest</tt>). RR will include euronode.cse.ucsc.edu when we activate the euronode.


The table <tt>hgcentral.gbNode</tt> contains the metadata for each node (e.g. domain, country and description). This is a copy of <tt>hgFixed.gbNodeReal</tt> (so we can modify on hgwdev).
The table <tt>hgcentral.gbNode</tt> contains the metadata for each node (e.g. domain, country and description). This is a copy of <tt>hgFixed.gbNodeReal</tt> (so we can modify on hgwdev).


The table <tt>hgcentral.geoIpNode</tt> is used to map a user's IP to the appropriate node. If an IP is not found in geoIpNode, the system assumes the user belongs in node 1 (California). hgcentral.geoIpNode is based on <tt>hgFixed.geoIpNode</tt>, which was built by Galt using a publicly available IP => country mapping database (http://software77.net/geo-ip). The copy in hgcentral may be modified (e.g. to add the IP used by QA team for testing).
The table <tt>hgcentral.geoIpNode</tt> is used to map a user's IP to the appropriate node. If an IP is not found in geoIpNode, the system assumes the user belongs in node 1 (California). hgcentral.geoIpNode is based on <tt>hgFixed.geoIpNodeReal</tt>, which was built by Galt using a publicly available IP => country mapping database (http://software77.net/geo-ip). The copy in hgcentral may be modified (e.g. to add the IP used by QA team for testing).


Note that the IPs are stored in the tables using a numeric representation (for speed). mysql has a built in function, <tt>INET_ATON</tt>, which can be used to convert the dotted-quad notation to the numeric form; e.g. to add a new entry to geoIpNode:
Note that the IPs are stored in the tables using a numeric representation (for speed). mysql has a built in function, <tt>INET_ATON</tt>, which can be used to convert the dotted-quad notation to the numeric form; e.g. to add a new entry to geoIpNode:
Line 29: Line 29:
== When does redirection happen? ==
== When does redirection happen? ==


An important limitation of these mirrors is that sessions are NOT shared between the different nodes.
If a european goes to <tt>http://genome.ucsc.edu/cgi-bin/hgGateway</tt>, they will be redirected to <tt>http://genome-euro.ucsc.edu/cgi-bin/hgGateway</tt>. If a non-european goes to <tt>http://genome-euro.ucsc.edu/cgi-bin/hgGateway</tt>, they will be redirected to <tt>http://genome.ucsc.edu/cgi-bin/hgGateway</tt>.


Users are only redirected via hgGateway; e.g. if a european goes directly to <tt>http://genome.ucsc.edu/cgi-bin/hgTracks</tt>, they will not be redirected back to europe. The rationale for this is that such links may include sessions that are specific to the California mirror.
Users are only redirected via hgGateway; e.g. if a european goes directly to <tt>http://genome.ucsc.edu/cgi-bin/hgTracks</tt>, they will not be redirected back to europe. The rationale for this is that such links may include sessions that are specific to the California mirror.


The first time a user is redirected, they will see a little dialog box at the top of hgGateway that let's them override the redirect; e.g. if user X is redirected to euronode, they may click on the link in the box to get back to the Californian hgGateway. This action is remembered with a cookie stored in the user's web browser (the value of the cookie will be <tt>redirect=manual</tt>). This cookie is not cleared by a cartReset (this is intentional). The user will always be able to switch which mirror they are using via a to-be-designed UI.
The first time a user is redirected, they will see a little dialog box at the top of hgGateway that let's them override the redirect (see screenshot below); e.g. if user X is redirected to euronode, they may click on the link in the box to get back to the Californian hgGateway. This manual redirect back to California is remembered with a cookie stored in the user's web browser (the value of the cookie will be <tt>redirect=manual</tt>). This cookie is not cleared by a cartReset (this is intentional). The user will always be able to switch which mirror they are using via a to-be-designed UI.


[[File:euronode.jpg]]
An important limitation of the geo-mirrors is that sessions are NOT shared between the different nodes.
 
[[File:euronode.jpg|800px|center]]


== How do I test this? ==
== How do I test this? ==


(THE following paragraph is outdated and no longer needed?)
We are putting the IP of hgroaming.cse.ucsc.edu (128.114.49.36) into the geoIpNode table as node == 2. So if you configure your web browser to use the hgroaming proxy, the site will "think" you are coming from europe. You can contact cluster-admin to get an hgroaming account.
We are putting the IP of hgroaming.cse.ucsc.edu (128.114.49.36) into the geoIpNode table as node == 2. So if you configure your web browser to use the hgroaming proxy, the site will "think" you are coming from europe. You can contact cluster-admin to get an hgroaming account.
You can use the kent/src utility '''geoMirrorNode''' to determine the ip => node mapping for a list of hosts or ip addresses.
New testing feature: browser.geoSuffix=
You can make a test say between demo1 and demo2 by adding browser.geoSuffix=Test
or some value that will be appended to the regular tables gbNode and geoIpNode
so that it really uses gbNodeTest and geoIpNodeTest which can be set up
however you like.  gbNodeTest should contain a copy of gbNode that has the
domain field (at least) modified to point to hgwdev-demo1.cse.ucs.edu and hgwdev-demo2.cse.ucs.edu
respectively.  If somebody else also needs to test redirection, simply create
copies of the two tables with a different suffix and use that in the geoSuffix setting.
One good trick (ONLY FOR THE TEST VERSION OF THE TABLES and not geoIpNode table itself)
is to do delete all rows from geoIpNodeTest (or whatever your suffix is) and then
simply insert a row for the specific desktops that you wish them to be seen as being in node 2:
hgsql hgcentraltest -e "insert into geoIpNodeTest VALUES (INET_ATON('128.114.49.36'), INET_ATON('128.114.49.36'), 2)"
where '128.114.49.36' above should be replaced the ip address of your test box that
you are running your browser from to test the redirection. 
You can use hostname <mybox> to find out your IP address.
Note: if the geoIpNode{geoSuffix} table is empty or does not contain any matching row
whose range encompasses your IP, then it just treats the IP as node=1, the default.
Cookies: You can find cookies for "ucsc.edu" in FireFox or whatever browser
that relate to the redirection, it is called "redirect".  You can delete the "redirect"
cookie to reset your test browser back to test again as if it were the first time.
In the case of FireFox, you can actually just leave the cookie-editing window open
for easy cookie access if you need to test and delete the redirect cookie repeateadly.

Latest revision as of 16:43, 29 August 2018


This pages describes the way the euronode mirror works

We are moving to a system where we have geographically specific servers (nodes); we will try to direct users to the node which is appropriate for them. I call this geoMirroring to avoid confusion with our traditional mirrors.

The node for a given server is specified in the hg.conf file; e.g. the machines in California will have the following in their hg.conf file:

browser.node=1

The euronode will have:

browser.node=2

We may add other nodes in the future (e.g. in asia). If this configuration variable is not found, the redirection code is not executed (this will be the case in traditional mirrors).

In the following discussion, hgcentral applies to RR (the relevant table for testing on hgwdev is hgcentraltest). RR will include euronode.cse.ucsc.edu when we activate the euronode.

The table hgcentral.gbNode contains the metadata for each node (e.g. domain, country and description). This is a copy of hgFixed.gbNodeReal (so we can modify on hgwdev).

The table hgcentral.geoIpNode is used to map a user's IP to the appropriate node. If an IP is not found in geoIpNode, the system assumes the user belongs in node 1 (California). hgcentral.geoIpNode is based on hgFixed.geoIpNodeReal, which was built by Galt using a publicly available IP => country mapping database (http://software77.net/geo-ip). The copy in hgcentral may be modified (e.g. to add the IP used by QA team for testing).

Note that the IPs are stored in the tables using a numeric representation (for speed). mysql has a built in function, INET_ATON, which can be used to convert the dotted-quad notation to the numeric form; e.g. to add a new entry to geoIpNode:

hgsql hgcentral -s -e "insert into geoIpNode VALUES (INET_ATON('128.114.49.36'), INET_ATON('128.114.49.36'), 2)"

The built-in function INET_NTOA can be used to convert back from the numeric representation to the dotted-quad notation.

When does redirection happen?

If a european goes to http://genome.ucsc.edu/cgi-bin/hgGateway, they will be redirected to http://genome-euro.ucsc.edu/cgi-bin/hgGateway. If a non-european goes to http://genome-euro.ucsc.edu/cgi-bin/hgGateway, they will be redirected to http://genome.ucsc.edu/cgi-bin/hgGateway.

Users are only redirected via hgGateway; e.g. if a european goes directly to http://genome.ucsc.edu/cgi-bin/hgTracks, they will not be redirected back to europe. The rationale for this is that such links may include sessions that are specific to the California mirror.

The first time a user is redirected, they will see a little dialog box at the top of hgGateway that let's them override the redirect (see screenshot below); e.g. if user X is redirected to euronode, they may click on the link in the box to get back to the Californian hgGateway. This manual redirect back to California is remembered with a cookie stored in the user's web browser (the value of the cookie will be redirect=manual). This cookie is not cleared by a cartReset (this is intentional). The user will always be able to switch which mirror they are using via a to-be-designed UI.

An important limitation of the geo-mirrors is that sessions are NOT shared between the different nodes.

Euronode.jpg

How do I test this?

(THE following paragraph is outdated and no longer needed?) We are putting the IP of hgroaming.cse.ucsc.edu (128.114.49.36) into the geoIpNode table as node == 2. So if you configure your web browser to use the hgroaming proxy, the site will "think" you are coming from europe. You can contact cluster-admin to get an hgroaming account.

You can use the kent/src utility geoMirrorNode to determine the ip => node mapping for a list of hosts or ip addresses.

New testing feature: browser.geoSuffix=

You can make a test say between demo1 and demo2 by adding browser.geoSuffix=Test or some value that will be appended to the regular tables gbNode and geoIpNode so that it really uses gbNodeTest and geoIpNodeTest which can be set up however you like. gbNodeTest should contain a copy of gbNode that has the domain field (at least) modified to point to hgwdev-demo1.cse.ucs.edu and hgwdev-demo2.cse.ucs.edu respectively. If somebody else also needs to test redirection, simply create copies of the two tables with a different suffix and use that in the geoSuffix setting.

One good trick (ONLY FOR THE TEST VERSION OF THE TABLES and not geoIpNode table itself) is to do delete all rows from geoIpNodeTest (or whatever your suffix is) and then simply insert a row for the specific desktops that you wish them to be seen as being in node 2:

hgsql hgcentraltest -e "insert into geoIpNodeTest VALUES (INET_ATON('128.114.49.36'), INET_ATON('128.114.49.36'), 2)"

where '128.114.49.36' above should be replaced the ip address of your test box that you are running your browser from to test the redirection. You can use hostname <mybox> to find out your IP address.

Note: if the geoIpNode{geoSuffix} table is empty or does not contain any matching row whose range encompasses your IP, then it just treats the IP as node=1, the default.

Cookies: You can find cookies for "ucsc.edu" in FireFox or whatever browser that relate to the redirection, it is called "redirect". You can delete the "redirect" cookie to reset your test browser back to test again as if it were the first time. In the case of FireFox, you can actually just leave the cookie-editing window open for easy cookie access if you need to test and delete the redirect cookie repeateadly.