Reverse proxying socket.io websocket with URL rewriting - nginx

Question

I am trying to reverse proxy several socket.io websockets and also rewrite the URLs so that I can have a front end server that can connect to several backend tty.js instances.
For example:
http://mysite.com/server1 reverse proxies to http://server1/ which is running tty.js
http://mysite.com/server2 reverse proxies to http://server2/ which is running tty.js
For HTTP any old proxy works fine. However when I rewrite the URLs instead of the websocket looking like:
http://mysite.com/server1/socket.io/1/?t=1354135029745 -> http://server1/socket.io/1/?t=1354135029745
it looks like:
http://mysite.com/socket.io/1/?t=1354135029745
This of course goes unhandled. I assume the rewrite happens and so the socket assumes it's "connection" is at /socket.io/1/?t=1354135029745 instead of /server1/socket.io/1/?t=1354135029745
Is there some way to do URL rewriting as well as proxy a websocket?
So far I have tried both Varnish, node-http-proxy and Nginx using the TCP proxy module and I haven't been able to figure out how to get this to work.
Right now my Varnish config looks like:
backend backend1 {
.host = "1.1.1.1";
.port = "8080";
.connect_timeout = 1s;
.between_bytes_timeout = 60s;
.max_connections = 800;
}
backend backend2 {
.host = "2.2.2.2";
.port = "8080";
.connect_timeout = 1s;
.between_bytes_timeout = 60s;
.max_connections = 800;
}
sub vcl_recv {
set req.grace = 120s
if(req.url ~ "^/server1/") {
set req.url = regsub(req.url, "^/server1/", "/");
set req.backend = backend1;
return(pipe);
}
if(req.url ~ "^/server2/") {
set req.url = regsub(req.url, "^/server2/", "/");
set req.backend = backend2;
return(pipe);
}
}
sub vcl_pipe {
if(req.http.upgrade) {
set bereq.http.upgrade = req.http.upgrade;
} else {
set bereq.http.connection = "close";
}
return(pipe);
}
I am forced to leave out the obligatory:
if(req.url ~ "^/socket.io/") {
set req.backend = backend1;
return(pipe);
}
since I have multiple backends. Is there a way to get this to work using node-http-proxy, Nginx or Varnish? Or if none of those have the ability, would something like haproxy or really anything else be able to?
My first inclination would be to set a custom header so when the URL does get rewritten it knows which backend it should hit, but I'm not sure if/how that would work.
Edit 1: I was able to sort of fix this issues by also pulling the req.http.referer and adding that to the socket.io statement:
if(req.url ~ "^/socket.io/" && req.http.referer ~ "server1"){
set req.backend = backend1;
return(pipe);
}
However it appears to be a little wonky and seems to revert socket.io to xhr polling which is not ideal but better than nothing.
Edit 2: After testing it out with a few backends what happens is once long polling starts it breaks the rest of the proxy. As soon as I refresh the page now the reverse proxy instead of going:
http://mysite.com/server1 -> http://server1
goes:
http://mysite.com/ -> http://server1
until I reload varnish and then goes back.

Answer 1

Most likely you also need to change the Host: header as well in your vcl_recv():
sub vcl_recv {
set req.grace = 120s
if(req.url ~ "^/server1/") {
set req.url = regsub(req.url, "^/server1/", "/");
set req.http.host = "server1";
set req.backend = backend1;
return(pipe);
}
if(req.url ~ "^/server2/") {
set req.url = regsub(req.url, "^/server2/", "/");
set req.http.host = "server2";
set req.backend = backend2;
return(pipe);
}
}
That should take care of your initial problem.
However using pipe in Varnish has it's pitfalls which you might want to consider. The most important of which is the lack of X-Forwarded-For header after the first request.