I'm using Go (Golang) 1.4.2 with Gorilla WebSockets behind an nginx 1.4.6 reverse proxy. My WebSockets are disconnecting after about a minute of having the page open. Same behavior occurs on Chrome and Firefox.
At first, I had problems connecting the server and client with WebSockets. Then, I read that I needed to tweak my nginx configuration. This is what I have.
server {
listen 80;
server_name example.com;
proxy_pass_header Server;
location / {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forward-Proto $scheme;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_pass http://127.0.0.1:1234;
}
}
My Go code is basically echoing back the client's message. (Errors omitted for brevity). This is my HandleFunc.
var up = websocket.Upgrader{
ReadBufferSize: 1024,
WriteBufferSize: 1024,
}
ws, _ := up.Upgrade(resp, req, nil)
defer ws.Close()
var s struct {
Foo string
Bar string
}
for {
ws.ReadJSON(&s)
ws.WriteJSON(s)
}
The JavaScript is pretty simple as well.
var ws = new WebSocket("ws://example.com/ws/");
ws.addEventListener("message", function(evnt) {
console.log(JSON.parse(evnt.data));
});
var s = {
Foo: "hello",
Bar: "world"
};
ws.send(JSON.stringify(s));
Go is reporting websocket: close 1006 unexpected EOF. I know that when I leave or refresh the page ReadJSON returns EOF, but this appears to be a different error. Also, the unexpected EOF happens by itself after about a minute of having the page open.
I have an onerror function in JavaScript. That event doesn't fire, but onclose fires instead.
I had the same issue, the problem is the nginx configuration. It defaults to a 1 minute read timeout for proxy_pass:
Syntax: proxy_read_timeout time;
Default: proxy_read_timeout 60s;
Context: http, server, location
See http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_read_timeout
In my case I've increased the timeout to 10 hours:
proxy_read_timeout 36000s;
Related
i am working with NGINX configuration right now.
I am stuck at some strange point.
NGINX allow me to connect via SSE but not via Websocket.
This is my config:
server {
listen 80;
server_name _;
location / {
proxy_pass http://127.0.0.1:5000;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection keep-alive;
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
}
This is error from browser console via WEBSOCKET:
[2022-08-30T07:49:22.667Z] Error: Failed to start the transport 'WebSockets': Error: WebSocket failed to connect. The connection could not be found on the server, either the endpoint may not be a SignalR endpoint, the connection ID is not present on the server, or there is a proxy blocking WebSockets. If you have multiple servers check that sticky sessions are enabled.
This is WEBSOCKET: Browser Error
This is SSE: SSE
##EDIT:
This is section which fail:
try {
await this._startTransport(connectUrl, requestedTransferFormat);
this.connectionId = negotiate.connectionId;
return;
}
catch (ex) {
this._logger.log(LogLevel.Error, `Failed to start the transport '${endpoint.transport}': ${ex}`);
negotiate = undefined;
transportExceptions.push(new FailedToStartTransportError(`${endpoint.transport} failed: ${ex}`, HttpTransportType[endpoint.transport]));
if (this._connectionState !== "Connecting" /* Connecting */) {
const message = "Failed to select transport before stop() was called.";
this._logger.log(LogLevel.Debug, message);
return Promise.reject(new Error(message));
}
###UPDATE
I FOUND THE SOLUTION. MY ANSWER IS CORRECT ANSWER! :)
I am comming back with the solution.
It is easy.
First thing is:
You need to add in nginx.conf in http { } section this:
map $http_upgrade $connection_upgrade
{
default upgrade;
'' close;
}
and this is config:
server {
listen 50150;
server_name _;
location / {
proxy_pass http://127.0.0.1:5000;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_cache off;
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
}
and you are good to go!
[2022-08-30T10:30:48.900Z] Information: WebSocket connected to ws:~~/~hub?id=Zk-~~.
I hope someone will get use of it.
Best regards
I have a very weird problem with socket.io and I was hoping someone can help me out.
For some reason few clients cannot connect to the server no matter what, when I am using https.
I am getting the following error code: ERR_CRYPTO_OPERATION_FAILED (see the detailed log below)
Again, most of the time the connection is perfectly fine, only some (random) clients seem to have this problem.
I have created a super simple server.js and client.js to make it easy to test.
I am using socket.io#2.4.1, and socket.io-client#2.4.0
Unfortunately version 3.x.x is not an option.
The OS is Ubuntu 18.04, both on the server, and the client side.
Nginx:
server {
listen 80;
server_name example.domain.com;
return 301 https://example.domain.com$request_uri;
}
server {
listen 443 ssl http2;
server_name example.domain.com;
ssl_certificate /path/to/cert.pem;
ssl_certificate_key /path/to/cert.key;
ssl_protocols TLSv1.2 TLSv1.3;
location /
{
proxy_pass http://127.0.0.1:8000;
include /etc/nginx/proxy_params;
}
location /socket.io {
proxy_http_version 1.1;
proxy_buffering off;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_connect_timeout 30s;
proxy_read_timeout 30s;
proxy_send_timeout 30s;
proxy_pass http://127.0.0.1:8000/socket.io;
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root /usr/share/nginx/html;
internal;
}
}
client.js:
const client = io.connect("https://example.domain.com", {
origins: '*:*',
transportOptions: {
polling: {
extraHeaders: {
'Authorization': token
}
}
},
});
Tried adding secure: true, reconnect: true, and rejectUnauthorized : false, but no difference.
Also, I tested it with and without the transportOptions.
server.js:
const port = 5000;
const app = express();
const server = app.listen(port, () => {
console.log(`Listening on port: ${port}`);
});
const io = socket(server);
io.on("connection", (socket) => {
console.log("Client connected", socket.id);
});
Of course, when I remove the redirect in nginx and use plain old http to connect, then everything is fine.
When I run DEBUG=* node client.js, I get the following:
socket.io-client:url parse https://example.domain.com/ +0ms
socket.io-client new io instance for https://example.domain.com/ +0ms
socket.io-client:manager readyState closed +0ms
socket.io-client:manager opening https://example.domain.com/ +1ms
engine.io-client:socket creating transport "polling" +0ms
engine.io-client:polling polling +0ms
engine.io-client:polling-xhr xhr poll +0ms
engine.io-client:polling-xhr xhr open GET: https://example.domain.com/socket.io/?EIO=3&transport=polling&t=NVowV1t&b64=1 +2ms
engine.io-client:polling-xhr xhr data null +2ms
engine.io-client:socket setting transport polling +61ms
socket.io-client:manager connect attempt will timeout after 20000 +66ms
socket.io-client:manager readyState opening +3ms
engine.io-client:socket socket error {"type":"TransportError","description":{"code":"ERR_CRYPTO_OPERATION_FAILED"}} +12ms
socket.io-client:manager connect_error +9ms
socket.io-client:manager cleanup +1ms
socket.io-client:manager will wait 1459ms before reconnect attempt +3ms
engine.io-client:socket socket close with reason: "transport error" +6ms
engine.io-client:polling transport not open - deferring close +74ms
socket.io-client:manager attempting reconnect +1s
...
Searching for ERR_CRYPTO_OPERATION_FAILED error, only leads me to the node.js errors page
which has only the following description:
Added in: v15.0.0
A crypto operation failed for an otherwise unspecified reason.
I am using Let's Encrypt certificate.
I don't get it. If it is an SSL issue, why am I getting this error only for few clients?
Maybe I am missing something in nginx?
Any help is much appreciated.
I've seem similar error with node-apn. My solution was to downgrade to nodejs v14. Maybe give that a try?
two step
the version of node must be 14.x
add this config when connect rejectUnauthorized: false
im trying to implement a system that comprise of nginx , daphne and gunicorn. So far , i have gotten gunicorn to work , however im facing an issue connecting the websockets for daphne. This issue does not arise when im in developing mode , only when i utilize nginx. This is the error code from my console :
reconnecting-websocket.js:199 WebSocket connection to 'ws://192.168.8.31/' failed: Error during WebSocket handshake: Unexpected response code: 200
This is my nginx config file :
upstream crm_server { server unix:/home/user/project/venv/run/gunicorn.sock fail_timeout=0;
}
upstream channels-backend {
server localhost:8001;
}
server {
listen 80;
# add here the ip address of your server
# or a domain pointing to that ip (like example.com or www.example.com)
server_name 192.168.8.31;
keepalive_timeout 5;
client_max_body_size 4G;
access_log /home/user/project/venv/logs/nginx-access.log;
error_log /home/user/project/venv/logs/nginx-error.log;
location /staticfiles/ {
root /home/user/roject/crm/staticfiles/;
}
# checks for static file, if not found proxy to app
location / {
try_files $uri #proxy_to_app; }
location /ws/ {
try_files $uri #proxy_to_ws;
}
location #proxy_to_ws {
include proxy_params;
proxy_pass http://channels-backend;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $server_name;
}
location #proxy_to_app {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_buffering off;
proxy_redirect off;
proxy_pass http://crm_server;
}
This is my front end thats generating the socket
<script type="text/javascript">
var loc = window.location
var wsStart = 'ws://'
if (loc.protocol == 'https'){
wsStart = 'wss://' } var endpoint = wsStart + loc.host + loc.pathname var socket = new ReconnectingWebSocket(endpoint)
This is my routing:
from channels.routing import ProtocolTypeRouter , URLRouter
from django.urls import path
from rnd.consumers import NotificationConsumer
from django.conf.urls import url
from channels.auth import AuthMiddlewareStack
from channels.security.websocket import AllowedHostsOriginValidator, OriginValidator
application = ProtocolTypeRouter({
"websocket": AllowedHostsOriginValidator(
AuthMiddlewareStack(
URLRouter(
[
path('',NotificationConsumer),
] ) )
) })
I would greatly appreciate any form of feed back!
When you proxy pass to open a Web-socket connection you need to proxy pass all of the web-socket http headers. https://developer.mozilla.org/en-US/docs/Web/HTTP/Protocol_upgrade_mechanism#WebSocket-specific_headers you will also need to pass the Origin header since you are using AllowedHostsOriginValidator.
--
also to debug these I suggest using a api client directly rather than the browser. Paw or Insomnia can be used to hit api to check that your Nginx config is routing to channels and not to Gunicorn.
Not all API operations are retriable, so backend send a header retriable = true if the operations can be retried.
How can I say Nginx: retry if the upstream response contains the retriable header?
upstream mash {
ip_hash;
server 127.0.0.1:8081;
server 192.168.0.11:8081;
}
server {
location / {
if ($request_method = POST ) {
proxy_next_upstream error;
}
if ($request_method = PUT ) {
proxy_next_upstream error;
}
proxy_pass http://mash/;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
}
The logic I want to implement is the following:
if(response.hasHeader('Retriable') {
do_retry
} else{
return_backend_response;
}
The Retrier Service
Create a second upstream retrier with a very simple web server running that forwards all requests back to nginx on a special port. Then nginx will forward the requests to the mash upstream. When the response comes back to nginx, the response will get sent to retrier and the logic for retrying can be executed there.
I'm using nginx 1.12.2 (downloaded from official site) on windows 8.1 as a reverse proxy server. I got a problem that one of every two request is ignored.
nginx config:
server {
listen 80;
server_name my_fake_domain.com;
access_log C:/nginxlogs/access.txt;
location / {
proxy_pass http://localhost:8080;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_read_timeout 1800;
proxy_connect_timeout 1800;
}
}
my nodejs server:
'use strict';
const http = require('http');
const httpServer = http.createServer((req, res) => {
res.writeHead(200, { 'Content-Type': 'text/plain' });
res.end('okay');
});
httpServer.listen(8080, '0.0.0.0');
I'm using postman to send test requests. A request to http://my_fake_domain.com after a succeeded request will always loading. This always loading request does not actually ignored. If I cancel it, it will show up in the access log.
Note that a request to http://localhost:8080 always succeed.
Where might I get wrong?