I am using Jelastic for my development environment (not yet in production).
My application is running with Unicorn but I discovered websockets with ActionCable and integrated it in my application.
Everything is working fine in local, but when deploying to my Jelastic environment (with the default NGINX/Unicorn configuration), I am getting this message in my javascript console and I see nothing in my access log
WebSocket connection to 'ws://dev.myapp.com:8080/' failed: WebSocket is closed before the connection is established.
I used to have on my local environment and I solved it by adding the needed ActionCable.server.config.allowed_request_origins in my config file. So I double-checked my development config for this and it is ok.
That's why I was wondering if there is something specific for NGINX config, else than what is explained on ActionCable git page
bundle exec puma -p 28080 cable/config.ru
For my application, I followed everything from enter link description here but nothing's mentioned about NGINX configuration
I know that websocket with ActionCable is quite new but I hope someone would be able to give me a lead on that
Many thanks
Ok so I finally managed to fix my issue. Here are the different steps which allowed to make this work:
1.nginx : I don't really know if this is needed but as my application is running with Unicorn, I added this into my nginx conf
upstream websocket {
server 127.0.0.1:28080;
}
server {
location /cable/ {
proxy_pass http://websocket/;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
}
}
And then in my config/environments/development.rb file:
config.action_cable.url = "ws://my.app.com/cable/"
2.Allowed request origin: I have then noticed that my connection was refused even if I was using ActionCable.server.config.allowed_request_origins in my config/environments/development.rb file. I am wondering if this is not due to the development default as http://localhost:3000 as stated in the documentation. So I have added this:
ActionCable.server.config.disable_request_forgery_protection = true
I have not yet a production environment so I am not yet able to test how it will be.
3.Redis password: as stated in the documentation, I was using a config/redis/cable.yml but I was having this error:
Error raised inside the event loop: Replies out of sync: #<RuntimeError: ERR operation not permitted>
/var/www/webroot/ROOT/public/shared/bundle/ruby/2.2.0/gems/em-hiredis-0.3.0/lib/em-hiredis/base_client.rb:130:in `block in connect'
So I understood the way I was setting my password for my redis server was not good.
In fact your have to do something like this:
development:
<<: *local
:url: redis://user:password#my.redis.com:6379
:host: my.redis.com
:port: 6379
And now everything is working fine and Actioncable is really impressive.
Maybe some of my issues were trivial but I am sharing them and how I resolved them so everyone can pick something if needed
Related
I am running nginx as part of the docker-compose template.
In nginx config I am referring to other services by their docker hostnames (e.g. backend, ui).
That works fine until I do that trick:
docker stop backend
docker stop ui
docker start ui
docker start backend
which makes backend and ui containers to exchange IP addresses (docker provides private network IPs on a basis of giving the next IP available in CIDR to each new requester). This 4 commands executed imitate some rare cases when both upstream containers got restarted at the same time but the nginx container did not. Also, I believe, this should be a very common situation when running pods on Kubernetes-based clusters.
Now nginx resolves backend host to ui's IP and ui to backend's IP.
Reloading nginx' configuration does help (nginx -s reload).
Also, if I do nslookup from within the nginx container - the IPs are always resolved correctly.
So this isolates the problem to be a pure nginx issue around the DNS caching.
The things I tried:
I have the resolver set under the http {} block in nginx config:
resolver 127.0.0.11 ipv6=off valid=10s;
Most common solution proposed by the folks on the internet to use variables in proxy-pass (this helps to prevent nginx to resolve and cache DNS records on start) - that did not make ANY difference at all:
server {
<...>
set $mybackend "backend:3000";
location /backend/ {
proxy_pass http://$mybackend;
}
}
Tried adding resolver line into the location itself
Tried setting the variable on the http{} block level, using map:
http {
map "" $mybackend {
default backend:3000;
}
server {
...
}
}
Tried to use openresty fork of nginx (https://hub.docker.com/r/openresty/openresty/) with resolver local=true
None of the solutions gave any effect at all. The DNS caches are only wiped if I reload nginx configuration inside of the container OR restart the container manually.
My current workaround is to use static docker network declared in docker-compose.yml. But this has its cons too.
Nginx version used: 1.20.0 (latest as of now)
Openresty versions used: 1.13.6.1 and 1.19.3.1 (latest as of now)
Would appreciate any thoughts
UPDATE 2021-09-08: Few months later I am back to solving this same issue and still no luck. Really looks like the bug in nginx - I can not make nginx to re-resolve the dns names. There seems to be no timeout to nginx' dns cache and none of the options listed above to introduce timeouts or trigger dns flush work.
UPDATE 2022-01-11: I think the problem is really in the nginx. I tested my config in many ways a couple months ago and it looks like something else in my nginx.conf prevents the valid parameter of the resolver directive from working properly. It is either the limit_req_zone or the proxy_cache_path directives used for request rate limiting and caching respectively. These just don't play nicely with the valid param for some reason. And I could not find any information about this anywhere in nginx docs.
I will get back to this later to confirm my hypothesis.
Maybe it's because nginx's DNS resolver for upstream servers only works in the commercial version, nginx plus?
https://www.nginx.com/products/nginx/load-balancing/#service-discovery
TLDR: Your Internet Provider may be caching dnses with no respect to tiny TTL values (like 1 second).
I've been trying to retest locally the same thing.
Your docker might be using local resolver (127.0.0.11)
Then Dns might be cached by your OS (which you may clean - that's OS specific)
Then you might have it cached on your WIFI/router (yes!)
Later it goes to your ISP and is beyond your control.
But nslookup is your friend, you can query each dns server between nginx and root DNS server.
Something very easy to reproduce (without setting up local dns server)
Create route 53 'A' entry with TTL of 1 second and try to query AWS dns server in your hosted zone (it will be sth. like ns-239.awsdns-29.com)
Play around with dig / nslookup command
nslookup
set type=a
server ns-239.awsdns-29.com
your.domain.com
It will return IP you have set
Change the Route53 'A' entry to some other ip.
use dig / nslookup and make sure you see changes immediately
Then set resolver in nginx to AWS dns name (for testing purposes only).
If that works it means that DNS is cached elsewere and this is no longer nginx issue!
In my case it was sunrise WIFI router which began to see new IP only after I restarted it (I assume things would resolve after some longer value).
Great help when debugging this is when your nginx is compiled with
--with-debug
Then in nginx logs you see whether given dns was resolved and to what IP.
My whole config looks like this (here with standard docker resolver which has to be set if you are using variables in proxy_pass!)
server {
listen 0.0.0.0:8888;
server_name nginx.my.custom.domain.in.aws;
resolver 127.0.0.11 valid=1s;
location / {
proxy_ssl_server_name on;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-Proto https;
proxy_set_header Host $host;
set $backend_servers my.custom.domain.in.aws;
proxy_pass https://$backend_servers$request_uri;
}
}
Then you can try to test it with
curl -L http://nginx.my.custom.domain.in.aws --resolve nginx.my.custom.domain.in.aws 0.0.0.0:8888
Was struggling on the same thing exactly for the same thing (Docker Swarm) and actually to make it work I required to let the upstream away from my configuration.
Something that works well (tested 5' ago on NGINX 2.22) :
location ~* /api/parameters/(.*)$ {
resolver 127.0.0.11 ipv6=off valid = 1s;
set $bck_parameters parameters:8000;
proxy_pass http://$bck_parameters/api/$1$is_args$args;
}
where $bck_parameters is NOT an upstream but the real server behind.
Doing same thing with upstream will fail.
I have GraphDb (standlone server version) in Ubuntu Server 16 running in localhost (with the command ./graphdb -d in /etc/graphdb/bin). But I only have ssh access to the server in the terminal, can't open Worbench in localhost:7200 locally.
I have many websites running on this machine with Ningx. If I try to access the machine's main IP with the port 7200 through the external web it doesn't work (e.g. http://193.133.16.72:7200/ = "connection timmed out").
I have tried to make a reverse proxy with Nginx with this code ("xxx" = domain):
listen 7200;
listen [::]:7200;
server_name sparql.xxx.com;
location / {
proxy_pass http://127.0.0.1:7200;
}
}
But all this fails. I checked and port 7200 is open in firewall (ufw). In the logs I get info that GraphDB is working locally in some testes. But I need Workbench access to import and create repositories (not sure how to do it or if it is possible without the Workbench GUI).
Is there a way to connect through the external web to the Workbench using a domain/IP and/or Nginx?
Read all the documentation and searched all day, but could not find a way to deal with this sorry. I only used GraphDB locally (the simple installer version), never used the standalone server in production before, sorry.
PS: two extra questions related:
a) For creating an URI endpoint it is the same procedure?
b) For GraphDB daemon to start automattically at boot time (with the command ./graphdb -d in graph/bin folder), what is the recommended way and configuration? (tryed the line "/etc/graphdb/bin ./graphdb -d" in rc.local but it didn't worked).
In case it is usefull for someone, I was able to make it work with this Nginx configuration:
server {
listen 80;
server_name sparql.xxxxxx.com;
location / {
proxy_pass http://localhost:7200;
proxy_set_header Host $host;
}
}
I think it was the "proxy_set_header Host $host;" that solved it (tried the rest without it before and didn't worked). I think GraphDB uses some headers to set configurations, and they were not passing.
I wounder if I am forgetting something else important to forward in the proxy, but in this moment the Worbench seams to work and opens in the domain used "sparql.xxxxxx.com".
I am using meteor with the BrowerPolicy package and Meteor Up with the abernix/meteord:base docker image to deploy my app to a EC2 instance. I use HTTPS using nginx all on the same server. The trouble comes when I allow connections to an AWS S3 bucket using the following line:
BrowserPolicy.content.allowOriginForAll('*.s3-us-west-2.amazonaws.com');
It works locally but when I deploy to the EC2 server, I get a 502 bad gateway error for the entire app.
I have read that this problem can sometimes be due to the header size being too large and that it can be fixed by changing proxy_buffer_size 8k; in the /var/lib/docker/aufs/mnt/CHECKEDID/opt/nginx/conf/nginx.conf file. I checked and my header size is 499 for a random svg that I have S3.
If indeed I need to make a change to the docker image to have this larger header size, how do I do that? I believe that this is the source repo for the docker image. If I am totally off base and there is a different problem, please let me know that too.
Thanks!
I ended up figuring it out. So it turns out to be a configuration error with nginx. I configured my EC2 instance using this guide. In order to fix nginx, I first logged into my cluster and opened this file:
sudo vi /etc/nginx/sites-available/default
I then added the proxy_buffer_size 8k; line to the server block of the configuration file. Finally, I checked the syntax with sudo nginx -t and restarted nginx nginx restart. That was it!
The best part is that since I configured my nginx instance manually and deploy my meteor instance on top of this running on port 3000, these settings persist even after I deploy new versions of my app.
I am trying to deploy an Action Cable -enabled-application to a VPS using Capistrano. I am using Puma, Nginx, and Redis (for Cable). After a couple hurdles, I was able to get it working in a local developement environment. I'm using the default in-process /cable URL. But, when I try deploying it to the VPS, I keep getting these two errors in the JS-log:
Establishing connection to host ws://{server-ip}/cable failed.
Connection to host ws://{server-ip}/cable was interrupted while loading the page.
And in my app-specific nginx.error.log I'm getting these messages:
2016/03/10 16:40:34 [info] 14473#0: *22 client 90.27.197.34 closed keepalive connection
Turning on ActionCable.startDebugging() in the JS-prompt shows nothing of interest. Just ConnectionMonitor trying to reopen the connection indefinitely. I'm also getting a load of 301: Moved permanently -requests for /cable in my network monitor.
Things I've tried:
Using the async adapter instead of Redis. (This is what is used in the developement env)
Adding something like this to my /etc/nginx/sites-enabled/{app-name}:
location /cable/ {
proxy_pass http://puma;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
}
Setting Rails.application.config.action_cable.allowed_request_origins to the proper host (tried "http://{server-ip}" and "ws://{server-ip}")
Turning on Rails.application.config.action_cable.disable_request_forgery_protection
No luck. What is causing the issue?
$ rails -v
Rails 5.0.0.beta3
Please inform me of any additional details that may be useful.
Finally, I got it working! I've been trying various things for about a week...
The 301-redirects were caused by nginx actually trying to redirect the browser to /cable/ instead of /cable. This is because I had specified /cable/ instead of /cable in the location stanza! I got the idea from this answer.
I'm getting some problems to make spiderable work with PhantomJS on my Ubuntu server. I saw this troubleshooting on Meteorpedia:
Ensure that the ROOT_URL that your Meteor server is configured to use
is accessible from the server itself. (Since v0.8.1.3[1])
I think that this could be a possible answer to why it is not working. What is exactly the purpose of this environment variable?
My application is publicly accessible on http://gentlenode.com/ but my proxy_pass on nginx is set to http://gentlenode/.
# HTTPS Server
server {
listen 443;
server_name gentlenode.com;
# ...
location / {
proxy_pass http://gentlenode/;
proxy_http_version 1.1;
# ...
}
}
Should I set ROOT_URL to http://gentlenode.com/, to http://gentlenode/ or to http://localhost/?
You can find my nginx configuration here: https://gist.github.com/LeCoupa/9877434
The ROOT_URL environment variable should be set to the URL that clients will be accessing your application with. So in your case, it would be http://gentlenode.com or https://gentlenode.com.
The ROOT_URL environment variable is read by Meteor.absoluteUrl, which is used in many (core) packages. Thus, setting ROOT_URL may be a requirement if you use these packages. spiderable is one such package.
// Line 62 of spiderable_server.js
var url = Spiderable._urlForPhantom(Meteor.absoluteUrl(), req.url);
I'll admit that we don't use spiderable so I'm not 100% certain if this will fix your problem, but here's what we do...
We set ROOT_URL to the URL which clients will use to initially connect. In your case, the nginx config automatically upgrades all HTTP requests to HTTPS, so all requests will be seen by your app under https://gentlenode.com. I think you should start your server after:
export ROOT_URL=https://gentlenode.com
Your proxy_pass section may be correct. We manually spell out the name of the local port. So we'd write:
proxy_pass http://localhost:58080;
If you have something that works already, this may not be necessary. I don't know all the quirks of nginx well enough to say if that part matters.