I am trying to deploy an Action Cable -enabled-application to a VPS using Capistrano. I am using Puma, Nginx, and Redis (for Cable). After a couple hurdles, I was able to get it working in a local developement environment. I'm using the default in-process /cable URL. But, when I try deploying it to the VPS, I keep getting these two errors in the JS-log:
Establishing connection to host ws://{server-ip}/cable failed.
Connection to host ws://{server-ip}/cable was interrupted while loading the page.
And in my app-specific nginx.error.log I'm getting these messages:
2016/03/10 16:40:34 [info] 14473#0: *22 client 90.27.197.34 closed keepalive connection
Turning on ActionCable.startDebugging() in the JS-prompt shows nothing of interest. Just ConnectionMonitor trying to reopen the connection indefinitely. I'm also getting a load of 301: Moved permanently -requests for /cable in my network monitor.
Things I've tried:
Using the async adapter instead of Redis. (This is what is used in the developement env)
Adding something like this to my /etc/nginx/sites-enabled/{app-name}:
location /cable/ {
proxy_pass http://puma;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
}
Setting Rails.application.config.action_cable.allowed_request_origins to the proper host (tried "http://{server-ip}" and "ws://{server-ip}")
Turning on Rails.application.config.action_cable.disable_request_forgery_protection
No luck. What is causing the issue?
$ rails -v
Rails 5.0.0.beta3
Please inform me of any additional details that may be useful.
Finally, I got it working! I've been trying various things for about a week...
The 301-redirects were caused by nginx actually trying to redirect the browser to /cable/ instead of /cable. This is because I had specified /cable/ instead of /cable in the location stanza! I got the idea from this answer.
Related
I installed on Ubuntu 16.04, 4 cores, 8Gb RAM. I ran the cerbot command and it returned a congratulatory message that it's successful.
This is my first time installing BigBlueButton. I followed the process and all seemed fine until I tried running it on HTTPS https://live.oltega.com, and it returned
This site can’t be reached
live.oltega.com refused to connect.
Try: Checking the connection
Checking the proxy and the firewall
ERR_CONNECTION_REFUSED
when I served the same on HTTP http://live.oltega.com it worked well but displays a blue screen because it can only work on HTTPS. What can I try next?
After obtaining a Let's Encrypt certificate, you should configure the BBB components like Nginx and Freeswith to use HTTPS.
Follow the instructions mentioned here.
The summary is :
1-Configure FreeSWITCH for using SSL
Edit the file /etc/bigbluebutton/nginx/sip.nginx and change the protocol and port on the proxy_pass line as shown bellow
location /ws {
proxy_pass https://203.0.113.1:7443;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
proxy_read_timeout 6h;
proxy_send_timeout 6h;
client_body_timeout 6h;
send_timeout 6h;
}
2- Configure BigBlueButton to load session via HTTPS
Edit /usr/share/bbb-web/WEB-INF/classes/bigbluebutton.properties and update the property bigbluebutton.web.serverURL to use HTTPS:
#----------------------------------------------------
# This URL is where the BBB client is accessible. When a user successfully
# enters a name and password, she is redirected here to load the client.
bigbluebutton.web.serverURL=https://bigbluebutton.example.com
Next, edit the file /usr/share/red5/webapps/screenshare/WEB-INF/screenshare.properties and update the property jnlpUrl and jnlpFile to HTTPS:
streamBaseUrl=rtmp://bigbluebutton.example.com/screenshare
jnlpUrl=https://bigbluebutton.example.com/screenshare
jnlpFile=https://bigbluebutton.example.com/screenshare/screenshare.jnlp
Next, you should do the following command:
$ sudo sed -e 's|http://|https://|g' -i /var/www/bigbluebutton/client/conf/config.xml
Open /usr/share/meteor/bundle/programs/server/assets/app/config/settings.yml editing and change:
kurento:
wsUrl: ws://bbb.example.com/bbb-webrtc-sfu
to
kurento:
wsUrl: wss://bbb.example.com/bbb-webrtc-sfu
and also
note:
enabled: true
url: http://bbb.example.com/pad
to
note:
enabled: true
url: https://bbb.example.com/pad
3- Next, modify the creation of recordings so they are served via HTTPS
Edit /usr/local/bigbluebutton/core/scripts/bigbluebutton.yml and change the value for playback_protocol as follows:
playback_protocol: https
4-If you have installed the API demos, edit /var/lib/tomcat7/webapps/demo/bbb_api_conf.jsp and change the value of BigBlueButtonURL use HTTPS.
// This is the URL for the BigBlueButton server
String BigBlueButtonURL = "https://bigbluebutton.example.com/bigbluebutton/";
5-Finally, to apply all of the configuration changes made, you must restart all components of BigBlueButton:
$ sudo bbb-conf --restart
I have a jetty web app running under k8s. This web app has a websocket end point. The service deployed is exposed via an nginx ingress on https.
Everything works fine, I have the web app running and the websockets work fine (ie messages get pushed and received) but the websockets close with a 1006 error code, which to be honest doesn't stop my code from working but doesn't look good either.
The websocket is exposed # /notifications. In a "normal" config, ie not k8s, just plain software installed on a VM, I would need to add the following to nginx.conf
location /notifications {
proxy_pass http://XXX/notifications;
proxy_read_timeout 3700s;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
proxy_set_header Origin '';
}
I tried doing this via the ingress
nginx.ingress.kubernetes.io/configuration-snippet: |
location /notifications {
proxy_pass http://webapp:8080/notifications;
proxy_http_version 1.1;
proxy_set_header Upgrade "websocket";
proxy_set_header Connection "Upgrade";
}
But it has no effect, ie I checked the nginx.conf generated and there is no such block added...
Anybody has had issues like this before? any clue on how to solve the 1006 issue?
1006 meaning
As per RFC-6455 1006 means Abnormal Closure:
Used to indicate that a connection was closed abnormally (that is, with no close frame being sent) when a status code is expected.
Also see CloseReason.CloseCodes (Java(TM) EE 7 Specification APIs)
There are so many possible causes either on server or on client.
Client errors: try websocket.org Echo Test
To isolate and debug client's errors, you may use websocket.org Echo Test
As for server error
Jetty
Jetty-related discussion is here: Question regarding abnormal · Issue #604 · eclipse/jetty.project. But it doesn't contain any solutions.
Race detector for golang server code
If your server is written on golang, you may try Data Race Detector - The Go Programming Language
Data races are among the most common and hardest to debug types of bugs in concurrent systems. A data race occurs when two goroutines access the same variable concurrently and at least one of the accesses is a write. See the The Go Memory Model for details.
Here is an example of a data race that can lead to crashes and memory corruption:
func main() {
c := make(chan bool)
m := make(map\[string\]string)
go func() {
m\["1"\] = "a" // First conflicting access.
c <- true
}()
m\["2"\] = "b" // Second conflicting access.
<-c
for k, v := range m {
fmt.Println(k, v)
}
}
Case for PHP code
The case for PHP code discussed here: Unclean a closed connection by close() websocket's method (1006) · Issue #236 · walkor/Workerman
We are currently experiencing an issue when we run our dotnet core server setup on Production. We publish it in Bamboo and run it from an AWS linux server, and it sits behind an nginx reverse proxy.
Essentially, every few days our dotnet core server process will go mute. It silently accepts and hangs on web requests, and even silently ignores our (more polite) attempts to stop it. We have verified that it is actually the netcore process that hangs by sending curl requests directly to port 5000 from within the server. We've replicated our production deployment to the best of our ability to our test environment and have not been able to reproduce this failure mode.
We've monitored the server with NewRelic and have inspected it at times when it's gone into failure mode. We've not been able to correlate this behaviour with any significant level of traffic, RAM usage, CPU usage, or open file descriptor usage. Indeed, these measurements all seem to stay at very reasonable levels.
My team and I are a bit stuck as to what might be causing our hung server, or even what we can do next to diagnose it. What might be causing our server process to hang? What further steps can we take to diagnose the issue?
Extra Information
Our nginx conf template:
upstream wfe {
server 127.0.0.1:5000;
server 127.0.0.1:5001;
}
server {
listen 80 default_server;
location / {
proxy_set_header Host $http_host;
proxy_pass http://wfe;
proxy_read_timeout 20s;
# Attempting a fix suggested by:
# https://medium.com/#mshanak/soved-dotnet-core-too-many-open-files-in-system-when-using-postgress-with-entity-framework-c6e30eeff6d1
proxy_buffering off;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection keep-alive;
proxy_cache_bypass $http_upgrade;
fastcgi_buffers 16 16k;
fastcgi_buffer_size 32k;
}
}
Our Program.cs:
using System.Diagnostics.CodeAnalysis;
using System.IO;
using System.Net;
using Microsoft.AspNetCore;
using Microsoft.AspNetCore.Hosting;
using Microsoft.Extensions.Logging;
using Serilog;
namespace MyApplication.Presentation
{
[ExcludeFromCodeCoverage]
public class Program
{
public static void Main(string[] args)
{
IWebHost host = WebHost.CreateDefaultBuilder(args)
#if DEBUG
.UseKestrel(options => options.Listen(IPAddress.Any, 5000))
#endif
.UseStartup<Startup>()
.UseSerilog()
.Build();
host.Run();
}
}
}
During our CD build process, we publish our application for deployment with:
dotnet publish --self-contained -c Release -r linux-x64
We then deploy the folder bin/Release/netcoreapp2.0/linux-x64 to our server, and run publish/<our-executable-name> from within.
EDIT: dotnet --version outputs 2.1.4, both on our CI platform and on the production server.
When the outage starts, nginx logs show that server responses to requests change from 200 to 502, with a single 504 being emitted at the time of the outage.
At the same time, logs from our server process just stop. And there are warnings there, but they're all explicit warnings that we've put into our application code. None of them indicate that any exceptions have been thrown.
After a few days of investigation I've found the reason of that issue. It is being caused by glibc >= 2.27, which lead to GC hang at some conditions, so there is almost nothing to do about it. However you have a few options:
Use Alpine Linux. It doesn't rely on glibc.
Use older distro like Debian 9, Ubuntu 16.04 or any other with glibc < 2.27
Try to patch glibc by yourself at your own risk: https://sourceware.org/bugzilla/show_bug.cgi?id=25847
Or wait for the glibc patch to be reviewed by community and included in your favorite distro.
More information can be found here: https://github.com/dotnet/runtime/issues/47700
This question already has answers here:
Add a prefix to all Flask routes
(15 answers)
Closed 5 years ago.
I already asked this in the Arch Linux boards but didn't get no answer. So I am trying my luck over here:
I am trying to set up nginx + gunicorn on my Arch Linux server to run multiple Flask apps. However I am seemingly failing in configuring nginx the right way to do so.
When I just got one Flask app up and running everything seems to work fine. I included the /etc/nginx/sites-available and /etc/nginx/sites-enabled in my /etc/nginx/nginx.conf.
I created a file "flask_settings" inside /etc/nginx/sites/available and linked it to /etc/nginx/sites-enabled. The file looks like this:
server {
location /{
proxy_pass http://127.0.0.1:8000;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
}
}
I have a folder containing my Flask app (sample App, hellp.py) which i run with gunicorn in a virtual environment. I just run it using
gunicorn hello:app
If i visit my servers IP I can access the file and the different routes.
Now I tried to set up another app creating another file in /etc/nginx/sites-enabled called flask2. It looks like this:
server {
location /hello {
proxy_pass http://127.0.0.1:8001;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
}
}
I then try to run the app inside its very own virtual environment with
gunicorn --bind 127.0.0.1:8001 hello:app
When I restart nginx afterwards, I am still able to access the first app and all of its routes, but if I try to access the other one by entering my servers IP + the router (after the "/"), nginx always tells me, that the sites cannot be found. Am I missing anything here?
Any help is highly appreciated.
Thanks in advance!
You should have seperate proxy location for both apps.
i.e, have one nginx conf file but multiple locations for each route or you can have seperate conf file for each.
For example:
h1.example.com proxy to a location to the required address with the port number
h2.example.com proxy to the location of the second app.
Using the approach you have, is not the recommended way to do it. Consider making a .sock file as explained in this tutorial(Although this is for Ubuntu, It can be adapted to Arch).
Another thing, (probably more important). Your server block is missing the server name. So Nginx does not know which URL to respond to.
Unless you have the server on the same computer as your local machine, 127.0.0.1 should not be accessible from the browser. Consider passing the parameter as : app.run(host = '0.0.0.0').
When it comes to the second site file. The location of the root (e.g: https://google.com/) is not mentioned. So Nginx cannot even reach the root. Again, the server name block should be mentioned for this one too.
Hope this helps.
I am using Jelastic for my development environment (not yet in production).
My application is running with Unicorn but I discovered websockets with ActionCable and integrated it in my application.
Everything is working fine in local, but when deploying to my Jelastic environment (with the default NGINX/Unicorn configuration), I am getting this message in my javascript console and I see nothing in my access log
WebSocket connection to 'ws://dev.myapp.com:8080/' failed: WebSocket is closed before the connection is established.
I used to have on my local environment and I solved it by adding the needed ActionCable.server.config.allowed_request_origins in my config file. So I double-checked my development config for this and it is ok.
That's why I was wondering if there is something specific for NGINX config, else than what is explained on ActionCable git page
bundle exec puma -p 28080 cable/config.ru
For my application, I followed everything from enter link description here but nothing's mentioned about NGINX configuration
I know that websocket with ActionCable is quite new but I hope someone would be able to give me a lead on that
Many thanks
Ok so I finally managed to fix my issue. Here are the different steps which allowed to make this work:
1.nginx : I don't really know if this is needed but as my application is running with Unicorn, I added this into my nginx conf
upstream websocket {
server 127.0.0.1:28080;
}
server {
location /cable/ {
proxy_pass http://websocket/;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
}
}
And then in my config/environments/development.rb file:
config.action_cable.url = "ws://my.app.com/cable/"
2.Allowed request origin: I have then noticed that my connection was refused even if I was using ActionCable.server.config.allowed_request_origins in my config/environments/development.rb file. I am wondering if this is not due to the development default as http://localhost:3000 as stated in the documentation. So I have added this:
ActionCable.server.config.disable_request_forgery_protection = true
I have not yet a production environment so I am not yet able to test how it will be.
3.Redis password: as stated in the documentation, I was using a config/redis/cable.yml but I was having this error:
Error raised inside the event loop: Replies out of sync: #<RuntimeError: ERR operation not permitted>
/var/www/webroot/ROOT/public/shared/bundle/ruby/2.2.0/gems/em-hiredis-0.3.0/lib/em-hiredis/base_client.rb:130:in `block in connect'
So I understood the way I was setting my password for my redis server was not good.
In fact your have to do something like this:
development:
<<: *local
:url: redis://user:password#my.redis.com:6379
:host: my.redis.com
:port: 6379
And now everything is working fine and Actioncable is really impressive.
Maybe some of my issues were trivial but I am sharing them and how I resolved them so everyone can pick something if needed