nginx / uwsgi-flask is changing port after form submission - nginx

I set up a flask/uwsgi and a nginx container.
First of all, I could not access the site having nginx listen on port 80 and defining the docker port 80:80. No idea why this would not work.
So I made nginx listen on port 8090 and opening port 80:8090 in docker:
nginx:
container_name: fa_nginx
build:
context: ./nginx
dockerfile: Dockerfile
restart: unless-stopped
ports:
- 80:8090
networks:
- fa_nginx_net
I am not exposing any ports in the dockerfile, there I copy just the conf file.
## nginx conf
server {
listen 8090;
location / {
include /etc/nginx/uwsgi_params;
uwsgi_pass web:8087;
}
Like that, I can access the site at http://localhost and browser around e.g. to http://localhost/faq.
However, when I submit a form (login), the nginx changes to another port and the redirected url looks like http://localhost:8090/auth/login.
the redirect in flask's login view after successful form validation is simply return redirect(url_for('main.profile')).
Here is the flask/uwsgi setup (shortened version without all then env etc):
web:
container_name: fa_web
build:
context: ./web
dockerfile: Dockerfile
expose:
- "8087"
networks:
- fa_nginx_net
[uwsgi]
plugin = python3
## python file where flask object app is defined.
wsgi-file = run.py
## The flask instance defined in run.py
callable = fapp
enable-threads = false
master = true
processes = 2
threads = 2
## socket on which uwsgi server should listen
protocol=http
socket = :8087
buffer-size=32768
## Gives permission to access the server.
chmod-socker = 660
vacuum = true
## various
die-on-term = true
## run.py
from app import create_app, db
import os
## init flask app
fapp = create_app()
if __name__ == "__main__":
if os.environ.get('FLASK_ENV') == 'development':
fapp.run(use_reloader=True, debug=True, host='0.0.0.0', port=8086)
else:
fapp.run(host='0.0.0.0', port=8087)
I have no idea why this is happening and how to solve this.

Ok, it is working now...
Basically I removed plugin=python3 and protocol=http from uwsgi.ini.
My setup looks now like this
## nginx default.conf
server {
listen 8090;
#server_name fa_web;
## To map the server block to an ip:
#server_name 1.2.3.4;
## To map the server block to a domain:
#server_name example.com www.example.com;
location / {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $http_host;
proxy_set_header X-NginX-Proxy true;
include /etc/nginx/uwsgi_params;
uwsgi_pass web:8087;
}
}
## uwsgi.ini
[uwsgi]
## python file where flask object app is defined.
wsgi-file = run.py
## The flask instance defined in myapp.py
callable = fapp
enable-threads = false
master = true
processes = 2
threads = 2
## socket on which uwsgi server should listen
socket = :8087
buffer-size=32768
#post-buffering=8192 ## workaround for consuming POST requests
## Gives permission to access the server.
chmod-socker = 660
vacuum = true
## various
die-on-term = true
The added headers in nginx conf didn't make a difference, just added them going forward configuring everything correctly.

Related

nginx get value of proxy_http_header

I add an traceId in my nginx config via $proxy_http_header. Here is the config:
location ~ /(xxx.*) {
proxy_set_header trace_id $connection-$connection_requests-$msec;
}
Now I need to print it in my Nginx log. How can I get it? It does not work if I get it via $http_trace_id. like
log_format normal '$remote_addr'
'"$http_trace_id"';

Unable to load Ruby 2.7.4 (Rails 6) app on AWS Elastic Beanstalk, AL2

We have a Rails app that was previously deployed on Ruby 2.6 platform of AWS Elastic Beanstalk. We've had to upgrade to Ruby 2.7.4 as the 2.6 platform was being deprecated by AWS. After the upgrade, we can see that the app is deployed successfully (from the eb-engine.log)... However, when I hit the EB url (XXXXX.YYYYY.us-east-1.elasticbeanstalk.com), nothing shows up. I have verified that both nginx and puma are running and the logs also show that the health check is fine. The environment indicator on the EB console also shows as OK. Any tips on what I need to look for to debug this issue?
On another note, if I deploy EB's sample rails app first to an env, and then upload my app to that env, it starts to work fine... However I do not want to go to production without first figuring out why I'm unable to access the app by deploying it directly onto a new env. Any help would be much appreciated. Below is my .ebextensions\nginx.config
files:
"/etc/nginx/conf.d/010_app_server.conf":
mode: "000644"
owner: root
group: root
content: |
# based on /etc/nginx/conf.d/webapp_healthd.conf
#
# and notes from "Getting to Know and Love AWS Elastic Beanstalk Configuration Files (.ebextensions)"
# (https://medium.com/trisfera/getting-to-know-and-love-aws-elastic-beanstalk-configuration-files-ebextensions-9a4502a26e3c)
#
# which suggested taking the approach of including a similar
# nginx conf file before webapp_healthd.conf, which has the
# effect of "overriding" the setttings from that file
upstream my_app_new_upstream {
server unix:///var/run/puma/my_app.sock;
}
log_format healthd_new_name '$msec"$uri"'
'$status"$request_time"$upstream_response_time"'
'$http_x_forwarded_for';
server {
listen 80;
server_name _ localhost; # need to listen to localhost for worker tier
if ($time_iso8601 ~ "^(\d{4})-(\d{2})-(\d{2})T(\d{2})") {
set $year $1;
set $month $2;
set $day $3;
set $hour $4;
}
access_log /var/log/nginx/access.log main;
access_log /var/log/nginx/healthd/application.log.$year-$month-$day-$hour healthd_new_name;
client_max_body_size 100m;
set $should_enforce_https 0;
if ($http_x_forwarded_proto != 'https') {
set $should_enforce_https 1;
}
if ($request_uri ~ ^/pages/health_check$) {
set $should_enforce_https 0;
}
if ($should_enforce_https = 1) {
return 301 https://$host$request_uri;
}
location / {
proxy_pass http://my_app_new_upstream; # match the name of upstream directive which is defined above
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
location /assets {
alias /var/app/current/public/assets;
gzip_static on;
gzip on;
expires max;
add_header Cache-Control public;
}
location /public {
alias /var/app/current/public;
gzip_static on;
gzip on;
expires max;
add_header Cache-Control public;
}
}
container_commands:
010_reload_nginx:
command: "sudo service nginx reload"
And here's my puma config
# Puma can serve each request in a thread from an internal thread pool.
# The `threads` method setting takes two numbers: a minimum and maximum.
# Any libraries that use thread pools should be configured to match
# the maximum value specified for Puma. Default is set to 5 threads for minimum
# and maximum; this matches the default thread size of Active Record.
#
max_threads_count = ENV.fetch("RAILS_MAX_THREADS") { 5 }
min_threads_count = ENV.fetch("RAILS_MIN_THREADS") { max_threads_count }
threads min_threads_count, max_threads_count
# Specifies the `worker_timeout` threshold that Puma will use to wait before
# terminating a worker in development environments.
#
worker_timeout 3600 if ENV.fetch("RAILS_ENV", "development") == "development"
bind "unix:///var/run/puma/my_app.sock"
pidfile "/var/run/puma/my_app.sock"
# Specifies the `port` that Puma will listen on to receive requests; default is 3000.
#
port ENV.fetch("PORT") { 3000 }
# Specifies the `environment` that Puma will run in.
#
environment ENV.fetch("RAILS_ENV") { "development" }
# Specifies the `pidfile` that Puma will use.
pidfile ENV.fetch("PIDFILE") { "tmp/pids/server.pid" }
# Specifies the number of `workers` to boot in clustered mode.
# Workers are forked web server processes. If using threads and workers together
# the concurrency of the application would be max `threads` * `workers`.
# Workers do not work on JRuby or Windows (both of which do not support
# processes).
#
# workers ENV.fetch("WEB_CONCURRENCY") { 2 }
# Use the `preload_app!` method when specifying a `workers` number.
# This directive tells Puma to first boot the application and load code
# before forking the application. This takes advantage of Copy On Write
# process behavior so workers use less memory.
#
# preload_app!
# Allow puma to be restarted by `rails restart` command.
plugin :tmp_restart

ERROR org.keycloak.adapters.OAuthRequestAuthenticator - failed to turn code into token

We have a problem with our application, Nginx and keycloak. There are 3 instances: instance 1 APP, Instance 2 NGINX (reverse proxy) and Instance 3 Keycloak.
When a user logs in, he creates the session in keycloak but when he returns to the SiAe application it is when he returns 403.
We enter the keycloak administration console and see that the session was successful and is open. But it is not possible that the return to the application works.
Logs
Nginx:
"GET /opensat/?state=dcc1c40f-3183-4c7b-8342-f7df620cf0b3&session_state=605ba79a-ee05-4918-a96d-71466e31210a&code=fe066a17-a97a-495c-94f9-b1e5e3d6ac1f.605ba79a-ee05-4918-a96d-71466e31210a.f91920a4-3267-4de5-9788-24093a32c217 HTTP/1.1" **403 405** "https://mydomain/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:71.0) Gecko/20100101 Firefox/71.0"
APP:
ERROR org.keycloak.adapters.OAuthRequestAuthenticator - failed to turn code into token
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:196)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:442)
at sun.security.ssl.InputRecord.read(InputRecord.java:480)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:934)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1332)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1359)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1343)
at org.apache.http.conn.ssl.SSLSocketFactory.createLayeredSocket(SSLSocketFactory.java:573)
at org.keycloak.adapters.SniSSLSocketFactory.createLayeredSocket(SniSSLSocketFactory.java:114)
at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:557)
at org.keycloak.adapters.SniSSLSocketFactory.connectSocket(SniSSLSocketFactory.java:109)
at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:414)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
Keycloak:
**INFO** [org.keycloak.storage.ldap.LDAPIdentityStoreRegistry] (default task-1) Creating new LDAP Store for the LDAP storage provider: 'ldap_pre', LDAP Configuration: {pagination=[true], fullSyncPeriod=[-1], usersDn=[ou=usuarios,dc=domain,dc=es], connectionPooling=[true], cachePolicy=[DEFAULT], useKerberosForPasswordAuthentication=[false], importEnabled=[true], enabled=[true], changedSyncPeriod=[86400], bindDn=[cn=admin,dc=domain,dc=es], usernameLDAPAttribute=[uid], lastSync=[1575269470], vendor=[other], uuidLDAPAttribute=[entryUUID], connectionUrl=[ldap://MIIP:389], allowKerberosAuthentication=[false], syncRegistrations=[false], authType=[simple], debug=[false], searchScope=[1], useTruststoreSpi=[ldapsOnly], priority=[0], userObjectClasses=[inetOrgPerson, organizationalPerson, person], rdnLDAPAttribute=[cn], editMode=[WRITABLE], validatePasswordPolicy=[false], batchSizeForSync=[1000]}, binaryAttributes: []
Configurations:
Nginx:
location / {
proxy_set_header Host $host;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_pass http://IP_KEYCLOAK:8081;
}
Keycloak:
....
<server name="default-server">
<http-listener name="default" socket-binding="http" redirect-socket="proxyhttps" proxy-address-forwarding="true" enable-http2="true"/>
......
<socket-binding name="proxy-https" port="443"/>
.....
APP .json:
{
"realm": "domain",
"auth-server-url": "https://domainkeycloak/",
"ssl-required": "none",
"resource": "sso",
"enable-cors" : true,
"credentials": {
"secret": "98236f9a-c8b1-488c-8b36-ace4f95b1aa6"
},
"confidential-port": 0,
"disable-trust-manager":true,
"allow-any-hostname" : true
}
Could anybody help us, please?
The error Connection reset means that the application cannot make request to Keycloak server. You should investigate the issue further by logging in to the application server and telnet keycloak domain with 443 port since you set auth-server-url to https://domainkeycloak/
The solution has been to separate keycloak with other Nginx instead of using one Nginx for APP and Keycloak. Now, we have 2 Nginx's and run OK keycloak with our APP.

Increase of requests count end up with 'Error: socket hang up'

Backend service/nginx proxy starts responding 'Error: socket hang up' when there is increased count of requests. The setup is as follows.
OS: CentOS 6
Express JS service -> nginx as a proxy -> flask app run by Gunicorn
JS app sends multiple requests at the same time to the other service, when the request count exceed ~100 it starts to return error responses. If the count is lower everything works fine.
I have followed example configuration of nginx which is in Gunicorn documentation + increasing timeout limits + increasing nginx open files limit. I have also tried keepalive option but the issue still remains. Gunicorn doesn't show any errors.
nginx configuration fragment:
upstream app_server {
server 127.0.0.1:8000 fail_timeout=0;
keepalive 100;
}
server {
listen 5001;
client_max_body_size 4G;
keepalive_timeout 300;
root /path/to/app/current/public; # static files
location / {
try_files $uri #proxy_to_app;
}
location #proxy_to_app {
# Timeouts
proxy_read_timeout 300;
proxy_connect_timeout 300;
proxy_send_timeout 300;
send_timeout 300;
proxy_http_version 1.1;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $http_host;
proxy_redirect off;
proxy_pass http://app_server;
}
}
Error response received from proxy:
{ RequestError: Error: socket hang up
at new RequestError (/home/pm2deploy/apps/app-backend/source/node_modules/request-promise-core/lib/errors.js:14:15)
at Request.plumbing.callback (/home/pm2deploy/apps/app-backend/source/node_modules/request-promise-core/lib/plumbing.js:87:29)
at Request.RP$callback [as _callback] (/home/pm2deploy/apps/app-backend/source/node_modules/request-promise-core/lib/plumbing.js:46:31)
at self.callback (/home/pm2deploy/apps/app-backend/source/node_modules/request/request.js:185:22)
at Request.emit (events.js:160:13)
at Request.onRequestError (/home/pm2deploy/apps/app-backend/source/node_modules/request/request.js:881:8)
at ClientRequest.emit (events.js:160:13)
at Socket.socketOnEnd (_http_client.js:423:9)
at Socket.emit (events.js:165:20)
at endReadableNT (_stream_readable.js:1101:12)
at process._tickCallback (internal/process/next_tick.js:152:19)
name: 'RequestError',
message: 'Error: socket hang up',
cause: { Error: socket hang up
at createHangUpError (_http_client.js:330:15)
at Socket.socketOnEnd (_http_client.js:423:23)
at Socket.emit (events.js:165:20)
at endReadableNT (_stream_readable.js:1101:12)
at process._tickCallback (internal/process/next_tick.js:152:19) code: 'ECONNRESET' },
error: { Error: socket hang up
at createHangUpError (_http_client.js:330:15)
at Socket.socketOnEnd (_http_client.js:423:23)
at Socket.emit (events.js:165:20)
at endReadableNT (_stream_readable.js:1101:12)
at process._tickCallback (internal/process/next_tick.js:152:19) code: 'ECONNRESET' },
options:
{ method: 'PUT',
uri: 'http://localhost:5001/transformers/segmentAvg',
qs:
{ stdMultiplier: 2,
segmentLeft: 1509366682333,
segmentRight: 1509367401685 },
body: { index: [Array], values: [Array] },
headers: {},
json: true,
callback: [Function: RP$callback],
transform: undefined,
simple: true,
resolveWithFullResponse: false,
transform2xxOnly: false },
response: undefined }
ADDED:
In the OS log was recorded following entry:
possible SYN flooding on port X. Sending cookies.
Kernel socket backlog reached the limit and dropped following requests.
Reason: Kernel dropping TCP connections due to LISTEN sockets buffer full in Red Hat Enterprise Linux
Increase kernel socket backlog limit
Check the current value:
# sysctl net.core.somaxconn
net.core.somaxconn = 128
Increase the value:
# sysctl -w net.core.somaxconn=2048
net.core.somaxconn = 2048
Confirm the change by viewing again:
# sysctl net.core.somaxconn
net.core.somaxconn = 2048
Persist the change:
echo "net.core.somaxconn = 2048" >> /etc/sysctl.conf
Increase application socket listen backlog
Configuration parameter of uWSGI
listen=1024
This solution was taken from https://access.redhat.com/solutions/30453

Serving Pyramid App with UWSGI and NGINX

I am new to NGINX, uWSGI AND Pyramid, and I am trying to serve a Pyramid app through uWSGI using nginx as a reverse proxy. I am really stuck at the moment and am hoping someone can make some suggestions for how to solve this. If you can explain a little what might be going on, that would be helpful too, as my understanding is very limited!
Currently, I am getting an `Internal Server Error' from uWSGI when I visit the reverse proxy URL. In the uWSGI error log, I am getting the error:
--- no python application found, check your startup logs for errors ---
The application works fine when I serve through uWSGI alone, launching with pserve. I can launch it from my virtual envelope as follows:
bin/pserve my-app/uwsgi.ini
But when I start nginx, and visit the proxy address, I get the Internal Server Error.
The settings I have in uwsgi.ini are as follows:
[app:main]
use = egg:myapp
pyramid.reload_templates = true
pyramid.debug_authorization = false
pyramid.debug_notfound = false
pyramid.debug_routematch = false
pyramid.default_locale_name = en
pyramid_debugtoolbar
[server:main]
use = egg:waitress#main
host = 0.0.0.0
port = 6543
[loggers]
keys = root, musiccircle
[handlers]
keys = console
[formatters]
keys = generic
[logger_root]
level = ERROR
handlers = console
[logger_musiccircle]
level = ERROR
handlers =
qualname = musiccircle
[handler_console]
class = StreamHandler
args = (sys.stderr,)
level = NOTSET
formatter = generic
[formatter_generic]
format = %(asctime)s %(levelname)-5.5s [%(name)s][%(threadName)s] %(message)s
[uwsgi]
socket = unix://home/usr/env/myapp/myapp.sock
master = true
processes = 48
cpu-affinity = 12
harakiri = 60
post-buffering = 8192
buffer-size = 65535
daemonize = ./uwsgi.log
pidfile = ./pid_5000.pid
listen = 32767
reload-on-as = 512
reload-on-rss = 192
limit-as = 1024
no-orphans = true
reload-mercy = 8
log-slow = true
virtualenv = /home/usr/env
And in the corresponding myapp.conf file in nginx, I have the following:
upstream myapp {
server 127.0.0.1:6543;
}
server {
listen 8080;
server_name myapp.local www.myapp.local;
location = /favicon.ico { access_log off; log_not_found off; }
location /static/ {
root /home/usr/env/myapp;
}
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
charset utf-8;
location / {
include uwsgi_params;
uwsgi_pass unix://home/usr/env/myapp/myapp.sock;
}
}
If you need to see anything else, please let me know. As you can see, Nginx is configured to serve at port 8080 (which it does), and the Pyramid app is being served by uWSGI to port 6543 (which it does).
Thanks in advance.
It seems Pyramid projects are intended to be installed (setup.py) and then run with a .ini configuration file with pserve. Pserve then passes in these config file details as **settings to your Pyramid app at run time.
This is different than, say, Flask which is not installed and generally has no configuration file. Such a Flask application can be run by uWSGI as needed, with all run-time configuration being handled by uWSGI or environment variables.
Since Pyramid usually needs a config file at run time, and relies on pserve to provide them when using a config file (ie production.ini), I think you'll have to run uwsgi --ini-paste production.ini (or if running with Pypy, uwsgi --pypy-paste production.ini) (thanks to #Sorrel)

Resources