my route is fetching user tokens
GET /tokens
average response time is around 180ms and response is json.
using Flask + nginx.
some requests, the response content is truncated at around 33kb, thus the JSON is malformed. some requests, with the same parameters, at nearly the same time, the response is ok at around 216kb.
My question is, why is this happening and why is this happening in an un consist way?
here is flask response code
class NormalResponse(Response):
def __init__(self, response):
super(NormalResponse, self).__init__(response, 200)
res = json.dumps(paginator.paginate(tokens))
return NormalResponse(res)
I found the issue related to nginx since the failed responses have this logs
2018/12/18 16:35:17 [crit] 16#16: *95010 open() "/var/tmp/nginx/uwsgi/1/42/0000000421" failed (13: Permission denied) while reading upstream, client: 172.31.72.76, server: , request: "GET /tokens?limit=501&offset=0&order=desc&owner_id=11111 HTTP/1.1", upstream: "uwsgi://unix:/run/server.socket:", host: "oauth.dev.bla_bla.com"
Seems like response overflows proxy_buffers and tries to temporarily save it to proxy_temp_path and your error message quite confirms it. You should check file permissions of nginx's user on that folder.
This problem has been resolved by adding
RUN chown -R www-data:www-data /var/tmp/nginx
to the Dockerfile
Related
I have a very simple flask app that is deployed on GKE and exposed via google external load balancer. And getting random 502 responses from the backend-service (added a custom headers on backend-service and nginx to make sure the source and I can see the backend-service's header but not nginx's)
The setup is;
LB -> backend-service -> neg -> pod (nginx -> uwsgi) where pod is the application built using flask and deployed via uwsgi and nginx.
The scenario is to handle image uploads in simple-secured way. Sender sends me a token with upload request.
My flask app
receive request and check the sent token via another service using "requests".
If token valid, proceed to handle the image and return 200
If token is not valid, stop and send back a 401 response.
First, I got suspicious about the 200 and 401's. And reverted all responses to 200. Following some of the expected responses, server starts to respond 502 and keep sending it. "Some of the messages at the very beginning succeeded".
nginx error logs contains below lines
2023/02/08 18:22:29 [error] 10#10: *145 readv() failed (104: Connection reset by peer) while reading upstream, client: 35.191.17.139, server: _, request: "POST /api/v1/imageUpload/image HTTP/1.1", upstream: "uwsgi://127.0.0.1:21270", host: "example-host.com"
my uwsgi.ini file is as below;
[uwsgi]
socket = 127.0.0.1:21270
master
processes = 8
threads = 1
buffer-size = 32768
stats = 127.0.0.1:21290
log-maxsize = 104857600
logdate
log-reopen
log-x-forwarded-for
uid = image_processor
gid = image_processor
need-app
chdir = /server/
wsgi-file = image_processor_application.py
callable = app
py-auto-reload = 1
pidfile = /tmp/uwsgi-imgproc-py.pid
my nginx.conf is as below
location ~ ^/api/ {
client_max_body_size 15M;
include uwsgi_params;
uwsgi_pass 127.0.0.1:21270;
}
Lastly, my app has a healthcheck method with simple JSON response. It does no extra stuff and simply returns. This never fails as explained above.
Edit : my nginx access logs in the pod shows the response as 401 while the client receives 502.
for those who gonna face with the same issue, the problem was post data reading (or not reading).
nginx was expecting to get post data read by the proxied, in our case uwsgi, app. But according to my logic I was not reading it in some cases and returning back the response.
Setting uwsgi post-buffering solved the issue.
post-buffering = %(16 * 1024 * 1024)
Which led me to this solution;
https://stackoverflow.com/a/26765936/631965
Nginx uwsgi (104: Connection reset by peer) while reading response header from upstream
Linux installation of Phusion Passenger(R) 6.0.14 on nginx/1.14.1. I'm not able to load a site (on a simplified nginx.conf with the conf.d/passenger.conf and conf.d/site1.conf includes. That error:
2022/08/03 19:24:34 [alert] 55601#0: *3 Error opening '/home/user3/sites/Passengerfile.json' for reading: Permission denied (errno=13);
This error means that the Nginx worker process (PID 55601, running as UID 992) does not have permission to access this file.
Please read this page to learn how to fix this problem: https://www.phusionpassenger.com/library/admin/nginx/troubleshooting/?a=upon-accessing-the-web-app-nginx-reports-a-permission-denied-error;
Extra info, client: 192.168.1.4, server: domain1.com, request: "GET / HTTP/1.1", host: "server_f.local"
I don't even know what to ask, other than how can I get this to work? That file doesn't exist. I've restarted nginx many times, and this is the only feedback that I get. I've checked over that page, which tells me to look at the error log it's reported in already. The host and server are correct, and I am on my LAN.
I have NGINX working as a cache engine and can confirm that pages are being cached as well as being served from the cache. But the error logs are getting filled with this error:
2018/01/19 15:47:19 [crit] 107040#107040: *26 chmod()
"/etc/nginx/cache/nginx3/c0/1d/61/ddd044c02503927401358a6d72611dc0.0000000007"
failed (1: Operation not permitted) while reading upstream, client:
xx.xx.xx.xx, server: *.---.com, request: "GET /support/applications/
HTTP/1.1", upstream: "http://xx.xx.xx.xx:80/support/applications/",
host: "---.com"
I'm not really sure what the source of this error could be since NGINX is working. Are these errors that can be safely ignored?
It looks like you are using nginx proxy caching, but nginx does not have the ability to manipulate files in it's cache directory. You will need to get the ownership/permissions correct on the cache directory.
Not explained in the original question is that the mounted storage is an Azure file share. So in the FSTAB I had to include the gid= and uid= for the desired owner. This then removed the need for chown and chmod also became unnecessary. This removed the chmod() error but introduced another.
Then I was getting errors on rename() without permission to perform this. At this point I scrapped what I was doing, moved to a different type of Azure storage (specifically a Disk attached to the VM) and all these problems went away.
So I'm offering this as an answer but realistically, the problem was not solved.
We noticed the same problem. Following the guide from Microsoft # https://learn.microsoft.com/en-us/azure/aks/azure-files-dynamic-pv#create-a-storage-class seems to have fixed it.
In our case the nginx process was using a different user for the worker threads, so we needed to find that user's uid and gid and use that in the StorageClass definition.
i am new to nginx, yesterday i installed lemp on centos and hosted a website on. today i get this in error message. please tell me if this is a security bug of nginx? Thank you!
cat /var/log/nginx/error.log
2017/07/01 03:52:18 [error] 6073#0: *23151 open() "/usr/share/nginx/html/testproxy.php" failed (2: No such file or directory), client: 91.196.50.33, server: _, request: "GET http://testp3.pospr.waw.pl/testproxy.php HTTP/1.1", host: "testp3.pospr.waw.pl"
It's not an nginx bug: If you're currently running a site that's live and accessible on the internet, bot scripts will try to find common vulnerabilities by hitting your server with various automated requests of common URLs, paths and patterns (e.g. admin.php, /wp-login.php for WordPress blogs...). Just make sure you set up proper security for your server and you'll be fine.
I'm using named pipes as the log file for the access_log of my nginx, I want to know what happens internally in nginx when I delete and recreate the pipe. What I note is that nginx keep working but stop logging.
Even if i don't create again the pipe nginx didn't try to create a regular file for logging.
I don't want to lose my logs, but apparently the only option is to restart nginx, can I enforce nginx to check again for the log file?
The error log only says this, even if the pipe doesn't exists or the pipe is recreated:
2012/02/27 22:45:13 [alert] 24537#0: *1097 write() to "/tmp/access.log.fifo" failed (32: Broken pipe) while logging request, client: 127.0.0.1, server: , request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8000/", host: "localhost:8002"
Thanks.
AFIAK, you need to send nginx a USR1 signal to instruct it to reopen the log files. Basically nginx will keep trying to write to the file-descriptor for the old files (that's why you are seeing the Broken Pipe error). More info here:
http://wiki.nginx.org/LogRotation (also click through the other links at the bottom of that page).
hth