Nginx memory leak issue - nginx

Question

We are facing issue with nginx memory leak it seems.
Setup:
Nginx running as deployment in GKE
Nginx version 1.20.2
Nginx is used to stream HLS. We write chunk file to a google filestore(NFS service). It is mounted on /var/www/html/.
Nginx never ever recovers memory it just grows on increasing. Nginx confiuration
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log notice;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
#tcp_nopush on;
keepalive_timeout 65;
#gzip on;
include /etc/nginx/conf.d/*.conf;
}
Default.conf
server {
listen 80;
server_name localhost;
proxy_buffering off;
proxy_buffer_size 4k;
proxy_buffers 64 4k;
proxy_busy_buffers_size 16k;
proxy_cache_valid 200 302 1m;
proxy_cache_valid 404 60m;
proxy_cache_use_stale error timeout invalid_header updating;
proxy_redirect off;
#charset koi8-r;
#access_log /var/log/nginx/host.access.log main;
location / {
root /usr/share/nginx/html;
index index.html index.htm;
add_header 'Access-Control-Allow-Origin' '*';
add_header 'Access-Control-Allow-Credentials' 'true';
add_header 'Access-Control-Allow-Methods' '*';
add_header 'Access-Control-Allow-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type';
add_header 'Access-Control-Max-Age' 1728000;
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root /usr/share/nginx/html;
}
location /stub_status {
stub_status on;
access_log on;
allow all;
}
}
}
Screenshot of our internal monitoring system

Answer 1

First of all, what does your chart show? Memory usage of nginx worker processes? Or memory utilization of whole system?
In case of nginx memory growth it may relate to known issue (basically with OpenSSL either), see https://trac.nginx.org/nginx/ticket/2316
So either try to apply patch suggested by Maxim in that issue, or try the workaround he suggested in the last comment or upgrade to newer version of OpenSSL (PKCS11 engine) or even nginx (especially if it is linked statically).
There are enough OpenSSL-related leak issues, see also for example https://github.com/kubernetes/ingress-nginx/issues/7647 or linked within. So to veryfy it is not affected by OpenSSL, try to test it without SSL/TLS/https and check whether you'd see growth of memory usage.
Although I don't see any memory leak trying vanilla nginx 1.20.2 (without any patch, built with OpenSSL 1.1.1k) testing similar configuration (I don't see proxy_pass directive in your config so I was simply proxying to http/https upstream too). No leak reproducible at all.
In case of high system memory usage, it may be common OS cache or even some buffering of NFS, see https://askubuntu.com/a/1393696/1384131 for similar question.