NGINX keeps file descriptors open for deleted files - nginx

Question

I have a server running nginx proxying to a python API, a user will request an mp4 file which will be generated and written to /tmp/cache and served with nginx. Some time later (~ 20 seconds) the file will be deleted by the python process but that file remains held by nginx meaning the space is unusable and eventually the user runs out of cache space causing the entire application to fail.
Relevant parts of nginx config:
http {
include mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /dev/stdout main;
# send headers in one piece, it is better than sending them one by one
tcp_nopush on;
sendfile on;
keepalive_timeout 65;
gzip on;
gzip_comp_level 6;
gzip_types text/plain text/css application/json application/x-javascript application/javascript text/javascript image/svg+xml image/x-icon image/bmp image/png image/gif image/jpeg image/jpg;
gzip_proxied no-cache no-store private expired auth;
gzip_vary on;
server {
listen 5000;
location /cache/ {
internal; # This tells nginx it's not accessible from the outside
alias /tmp/cache/;
}
location /api/ {
add_header Cache-Control "no-store";
expires off;
add_header 'Access-Control-Allow-Origin' '*';
add_header 'Access-Control-Allow-Methods' 'GET, POST, PUT, DELETE, OPTIONS';
proxy_pass http://frigate_api/;
proxy_pass_request_headers on;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
}
}
relevant python api code:
#bp.route("/<camera_name>/start/<float:start_ts>/end/<float:end_ts>/clip.mp4")
def recording_clip(camera_name, start_ts, end_ts):
download = request.args.get("download", type=bool)
...
file_name = f"clip_{camera_name}_{start_ts}-{end_ts}.mp4"
path = f"/tmp/cache/{file_name}"
response = make_response()
response.headers["Content-Description"] = "File Transfer"
response.headers["Cache-Control"] = "no-cache"
response.headers["Content-Type"] = "video/mp4"
if download:
response.headers["Content-Disposition"] = "attachment; filename=%s" % file_name
response.headers["Content-Length"] = os.path.getsize(path)
response.headers[
"X-Accel-Redirect"
] = f"/cache/{file_name}" # nginx: http://wiki.nginx.org/NginxXSendfile
return response
we can see that after the file has been deleted by python, nginx keeps the file open meaning the space is still being used
# lsof -n | grep /tmp |grep deleted
nginx 130 root 27r REG 0,122 561200658 60 /tmp/cache/clip_front_doorbell_cam_1674691364.607239-1674692415.428497.mp4 (deleted)
nginx 130 152 nginx root 27r REG 0,122 561200658 60 /tmp/cache/clip_front_doorbell_cam_1674691364.607239-1674692415.428497.mp4 (deleted)
nginx 130 156 nginx root 27r REG 0,122 561200658 60 /tmp/cache/clip_front_doorbell_cam_1674691364.607239-1674692415.428497.mp4 (deleted)
nginx 130 159 nginx root 27r REG 0,122 561200658 60 /tmp/cache/clip_front_doorbell_cam_1674691364.607239-1674692415.428497.mp4 (deleted)
nginx 130 163 nginx root 27r REG 0,122 561200658 60 /tmp/cache/clip_front_doorbell_cam_1674691364.607239-1674692415.428497.mp4 (deleted)
nginx 130 166 nginx root 27r REG 0,122 561200658 60 /tmp/cache/clip_front_doorbell_cam_1674691364.607239-1674692415.428497.mp4 (deleted)
nginx 130 169 nginx root 27r REG 0,122 561200658 60 /tmp/cache/clip_front_doorbell_cam_1674691364.607239-1674692415.428497.mp4 (deleted)
nginx 130 173 nginx root 27r REG 0,122 561200658 60 /tmp/cache/clip_front_doorbell_cam_1674691364.607239-1674692415.428497.mp4 (deleted)
nginx 130 177 nginx root 27r REG 0,122 561200658 60 /tmp/cache/clip_front_doorbell_cam_1674691364.607239-1674692415.428497.mp4 (deleted)
nginx 130 181 nginx root 27r REG 0,122 561200658 60 /tmp/cache/clip_front_doorbell_cam_1674691364.607239-1674692415.428497.mp4 (deleted)
nginx 130 185 nginx root 27r REG 0,122 561200658 60 /tmp/cache/clip_front_doorbell_cam_1674691364.607239-1674692415.428497.mp4 (deleted)
nginx 130 189 nginx root 27r REG 0,122 561200658 60 /tmp/cache/clip_front_doorbell_cam_1674691364.607239-1674692415.428497.mp4 (deleted)
nginx 130 194 nginx root 27r REG 0,122 561200658 60 /tmp/cache/clip_front_doorbell_cam_1674691364.607239-1674692415.428497.mp4 (deleted)
nginx 130 197 nginx root 27r REG 0,122 561200658 60 /tmp/cache/clip_front_doorbell_cam_1674691364.607239-1674692415.428497.mp4 (deleted)
nginx 130 203 nginx root 27r REG 0,122 561200658 60 /tmp/cache/clip_front_doorbell_cam_1674691364.607239-1674692415.428497.mp4 (deleted)
nginx 130 211 nginx root 27r REG 0,122 561200658 60 /tmp/cache/clip_front_doorbell_cam_1674691364.607239-1674692415.428497.mp4 (deleted)
nginx 130 216 nginx root 27r REG 0,122 561200658 60 /tmp/cache/clip_front_doorbell_cam_1674691364.607239-1674692415.428497.mp4 (deleted)
nginx 130 221 nginx root 27r REG 0,122 561200658 60 /tmp/cache/clip_front_doorbell_cam_1674691364.607239-1674692415.428497.mp4 (deleted)
nginx 130 225 nginx root 27r REG 0,122 561200658 60 /tmp/cache/clip_front_doorbell_cam_1674691364.607239-1674692415.428497.mp4 (deleted)
nginx 130 230 nginx root 27r REG 0,122 561200658 60 /tmp/cache/clip_front_doorbell_cam_1674691364.607239-1674692415.428497.mp4 (deleted)
nginx 130 235 nginx root 27r REG 0,122 561200658 60 /tmp/cache/clip_front_doorbell_cam_1674691364.607239-1674692415.428497.mp4 (deleted)
nginx 130 240 nginx root 27r REG 0,122 561200658 60 /tmp/cache/clip_front_doorbell_cam_1674691364.607239-1674692415.428497.mp4 (deleted)
nginx 130 244 nginx root 27r REG 0,122 561200658 60 /tmp/cache/clip_front_doorbell_cam_1674691364.607239-1674692415.428497.mp4 (deleted)
nginx 130 247 nginx root 27r REG 0,122 561200658 60 /tmp/cache/clip_front_doorbell_cam_1674691364.607239-1674692415.428497.mp4 (deleted)
nginx 130 251 nginx root 27r REG 0,122 561200658 60 /tmp/cache/clip_front_doorbell_cam_1674691364.607239-1674692415.428497.mp4 (deleted)
nginx 130 254 nginx root 27r REG 0,122 561200658 60 /tmp/cache/clip_front_doorbell_cam_1674691364.607239-1674692415.428497.mp4 (deleted)
nginx 130 257 nginx root 27r REG 0,122 561200658 60 /tmp/cache/clip_front_doorbell_cam_1674691364.607239-1674692415.428497.mp4 (deleted)
nginx 130 261 nginx root 27r REG 0,122 561200658 60 /tmp/cache/clip_front_doorbell_cam_1674691364.607239-1674692415.428497.mp4 (deleted)
nginx 130 263 nginx root 27r REG 0,122 561200658 60 /tmp/cache/clip_front_doorbell_cam_1674691364.607239-1674692415.428497.mp4 (deleted)
nginx 130 266 nginx root 27r REG 0,122 561200658 60 /tmp/cache/clip_front_doorbell_cam_1674691364.607239-1674692415.428497.mp4 (deleted)
nginx 130 269 nginx root 27r REG 0,122 561200658 60 /tmp/cache/clip_front_doorbell_cam_1674691364.607239-1674692415.428497.mp4 (deleted)
nginx 130 272 nginx root 27r REG 0,122 561200658 60 /tmp/cache/clip_front_doorbell_cam_1674691364.607239-1674692415.428497.mp4 (deleted)
nginx 130 276 nginx root 27r REG 0,122 561200658 60 /tmp/cache/clip_front_doorbell_cam_1674691364.607239-1674692415.428497.mp4 (deleted)
is there any way to disable this behavior?