turning on gzip cuts the response from upstream - nginx

Question

My upstream server returns extremely large JSON responses (5~8GB).
I'm trying to condense those responses by enabling gzip on nginx. This is my config
server {
listen 0.0.0.0:8080;
location / {
gzip on;
gzip_comp_level 1;
gzip_types *;
gzip_proxied any;
proxy_pass http://localhost:8081;
}
}
This config technically works. At least, it works for smaller responses (~150MB before compression). When I try to download large response (~7.5GB before compression)
curl -v --compressed --output /path_to_file -X POST http://localhost:8080 --data '{data}'
It gets cut off in the middle, i.e. I see this message from curl
curl: (18) transfer closed with outstanding read data remaining
and the response itself is incomplete (on average it only downloads ~5.3GB out of ~7.5GB)
I also see this log from nginx
2022/04/20 01:18:45 [error] 37#37: *135 upstream prematurely closed connection while reading upstream, client: 127.0.0.1, server: , request: "POST / HTTP/1.1", upstream: "http://127.0.0.1:8081/", host: "localhost:8080"
I tried increasing proxy_max_temp_file_size, and, I tried disabling buffering. nothing works
Any ideas?
Edit: this is the nginx.conf that's built in in the docker image I'm using
worker_processes auto;
error_log "/opt/bitnami/nginx/logs/error.log";
pid "/opt/bitnami/nginx/tmp/nginx.pid";
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] '
'"$request" $status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log "/opt/bitnami/nginx/logs/access.log" main;
add_header X-Frame-Options SAMEORIGIN;
client_body_temp_path "/opt/bitnami/nginx/tmp/client_body" 1 2;
proxy_temp_path "/opt/bitnami/nginx/tmp/proxy" 1 2;
fastcgi_temp_path "/opt/bitnami/nginx/tmp/fastcgi" 1 2;
scgi_temp_path "/opt/bitnami/nginx/tmp/scgi" 1 2;
uwsgi_temp_path "/opt/bitnami/nginx/tmp/uwsgi" 1 2;
sendfile on;
tcp_nopush on;
tcp_nodelay off;
gzip on;
gzip_http_version 1.0;
gzip_comp_level 2;
gzip_proxied any;
gzip_types text/plain text/css application/javascript text/xml application/xml+rss;
keepalive_timeout 65;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;
ssl_ciphers HIGH:!aNULL:!MD5;
client_max_body_size 80M;
server_tokens off;
absolute_redirect off;
port_in_redirect off;
include "/opt/bitnami/nginx/conf/server_blocks/*.conf";
# HTTP Server
server {
# Port to listen on, can also be set in IP:PORT format
listen 8080;
include "/opt/bitnami/nginx/conf/bitnami/*.conf";
location /status {
stub_status on;
access_log off;
allow 127.0.0.1;
deny all;
}
}
}

Answer 1

in similar case this line in location block solved the problem
proxy_http_version 1.1;
it seems like gzip'ed response somehow incompatible with http2