I would like to replace parts of response bodies coming from an upstream HTTP server, which I am reverse proxying with Ngninx.
There the HttpSubsModule with the sub_filter directive, but here's the problem
This module just works with plain text. If the response is compressed, it can't uncompress the response and will ignore this response. This module can be compatible with gzip filter module. But it will not work with proxy compressed response. You can disable the compressed response like this:
proxy_set_header Accept-Encoding "";
Hmph.
Are there any alternatives that would allow me to modify proxied content without loosing upstream compression?
We had a similar issue in our case and we used "gunzip" module from ngnix which helped us to uncompress the compressed response and update the response as per the requirement.
Related
Is there any way to tell nginx to ignore missing HTTP headers when proxying requests?
There is an existing proprietary HTTP Server sending requests without any header. The Server can not be configured. I need various endpoints from this server in a web application. Therefore I want to setup my nginx to proxy requests to this server. I have location configuration in my regular server.
location /api/ {
proxy_pass http://localhost:80/;
}
When calling corresponding URIs nginx complains:
upstream sent no valid HTTP/1.0 header while reading response header from upstream, client: 127.0.0.1, server: localhost, request: ....
Is there any way to tell nginx not to expect any headers, and just to forward the received payload?
Thanks for your help!
Kind regards,
Andreas
Edit: Ok, found server is using http 0.9, when calling curl directly to the server threw an error:
curl: (1) Received HTTP/0.9 when not allowed
Using the option --http0.9 got the desired result. (Which is received in a browser without further ado). Any chance to tell nginx to proxy to an http 0.9 server?
You can configure nginx to ignore missing headers by setting the proxy_ignore_headers directive to "X-Accel-Expires" and "Expires". This will tell nginx to ignore those specific headers and not expect them to be present when proxying requests.
You can add this to the location block in your nginx configuration:
proxy_ignore_headers X-Accel-Expires Expires;
Additionally, proxy_pass_request_headers off; could be used, this tells nginx to ignore all headers passed to the proxied server.
As for the payload, as long as the underlying protocol is HTTP, it should be fine, as the payload will be sent in the body of the HTTP request.
So your location block could look like this
location /api/ {
proxy_pass http://localhost:80/;
proxy_pass_request_headers off;
}
Please note that this is may not be recommended, as the missing headers may contain important information like authentication and should be handled correctly.
when several clients request the same file, which is responsed by the gzip function in nginx . I hope that other responses could use the cached gzip content . How to config ?
There was a discussion of the same in NGINX forums.
I find that this suggestion makes the most sense. However, it mostly applies to when you do proxy with NGINX and not fastcgi cache.
Essentially you will ensure Accept-Encoding: gzip is sent to your backend to ensure that you always generate/cache gzipped content, and then use gunzip module for clients that don't request gzip encoding.
The Upstream server is wowza , which does not accept the custom headers if I don't enable them on application level.
Nginx is working as a proxy server, from the browser I want to send few custom headers which should be received and logged by Nginx Proxy but before forwarding request to upstream server those headers should be removed from the request.
So upstream server never come to know that there where any custom headers.
I tried proxy_hide_header as well as proxy_set_header "<header>" "" , but seems like they apply to response headers not the request headers.
And even if I accept to enable the headers on wowza, then again I am not able to find a way to enable headers at server level for all application. Currenlty I have to add headers to each newly created application which is not feasible for me to do.
Any help would be appreciated.
The proxy_set_header HEADER "" does exactly what you expect. See https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_set_header.
If the value of a header field is an empty string then this field will not be passed to a proxied server:
proxy_set_header Accept-Encoding "";
I have just confirmed this is working as documented, I used Nginx v1.12.
It's said that Transport will handle the Content-Encoding automatically (like auto decompressing when reading from resp.Body).
It's also said that, Content-Encoding is an end-to-end HTTP header, not a hop-by-hop one.
Therefore, if a proxy copied Content-Encoding back to client's response header, and this proxy also io.Copy the upstream response body (which may decompressing automatically since io.Copy will read from resp.Body), won't it be inconsistent to client? (Content-Encoding copied from upstream response, but body has been decompressed)
In general, the Content-Encoding response header should not be altered by a proxy.
Different encodings of the same URI are deemed to be different representations and have different ETags. So, changing Content-Encoding would not play well with caching.
But if it's your own proxy and client in your own ecosystem, you could do it, since you know what's going on, so if your proxy is decompressing data back to the client you'd need to strip the Content-Encoding header.
I'm trying to upload some files with compression to a server. The files will be fairly large and the server is a standard HTTP server where the interface defines that they're not compressed. Is it possible to use something like Content-Encoding to indicate that the upload request is compressed, much like it is used for downstream compression?
Apache supports it with the mod_deflate module but I does not look a common web server feature. If you have access to the server you can enable this module or rewrite the server side code to handle your compressed data (e.g. a special servlet/php which call the original servlet/php with the decompressed data).