Adding HTML to backend response body by NGINX Kubernetes Ingress - nginx

Question

Background
We're hosting a proprietary web application, which we like to customize rudimentary to our CD (mainly colors). Since the application doesn't support this and I don't have access to the source, I'd like to create a custom css stylesheet and include it in the app by manipulating it's ingress (= injecting css/stylesheets into the DOM).
http_sub_module of the Nginx Ingress Controller for Kubernetes
The http_sub_module seems similar to Apaches mod_substitute. When I exec nginx -V inside the nginx ingress pod, there is --with-http_sub_module listed in the configure arguments, so it must be avaliable in the currently deployed 1.19.2 version.
I found a similar question using subs_filter (instead of sub_filter). It seems that the one with s is from nginx plus, also documented here. All linked examples there use subs_, where the regular community documentation uses sub_filter. Both of them worked without an error, I guess the plus one is an alias if no plus subscription is avaliable.
Since sub_filter doesn't seem to work, I tried both of them without success:
annotations:
kubernetes.io/ingress.class: "nginx"
nginx.ingress.kubernetes.io/rewrite-target: /
nginx.ingress.kubernetes.io/configuration-snippet: |
#subs_filter_once on;
#subs_filter_types 'text/html';
# subs_filter 'en' 'de';
# subs_filter '</body>' '<h1>TEST</h1></body>';
sub_filter '</body>' '<h1>TEST</h1></body>';
Further things I've checked/tried
The configuration snippet is applied. I looked into the nginx ingress pod, the server vhost block for my application has the sub_filter/subs_filter directive inside the / location block.
The default type for filtering is sub_filter_types text/html. The website returns Content-Type: text/html so it should match. To be sure, I also set sub_filter_types "text/html"; in the snippet. And even * which matches any mime type according to the docs, both without any difference.
Doesn't this module work with mod_proxy used by k8s? I can't imagine that since the module is relatively old and I see no reason why it shouldn't work when nginx acts as reverse proxy, since it needs to have access to the HTML header/body too.

Answer 1

Found out that the problem was something completely different: The application supports gzip, so this compression was enabled. But the http_sub_module doesn't support gzip, as documented here. It only works with plain text reasponses. If the response is compressed, the module just do nothing. This explains why it doesn't work and there was no error.
Luckily, the compression could be easily disabled without any modifications on the application by setting the following header:
proxy_set_header Accept-Encoding "";
If this is added to the nginx.ingress.kubernetes.io/configuration-snippet section, it accepts only plain text and every application which is compilant to the specification will respect that.
But it doesn't mean that we can't use any compression. The Gzip filter can be used from the ingress to the user, it's only not supported from the ingress to the application.
Full working example annotation snippet
ingres:
# ...
annotations:
kubernetes.io/ingress.class: "nginx"
# http://nginx.org/en/docs/http/ngx_http_sub_module.html
nginx.ingress.kubernetes.io/configuration-snippet: |
sub_filter "</body>" "<style>body{ color: red !important; }</style></body>";
# The http_sub_module doesn't support compression from the ingress to the backend application
proxy_set_header Accept-Encoding "";
This would apply the style block before the closing body tag like this:
<style>body{ color: red !important; }</style></body></html>
For productive usage, we can include a link to a custom css file here, which overrides the applications design to our needs.