Increase URL length limit for K8S ingress - http

I have a GET request URL into a service on my kubernetes that's ~9k long and it seems like the request is getting stuck in Kubernetes's ingress. When I tried calling the url from within the docker or from other docker in the cluster it works fine. However when I go through a domain name I'm getting the following response header:

I think the parameter you must modify is Client Body Buffer Size
Sets buffer size for reading client request body per location. In case
the request body is larger than the buffer, the whole body or only its
part is written to a temporary file. By default, buffer size is equal
to two memory pages. This is 8K on x86, other 32-bit platforms, and
x86-64. It is usually 16K on other 64-bit platforms. This annotation
is applied to each location provided in the ingress rule
nginx.ingress.kubernetes.io/client-body-buffer-size: "1000" # 1000 bytes
nginx.ingress.kubernetes.io/client-body-buffer-size: 1k # 1 kilobyte
nginx.ingress.kubernetes.io/client-body-buffer-size: 1K # 1 kilobyte
nginx.ingress.kubernetes.io/client-body-buffer-size: 1m # 1 megabyte
So you must add an annotation to your nginx ingress config.

In my case, I had to set http2_max_header_size and http2_max_field_size in my ingress server-snippet annotation. For example:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/server-snippet: |
http2_max_header_size 16k;
http2_max_field_size 16k;
I was getting ERR_CONNECTION_CLOSED and ERR_FAILED in Google Chrome and "empty response" using curl, but the backend would work if accessed directly from the cluster network.
Assigning client-header-buffer-size or large-client-header-buffers in the ingress controller ConfigMap didn't seem to work for me either, but I realized that curl would do it if using HTTP 1.1 (curl --http1.1)

Find the configmap name in the nginx ingress controller pod describition
kubectl -n utility describe pods/test-nginx-ingress-controller-584dd58494-d8fqr |grep configmap
--configmap=test-namespace/test-nginx-ingress-controller
Note: In my case, the namespace is "test-namespace" and the configmap name is "test-nginx-ingress-controller"
Create a configmap yaml
cat << EOF > test-nginx-ingress-controller-configmap.yaml
kind: ConfigMap
apiVersion: v1
metadata:
name: test-nginx-ingress-controller
namespace: test-namespace
data:
large-client-header-buffers: "4 16k"
EOF
Note: Please replace the namespace and configmap name as per finding in the step 1
Deploy the configmap yaml
kubectl apply -f test-nginx-ingress-controller-configmap.yaml
Then you will see the change is updated to nginx controller pod after mins
i.g.
kubectl -n test-namespace exec -it test-nginx-ingress-controller-584dd58494-d8fqr -- cat /etc/nginx/nginx.conf|grep large
large_client_header_buffers 4 16k;
Thanks to the sharing by NeverEndingQueue in How to use ConfigMap configuration with Helm NginX Ingress controller - Kubernetes

Related

How to implement global rate limiting with Kubernetes NGINX ingress controller

I am looking to implement global rate limiting to a production deployment on Azure in order to ensure that my application do not become unstable due to an uncontrollable volume of traffic(I am not talking about DDoS, but a large volume of legitimate traffic). Azure Web Application Firewall supports only IP based rate limiting.
I've looked for alternatives without to do this without increasing the hop count in the system. The only solution I've found is using limit_req_zone directive in NGINX. This does not give actual global rate limits, but it can be used to impose a global rate limit per pod. Following configmap is mounted to the Kubernetes NGINX ingress controller to achieve this.
kind: ConfigMap
apiVersion: v1
metadata:
name: nginx-ingress-ingress-nginx-controller
namespace: ingress-basic
data:
http-snippet : |
limit_req_zone test zone=static_string_rps:5m rate=10r/m ;
location-snippet: |
limit_req zone=static_string_rps burst=20 nodelay;
limit_req_status 429;
static_string_rps is a constant string and due to this all the requests are counted under a single keyword which provides global rate limits per pod.
This seems like a hacky way to achieve global rate limiting. Is there a better alternative for this and does Kubernetes NGINX ingress controller officially support this approach?(Their documentation says they support mounting configmaps for advanced configurations but there is no mention about using this approach without using an additional memcached pod for syncing counters between pods)
https://www.nginx.com/blog/rate-limiting-nginx/#:~:text=One%20of%20the%20most%20useful,on%20a%20log%E2%80%91in%20form.
https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#global-rate-limiting
According to Kubernetes slack community, anything that requires global coordination for rate limiting is going to have a potentially severe bottleneck for performance and will create a single point of failure. Therefore even if we do use an external solution to this would cause bottlenecks and hence it is not recommended.(However this is not mentioned in the docs)
According to them using limit_req_zone is a valid approach and it is officially supported by the Kubernetes NGINX Ingress controller community which means that it is production ready.
I suggest you use this module if you want to apply global rate limiting(Although its not exact global rate limiting). If you have multiple ingresses in your cluster, you can use the following approach to apply global rate limits per ingress.
Deploy the following ConfigMap in the namespace in which your K8 NGINX Ingress controller is present. This will create 2 counters with the keys static_string_ingress1 and static_string_ingress2.
NGINX Config Map
kind: ConfigMap
apiVersion: v1
metadata:
name: nginx-ingress-ingress-nginx-controller
namespace: ingress-basic
data:
http-snippet : |
limit_req_zone test zone=static_string_ingress1:5m rate=10r/m ;
limit_req_zone test zone=static_string_ingress2:5m rate=30r/m ;
Ingress Resource 1
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: test-ingress-1
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/affinity: cookie
nginx.ingress.kubernetes.io/backend-protocol: HTTPS
nginx.ingress.kubernetes.io/configuration-snippet: |
limit_req zone=static_string_ingress1 burst=5 nodelay;
limit_req_status 429;
spec:
tls:
- hosts:
- test.com
rules:
- host: test.com
http:
paths:
- path: /
backend:
serviceName: test-service
servicePort: 9443
Similary you can add a separate limit to the ingress resource 2 by adding the following configuration snippet to ingress resource 2 annotations.
Ingress resource 2
annotations:
nginx.ingress.kubernetes.io/configuration-snippet: |
limit_req zone=static_string_ingress2 burst=20 nodelay;
limit_req_status 429;
Note that the keys static_string_ingress1 and static_string_ingress2 are static strings and all requests passing through the relevan ingress will be counted using one of they keys which will create the global rate limiting effect.
However, these counts are maintained separately by each NGINX Ingress controller pod. Therefore the actual rate limit will be defined limit * No. of NGINX pods
Further I have monitored the pod memory and CPU usage when using limit_req_zone module counts and it does not create a considerable increase in resource usage.
More information on this topic is available on this blog post I wrote: https://faun.pub/global-rate-limiting-with-kubernetes-nginx-ingress-controller-fb0453447d65
Please note that this explanation is valid for Kubernetes NGINX Ingress Controller(https://github.com/kubernetes/ingress-nginx) not to be confused with NGINX controller for Kubernetes(https://github.com/nginxinc/kubernetes-ingress)

Preserving Source IP in NGINX Ingress Controller

We're running an NGINX Ingress Controller as the 'front door' to our EKS cluster.
Our upstream apps need the client IP to be preserved, so I've had my ingress configmap configured to use the proxy protocol:
apiVersion: v1
kind: ConfigMap
metadata:
name: ingress-nginx-controller
namespace: ingress-nginx
data:
custom-http-errors: 404,503,502,500
ssl-redirect: "true"
ssl-protocols: "TLSv1.2"
ssl-ciphers: "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384:!ECDHE-RSA-AES256-SHA384:!ECDHE-RSA-AES128-SHA256"
use-proxy-protocol: "true"
proxy-real-ip-cidr: "0.0.0.0/0"
This sends the X-Forwarded-For header with the client IP to the upstream pods. This seemed like it was working well, but once our apps started to receive heavier traffic, our monitors would occasionally report connection timeouts when connecting to the apps on the cluster.
I was able reproduce the issue in a test environment using JMeter. Once I set use-proxy-protocol to false, the connection timeouts would no longer occur. I started to look into the use-proxy-protocol setting.
The ingress docs describe the use-proxy-protocol setting here
However the docs also mention the settings "enable-real-ip" and "forwarded-for-header"
At the link provided in the description for enable-real-ip, it says that I can set forwarded-for-header to value proxy_protocol.
Based on this, I've updated my Ingress configmap to:
apiVersion: v1
kind: ConfigMap
metadata:
name: ingress-nginx-controller
namespace: ingress-nginx-test
data:
custom-http-errors: 404,503,502,500
ssl-redirect: "true"
ssl-protocols: "TLSv1.2"
ssl-ciphers: "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384:!ECDHE-RSA-AES256-SHA384:!ECDHE-RSA-AES128-SHA256"
enable-real-ip: "true"
forwarded-for-header: "proxy_protocol"
proxy-real-ip-cidr: "0.0.0.0/0"
This configuration also properly sends the X-Forwarded-For header with the client IP to the upstream pods. However, it also seems to eliminate the connection timeout issues I was seeing. With this setup performance does not degrade anywhere near as badly as I ramp up the thread count in JMeter.
I would like to better understand the difference between these two configurations. I’d also like to know what is the best practice most widely adopted method of achieving this among Kubernetes shops since this is likely a common use-case.

Can we NOT set nginx.ingress.kubernetes.io/client-body-timeout: '120' using ingress annotations?

The default value of said annotation is 60 sec; I am looking to change its value to 120 sec. I added this as an annotation in ingress resource file but it doesn't seem to be working.
Since my request body is quite big, I am getting 408 from ingress http server immediately after 60 sec only;
Where else I can define this annotation if it is not allowed in ingress file itself?
The following page doesn't mention this annotation; Is it because it is not meant to be added as an annotation?
https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations
Ingress resource snippet:
kind: Ingress
metadata:
name: app-ingress
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /my-app
nginx.ingress.kubernetes.io/client-header-buffer-size: "1M"
nginx.ingress.kubernetes.io/client-header-timeout: "60"
nginx.ingress.kubernetes.io/client-body-buffer-size: "1M"
nginx.ingress.kubernetes.io/client-body-timeout: "120"
nginx.ingress.kubernetes.io/use-regex: "true"
nginx.ingress.kubernetes.io/configuration-snippet: |
proxy_set_header custom-header $1;
spec:
rules:
- http:
paths:
- path: /(UK)/my-app/(.*)$
backend:
serviceName: test
servicePort: 80
To summarize our conversation in comments:
There are two Nginx ingress controllers;
One nginx controller is maintained by kubernetes community and the other one by nginx (the company behind nginx product). Here is the github repo for Nginx ingress controller and and here for kubernetes nginx controller.
Nginx controller provided by kubernetes doesn't allow setting client-body-timeout with annotation. Here is a link to github repo with annotations code. This means that what you are left with is either
setting this parameter globally, or
opening feature request on github and waiting for someone to implement it.
client-body-timeout parameter can only be set through global config (as specified in documentation).
Adding to HelloWorlds answer, if someone is looking to provide this annotation globally with Kubernetes version of Ingress then following steps could be followed:
Check in which namespace ingress pod is running. Mostly the namespace name will be something like -ingress-some-string-.
$ kubectl get ns
Lets say the namespace is: 'ingress-nginx'
Now that namespace is known, check pods inside that namespace.
$ kubectl get pods -n ingress-nginx
Lets say you get a pod something like: 'ingress-nginx-controller-abcdefg'
Check the configmap this pod is using the following command:
$ kubectl get pod ingress-nginx-controller-abcdefg -n ingress-nginx -o yaml | grep configmap
You will get an output something like: --configmap=${POD_NAMESPACE}/nginx-configuration
Now, you have to create a config map with above name with required and supported configurations by Kubernetes Ingress.
$ cat global-configmap.yaml
apiVersion: v1
kind: ConfigMap
meta:
name: nginx-configuration
namespace: ingress-nginx
data:
client-body-timeout: "120" # default value is 60 seconds
Now, apply this config map yaml.
$ kubectl apply -f global-configmap.yaml

docker push into private Docker registry inside Kubernetes cluster fails with: 413 Request Entity Too Large

I have deployed a private Docker registry (image registry:2) in a Kubernetes cluster and exposed it via an Ingress. I am using the nginxinc/kubernetes-ingress (not: kubernetes/ingress-nginx) NGINX ingress controller.
curl https://my_registry/v2/_catalog works fine. But docker push into the registry runs into this error: Pushing ... 100.6MB/100.6MB ... 413 Request Entity Too Large.
For what I know, this can be mitigated by instructing the NGINX ingres controller to accept larger chunks of data. I have e.g. tried adding the annotation nginx.ingress.kubernetes.io/proxy-body-size: "200m" into my Ingress specification (as suggested here) but this has not worked so far.
So what is the right way for instructing an nginxinc/kubernetes-ingress NGINX ingress controller to accept sufficiently large chunks?
UPDATE I have meanwhile concluded that nginxinc/kubernetes-ingress does not take its configuration from annotations, but from a ConfigMap named nginx-config that resides in the same namespace as the NGINX ingress controller. I have now added such a ConfigMap with data client-max-body-size: "200m", but the problem still persists.
You need to set Annotation:
nginx.org/client-max-body-size "200m"
I have switched from NGINX Inc.'s to Kubernetes' NGINX ingress controller, and there adding the following annotation to the Ingress' metadata proved sufficient:
annotations:
nginx.ingress.kubernetes.io/proxy-body-size: 500m

kubernetes nginx ingress fails to redirect HTTP to HTTPS

I have a web app hosted in the Google Cloud platform that sits behind a load balancer, which itself sits behind an ingress. The ingress is set up with an SSL certificate and accepts HTTPS connections as expected, with one problem: I cannot get it to redirect non-HTTPS connections to HTTPS. For example, if I connect to it with the URL http://foo.com or foo.com, it just goes to foo.com, instead of https://foo.com as I would expect. Connecting to https://foo.com explicitly produces the desired HTTPS connection.
I have tried every annotation and config imaginable, but it stubbornly refuses, although it shouldn't even be necessary since docs imply that the redirect is automatic if TLS is specified. Am I fundamentally misunderstanding how ingress resources work?
Update: Is it necessary to manually install nginx ingress on GCP? Now that I think about it, I've been taking its availability on the platform for granted, but after coming across information on how to install nginx ingress on the Google Container Engine, I realized the answer may be a lot simpler than I thought. Will investigate further.
Kubernetes version: 1.8.5-gke.0
Ingress YAML file:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: https-ingress
annotations:
kubernetes.io/ingress.class: "nginx"
ingress.kubernetes.io/ssl-redirect: "true"
ingress.kubernetes.io/secure-backends: "true"
ingress.kubernetes.io/force-ssl-redirect: "true"
spec:
tls:
- hosts:
- foo.com
secretName: tls-secret
rules:
- host: foo.com
http:
paths:
- path: /*
backend:
serviceName: foo-prod
servicePort: 80
kubectl describe ing https-ingress output
Name: https-ingress
Namespace: default
Address:
Default backend: default-http-backend:80 (10.56.0.3:8080)
TLS:
tls-secret terminates foo.com
Rules:
Host Path Backends
---- ---- --------
foo.com
/* foo-prod:80 (<none>)
Annotations:
force-ssl-redirect: true
secure-backends: true
ssl-redirect: true
Events: <none>
The problem was indeed the fact that the Nginx Ingress is not standard on the Google Cloud Platform, and needs to be installed manually - doh!
However, I found installing it to be much more difficult than anticipated (especially because my needs pertained specifically to GCP), so I'm going to outline every step I took from start to finish in hopes of helping anyone else who uses that specific cloud and has that specific need, and finds generic guides to not quite fit the bill.
Get Cluster Credentials
This is a GCP specific step that tripped me up for a while - you're dealing with it if you get weird errors like
kubectl unable to connect to server: x509: certificate signed by unknown authority
when trying to run kubectl commands. Run this to set up your console:
gcloud container clusters get-credentials YOUR-K8s-CLUSTER-NAME --z YOUR-K8S-CLUSTER-ZONE
Install Helm
Helm by itself is not hard to install, and the directions can be found on GCP's own docs; what they neglect to mention, however, is that on new versions of K8s, RBAC configuration is required to allow Tiller to install things. Run the following after helm init:
kubectl create serviceaccount --namespace kube-system tiller
kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
kubectl patch deploy --namespace kube-system tiller-deploy -p '{"spec":{"template":{"spec":{"serviceAccount":"tiller"}}}}'
Install Nginx Ingress through Helm
Here's another step that tripped me up - rbac.create=true is necessary for the aforementioned RBAC factor.
helm install --name nginx-ingress-release stable/nginx-ingress --set rbac.create=true
Create your Ingress resource
This step is the simplest, and there are plenty of sample nginx ingress configs to tweak - see #JahongirRahmonov's example above. What you MUST keep in mind is that this step takes anywhere from half an hour to over an hour to set up - if you change the config and check again immediately, it won't be set up, but don't take that as implication that you messed something up! Wait for a while and see first.
It's hard to believe this is how much it takes just to redirect HTTP to HTTPS with Kubernetes right now, but I hope this guide helps anyone else stuck on such a seemingly simple and yet so critical need.
GCP has a default ingress controller which at the time of this writing cannot force https.
You need to explicitly manage an NGINX Ingress Controller.
See this article on how to do that on GCP.
Then add this annotation to your ingress:
kubernetes.io/ingress.allow-http: "false"
Hope it helps.

Resources