I have a deployment which comprises 2 pods with a webapp in them. Nginx ingress controller is installed to expose those 2 pods using tls-passthrough.
I have tried to use annotations to setup sticky-sessions but to no avail. No cookie is present in the headers hence no stickyness.
What could I do to make it work ?
Many thanks for your wise answers.
The Ingress I have created :
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress
annotations:
# nginx.ingress.kubernetes.io/secure-backends: "true"
# nginx.ingress.kubernetes.io/ssl-passthrough: "true"
# nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
# nginx.ingress.kubernetes.io/rewrite-target: /
nginx.ingress.kubernetes.io/affinity: "cookie"
nginx.ingress.kubernetes.io/affinity-mode: "persistent"
nginx.ingress.kubernetes.io/session-cookie-name: "wabam"
nginx.ingress.kubernetes.io/session-cookie-expires: "172800"
nginx.ingress.kubernetes.io/session-cookie-max-age: "172800"
namespace: default
spec:
rules:
- host: wabam.moulip.lan
http:
paths:
- backend:
serviceName: wabam-ingress-svc
servicePort: 443
path: /
I am posting this as a community wiki answer for better visibility.
As already mentioned in the comments: any other annotation will not work when used alongside the ssl-passthrough. This is also being mentioned by the official docs:
Because SSL Passthrough works on layer 4 of the OSI model (TCP) and
not on the layer 7 (HTTP), using SSL Passthrough invalidates all the
other annotations set on an Ingress object.
Related
I am trying to add authentication to my cluster by using an oauth2-proxy. Locally I tested and is working as expected: When I go to the landing page it redirects me to an IP such as GitHub. After login, I am redirected to my page and everything is working as expected. For some odd reason, when porting from local to the cluster living in the google cloud, I am not getting redirect to GitHub for authentication.
I have two ingresses, one for the oauth2-proxy (ingress-oauth2-proxy) and another for all of the apps (ingress-apps). I added the Nginx annotations and still nothing.
Here are the Ingress YAML file used for the creation of the ingress rules
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
acme.cert-manager.io/http01-edit-in-place: "true"
ingress.kubernetes.io/force-ssl-redirect: "true"
ingress.kubernetes.io/ssl-redirect: "true"
nginx.ingress.kubernetes.io/auth-url: "https://auth.example.com/oauth2/auth"
nginx.ingress.kubernetes.io/auth-signin: "https://auth.exmaple.com/oauth2/start?rd=$escaped_request_uri"
cert-manager.io/cluster-isuer: letsecnrypt-prod
nginx.ingress.kubernetes.io/rewrite-target: $/1
name: ingress-apps
namespace: default
spec:
rules:
- host: echo.example.com
http:
paths:
- path: /
backend:
serviceName: echo1
servicePort: 80
tls:
- hosts:
- echo.exmaple.com
secretName: echo-tls-cert
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: ingress-oauth2-proxy
namespace: default
annotations:
kubernetes.io/ingress.class: nginx
acme.cert-manager.io/http01-edit-in-place: "true"
ingress.kubernetes.io/force-ssl-redirect: "true"
ingress.kubernetes.io/ssl-redirect: "true"
spec:
rules:
- host: auth.exmaple.com
http:
paths:
- backend:
serviceName: oauth2-proxy
servicePort: 4180
path: /oauth2
tls:
- hosts:
- auth.exmaple.com
secretName: auth-tls-cert
I checked the logs of the OAuth proxy and when I go to echo.example.com nothing happens. If I make a request to auth.example.com/oauth2 I get redirected, as expected, to the IP login page which is GitHub in this case.
Am I missing something?
Note: I checked and both ingresses are being applied.
Ok, I figured it out: The problem was the Ingress controller used. I installed the Nginx controller from the repo helm.nginx.com/stable which is nginxinc and does not support the annotations. Therefore they were being ignored. To fix it I just used the kubernetes.github.io/ingress-nginx/deploy/#gce-gke. This is better explained here Nginx ingress controller authentication not working
Cheers ! :)
I would like to use some kubernetes ingress (doesn't matter the ingress type) on google cloud GKE 1.19.7-gke.1302, and nginx ingress deployed by nginx-ingress-0.8.0 helm chart, and my need is to fetch a cookie and use it later to authenticate to our service. When I use nginx as deployment it works, but when I use nginx ingress it seems to ignore the annotation.
I tried using this configuration snippet for the nginx ingress, which works on the nginx deployment but not on the ingress:
nginx.ingress.kubernetes.io/configuration-snippet: |
proxy_set_header "Authorization" "Basic $cookie_myToken";
And this is my entire ingress yaml:
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: my-nginx-ingress
namespace: default
annotations:
kubernetes.io/ingress.class: nginx
# enable backend redirections
kubernetes.io/tls-acme: "true"
# enable stickiness
nginx.ingress.kubernetes.io/affinity: "cookie"
nginx.ingress.kubernetes.io/session-cookie-name: "route"
nginx.ingress.kubernetes.io/session-cookie-expires: "172800"
nginx.ingress.kubernetes.io/session-cookie-max-age: "172800"
nginx.ingress.kubernetes.io/configuration-snippet: |
proxy_set_header "Authorization" "Basic $cookie_myToken";
spec:
tls:
- secretName: my-tls
hosts:
- "test.com"
defaultBackend:
service:
name: my-nodeport
port:
number: 80
rules:
- host: "test.com"
http:
paths:
- path: /
pathType: ImplementationSpecific
backend:
service:
name: my-nodeport
port:
number: 80
Any ideas?
Posting this answer to make it more visible as issue was identified and solved in comment section.
Root cause of this issue was Nginx Ingress Controller misconfiguration.
In the OP's setup Nginx Ingress Controller, v.0.8.0 was used. After redeploying Nginx Ingress to latest version (v0.44.0) from Nginx Ingress Documentation it started working properly.
Yes, reinstalling nginx ingress to the latest version with helm solved the problem.
I'd like to split incoming traffic in Kubernetes Nginx in the following way:
Client --> Nginx --> {Service A, Service B}
The problem I am facing is Service A is an internal service and does not support HTTPS therefore SSL should be terminated for Service A. On the other hand, Service B is an external service (hosted on example.com) and only works over HTTPS.
I cannot manage to get this work easily with Kubernetes Nginx. Here is what I have come with:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress-proxy
annotations:
nginx.ingress.kubernetes.io/backend-protocol: HTTPS
nginx.ingress.kubernetes.io/upstream-vhost: example.com
spec:
tls:
- hosts:
- proxy.com
secretName: secret
rules:
- host: proxy.com
http:
paths:
- path: /api/v1/endpoint
backend:
serviceName: service-a
servicePort: 8080
- path: /
backend:
serviceName: service-b
servicePort: 443
kind: Service
apiVersion: v1
metadata:
name: service-b
namespace: default
spec:
type: ExternalName
externalName: service-b.external
ports:
- port: 443
I have got a route for service-b.external:443 to point to example.com.
This solution only works if service-b is over HTTPS, but in my case, I cannot change to HTTPS for this service because of some other internal dependencies.
My problem is the backend-protocol annotation works for the whole kind and I cannot define it per path.
P.S: I am using AWS provider
Following the suggested solution and question from comments.
Yes, like mentioned below it is possible to have two ingress items. In your case
only one should have backend-protocol in it.
According to nginx ingress documentation:
Basic usage - host based routing¶
ingress-nginx can be used for many use cases, inside various cloud provider and supports a lot of configurations. In this section you can find a common usage scenario where a single load balancer powered by ingress-nginx will route traffic to 2 different HTTP backend services based on the host name.
First of all follow the instructions to install ingress-nginx. Then imagine that you need to expose 2 HTTP services already installed: myServiceA, myServiceB. Let's say that you want to expose the first at myServiceA.foo.org and the second at myServiceB.foo.org. One possible solution is to create two ingress resources:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: ingress-myservicea
annotations:
# use the shared ingress-nginx
kubernetes.io/ingress.class: "nginx"
spec:
rules:
- host: myservicea.foo.org
http:
paths:
- path: /
backend:
serviceName: myservicea
servicePort: 80
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: ingress-myserviceb
annotations:
# use the shared ingress-nginx
kubernetes.io/ingress.class: "nginx"
spec:
rules:
- host: myserviceb.foo.org
http:
paths:
- path: /
backend:
serviceName: myserviceb
servicePort: 80
When you apply this yaml, 2 ingress resources will be created managed by the ingress-nginx instance. Nginx is configured to automatically discover all ingress with the kubernetes.io/ingress.class: "nginx" annotation. Please note that the ingress resource should be placed inside the same namespace of the backend resource.
On many cloud providers ingress-nginx will also create the corresponding Load Balancer resource. All you have to do is get the external IP and add a DNS A record inside your DNS provider that point myServiceA.foo.org and myServiceB.foo.org to the nginx external IP. Get the external IP by running:
kubectl get services -n ingress-nginx
It is also possible to have separate nginx classes as mentioned here.
I'm trying to implement SSL Passtrough with nginx-ingress-controller. This is my Ingress Object:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/backend-protocol: HTTPS
nginx.ingress.kubernetes.io/ssl-passthrough: "true"
nginx.ingress.kubernetes.io/ssl-redirect: "true"
labels:
my-label: example
name: example
namespace: example
spec:
rules:
- host: '*.example.com'
http:
paths:
- backend:
serviceName: example
servicePort: 8443
path: /
The --enable-ssl-passtrough flag is present in the controller args.
When request is coming trough ingress controller to my underlying service I'm trying to parse the SNI to find out what domain was used to find out the certificate I should present, but the service cannot find the SNI and returns this error:
{"level":"debug","ts":1592992137.1836417,"msg":"Error getting server name","error":"No hostname"}
Does nginx-ingress-controller remove the SNI when parsing? Or what may be the reason for such behavior?
Thanks in advance for help
I contacted nginx-ingress developers directly and I got information that the reason this is not working is the wildcard domain, which is not supported by nginx-ingress.
Everything else is configured correctly and when changing *.example.com into something specific (e.g. whatever.example.com) it works correctly.
I try to port an ASP.NET Core 1 application with Identity to Kubernetes. The login doesn't work and I got different errors like The anti-forgery token could not be decrypted. The problem is that I'm using a deployment with three replica sets so that further request were served by different pods that don't know about the anti-forgery token. Using replicas: 3 it works.
In the same question I found a sticky session documentation which seems a solution for my problem. The cookie name .AspNetCore.Identity.Application is from my browser tools.
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: myapp-k8s-test
annotations:
nginx.ingress.kubernetes.io/affinity: "cookie"
nginx.ingress.kubernetes.io/session-cookie-name: ".AspNetCore.Identity.Application"
spec:
replicas: 3
template:
metadata:
labels:
app: myapp-k8s
spec:
containers:
- name: myapp-app
image: myreg/myapp:0.1
ports:
- containerPort: 80
env:
- name: "ASPNETCORE_ENVIRONMENT"
value: "Production"
imagePullSecrets:
- name: registrypullsecret
This doesn't work, either with or without leading dot at the cookie name. I also tried adding the following annotations
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/session-cookie-hash: sha1
What is required to allow sticky sessions on Kubernetes with ASP.NET Core?
Found out that I made two logical mistakes:
Sticky sessions doesn't work this way
I assumed that Kubernetes will look into the cookie and create some mapping of cookie hashes to pods. But instead, another session is generated and append to our http header. nginx.ingress.kubernetes.io/session-cookie-name is only the name of those generated cookie. So per default, it's not required to change them.
Scope to the right object
The annotations must be present on the ingress, NOT the deployment (stupid c&p mistake)
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: myapp-k8s-ingress
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/affinity: "cookie"
nginx.ingress.kubernetes.io/session-cookie-hash: sha1
spec:
tls:
- hosts:
- myapp-k8s.local
rules:
- host: myapp-k8s.local
http:
paths:
- path: /
backend:
serviceName: myapp-svc
servicePort: 80
This works as expected.