I have a domain at Cloudflare and some wildcards for subdomains
which both point to the load balancer of an nginx ingress of a Kubernetes cluster (GKE) of the GCP. Now, we have two pods and services running each (echo1 and echo2, which are essentially identical) and when I apply an ingress
kind: Ingress
metadata:
name: echo-ingress
annotations:
kubernetes.io/ingress.class: "nginx"
nginx.ingress.kubernetes.io/ssl-redirect: "false"
spec:
rules:
- host: "echo1.eu3.example.com"
http:
paths:
- pathType: Prefix
path: "/"
backend:
service:
name: echo1
port:
number: 80
- host: "echo2.example.com"
http:
paths:
- pathType: Prefix
path: "/"
backend:
service:
name: echo2
port:
number: 80
I can reach echo2 under echo2.example.com, but not echo1.eu3.example.com. My question is how I can make the second one reachable as well.
I can advise you to make some check.
Just set the Proxy status for "echo1.eu3.example.com" as DNS only. Then check the access. If ok - install certificates in kubernetes via cert manager. We faced some times with this issue and resolved by using 3 deep domains. For instance "echo1-eu3.example.com". It seems cloudfront does not like such domains :) Of course if someone write a solution how to work with deep domains in cloudfront - it would be good practice for us :)
Related
We have a Kubernetes cluster with an nginx controller.
We are using this nginx controller to route different paths to different services. Though,
when a service is on a different namespace, I cannot figure out how to redirect to it.
This is the nginx configuration
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: gateway-ingress
annotations:
kubernetes.io/ingress.class: "nginx"
kubernetes.io/tls-acme: 'true'
nginx.ingress.kubernetes.io/proxy-body-size: "0"
nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
nginx.ingress.kubernetes.io/enable-cors: "true"
nginx.ingress.kubernetes.io/cors-allow-methods: "*"
nginx.ingress.kubernetes.io/cors-allow-headers: "*"
nginx.ingress.kubernetes.io/cors-allow-origin: "*"
nginx.ingress.kubernetes.io/service-upstream: "true"
spec:
rules:
- host: MY_BACKEND.MY_DOMAIN.com
http:
paths:
- pathType: Prefix
path: "/"
backend:
service:
name: my-backend
port:
number: 3001
- host: MY_FRONTEND.MY_DOMAIN.com
http:
paths:
- pathType: Prefix
path: "/"
backend:
service:
name: my-frontend
port:
number: 3000
...
The above snippet works perfectly, but it is worth to mention that both services my-backend and my-frontend are running on default namespace.
We have then deployed a Jenkins instance on namespace jenkins. My first attempty was to simply add
- host: MY_JENKINS.MY_DOMAIN.com
http:
paths:
- pathType: Prefix
path: "/"
backend:
service:
name: jenkins-service
port:
number: 8080
Though when browsing on MY_JENKINS.MY_DOMAIN.com I get a 503 Service Temporary Unavailable. Could it be because it is running on a different namespace? Namely jenkins?
Normally in order to connect on different namespaces I just use the standard service.namespace. Though here when trying I get a standard console error/warning
The Ingress "gateway-ingress" is invalid: spec.rules[16].http.paths[0].backend.service.name: Invalid value: "WHATEVER_VALUE_WRITTEN": a DNS-1035 label must consist of lower case alphanumeric characters or '-', start with an alphabetic character, and end with an alphanumeric character (e.g. 'my-name', or 'abc-123', regex used for validation is '[a-z]([-a-z0-9]*[a-z0-9])?')
I tried: service.namespace, namespace.service, namespace/service, service/namespace. All the attempts and up on the same error message.
How can I instruct nginx-controller to connect to a service that is in a specific namespace?
You can reach a service from another namespace with: <service>.<namespace>; by bypassing the service you can also reach other pods by their ip address (even in another namespace).
But afaik all ingress controllers (also nginx) per default do not allow multi-namespace ingress out of security reasons.
For nginx you can configure ingress across multiple namespaces with a master/minion approach. You have to add the nginx.org/mergeable-ingress-type-annotation to your Ingress resources.
A master processes configuration on the host and minions provide the ingress resources.
Find more info here: https://docs.nginx.com/nginx-ingress-controller/configuration/ingress-resources/cross-namespace-configuration/
And here:
https://github.com/nginxinc/kubernetes-ingress/tree/v2.2.1/examples/mergeable-ingress-types
I am working with ingress-nginx in kubernetes to set up a server.
The issue is that the paths are not routing at all and I get a 404 error from the nginx server on any request I make.
Below is my code for ingress:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-srv
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/use-regex: "true"
spec:
# defaultBackend:
# service:
# name: auth-srv
# port:
# number: 3000
rules:
- host: app.dev
- http:
paths:
- pathType: Prefix
path: /api/auth/?(.*)
backend:
service:
name: auth-srv
port:
number: 3000
- path: /api/coms/?(.*)
pathType: Prefix
backend:
service:
name: coms-srv
port:
number: 3000
If I uncomment the default backend service I get a response but as soon as I remove it I get the 404 nginx error. So I know its connecting to the services I set.
I don't know where I'm going wrong how to go about fixing this as I'm copying straight from the docs. Any help or insight would be great. Thank you in advance!
Edit 1: I removed the regex from the path and commented out the /api/auth path so no requests should be going to the auth-srv. For some reason, all requests route to the auth-srv even though there is no mapping to it. NOTE: Both the auth and coms pods/services are running in the background, just ingress-nginx still isn't routing properly.
So the reason why this wasn't routing properly was because of the:
- host: app.dev
- http:
The "-" in front of the "http" made the controller think it was its own ruleset so the following routes had a host of "*". After I Removed the "-" in front of the "http", the rules were set to the proper host of app.dev and it started routing accordingly.
Thank you for your help everyone! What a long day it has been :')
I have below ingress-nginx configuration file.
It rewrites request for one.example.com. I have added another domain, but I don't want rewrite to happen for other domain.
I went through doc, but there is now rewrite example for multiple host setup.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-service
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/use-regex: "true"
nginx.ingress.kubernetes.io/rewrite-target: /$1
certmanager.k8s.io/cluster-issuer: "letsencrypt-example-prod"
nginx.ingress.kubernetes.io/ssl-redirect: "true"
spec:
tls:
- hosts:
- one.example.com
- two.example.com
secretName: super-secret
rules:
- host: one.example.com
http:
paths:
- path: /customer/?(.*)
pathType: Prefix
backend:
service:
name: customer-srv
port:
number: 3000
host: two.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: kube-prometheus-grafana
port:
number: 80
I could create two separate ingress files for each host.
What will happen in that case it will create two load balancers with different dns?
(cluster is hosted on aws)
You can create two separate ingress.
In that case, nothing will change your ingress controller IP would be same only for DNS.
What will happen in that case it will create two load balancers with
different dns?
No, it won't create two load balancers.
I'd like to split incoming traffic in Kubernetes Nginx in the following way:
Client --> Nginx --> {Service A, Service B}
The problem I am facing is Service A is an internal service and does not support HTTPS therefore SSL should be terminated for Service A. On the other hand, Service B is an external service (hosted on example.com) and only works over HTTPS.
I cannot manage to get this work easily with Kubernetes Nginx. Here is what I have come with:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress-proxy
annotations:
nginx.ingress.kubernetes.io/backend-protocol: HTTPS
nginx.ingress.kubernetes.io/upstream-vhost: example.com
spec:
tls:
- hosts:
- proxy.com
secretName: secret
rules:
- host: proxy.com
http:
paths:
- path: /api/v1/endpoint
backend:
serviceName: service-a
servicePort: 8080
- path: /
backend:
serviceName: service-b
servicePort: 443
kind: Service
apiVersion: v1
metadata:
name: service-b
namespace: default
spec:
type: ExternalName
externalName: service-b.external
ports:
- port: 443
I have got a route for service-b.external:443 to point to example.com.
This solution only works if service-b is over HTTPS, but in my case, I cannot change to HTTPS for this service because of some other internal dependencies.
My problem is the backend-protocol annotation works for the whole kind and I cannot define it per path.
P.S: I am using AWS provider
Following the suggested solution and question from comments.
Yes, like mentioned below it is possible to have two ingress items. In your case
only one should have backend-protocol in it.
According to nginx ingress documentation:
Basic usage - host based routing¶
ingress-nginx can be used for many use cases, inside various cloud provider and supports a lot of configurations. In this section you can find a common usage scenario where a single load balancer powered by ingress-nginx will route traffic to 2 different HTTP backend services based on the host name.
First of all follow the instructions to install ingress-nginx. Then imagine that you need to expose 2 HTTP services already installed: myServiceA, myServiceB. Let's say that you want to expose the first at myServiceA.foo.org and the second at myServiceB.foo.org. One possible solution is to create two ingress resources:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: ingress-myservicea
annotations:
# use the shared ingress-nginx
kubernetes.io/ingress.class: "nginx"
spec:
rules:
- host: myservicea.foo.org
http:
paths:
- path: /
backend:
serviceName: myservicea
servicePort: 80
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: ingress-myserviceb
annotations:
# use the shared ingress-nginx
kubernetes.io/ingress.class: "nginx"
spec:
rules:
- host: myserviceb.foo.org
http:
paths:
- path: /
backend:
serviceName: myserviceb
servicePort: 80
When you apply this yaml, 2 ingress resources will be created managed by the ingress-nginx instance. Nginx is configured to automatically discover all ingress with the kubernetes.io/ingress.class: "nginx" annotation. Please note that the ingress resource should be placed inside the same namespace of the backend resource.
On many cloud providers ingress-nginx will also create the corresponding Load Balancer resource. All you have to do is get the external IP and add a DNS A record inside your DNS provider that point myServiceA.foo.org and myServiceB.foo.org to the nginx external IP. Get the external IP by running:
kubectl get services -n ingress-nginx
It is also possible to have separate nginx classes as mentioned here.
I'm currently using Kubernetes on GKE to serve the various parts of my product on different subdomains with the Ingress resource. For example: api.mydomain.com, console.mydomain.com, etc.
ingress.yml (current):
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress
spec:
rules:
- host: api.mydomain.com
http:
paths:
- backend:
serviceName: api-service
servicePort: 80
- host: console.mydomain.com
http:
paths:
- backend:
serviceName: console-service
servicePort: 80
That works wonderfully, with the L7 GCE load balancer routing to the appropriate places. What I would like to do, however, is deploy many feature-branch deployments as subdomains to test and demonstrate new features before pushing to production. These could be something like new-stylesheet.console.mydomain.com or upgraded-algorithm.api.mydomain.com, inspired by GitLab CI's environments.
Here's a potential workflow for each deployment:
Create feature-api-deployment.yml
Create feature-api-service.yml
Update ingress.yml with new subdomain rule: feature.api.mydomain.com specifying serviceName: feature-api-service
But enumerating and maintaining all subdomain->service mappings will get messy with tearing down deployments, and create a ton of GCE backends (default quota is 5...) so it's not ideal.
Is there anything built in to Kubernetes that I'm overlooking to handle this? Something like this would be ideal to pick a target service based on a matched subdomain:
ingress.yml (wanted)
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress
spec:
rules:
- host: *.api.mydomain.com
http:
paths:
- backend:
serviceName: {value of *}-api-service
servicePort: 80
There certainly isn't anything like wildcard domains available in kubernetes, but you might be able to get want you want using Helm
With helm, you can template variables inside your manifests, so you can have the name of your branch be in the helm values file
From there, you can have gitlab-ci do the helm installation in a build pipeline and if you configure your chart correctly, you can specify a helm argument of the pipeline name.
There's a great blog post about this kind of workflow here - hopefully this'll get your where you need to go.
here is an example:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-wildcard-host
spec:
rules:
- host: "foo.bar.com"
http:
paths:
- pathType: Prefix
path: "/bar"
backend:
service:
name: service1
port:
number: 80
- host: "*.foo.com"
http:
paths:
- pathType: Prefix
path: "/foo"
backend:
service:
name: service2
port:
number: 80
and the source.
Services are available locally as
my-svc.svc.cluster.local
You can write a simple NGINX proxy which can forward which can parse the subdomain and proxy pass it to http://$service as long as the subdomain and service name matches as in this case.