I am doing some research on how to implement https secure connection between Nginx Ingress -> backend services. So far I have SSL setup in Nginx Ingress controller that uses Lets Encrypt cert manager to rotate certificate using http-01 challenge.
Here is my scenario:
Client from internet -> 2. Load balancer -> 3. Ingress Controller (that terminates TLS traffic) -> 4. Service (port 80) -> 5. Pod (port 80).
So my question is how can I secure communication between ingress controller and pod so that traffic is encrypted end to end? Do I need my own certificate authority to do that? If so, are there any open source solution that can handle certificate management just like Cert manager?
1. Nginx ingress controller + DAPR
I am not sure I can post here youtube urls(at least I have never seen anyone doing that) but.. I think this is 100% exactly what you want. Your scenario is discussed in 1st topic, you need watch only it. Plus as a benefit - you will see step-by-step installation there. Personally I found that video very helpful
Secure Ingress pods communication
2. You can achieve that with Istio itself.
Istio By Example!:Secure Ingress
3. Istio + Calico network policy for Istio
Enforce network policy for Istio
The Calico support for Istio service mesh has the following benefits:
-Pod traffic controls
Lets you restrict ingress traffic inside and outside pods and mitigate common threats to Istio-enabled apps.
-Supports security goals
Enables adoption of a zero trust network model for security, including traffic encryption, multiple enforcement points, and multiple identity criteria for authentication.
Replace let`s encrypt with aws certificates because they are free.
Validate your domains you use inside you cluster and then edit the main service of your ingress controller.Use this annotations if you like if you use aws.
annotations:
service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:XXXXXXXX"
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "http"
service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "https"
service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "60"
service.beta.kubernetes.io/aws-load-balancer-ssl-negotiation-policy:"ELBSecurityPolicy-TLS-1-2-2017-01"
Related
I've read everywhere that to set Https to access a kubernetes cluster you need to have an Ingress and not simply a LoadBalancer service which also exposes the cluster outside.
My question is pretty theoretical: if an Ingress (and it is) is composed of a LoadBalancer service, a Controller (a deployment/pod of an nginx image for example) and a set of Rules (in order to correctly proxy the incoming requests inside the cluster), why can't we set Https in front of a LoadBalancer instead of an Ingress?
As title of exercise I've built the three components separately by myself (a LoadBalancer, a Controller/API Gateway with some Rules): these three together already get the incoming requests and proxy them inside the cluster according to specific rules so, I can say, I have built an Ingress by myself. Can't I add https to this structure and do I need to set a redundant part (a k8s Ingress) in front of the cluster?
Not sure if I fully understood your question.
In Kubernetes you are exposing you cluster/application using service, which is well described here. Good compare of all services can be found in this article.
When you are creating service type LoadBalancer it creates L4 LoadBalancer. L4 is aware of information like source IP:port and destination IP:port, but don't have any information about application layer (Layer 7). HTTP/HTTPS LoadBalancers are on Layer 7, so they are aware of application. More information about Load Balancing can be found here.
Layer 4-based load balancing to direct traffic based on data from network and transport layer protocols, such as IP address and TCP or UDP port
Layer 7-based load balancing to add content-based routing decisions based on attributes, such as the HTTP header and the uniform resource identifier
Ingress is something like LoadBalancer with L7 support.
The Ingress is a Kubernetes resource that lets you configure an HTTP load balancer for applications running on Kubernetes, represented by one or more Services. Such a load balancer is necessary to deliver those applications to clients outside of the Kubernetes cluster.
Ingress also provides many advantages. For example if you have many services in your cluster you can create one LoadBalancer and Ingress which will be able to redirect traffic to proper service and allows you to cut costs of creating a few LoadBalancers.
In order for the Ingress resource to work, the cluster must have an ingress controller running.
The Ingress controller is an application that runs in a cluster and configures an HTTP load balancer according to Ingress resources. The load balancer can be a software load balancer running in the cluster or a hardware or cloud load balancer running externally. Different load balancers require different Ingress controller implementations.
In the case of NGINX, the Ingress controller is deployed in a pod along with the load balancer.
There are many Ingress Controllers, but the most popular is Nginx Ingress Controller
So my answer regarding:
why can't we set Https in front of a LoadBalancer instead of an Ingress?
It's not only about securing your cluster using HTTPS but also many capabilities and features which Ingress provides.
Very good documentation regarding HTTP(S) Load Balancing can be found on GKE Docs.
I am a kubernetes newbie, and no matter how much I read about it I cannot get my head around this issue.
I have a simple deployment , which is creating a pod with a not so complex app.
I know what an ingress and an ingress controller is doing ,but as far as I understand it is not required for me to expose my pod-app externally. Only a LoadBalancer service should be enough.
I do not have need of more than one rule for traffic routing.
Am I wrong about that?
Traditionally, you would create a LoadBalancer service for each service you want to expose externally. This can get rather expensive. Ingress gives you a way to route requests to services based on the request host or path, centralizing a number of services into a single entrypoint.
Also load balancer provisioning takes time and works only in supported cloud providers such as AWS, GCP etc.
One more thing to consider is the need of L4(TCP/UDP) layer routing because kubernetes Ingress API is primarily L7 layer but some of the ingress controllers such as traefik, nginx supports L4 layer(TCP/UDP) along with L7 layer(HTTP) routing.
So answer to your question is it depends based on your environment and use cases.
Ingress and IngressControllers are used for traffic routing at layer 7 i.e., if your backends talk L7 protocols like HTTP, GRPC etc. You can route requests based on request path to different backend services using Ingress.
If your app doesn't operate at Layer 7, you might not need an Ingress.
Another question you could ask yourself if your migrating your app from non-kubernetes environment to kuberneters is - are you using a reverse proxy like nginx already? If so, you might want to use Ingress. I say might because it is not necessary. You can achieve the same affect of using an Ingress by running the nginx container as a pod, writing the nginx.conf yourself and making it available externally(using a LoadBalancer service for example). Instead by using IngressController you need not maintain the nginx pod or write nginx.conf. You can instead express the same configuration as Ingress resource which is much simpler.
Ingress is needed if you want to expose your service to external. Especially for layer 7 in OSI model (HTTP transport). Ingress also provide the mechanism to enable TLS support in your load balancer. Traffic routing is controlled by rules defined on the Ingress resource. An Ingress can be configured to give Services externally-reachable URLs, load balance traffic, terminate SSL / TLS, and offer name based virtual hosting. An Ingress controller is responsible for fulfilling the Ingress, usually with a load balancer, though it may also configure your edge router or additional frontends to help handle the traffic.
By default Ingress controller depends on which kind of cloud provider you're using, or if you're using on premise you'll need to set it up based on what you need. In one cluster, you can create multiple Ingress controller as well. There's many kinds of Ingress controller, you can take a look on this article.
I think this article about Load Balancer and Ingress may also help.
The Problem
I need to expose a Kubernetes NodePort service externally over https.
The Setup
I've deployed Kubernetes on bare-metal and have deployed Polyaxon on the cluster via Helm
I need to access Polyaxon's dashboard via the browser, using a virtual machine that's external to the cluster
The dashboard is exposed as a NodePort service, and I'm able to connect to it over http. I am not able to connect over https, which is a hard requirement in my case.
Following an initial "buildout" period, both the cluster and the virtual machine will not have access to the broader internet. They will connect to one another and that's it.
Polyaxon supposedly supports SSL/TLS through its own configs, but there's very little documentation on this. I've made my best attempts to solve the issue that way and also bumped an issue on their github, but haven't had any luck so far.
So I'm now wondering if there might be a more general Kubernetes hack that could help me here.
The Solutions
I'm looking for the simplest solution, rather than the most elegant or scalable. There are also some things that might make my situation simpler than the average user who would want https, namely:
It would be OK to support https on just one node, rather than every node
I don't need (or really want) a domain name; connecting at https://<ip_address>:<port> is not just OK but preferred
A self-signed certificate is also OK
So I'm hoping there's some way to manipulate the NodePort service directly such that https will work on the virtual machine. If that's not possible, other solutions I've considered are using an Ingress Controller or some sort of proxy, but those solutions are both a little half-baked in my mind. I'm a novice with both Kubernetes and networking ideas in general, so if you're going to propose something more complex please speak very slowly :)
Thanks a ton for your help!
Ingress-controller it's a standard way to expose HTTP backend over TLS connection from cluster to client.
Existing NodePort service has ClusterIP which can be used as a backend for Ingress. ClusterIP type of service is enough, so you can change service type later to prevent HTTP access via nodeIP:nodePort.
Ingress-controller allows you to teminate TLS connection or pass-through TLS traffic to the backend.
You can use self-signed certificate or use cert-manager with Let's encrypt service.
Note, that starting from 0.22.0 version Nginx-ingress rewrite syntax has changed and some examples in the articles may be outdated.
Check the links:
TLS termination
TLS/HTTPS
How to get Kubernetes Ingress to terminate SSL and proxy to service?
Configure Nginx Ingress Controller for TLS termination on Kubernetes on Azure
My problem is simple. I have an AKS deployment with a LoadBalancer service that needs to use HTTPS with a certificate.
How do I do this?
Everything I'm seeing online involves Ingress and nginx-ingress in particular.
But my deployment is not a website, it's a Dropwizard service with a REST API on one port and an admin service on another port. I don't want to map the ports to a path on port 80, I want to keep the ports as is. Why is HTTPS tied to ingress?
I just want HTTPS with a certificate and nothing more changed, is there a simple solution to this?
A sidecar container with nginx with the correct certificates (possible loaded off a Secret or a ConfigMap) will do the job without ingress. This seems to be a good example, using nginx-ssl-proxy container.
Yes, that's right as of this writing an Ingress will currently work either on port 80 or port 443, potentially it can be extended to use any port because nginx, Traefik, haproxy, etc can all listen on different ports.
So you are down to either a LoadBalancer or a NodePort type of service. Type LoadBalancer will not work directly with TLS since the Azure load balancers are layer 4. So you will have to use Application Gateway and it's preferred to use an internal load balancer for security reasons.
Since you are using Azure you can run something like this (assuming that your K8s cluster is configured the right way to use the Azure cloud provider, either the --cloud-provider option or the cloud-controller-manager):
$ cat <<EOF
apiVersion: v1
kind: Service
metadata:
name: your-app
annotations:
service.beta.kubernetes.io/azure-load-balancer-internal: "true"
spec:
type: LoadBalancer
ports:
- port: <your-port>
selector:
app: your-app
EOF | kubectl apply -f -
and that will create an Azure load balancer on the port you like for your service. Behind the scenes, the load balancer will point to a port on the nodes and within the nodes, there will be firewall rules that will route to your container. Then you can configure Application Gateway. Here's a good article describing it but using port 80, you will have to change it use port 443 and configuring the TLS certs, and the Application Gateway also supports end to end TLS in case you want to terminate TLS directly on your app too.
The other option is NodePort, and you can run something like this:
$ kubectl expose deployment <deployment-name> --type=NodePort
Then Kubernetes will pick a random port on all your nodes where you can send traffic to your service listening on <your-port>. So, in this case, you will have to manually create a load balancer with TLS or a traffic source that listens on TLS <your-port> and forwards it to a NodePort on all your nodes, this load balancer can be anything like haproxy, nginx, Traefik or something else that supports terminating TLS. And you can also use the Application Gateway to forward directly to your node ports, in other words, define a listener that listens on the NodePort of your cluster.
I have a specific use case that I can not seem to solve.
A typical gcloud setup:
A K8S cluster
A gcloud storage bucket
A gcloud loadbalancer
I managed to get my domain https://cdn.foobar.com/uploads/ to points to a google storage backend without any issue: I can access files. Its the backend service one that fails.
I would like the CDN to act as a cache, when a HTTP request hits it such as https://cdn.foobar.com/assets/x.jpg, if it does not have a copy of the asset it should query an other domain https://foobar.com/assets/x.jpg.
I understood that this what was load balancers backend-service were for. (Right?)
The backend-service is pointing to the instance group of the k8s cluster and requires a port. I guessed that I needed to allow the firewall to expose the Nodeport of my web application service for the loadbalancer to be able to query it.
Cloud CDN
Load balancing
Failing health-checks.
The backend service is pointing to the instance group of the k8s cluster and requires some ports (default 80?) 80 failed. I guessed that I needed to allow the firewall to expose the 32231 Nodeport of my web application service for the loadbalancer to be able to query it. That still failed with a 502.
?> kubectl describe svc
Name: backoffice-service
Namespace: default
Labels: app=backoffice
Selector: app=backoffice
Type: NodePort
IP: 10.7.xxx.xxx
Port: http 80/TCP
NodePort: http 32231/TCP
Endpoints: 10.4.x.x:8500,10.4.x.x:8500
Session Affinity: None
No events.
I ran out of ideas at this point.
Any hints int the right direction would be much appreciated.
When deploying your service as type 'NodePort', you are exposing the service on each Node's IP, but the service is not reachable to the exterior, so you need to expose your service as 'LoadBalancer'
Since you're looking to use an HTTP(s) Load Balancer, I'll recommend using a Kubernetes Ingress resource. This resource will be in charge of configuring the HTTP(s) load balancer and the required ports that your service is using, as well as the health checks on the specified port.
Since you're securing your application, you will need to configure a secret object for securing the Ingress.
This example will help you getting started on an Ingress with TLS termination.