We make use of Ingress to create HTTPS load balancers that forward directly to our (typically nodejs) services. However, recently we have wanted more control of traffic in front of nodejs which the Google load balancer doesn't provide.
Standardised, custom error pages
Standard rewrite rules (e.g redirect http to https)
Decouple pod readinessProbes from load balancer health checks (so we can still serve custom error pages when there are no healthy pods).
We use nginx in other parts of our stack so this seems like a good choice, and I have seen several examples of nginx being used to front services in Kubernetes, typically in one of two configurations.
An nginx container in every pod forwarding traffic directly to the application on localhost.
A separate nginx Deployment & Service, scaled independently and forwarding traffic to the appropriate Kubernetes Service.
What are the pros/cons of each method and how should I determine which one is most appropriate for our use case?
Following on from Vincent H, I'd suggest pipelining the Google HTTPS Load Balancer to an nginx ingress controller.
As you've mentioned, this can scale independently; have it's own health checks; and you can standardise your error pages.
We've achieved this by having a single kubernetes.io/ingress.class: "gce" ingress object, which has a default backend of our nginx ingress controller. All our other ingress objects are annotated with kubernetes.io/ingress.class: "nginx".
We're using the controller documented here: https://github.com/kubernetes/ingress/tree/master/controllers/nginx. With a custom /etc/nginx/template/nginx.tmpl allowing complete control over the ingress.
For complete transparency, we haven't (yet) setup custom error pages in the nginx controller, however the documentation appears straight forward.
One of the requirements listed is that you want to decouple the pod readinessProbes so you can serve custom error pages. If you are adding a nginx container to every pod this isn't possible. Then the pod will be restarted when one of the containers in the pod fails to adhere to the liveness/readiness probes. Personally I also prefer to decouple as much as you can so you can scale independent pods, assign custom machine types if needed, and it saves you some resources by starting only the amount of nginx instances you really need (mostly memory).
Related
So if I have 10 services that I need to expose to the outside world and use path-based routing to connect to different services, I can create an Nginx pod and service type LoadBalancer
I can then create Nginx configurations and can redirect to different services depending upon the URL path. After exploring more, I came to know about Nginx ingress which can also do the same. What is the difference between the two approaches and which approach is better?
In both cases, you are running an Nginx reverse proxy in a Kubernetes pod inside the cluster. There is not much technical difference between them.
If you run the proxy yourself, you have complete control over the Nginx version and the configuration, which could be desirable if you have very specific needs and you are already an Nginx expert. The most significant downside is that you need to manually reconfigure and redeploy the central proxy if you add or change a service.
If you use Kubernetes ingress, the cluster maintains the Nginx configuration for you, and it regenerates it based on Ingress objects that can be deployed per-service. This is easier to maintain if you are not an Nginx expert, and you can add and remove services without touching the centralized configuration.
The Kubernetes ingress system in principle can also plug in alternate proxies; it is not limited to Nginx. However, its available configuration is somewhat limited, and some of the other proxies I've looked at recommend using their own custom Kubernetes resources instead of the standard Ingress objects.
I followed this guide: https://www.digitalocean.com/community/tutorials/how-to-set-up-an-nginx-ingress-with-cert-manager-on-digitalocean-kubernetes on how to setup an Nginx Ingress with Cert Manager with Kubernetes having DigitalOcean as a cloud provider.
The tutorial worked fine, I was able to setup everything according to what it was written. Though, (as it is stated) following the tutorial one ends up with three pods of which only one is in "Running 1/1", while the other two are "Down". Also when checking the comments section, it seems that it is quite a problem. Since if all the traffic gets routed to only 1 pods, it is not really scalable. Or am I missing something? Quoting from their tutorial:
Note: By default the Nginx Ingress LoadBalancer Service has
service.spec.externalTrafficPolicy set to the value Local, which
routes all load balancer traffic to nodes running Nginx Ingress Pods.
The other nodes will deliberately fail load balancer health checks so
that Ingress traffic does not get routed to them.
Mainly my question is: Is there a best practice that I am missing in order to have Kubernetes hosting my website? It seems I have to choose either scalability (having all the pods healthy and running) or getting IP of the client visitor.
And for whoever will ever find himself/herself in my situation, this is the reply I got from the DigitaOcean Support:
Unfortunately with that Kubernetes setup it would show those other
nodes as down without additional traffic configuration. It is possible
to skip the nginx ingress part and just use a DigitalOcean load
balancer but this again does require a good deal of setup and can be
more difficult then easy.
The suggestion to have a website with analytics (IP) and scalable was to setup a droplet with Nginx and setup a LoadBalancer to it. More specifically:
As for using a droplet this would be a normal website configuration
with Nginx as your webserver configured to serve content to your app.
You would have full access to your application and the Nginx logs on
the droplet itself. Putting a load balancer in front of this would
require additional configuration as load balancers do not pass the
x-forward header so the IP addresses of clients would not show up in
the logs by default. You would need to configured proxy protocol on
the load balancer and in your nginx configuration to be able to obtain
those IPs.
https://www.digitalocean.com/blog/load-balancers-now-support-proxy-protocol/
This is also a bit more complex unfortunately.
Hope it might save some time to someone
All the tutorials I've found on the Internet demonstrate how to deploy a web site/app in an nginx container and set up an Ingress to proxy related requests to that container.
That seems kind of redundant to me. Why not to place the content on a volume, mount it to ingress-nginx-controller and serve static assets from there. It seems to be better from performance perspective: no additional proxy and no additional container (with nginx) that consumes compute resources.
An Ingress resource is an abstract concept that was designed to route HTTP/S requests to kubernetes services. It defines a limited subset of that functionality to remain generic and be implemented by many types of ingress controller.
ingress-nginx is only one implementation of an ingress controller, that happens to also be good at serving static content. HAProxy, Traefik or the AWS lb are at the other end of the scale.
An nginx based ingress controller could add custom annotations to support something custom like what you propose, but I imagine you would have a hard time arguing that into the project. Internally, they already add a second nginx instance for the default backend.
If one proxy hop is a critical differentiator in performance then you would probably get more benefit hosting the static content on an edge CDN/device. Another alternative is to not rely on an Ingress and publish the service directly to the outside world.
You need a container to run the application. Ingress service is just used to access that application outside the Kubernetes cluster. You cannot just keep the content in a volume and mount it to the ingress controller. One Ingress controller is used to route requests for multiple applications.
I am a kubernetes newbie, and no matter how much I read about it I cannot get my head around this issue.
I have a simple deployment , which is creating a pod with a not so complex app.
I know what an ingress and an ingress controller is doing ,but as far as I understand it is not required for me to expose my pod-app externally. Only a LoadBalancer service should be enough.
I do not have need of more than one rule for traffic routing.
Am I wrong about that?
Traditionally, you would create a LoadBalancer service for each service you want to expose externally. This can get rather expensive. Ingress gives you a way to route requests to services based on the request host or path, centralizing a number of services into a single entrypoint.
Also load balancer provisioning takes time and works only in supported cloud providers such as AWS, GCP etc.
One more thing to consider is the need of L4(TCP/UDP) layer routing because kubernetes Ingress API is primarily L7 layer but some of the ingress controllers such as traefik, nginx supports L4 layer(TCP/UDP) along with L7 layer(HTTP) routing.
So answer to your question is it depends based on your environment and use cases.
Ingress and IngressControllers are used for traffic routing at layer 7 i.e., if your backends talk L7 protocols like HTTP, GRPC etc. You can route requests based on request path to different backend services using Ingress.
If your app doesn't operate at Layer 7, you might not need an Ingress.
Another question you could ask yourself if your migrating your app from non-kubernetes environment to kuberneters is - are you using a reverse proxy like nginx already? If so, you might want to use Ingress. I say might because it is not necessary. You can achieve the same affect of using an Ingress by running the nginx container as a pod, writing the nginx.conf yourself and making it available externally(using a LoadBalancer service for example). Instead by using IngressController you need not maintain the nginx pod or write nginx.conf. You can instead express the same configuration as Ingress resource which is much simpler.
Ingress is needed if you want to expose your service to external. Especially for layer 7 in OSI model (HTTP transport). Ingress also provide the mechanism to enable TLS support in your load balancer. Traffic routing is controlled by rules defined on the Ingress resource. An Ingress can be configured to give Services externally-reachable URLs, load balance traffic, terminate SSL / TLS, and offer name based virtual hosting. An Ingress controller is responsible for fulfilling the Ingress, usually with a load balancer, though it may also configure your edge router or additional frontends to help handle the traffic.
By default Ingress controller depends on which kind of cloud provider you're using, or if you're using on premise you'll need to set it up based on what you need. In one cluster, you can create multiple Ingress controller as well. There's many kinds of Ingress controller, you can take a look on this article.
I think this article about Load Balancer and Ingress may also help.
I want to use Nginx to load balance a kubernetes deployment.
The deployment is part of a service. It contains pod which can be scaled. I want NGINX to be part of the service without being scaled.
I know that I can use NGINX as an external load balancer by configuring it with external dns resolver. With that it can get the IP of the pods scaled and apply its own load balanced rules.
Is it possible to make NGINX part of the service? Then how to do the DNS resolution to the pods? In that case, which pods the service name is refered to?
I would like to avoid the declaration of two services to keep a single definition of the setup which represent a microservice.
More generally, how can I declare in a same service:
a unit which is scaled
a backend, not scaled
a database, not scaled
Thanks all
You can't have NGINX as part of the service. Service doesn't contain any pods, deployment does. It sounds like you want an ingress service, that would be a load balancer any and all services on the cluster
EDIT:
An ingress controller in essence is a deployment of NGINX exposed publicly as a service acting as a load balancer/fan out. The deployment scans your cluster for ingress resources and reconfigures NGINX to forward requests to appropriate services.
Typically people deploy a single controller that acts as the load balancer for all of your microservices. You can fan out based on DNS, URI, other headers and so on. You can also do TLS termination, add basic auth to specific services, it's even possible to splice in NGINX config snippets directly into the ingress resources.