I have deployed a Linkerd Service mesh and my Kubernetes cluster is configured with the Nginx ingress controller as a DaemonSet and all the ingresses are working fine also the Linkerd. Recently, I have added a traffic split functionality to run my blue/green setup I can reach through to these services with separate ingress resources. I have created an apex-web service as described here. If I reached you this service internally it perfectly working. I have created another ingress resources and I'm not able to test the blue/green functionality outside of my cluster. I'd like to mention that I have meshed (injected the Linkerd proxy) to all my Nginx pods but it is returning "503 Service Temporarily Unavailable" message from the Nginx.
I went through the documentation and I have created ingress following this, I can confirm that below annotations were added to the ingress resources.
annotations:
kubernetes.io/ingress.class: "nginx"
nginx.ingress.kubernetes.io/configuration-snippet: |
proxy_set_header l5d-dst-override $service_name.$namespace.svc.cluster.local:$service_port;
grpc_set_header l5d-dst-override $service_name.$namespace.svc.cluster.local:$service_port;
but still no luck with the out side of the cluster.
I'm testing with the given emojivoto app and all the traffic split and the apex-web services are in this training repository.
I'm not quite sure what went wrong and how to fix this outside from the cluster. I'd really appreciate if anyone assist me to fix this Linkerd, Blue/Green issue.
I have raised this question in the Linkerd Slack channel and got this fixed with the wonderful support from the community. Seems Nginx doesn't like the service which doesn't have an endpoint. My configuration was correct and asked to change the service pointed in the traffic split to a service with an endpoint and it fixed the issue.
In a nutshell, my traffic split was configured with web-svc and web-svc-2 services. I have changed the traffic split spec.service to the same web-svc and it worked
Here is the traffic split configuration after the update.
apiVersion: split.smi-spec.io/v1alpha1
kind: TrafficSplit
metadata:
name: web-svc-ts
namespace: emojivoto
spec:
# The root service that clients use to connect to the destination application.
service: web-svc
# Services inside the namespace with their own selectors, endpoints and configuration.
backends:
- service: web-svc
# Identical to resources, 1 = 1000m
weight: 500m
- service: web-svc-2
weight: 500m
Kudos to the Linkerd team who supported me to fix this issue. It is working like a charm.
tl;dr: The nginx ingress requires a Service resource to have an Endpoint resource in order to be considered a valid destination for traffic. The architecture in the repo creates three Service resources, one of which acts as an apex and has no Endpoint resources because it has no selectors, so the nginx ingress won't send traffic to it, and the leaf services will not get traffic as a result.
The example in the repo follows the SMI Spec by defining a single apex service and two leaf services. The web-apex service does not have any endpoints, so nginx will not send traffic to it.
According to the SMI Spec services can be self-referential, which means that a service can be both an apex and a leaf service, so to use the nginx ingress with this example, you can modify the TrafficSplit definition to change the spec.service value from web-apex to web-svc:
apiVersion: split.smi-spec.io/v1alpha1
kind: TrafficSplit
metadata:
name: web-svc-ts
namespace: emojivoto
spec:
# The root service that clients use to connect to the destination application.
service: web-svc
# Services inside the namespace with their own selectors, endpoints and configuration.
backends:
- service: web-svc
# Identical to resources, 1 = 1000m
weight: 500m
- service: web-svc-2
weight: 500m
Related
I am trying to understand the difference between Nginx ingress controller
kind:service
vs
kind: Ingress
vs
kind: configMap
in Kubernetes but a little unclear.
Is kind: Service same as Kind: Ingress in Service and Ingress?
kind represents the type of Kubernetes objects to be created while using the yaml file.
Kubernetes objects are persistent entities in the Kubernetes
system. Kubernetes uses these entities to represent the state of your
cluster. Specifically, they can describe:
What containerized applications are running (and on which nodes)
The resources available to those applications
The policies around how those applications behave, such as restart policies, upgrades, and fault-tolerance
ConfigMap Object: A ConfigMap is an API object used to store
non-confidential data in key-value pairs. Pods can consume ConfigMaps
as environment variables, command-line arguments, or as configuration
files in a volume.
Ingress Object: An API object that manages external access to the
services in a cluster, typically HTTP. Ingress may provide load
balancing, SSL termination and name-based virtual hosting.
Service Object: In Kubernetes, a Service is an abstraction which
defines a logical set of Pods and a policy by which to access them
(sometimes this pattern is called a micro-service).
We have two AKS clusters for different environments. Both use a Nginx server as a custom ingress. By that I mean that it acts like an ingress, but it is just a normal Nginx deployment behind a service. There are several good reasons for that setup, the main one being that ingress did not exist in AKS when we started.
The services are defined like this:
apiVersion: v1
kind: Service
metadata:
name: <our name>
namespace: <our namespace>
spec:
ports:
- port: 443
targetPort: 443
selector:
app: <our app>
loadBalancerIP: <our ip>
type: LoadBalancer
externalTrafficPolicy: Local
We have configured Nginx with the real ip module like this:
real_ip_header X-Original-Forwarded-For;
set_real_ip_from 10.0.0.0/8; # or whatever ip is correct
One environment uses the old basic networking, networkPlugin=kubenet. There Nginx logs the real client IP addresses in the log and can use them for access controls. The other uses advanced networking, networkPlugin=azure. There Nginx logs the IP address of one of the nodes, which is useless. Both the X-Original-Forwarded-For and the standard X-Forwarded-For headers are empty and of course the source IP is from the node, not from the client.
Is there a way around this? If at all possible we would like to avoid defining a "real" ingress as our own Nginx server contains custom configuration that would be hard to duplicate in such a setup, plus it is not clear that a standard ingress would help either?
Microsoft should have fixed this by now for real ingresses. However, apparently the fix doesn't cover our case where Nginx runs as a pod behind a service with advanced networking. We were told to use the workaround posted by denniszielke in https://github.com/Azure/AKS/issues/607 where the iptables for all nodes are updated regularly. Quite dirty in my view, but it works.
We still have the service defined as above with "externalTrafficPolicy: Local" and we have installed the ConfigMap and DaemonSet from the link. I changed the script to reduce logging a bit and moved both to another namespace.
I'm running Kubernetes on Google Compute Engine (GCE). I have an Ingress set up. Everything works perfectly except when I upload large files, the L7 HTTPS Load Balancer terminates the connection after 30 seconds. I know that I can bump this up manually in the "Backend Service", but I'm wondering if there is a way to do this from the Ingress spec. I worry that my manual tweak will get changed back to 30s later on.
The nginx ingress controller has a number of annotations that can be used to configure nginx. Does the GCE L7 Load Balancer have something similar?
This can now be configured within GKE, by using a custom resource BackendConfig.
apiVersion: cloud.google.com/v1beta1
kind: BackendConfig
metadata:
name: my-bconfig
spec:
timeoutSec: 60
And then configuring your Service to use this configuration with an annotation:
apiVersion: v1
kind: Service
metadata:
name: my-service
annotations:
beta.cloud.google.com/backend-config: '{"ports": {"80":"my-bconfig"}}'
spec:
ports:
- port: 80
.... other fields
See Configuring a backend service through Ingress
For anyone else looking for the solution to this problem, timeout and other settings (e.g. enable CDN) can only be configured manually at the moment.
Follow this kubernetes/ingress-gce issue for the latest updates on a long-term solution.
I have a specific use case that I can not seem to solve.
A typical gcloud setup:
A K8S cluster
A gcloud storage bucket
A gcloud loadbalancer
I managed to get my domain https://cdn.foobar.com/uploads/ to points to a google storage backend without any issue: I can access files. Its the backend service one that fails.
I would like the CDN to act as a cache, when a HTTP request hits it such as https://cdn.foobar.com/assets/x.jpg, if it does not have a copy of the asset it should query an other domain https://foobar.com/assets/x.jpg.
I understood that this what was load balancers backend-service were for. (Right?)
The backend-service is pointing to the instance group of the k8s cluster and requires a port. I guessed that I needed to allow the firewall to expose the Nodeport of my web application service for the loadbalancer to be able to query it.
Cloud CDN
Load balancing
Failing health-checks.
The backend service is pointing to the instance group of the k8s cluster and requires some ports (default 80?) 80 failed. I guessed that I needed to allow the firewall to expose the 32231 Nodeport of my web application service for the loadbalancer to be able to query it. That still failed with a 502.
?> kubectl describe svc
Name: backoffice-service
Namespace: default
Labels: app=backoffice
Selector: app=backoffice
Type: NodePort
IP: 10.7.xxx.xxx
Port: http 80/TCP
NodePort: http 32231/TCP
Endpoints: 10.4.x.x:8500,10.4.x.x:8500
Session Affinity: None
No events.
I ran out of ideas at this point.
Any hints int the right direction would be much appreciated.
When deploying your service as type 'NodePort', you are exposing the service on each Node's IP, but the service is not reachable to the exterior, so you need to expose your service as 'LoadBalancer'
Since you're looking to use an HTTP(s) Load Balancer, I'll recommend using a Kubernetes Ingress resource. This resource will be in charge of configuring the HTTP(s) load balancer and the required ports that your service is using, as well as the health checks on the specified port.
Since you're securing your application, you will need to configure a secret object for securing the Ingress.
This example will help you getting started on an Ingress with TLS termination.
I exposed a service with a static IP and an Ingress through an nginx controller as one of the examples of the kubernetes/ingress repository. I have a second LoadBalancer service, that is not managed by any Ingress resource that is no longer properly exposed after the adding the new resources for the first service (I do not understand why this is the case).
I tried to add a second Ingress and LoadBalancer service to assign the second static IP, but I cant get it to work.
How would I go about exposing the second service, preferably with an Ingress? Do I need to add a second Ingress resource or do I have to reconfigure the one I already have?
Using a Service with type: LoadBalancer and using an Ingress are usually mutually exclusive ways to expose your application.
When you create a Service with type: LoadBalancer, Kubernetes creates a LoadBalancer in your cloud account that has an IP, opens the ports on that LoadBalancer that match your Service, and then directs all traffic to that IP to the 1 Service. So if you have 2 Service objects, each with 'type: LoadBalancer' for 2 different Deployments, then you have 2 IPs as well (one for each Service).
The Ingress model is based on directing traffic through a single Ingress Controller which is running something like nginx. As the Ingress resources are added, the Ingress Controller reconfigures nginx to include the new Ingress details. In this case, there will be a Service for the Ingress Controller (e.g. nginx) that is type: LoadBalancer, but all of the services that the Ingress resources point to should be type: ClusterIP. Traffic for all the Ingress objects will flow through the same public IP of the LoadBalancer for the Ingress Controller Service to the Ingress Controller (e.g. nginx) Pods. The configuration details from the Ingress object (e.g. virtual host or port or route) will then determine which Service will get the traffic.