We have two AKS clusters for different environments. Both use a Nginx server as a custom ingress. By that I mean that it acts like an ingress, but it is just a normal Nginx deployment behind a service. There are several good reasons for that setup, the main one being that ingress did not exist in AKS when we started.
The services are defined like this:
apiVersion: v1
kind: Service
metadata:
name: <our name>
namespace: <our namespace>
spec:
ports:
- port: 443
targetPort: 443
selector:
app: <our app>
loadBalancerIP: <our ip>
type: LoadBalancer
externalTrafficPolicy: Local
We have configured Nginx with the real ip module like this:
real_ip_header X-Original-Forwarded-For;
set_real_ip_from 10.0.0.0/8; # or whatever ip is correct
One environment uses the old basic networking, networkPlugin=kubenet. There Nginx logs the real client IP addresses in the log and can use them for access controls. The other uses advanced networking, networkPlugin=azure. There Nginx logs the IP address of one of the nodes, which is useless. Both the X-Original-Forwarded-For and the standard X-Forwarded-For headers are empty and of course the source IP is from the node, not from the client.
Is there a way around this? If at all possible we would like to avoid defining a "real" ingress as our own Nginx server contains custom configuration that would be hard to duplicate in such a setup, plus it is not clear that a standard ingress would help either?
Microsoft should have fixed this by now for real ingresses. However, apparently the fix doesn't cover our case where Nginx runs as a pod behind a service with advanced networking. We were told to use the workaround posted by denniszielke in https://github.com/Azure/AKS/issues/607 where the iptables for all nodes are updated regularly. Quite dirty in my view, but it works.
We still have the service defined as above with "externalTrafficPolicy: Local" and we have installed the ConfigMap and DaemonSet from the link. I changed the script to reduce logging a bit and moved both to another namespace.
Related
I have a single-node k8s cluster on DigitalOcean. I used to use NGINX Ingress Controller with ClusterIP service, but then it stopped working for some reason, so I updated it to use LoadBalancer service instead, but it costs me $12, which is too much for my pet projects.
So as a temporary measure I am using NodePort service with a hardcoded external IP of the machine. Here's how the helm release configuration looks now:
controller:
enableExternalDNS: true
service:
externalIPs:
- 46.XXX.XXX.XXX
type: NodePort
rbac:
create: true
But I don't like it being hardcoded like that. Is there a way to tell NGINX Ingress Controller to use node external IP without hardcoding it?
Or maybe there's some other way to expose the service without using LoadBalancer?
I have deployed a Linkerd Service mesh and my Kubernetes cluster is configured with the Nginx ingress controller as a DaemonSet and all the ingresses are working fine also the Linkerd. Recently, I have added a traffic split functionality to run my blue/green setup I can reach through to these services with separate ingress resources. I have created an apex-web service as described here. If I reached you this service internally it perfectly working. I have created another ingress resources and I'm not able to test the blue/green functionality outside of my cluster. I'd like to mention that I have meshed (injected the Linkerd proxy) to all my Nginx pods but it is returning "503 Service Temporarily Unavailable" message from the Nginx.
I went through the documentation and I have created ingress following this, I can confirm that below annotations were added to the ingress resources.
annotations:
kubernetes.io/ingress.class: "nginx"
nginx.ingress.kubernetes.io/configuration-snippet: |
proxy_set_header l5d-dst-override $service_name.$namespace.svc.cluster.local:$service_port;
grpc_set_header l5d-dst-override $service_name.$namespace.svc.cluster.local:$service_port;
but still no luck with the out side of the cluster.
I'm testing with the given emojivoto app and all the traffic split and the apex-web services are in this training repository.
I'm not quite sure what went wrong and how to fix this outside from the cluster. I'd really appreciate if anyone assist me to fix this Linkerd, Blue/Green issue.
I have raised this question in the Linkerd Slack channel and got this fixed with the wonderful support from the community. Seems Nginx doesn't like the service which doesn't have an endpoint. My configuration was correct and asked to change the service pointed in the traffic split to a service with an endpoint and it fixed the issue.
In a nutshell, my traffic split was configured with web-svc and web-svc-2 services. I have changed the traffic split spec.service to the same web-svc and it worked
Here is the traffic split configuration after the update.
apiVersion: split.smi-spec.io/v1alpha1
kind: TrafficSplit
metadata:
name: web-svc-ts
namespace: emojivoto
spec:
# The root service that clients use to connect to the destination application.
service: web-svc
# Services inside the namespace with their own selectors, endpoints and configuration.
backends:
- service: web-svc
# Identical to resources, 1 = 1000m
weight: 500m
- service: web-svc-2
weight: 500m
Kudos to the Linkerd team who supported me to fix this issue. It is working like a charm.
tl;dr: The nginx ingress requires a Service resource to have an Endpoint resource in order to be considered a valid destination for traffic. The architecture in the repo creates three Service resources, one of which acts as an apex and has no Endpoint resources because it has no selectors, so the nginx ingress won't send traffic to it, and the leaf services will not get traffic as a result.
The example in the repo follows the SMI Spec by defining a single apex service and two leaf services. The web-apex service does not have any endpoints, so nginx will not send traffic to it.
According to the SMI Spec services can be self-referential, which means that a service can be both an apex and a leaf service, so to use the nginx ingress with this example, you can modify the TrafficSplit definition to change the spec.service value from web-apex to web-svc:
apiVersion: split.smi-spec.io/v1alpha1
kind: TrafficSplit
metadata:
name: web-svc-ts
namespace: emojivoto
spec:
# The root service that clients use to connect to the destination application.
service: web-svc
# Services inside the namespace with their own selectors, endpoints and configuration.
backends:
- service: web-svc
# Identical to resources, 1 = 1000m
weight: 500m
- service: web-svc-2
weight: 500m
I have an ASP.NET Core webapi running in an on-prem bare-metal Kubernetes cluster. There's no external load-balancer, and I'm using NGINX ingress.
I want to get the users' IP address, and am using HttpContext.Connection.RemoteIpAddress in the .NET code.
Unfortunately, this is picking up the IP address of the nginx ingress (possibly the controller given the namespace)...
::ffff:10.244.1.85
Doing a reverse DNS lookup resolves that to...
10-244-1-85.ingress-nginx.ingress-nginx.svc.cluster.local
After a little bit of Googling, I tried adding externalTrafficPolicy: "Local" to my service definition, but that didn't make a difference.
This seems like something that should be really trivial and quite a common requirement. Any ideas?
First try to get ip from header X-Forwarded-For as shown below, if it's null then you can use Connection.RemoteIpAddress. Also make sure your nginx configmap has proxy enabled as per screenshot:
var ip = IPAddress.Parse(_accessor.ActionContext.HttpContext.Request.Headers["X-Forwarded-For"].FirstOrDefault());
if (string.IsNullOrEmpty(ip.ToString()))
{
ip = _accessor.ActionContext.HttpContext.Connection.RemoteIpAddress.MapToIPv4();
}
My problem is simple. I have an AKS deployment with a LoadBalancer service that needs to use HTTPS with a certificate.
How do I do this?
Everything I'm seeing online involves Ingress and nginx-ingress in particular.
But my deployment is not a website, it's a Dropwizard service with a REST API on one port and an admin service on another port. I don't want to map the ports to a path on port 80, I want to keep the ports as is. Why is HTTPS tied to ingress?
I just want HTTPS with a certificate and nothing more changed, is there a simple solution to this?
A sidecar container with nginx with the correct certificates (possible loaded off a Secret or a ConfigMap) will do the job without ingress. This seems to be a good example, using nginx-ssl-proxy container.
Yes, that's right as of this writing an Ingress will currently work either on port 80 or port 443, potentially it can be extended to use any port because nginx, Traefik, haproxy, etc can all listen on different ports.
So you are down to either a LoadBalancer or a NodePort type of service. Type LoadBalancer will not work directly with TLS since the Azure load balancers are layer 4. So you will have to use Application Gateway and it's preferred to use an internal load balancer for security reasons.
Since you are using Azure you can run something like this (assuming that your K8s cluster is configured the right way to use the Azure cloud provider, either the --cloud-provider option or the cloud-controller-manager):
$ cat <<EOF
apiVersion: v1
kind: Service
metadata:
name: your-app
annotations:
service.beta.kubernetes.io/azure-load-balancer-internal: "true"
spec:
type: LoadBalancer
ports:
- port: <your-port>
selector:
app: your-app
EOF | kubectl apply -f -
and that will create an Azure load balancer on the port you like for your service. Behind the scenes, the load balancer will point to a port on the nodes and within the nodes, there will be firewall rules that will route to your container. Then you can configure Application Gateway. Here's a good article describing it but using port 80, you will have to change it use port 443 and configuring the TLS certs, and the Application Gateway also supports end to end TLS in case you want to terminate TLS directly on your app too.
The other option is NodePort, and you can run something like this:
$ kubectl expose deployment <deployment-name> --type=NodePort
Then Kubernetes will pick a random port on all your nodes where you can send traffic to your service listening on <your-port>. So, in this case, you will have to manually create a load balancer with TLS or a traffic source that listens on TLS <your-port> and forwards it to a NodePort on all your nodes, this load balancer can be anything like haproxy, nginx, Traefik or something else that supports terminating TLS. And you can also use the Application Gateway to forward directly to your node ports, in other words, define a listener that listens on the NodePort of your cluster.
I'm running Kubernetes on Google Compute Engine (GCE). I have an Ingress set up. Everything works perfectly except when I upload large files, the L7 HTTPS Load Balancer terminates the connection after 30 seconds. I know that I can bump this up manually in the "Backend Service", but I'm wondering if there is a way to do this from the Ingress spec. I worry that my manual tweak will get changed back to 30s later on.
The nginx ingress controller has a number of annotations that can be used to configure nginx. Does the GCE L7 Load Balancer have something similar?
This can now be configured within GKE, by using a custom resource BackendConfig.
apiVersion: cloud.google.com/v1beta1
kind: BackendConfig
metadata:
name: my-bconfig
spec:
timeoutSec: 60
And then configuring your Service to use this configuration with an annotation:
apiVersion: v1
kind: Service
metadata:
name: my-service
annotations:
beta.cloud.google.com/backend-config: '{"ports": {"80":"my-bconfig"}}'
spec:
ports:
- port: 80
.... other fields
See Configuring a backend service through Ingress
For anyone else looking for the solution to this problem, timeout and other settings (e.g. enable CDN) can only be configured manually at the moment.
Follow this kubernetes/ingress-gce issue for the latest updates on a long-term solution.