I am having an issue with Kubernetes on GKE. I am unable to resolve services by name. I got an drone-server service running which is connected to a single pod. The ingress connected to the service is successfully connecting but when trying to do for example a nslookup from a busybox pod is it unable to resolve the hostname.
Services:
$ k get services -n drone
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
drone-server ClusterIP 10.39.242.23 <none> 80/TCP 2d
drone-vault ClusterIP 10.39.248.166 <none> 80/TCP 40m
Busybox nslookup:
$ kubectl exec -ti busybox -- nslookup drone-server
Server: 10.39.240.10
Address 1: 10.39.240.10 kube-dns.kube-system.svc.cluster.local
nslookup: can't resolve 'drone-server'
When i try to lookup kubernetes.default am I getting a local address back:
$ kubectl exec -ti busybox -- nslookup kubernetes.default
Server: 10.39.240.10
Address 1: 10.39.240.10 kube-dns.kube-system.svc.cluster.local
Name: kubernetes.default
Address 1: 10.39.240.1 kubernetes.default.svc.cluster.local
Resolv config:
/etc/resolv.conf seems to be configured correctly (the nameserver is matching the kube-dns service cluster ip).
$ kubectl exec -ti busybox -- cat /etc/resolv.conf
nameserver 10.39.240.10
search default.svc.cluster.local svc.cluster.local cluster.local europe-west3-a.c.cluster-a8e6d9e252b63e03.internal c.cluster-a8e6d9e252b63e03.internal google.internal
options ndots:5
Your drone-server service is in the drone namespace and you're trying to nslookup from default namespace. You need to provide the namespace also in command as follows:
kubectl exec -ti busybox -- nslookup drone-server.drone
This is because your busybox in the default namespace and it tries to look drone-server in same namespace.
Related
EDIT:
I deleted minikube, enabled kubernetes in Docker desktop for Windows and installed ingress-nginx manually.
$helm upgrade --install ingress-nginx ingress-nginx --repo https://kubernetes.github.io/ingress-nginx --namespace ingress-nginx --create-namespace
Release "ingress-nginx" does not exist. Installing it now.
Error: rendered manifests contain a resource that already exists. Unable to continue with install: ServiceAccount "ingress-nginx" in namespace "ingress-nginx" exists and cannot be imported into the current release: invalid ownership metadata; annotation validation error: missing key "meta.helm.sh/release-name": must be set to "ingress-nginx"; annotation validation error: missing key "meta.helm.sh/release-namespace": must be set to "ingress-nginx"
It gave me an error but I think it's because I did it already before because:
$kubectl get svc -n ingress-nginx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx-controller LoadBalancer 10.106.222.233 localhost 80:30199/TCP,443:31093/TCP 11m
ingress-nginx-controller-admission ClusterIP 10.106.52.106 <none> 443/TCP 11m
Then applied all my yaml files again but this time ingress is not getting any address:
$kubectl get ing
NAME CLASS HOSTS ADDRESS PORTS AGE
myapp-ingress <none> myapp.com 80 10m
I am using docker desktop (windows) and installed nginx-ingress controller via minikube addons enable command:
$kubectl get pods -n ingress-nginx
NAME READY STATUS RESTARTS AGE
ingress-nginx-admission-create--1-lp4md 0/1 Completed 0 67m
ingress-nginx-admission-patch--1-jdkn7 0/1 Completed 1 67m
ingress-nginx-controller-5f66978484-6mpfh 1/1 Running 0 67m
And applied all my yaml files:
$kubectl get svc --all-namespaces -o wide
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
default event-service-svc ClusterIP 10.108.251.79 <none> 80/TCP 16m app=event-service-app
default kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 16m <none>
default mssql-clusterip-srv ClusterIP 10.98.10.22 <none> 1433/TCP 16m app=mssql
default mssql-loadbalancer LoadBalancer 10.109.106.174 <pending> 1433:31430/TCP 16m app=mssql
default user-service-svc ClusterIP 10.111.128.73 <none> 80/TCP 16m app=user-service-app
ingress-nginx ingress-nginx-controller NodePort 10.101.112.245 <none> 80:31583/TCP,443:30735/TCP 68m app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
ingress-nginx ingress-nginx-controller-admission ClusterIP 10.105.169.167 <none> 443/TCP 68m app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
kube-system kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 72m k8s-app=kube-dns
All pods and services seems to be running properly. Checked the pod logs, all migrations etc. has worked and app is up and running. But when I try to send an HTTP request, I get a socket hang up error. I've checked all the logs for all pods, couldn't find anything useful.
$kubectl get ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
myapp-ingress nginx myapp.com localhost 80 74s
This one is also a bit weird, I was expecting ADRESS to be set to an IP not to localhost. So adding 127.0.0.1 entry for myapp.com in /etc/hosts also didn't seem so right.
My question here is what I might be doing wrong? Or how can I even trace where are my requests are being forwarded to?
ingress-svc.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: myapp-ingress
spec:
rules:
- host: myapp.com
http:
paths:
- path: /api/Users
pathType: Prefix
backend:
service:
name: user-service-svc
port:
number: 80
- path: /api/Events
pathType: Prefix
backend:
service:
name: event-service-svc
port:
number: 80
events-depl.yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
name: event-service-app
labels:
app: event-service-app
spec:
replicas: 1
selector:
matchLabels:
app: event-service-app
template:
metadata:
labels:
app: event-service-app
spec:
containers:
- name: event-service-app
image: ghcr.io/myapp/event-service:master
imagePullPolicy: Always
ports:
- containerPort: 80
imagePullSecrets:
- name: myapp
---
apiVersion: v1
kind: Service
metadata:
name: event-service-svc
spec:
selector:
app: event-service-app
ports:
- protocol: TCP
port: 80
targetPort: 80
Reproduction
I reproduced the case using minikube v1.24.0, Docker desktop 4.2.0, engine 20.10.10
First, localhost in ingress appears due to logic, it doesn't really matter what IP address is behind the domain in /etc/hosts, I added a different one for testing and still it showed localhost. Only metallb will provide an IP address from set up network.
What happens
When minikube driver is docker, minikube creates a big container (VM) where kubernetes components are run. This can be checked by running docker ps command in host system:
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
f087dc669944 gcr.io/k8s-minikube/kicbase:v0.0.28 "/usr/local/bin/entr…" 16 minutes ago Up 16 minutes 127.0.0.1:59762->22/tcp, 127.0.0.1:59758->2376/tcp, 127.0.0.1:59760->5000/tcp, 127.0.0.1:59761->8443/tcp, 127.0.0.1:59759->32443/tcp minikube
And then minikube ssh to get inside this container and run docker ps to see all kubernetes containers.
Moving forward. Before introducing ingress, it's already clear that even NodePort doesn't work as intended. Let's check it.
There are two ways to get minikube VM IP:
run minikube IP
kubectl get nodes -o wide and find the node's IP
What should happen next with NodePort is requests should go to minikube_IP:Nodeport while it doesn't work. It happens because docker containers inside the minikube VM are not exposed outside of the cluster which is another docker container.
On minikube to access services within cluster there is a special command - minikube service %service_name% which will create a direct tunnel to the service inside the minikube VM (you can see that it contains service URL with NodePort which is supposed to be working):
$ minikube service echo
|-----------|------|-------------|---------------------------|
| NAMESPACE | NAME | TARGET PORT | URL |
|-----------|------|-------------|---------------------------|
| default | echo | 8080 | http://192.168.49.2:32034 |
|-----------|------|-------------|---------------------------|
* Starting tunnel for service echo.
|-----------|------|-------------|------------------------|
| NAMESPACE | NAME | TARGET PORT | URL |
|-----------|------|-------------|------------------------|
| default | echo | | http://127.0.0.1:61991 |
|-----------|------|-------------|------------------------|
* Opening service default/echo in default browser...
! Because you are using a Docker driver on windows, the terminal needs to be open to run it
And now it's available on host machine:
$ curl http://127.0.0.1:61991/
StatusCode : 200
StatusDescription : OK
Adding ingress
Moving forward and adding ingress.
$ minikube addons enable ingress
$ kubectl get svc -A
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default echo NodePort 10.111.57.237 <none> 8080:32034/TCP 25m
ingress-nginx ingress-nginx-controller NodePort 10.104.52.175 <none> 80:31041/TCP,443:31275/TCP 2m12s
Trying to get any response from ingress by hitting minikube_IP:NodePort with no luck:
$ curl 192.168.49.2:31041
curl : Unable to connect to the remote server
At line:1 char:1
+ curl 192.168.49.2:31041
Trying to create a tunnel with minikube service command:
$ minikube service ingress-nginx-controller -n ingress-nginx
|---------------|--------------------------|-------------|---------------------------|
| NAMESPACE | NAME | TARGET PORT | URL |
|---------------|--------------------------|-------------|---------------------------|
| ingress-nginx | ingress-nginx-controller | http/80 | http://192.168.49.2:31041 |
| | | https/443 | http://192.168.49.2:31275 |
|---------------|--------------------------|-------------|---------------------------|
* Starting tunnel for service ingress-nginx-controller.
|---------------|--------------------------|-------------|------------------------|
| NAMESPACE | NAME | TARGET PORT | URL |
|---------------|--------------------------|-------------|------------------------|
| ingress-nginx | ingress-nginx-controller | | http://127.0.0.1:62234 |
| | | | http://127.0.0.1:62235 |
|---------------|--------------------------|-------------|------------------------|
* Opening service ingress-nginx/ingress-nginx-controller in default browser...
* Opening service ingress-nginx/ingress-nginx-controller in default browser...
! Because you are using a Docker driver on windows, the terminal needs to be open to run it.
And getting 404 from ingress-nginx which means we can send requests to ingress:
$ curl http://127.0.0.1:62234
curl : 404 Not Found
nginx
At line:1 char:1
+ curl http://127.0.0.1:62234
Solutions
Above I explained what happens. Here are three solutions how to get it work:
Use another minikube driver (e.g. virtualbox. I used hyperv since my laptop has windows 10 pro)
minikube ip will return "normal" IP address of virtual machine and all network functionality will work just fine. You will need to add this IP address into /etc/hosts for domain used in ingress rule
Note! Even though localhost was shown in kubectl get ing ingress output in ADDRESS.
Use built-in kubernetes feature in Docker desktop for Windows.
You will need to manually install ingress-nginx and change ingress-nginx-controller service type from NodePort to LoadBalancer so it will be available on localhost and will be working. Please find my another answer about Docker desktop for Windows
(testing only) - use port-forward
It's almost exactly the same idea as minikube service command. But with more control. You will open a tunnel from host VM port 80 to ingress-nginx-controller service (eventually pod) on port 80 as well. /etc/hosts should contain 127.0.0.1 test.domain entity.
$ kubectl port-forward service/ingress-nginx-controller -n ingress-nginx 80:80
Forwarding from 127.0.0.1:80 -> 80
Forwarding from [::1]:80 -> 80
And testing it works:
$ curl test.domain
StatusCode : 200
StatusDescription : OK
Update for kubernetes in docker desktop on windows and ingress:
On modern ingress-nginx versions .spec.ingressClassName should be added to ingress rules. See last updates, so ingress rule should look like:
apiVersion: networking.k8s.io/v1
kind: Ingress
...
spec:
ingressClassName: nginx # can be checked by kubectl get ingressclass
rules:
- host: myapp.com
http:
...
I need to specify External IP that will be associated to Nginx service upon the Ingress Controller creation using the HELM install:
helm install nginx-ingress ingress-nginx/ingress-nginx -f internal-ingress.yaml
--set controller.nodeSelector."beta\.kubernetes\.io/os"=linux
--set defaultBackend.nodeSelector."beta\.kubernetes\.io/os"=linux
So I want to set statically EXTERNAL-IP address that can be seen upon this creation:
kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx-ingress-ingress-nginx-controller LoadBalancer 10.0.36.81 10.33.27.35 80:31312/TCP,443:30653/TCP 5d
I need this because DNS configuration has already been set up for this IP and I would like to avoid this kind of configuration again.
You can add flag to installation command:
--set controller.service.loadBalancerIP=XXXX
where XXXX is static EXTERNAL IP you want to use.
Remember that IP needs to be regional and in the same region as the cluster.
See: external-ip-ingress-controller.
I have an nginx pod in the default namespace and a ClusterIP service exposing the pod.
$ kubectl run nginx-pod --image=nginx
$ kubectl expose po nginx-pod --name=nginx-service --port=8080 --target-port=80 --type=ClusterIP
I can access the service via its internal IP from inside the cluster.
$ kubectl get svc nginx-service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx-service ClusterIP 10.100.5.218 <none> 8080/TCP 2m49s
$ wget -O- 10.100.5.218:8080
--> 200 OK
I can access the service by name from inside a pod.
$ kubectl run tmp -it --rm --image=busybox --command -- /bin/sh -c 'wget -O- nginx-service:8080'
--> 200 OK
However, why can't I access the service by name from outside a pod?
$ wget -O- nginx-service:8080
or
$ wget -O- nginx-service.default.svc.cluster.local:8080
--> wget: unable to resolve host address ‘nginx-service’
The magic service hostnames (and pod hostnames) are provided by the "cluster DNS" service, usually CoreDNS these days. A resolv.conf aimed at the internal CoreDNS is automatically injected into all pods. But I'm guessing by "outside of a pod" you mean on the underlying host which has no such entries in its resolv.conf.
Please see this.
https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services-service-types
You either need NodePort or LoadBalancer service type.
I can't access to Network IP assigned by MetalLB load Balancer
I created a Kubernetes cluster in k3s. Its 1 master and 1 workers. Each one has its own Private IP.
Master 192.168.0.13
Worker 192.168.0.13
I Installed k3s with INSTALL_K3S_EXEC=" --no-deploy servicelb --no-deploy traefik"
Now I am trying to deploy a app using MetalLB and nginx ingress
--set configInline.address-pools[0].name=default \
--set configInline.address-pools[0].protocol=layer2 \
--set configInline.address-pools[0].addresses[0]=192.168.0.21-192.168.0.30
helm install nginx-ingress stable/nginx-ingress --namespace kube-system \
--set controller.image.repository=quay.io/kubernetes-ingress-controller/nginx-ingress-controller\
--set controller.image.tag=0.30.0 \
--set controller.image.runAsUser=33 \
--set defaultBackend.enabled=false
I Can see every pod up and running
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
coredns-d798c9dd-lsdnp 1/1 Running 5 37h 10.42.0.25 c271-k3s-ocrh <none> <none>
local-path-provisioner-58fb86bdfd-bcpl7 1/1 Running 5 37h 10.42.0.22 c271-k3s-ocrh <none> <none>
metrics-server-6d684c7b5-v9tmh 1/1 Running 5 37h 10.42.0.24 c271-k3s-ocrh <none> <none>
metallb-speaker-4kbmw 1/1 Running 0 4m7s 192.168.0.14 c271-k3s-agent <none> <none>
metallb-controller-75bf779d4f-nb47l 1/1 Running 0 4m7s 10.42.1.45 c271-k3s-agent <none> <none>
metallb-speaker-776p9 1/1 Running 0 4m7s 192.168.0.13 c271-k3s-ocrh <none> <none>
nginx-ingress-default-backend-5b967cf596-554bq 1/1 Running 0 98s 10.42.1.46 c271-k3s-agent <none> <none>
nginx-ingress-controller-674675d5b6-blndp 1/1 Running 0 98s 10.42.1.47 c271-k3s-agent <none> <none>
App getting IP 192.168.0.21
❯ kubectl get services -n kube-system -l app=nginx-ingress -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
nginx-ingress-default-backend ClusterIP 10.43.170.195 <none> 80/TCP 112s app=nginx-ingress,component=default-backend,release=nginx-ingress
nginx-ingress-controller LoadBalancer 10.43.220.166 192.168.0.21 80:31735/TCP,443:31566/TCP 111s app=nginx-ingress,component=controller,release=nginx-ingress
I Can access the app from master and worker by curl to nginx controller pod
HTTP/1.1 200 OK
Server: nginx/1.17.8
Date: Sat, 21 Mar 2020 10:43:34 GMT
Content-Type: text/html
Content-Length: 153
Connection: keep-alive
But the IP is not accessible from local 192.168.0.21
Diagnosis : DHCP is on, and 192.168.0.21-192.168.0.30 is absolutely free., When i try to allocate 192.168.0.21 to master or agent by netplan config they get the IP.
Please Guide me, What i am missing.
You need to make sure that the source IP address (external-ip assigned by metallb) is preserved. To achieve this, set the value of the externalTrafficPolicy field of the ingress-controller Service spec to Local. For example
apiVersion: v1
kind: Service
metadata:
name: my-app
labels:
helm.sh/chart: webapp-0.1.0
app.kubernetes.io/name: webapp
app.kubernetes.io/instance: my-app
app.kubernetes.io/version: "1.16.0"
app.kubernetes.io/managed-by: Helm
spec:
type: ClusterIP
ports:
- port: 80
targetPort: http
protocol: TCP
name: http
selector:
app.kubernetes.io/name: webapp
app.kubernetes.io/instance: my-app
externalTrafficPolicy: Local
The default value for externalTrafficPolicy field is 'Cluster'. So change it to Local
In my setup with Cilium and HAProxy ingress controller I'd to change externalTrafficPolicy from Local to Cluster
kubectl --namespace ingress-controller patch svc haproxy-ingress \
-p '{"spec":{"externalTrafficPolicy":"Cluster"}}'
from two years I've been using metalLb in my home-lab, and I didn't get the error (although I got other errors for example ., MetalLB fails to assign an IP address from the pool)
I want to share my current setup with folks who are still struggling on the internet.
helm install --create-namespace metallb metallb/metallb -n metallb-system -f values.yaml
configInline:
address-pools:
- name: default
protocol: layer2
addresses:
- 192.168.0.21/30
# can use series like 192.168.0.21-24 too.
Debugging - try to get logs from all the pod in namespace metallb.
kail -n metallb
K8S installed with calico using https://github.com/geerlingguy/ansible-role-kubernetes
Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.1", GitCommit:"3ddd0f45aa91e2f30c70734b175631bec5b5825a", GitTreeState:"clean", BuildDate:"2022-05-24T12:17:11Z", GoVersion:"go1.18.2", Compiler:"gc", Platform:"darwin/amd64"}
Kustomize Version: v4.5.4
Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.3", GitCommit:"aef86a93758dc3cb2c658dd9657ab4ad4afc21cb", GitTreeState:"clean", BuildDate:"2022-07-13T14:23:26Z", GoVersion:"go1.18.3", Compiler:"gc", Platform:"linux/amd64"}
Maybe switching externalTrafficPolicy to local/cluster may help however, I didn't try. my setup works out of the box.
Good Luck.
I am fairly new to Kubernetes and I have recently exposed a service using miniKube using NodePort type. I want to test the running of my application but I dont see any external ip but the port only. Here is the output of my:
$kubectl get service
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes 10.0.0.1 <none> 443/TCP 1h
kubernetes-bootcamp 10.0.0.253 <nodes> 8080:31180/TCP 20m
$kubectl describe services/kubernetes-bootcamp
Name: kubernetes-bootcamp
Namespace: default
Labels: run=kubernetes-bootcamp
Annotations: <none>
Selector: run=kubernetes-bootcamp
Type: NodePort
IP: 10.0.0.253
Port: <unset> 8080/TCP
NodePort: <unset> 31180/TCP
Endpoints: 172.17.0.2:8080
Session Affinity: None
Events: <none>
What is the External IP in this case so that I can use curl to get the output of my app exposed, I followed the tutorial while working on my laptop : https://kubernetes.io/docs/tutorials/kubernetes-basics/expose-interactive/.
P.S : What does that <nodes> means in the output of the get service command under External-IP?
As you are using minikube, the command minikube ip will return the IP you are looking for.
In case you are not using minikube, kubectl get nodes -o yaml will show you, amongst other data, the IP address of the node.