I am fairly new to Kubernetes and I have recently exposed a service using miniKube using NodePort type. I want to test the running of my application but I dont see any external ip but the port only. Here is the output of my:
$kubectl get service
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes 10.0.0.1 <none> 443/TCP 1h
kubernetes-bootcamp 10.0.0.253 <nodes> 8080:31180/TCP 20m
$kubectl describe services/kubernetes-bootcamp
Name: kubernetes-bootcamp
Namespace: default
Labels: run=kubernetes-bootcamp
Annotations: <none>
Selector: run=kubernetes-bootcamp
Type: NodePort
IP: 10.0.0.253
Port: <unset> 8080/TCP
NodePort: <unset> 31180/TCP
Endpoints: 172.17.0.2:8080
Session Affinity: None
Events: <none>
What is the External IP in this case so that I can use curl to get the output of my app exposed, I followed the tutorial while working on my laptop : https://kubernetes.io/docs/tutorials/kubernetes-basics/expose-interactive/.
P.S : What does that <nodes> means in the output of the get service command under External-IP?
As you are using minikube, the command minikube ip will return the IP you are looking for.
In case you are not using minikube, kubectl get nodes -o yaml will show you, amongst other data, the IP address of the node.
Related
I'm following the quickstart guide https://kubernetes.github.io/ingress-nginx/deploy/#aws to install it on an aws eks cluster. The cluster runs in a private subnet and will receive traffic via a cloudflare argo tunnel.
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.0/deploy/static/provider/aws/deploy.yaml
When I then check the service I can see that it is pending:
kubectl get svc --namespace=ingress-nginx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx-controller LoadBalancer 10.100.64.86 <pending> 80:31323/TCP,443:31143/TCP 2d5h
The service generated seems ok, with valid annotations:
kubectl describe svc ingress-nginx-controller --namespace=ingress-nginx
Name: ingress-nginx-controller
Namespace: ingress-nginx
Labels: app.kubernetes.io/component=controller
app.kubernetes.io/instance=ingress-nginx
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=ingress-nginx
app.kubernetes.io/version=1.1.0
helm.sh/chart=ingress-nginx-4.0.10
Annotations: service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: true
service.beta.kubernetes.io/aws-load-balancer-type: nlb
Selector: app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
Type: LoadBalancer
IP Family Policy: SingleStack
IP Families: IPv4
IP: 10.100.64.86
IPs: 10.100.64.86
Port: http 80/TCP
TargetPort: http/TCP
NodePort: http 31323/TCP
Endpoints: 192.168.193.149:80
Port: https 443/TCP
TargetPort: https/TCP
NodePort: https 31143/TCP
Endpoints: 192.168.193.149:443
Session Affinity: None
External Traffic Policy: Local
HealthCheck NodePort: 30785
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal EnsuringLoadBalancer 2m23s (x646 over 2d5h) service-controller Ensuring load balancer
Not sure how to troubleshoot or fix
what worked for me was to download the installation file (https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.0/deploy/static/provider/aws/deploy.yaml) and add an annotation to the controller-service and then reapply the installation.
service.beta.kubernetes.io/aws-load-balancer-internal: "true"
not sure why it doesnt work as-is.
i would suggest try applying this changes,
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-0.32.0/deploy/static/provider/aws/deploy.yaml
The change that you have is for internal load balancer that wont give you a public exposed IP or Loabalancer.
service.beta.kubernetes.io/aws-load-balancer-internal: "true"
You can follow this Guide and this will create the NLB loadbalancer for you and the ingress tutorial also.
Read more at : https://aws.amazon.com/blogs/opensource/network-load-balancer-nginx-ingress-controller-eks/
EDIT:
I deleted minikube, enabled kubernetes in Docker desktop for Windows and installed ingress-nginx manually.
$helm upgrade --install ingress-nginx ingress-nginx --repo https://kubernetes.github.io/ingress-nginx --namespace ingress-nginx --create-namespace
Release "ingress-nginx" does not exist. Installing it now.
Error: rendered manifests contain a resource that already exists. Unable to continue with install: ServiceAccount "ingress-nginx" in namespace "ingress-nginx" exists and cannot be imported into the current release: invalid ownership metadata; annotation validation error: missing key "meta.helm.sh/release-name": must be set to "ingress-nginx"; annotation validation error: missing key "meta.helm.sh/release-namespace": must be set to "ingress-nginx"
It gave me an error but I think it's because I did it already before because:
$kubectl get svc -n ingress-nginx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx-controller LoadBalancer 10.106.222.233 localhost 80:30199/TCP,443:31093/TCP 11m
ingress-nginx-controller-admission ClusterIP 10.106.52.106 <none> 443/TCP 11m
Then applied all my yaml files again but this time ingress is not getting any address:
$kubectl get ing
NAME CLASS HOSTS ADDRESS PORTS AGE
myapp-ingress <none> myapp.com 80 10m
I am using docker desktop (windows) and installed nginx-ingress controller via minikube addons enable command:
$kubectl get pods -n ingress-nginx
NAME READY STATUS RESTARTS AGE
ingress-nginx-admission-create--1-lp4md 0/1 Completed 0 67m
ingress-nginx-admission-patch--1-jdkn7 0/1 Completed 1 67m
ingress-nginx-controller-5f66978484-6mpfh 1/1 Running 0 67m
And applied all my yaml files:
$kubectl get svc --all-namespaces -o wide
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
default event-service-svc ClusterIP 10.108.251.79 <none> 80/TCP 16m app=event-service-app
default kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 16m <none>
default mssql-clusterip-srv ClusterIP 10.98.10.22 <none> 1433/TCP 16m app=mssql
default mssql-loadbalancer LoadBalancer 10.109.106.174 <pending> 1433:31430/TCP 16m app=mssql
default user-service-svc ClusterIP 10.111.128.73 <none> 80/TCP 16m app=user-service-app
ingress-nginx ingress-nginx-controller NodePort 10.101.112.245 <none> 80:31583/TCP,443:30735/TCP 68m app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
ingress-nginx ingress-nginx-controller-admission ClusterIP 10.105.169.167 <none> 443/TCP 68m app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
kube-system kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 72m k8s-app=kube-dns
All pods and services seems to be running properly. Checked the pod logs, all migrations etc. has worked and app is up and running. But when I try to send an HTTP request, I get a socket hang up error. I've checked all the logs for all pods, couldn't find anything useful.
$kubectl get ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
myapp-ingress nginx myapp.com localhost 80 74s
This one is also a bit weird, I was expecting ADRESS to be set to an IP not to localhost. So adding 127.0.0.1 entry for myapp.com in /etc/hosts also didn't seem so right.
My question here is what I might be doing wrong? Or how can I even trace where are my requests are being forwarded to?
ingress-svc.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: myapp-ingress
spec:
rules:
- host: myapp.com
http:
paths:
- path: /api/Users
pathType: Prefix
backend:
service:
name: user-service-svc
port:
number: 80
- path: /api/Events
pathType: Prefix
backend:
service:
name: event-service-svc
port:
number: 80
events-depl.yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
name: event-service-app
labels:
app: event-service-app
spec:
replicas: 1
selector:
matchLabels:
app: event-service-app
template:
metadata:
labels:
app: event-service-app
spec:
containers:
- name: event-service-app
image: ghcr.io/myapp/event-service:master
imagePullPolicy: Always
ports:
- containerPort: 80
imagePullSecrets:
- name: myapp
---
apiVersion: v1
kind: Service
metadata:
name: event-service-svc
spec:
selector:
app: event-service-app
ports:
- protocol: TCP
port: 80
targetPort: 80
Reproduction
I reproduced the case using minikube v1.24.0, Docker desktop 4.2.0, engine 20.10.10
First, localhost in ingress appears due to logic, it doesn't really matter what IP address is behind the domain in /etc/hosts, I added a different one for testing and still it showed localhost. Only metallb will provide an IP address from set up network.
What happens
When minikube driver is docker, minikube creates a big container (VM) where kubernetes components are run. This can be checked by running docker ps command in host system:
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
f087dc669944 gcr.io/k8s-minikube/kicbase:v0.0.28 "/usr/local/bin/entr…" 16 minutes ago Up 16 minutes 127.0.0.1:59762->22/tcp, 127.0.0.1:59758->2376/tcp, 127.0.0.1:59760->5000/tcp, 127.0.0.1:59761->8443/tcp, 127.0.0.1:59759->32443/tcp minikube
And then minikube ssh to get inside this container and run docker ps to see all kubernetes containers.
Moving forward. Before introducing ingress, it's already clear that even NodePort doesn't work as intended. Let's check it.
There are two ways to get minikube VM IP:
run minikube IP
kubectl get nodes -o wide and find the node's IP
What should happen next with NodePort is requests should go to minikube_IP:Nodeport while it doesn't work. It happens because docker containers inside the minikube VM are not exposed outside of the cluster which is another docker container.
On minikube to access services within cluster there is a special command - minikube service %service_name% which will create a direct tunnel to the service inside the minikube VM (you can see that it contains service URL with NodePort which is supposed to be working):
$ minikube service echo
|-----------|------|-------------|---------------------------|
| NAMESPACE | NAME | TARGET PORT | URL |
|-----------|------|-------------|---------------------------|
| default | echo | 8080 | http://192.168.49.2:32034 |
|-----------|------|-------------|---------------------------|
* Starting tunnel for service echo.
|-----------|------|-------------|------------------------|
| NAMESPACE | NAME | TARGET PORT | URL |
|-----------|------|-------------|------------------------|
| default | echo | | http://127.0.0.1:61991 |
|-----------|------|-------------|------------------------|
* Opening service default/echo in default browser...
! Because you are using a Docker driver on windows, the terminal needs to be open to run it
And now it's available on host machine:
$ curl http://127.0.0.1:61991/
StatusCode : 200
StatusDescription : OK
Adding ingress
Moving forward and adding ingress.
$ minikube addons enable ingress
$ kubectl get svc -A
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default echo NodePort 10.111.57.237 <none> 8080:32034/TCP 25m
ingress-nginx ingress-nginx-controller NodePort 10.104.52.175 <none> 80:31041/TCP,443:31275/TCP 2m12s
Trying to get any response from ingress by hitting minikube_IP:NodePort with no luck:
$ curl 192.168.49.2:31041
curl : Unable to connect to the remote server
At line:1 char:1
+ curl 192.168.49.2:31041
Trying to create a tunnel with minikube service command:
$ minikube service ingress-nginx-controller -n ingress-nginx
|---------------|--------------------------|-------------|---------------------------|
| NAMESPACE | NAME | TARGET PORT | URL |
|---------------|--------------------------|-------------|---------------------------|
| ingress-nginx | ingress-nginx-controller | http/80 | http://192.168.49.2:31041 |
| | | https/443 | http://192.168.49.2:31275 |
|---------------|--------------------------|-------------|---------------------------|
* Starting tunnel for service ingress-nginx-controller.
|---------------|--------------------------|-------------|------------------------|
| NAMESPACE | NAME | TARGET PORT | URL |
|---------------|--------------------------|-------------|------------------------|
| ingress-nginx | ingress-nginx-controller | | http://127.0.0.1:62234 |
| | | | http://127.0.0.1:62235 |
|---------------|--------------------------|-------------|------------------------|
* Opening service ingress-nginx/ingress-nginx-controller in default browser...
* Opening service ingress-nginx/ingress-nginx-controller in default browser...
! Because you are using a Docker driver on windows, the terminal needs to be open to run it.
And getting 404 from ingress-nginx which means we can send requests to ingress:
$ curl http://127.0.0.1:62234
curl : 404 Not Found
nginx
At line:1 char:1
+ curl http://127.0.0.1:62234
Solutions
Above I explained what happens. Here are three solutions how to get it work:
Use another minikube driver (e.g. virtualbox. I used hyperv since my laptop has windows 10 pro)
minikube ip will return "normal" IP address of virtual machine and all network functionality will work just fine. You will need to add this IP address into /etc/hosts for domain used in ingress rule
Note! Even though localhost was shown in kubectl get ing ingress output in ADDRESS.
Use built-in kubernetes feature in Docker desktop for Windows.
You will need to manually install ingress-nginx and change ingress-nginx-controller service type from NodePort to LoadBalancer so it will be available on localhost and will be working. Please find my another answer about Docker desktop for Windows
(testing only) - use port-forward
It's almost exactly the same idea as minikube service command. But with more control. You will open a tunnel from host VM port 80 to ingress-nginx-controller service (eventually pod) on port 80 as well. /etc/hosts should contain 127.0.0.1 test.domain entity.
$ kubectl port-forward service/ingress-nginx-controller -n ingress-nginx 80:80
Forwarding from 127.0.0.1:80 -> 80
Forwarding from [::1]:80 -> 80
And testing it works:
$ curl test.domain
StatusCode : 200
StatusDescription : OK
Update for kubernetes in docker desktop on windows and ingress:
On modern ingress-nginx versions .spec.ingressClassName should be added to ingress rules. See last updates, so ingress rule should look like:
apiVersion: networking.k8s.io/v1
kind: Ingress
...
spec:
ingressClassName: nginx # can be checked by kubectl get ingressclass
rules:
- host: myapp.com
http:
...
My my-app service exposes multiple ports:
/Mugen$ kubectl get endpoints
NAME ENDPOINTS AGE
my-app 172.17.0.7:80,172.17.0.7:8003,172.17.0.7:8001 + 3 more... 7m
kubernetes 192.168.99.100:8443 10h
mysql-server 172.17.0.5:3306 10h
When executing minikube service my-app -n default --url, I'm getting each port forwarded by minikube, however I can't tell which is which without querying them. Is there a simple way to print the mapping or to set the port forwarding myself?
/Mugen$ minikube service my-app -n default --url
http://192.168.99.100:30426
http://192.168.99.100:30467
http://192.168.99.100:31922
http://192.168.99.100:32008
http://192.168.99.100:30895
http://192.168.99.100:31602
You can easily check the port and TargetPort mapping in kubernetes service using:
kubectl descrive svc my-app
Name: my-app
Namespace: default
Labels: <none>
Annotations: <none>
Selector: app=MyApp
Type: NodePort
IP: 10.152.183.56
Port: http 80/TCP
TargetPort: 9376/TCP
NodePort: http 30696/TCP
Endpoints: <none>
Port: https 443/TCP
TargetPort: 9377/TCP
NodePort: https 32715/TCP
Endpoints: <none>
Session Affinity: None
External Traffic Policy: Cluster
Events: <none>
This way you can find port, targetport and endpoints mapping.
EDIT: The whole point of my setup is to achieve (if possible) the following :
I have multiple k8s nodes
When I contact an IP address (from my company's network), it should be routed to one of my container/pod/service/whatever.
I should be able to easily setup that IP (like in my service .yml definition)
I'm running a small Kubernetes cluster (built with kubeadm) in order to evaluate if I can move my Docker (old)Swarm setup to k8s. The feature I absolutely need is the ability to assign IP to containers, like I do with MacVlan.
In my current docker setup, I'm using MacVlan to assign IP addresses from my company's network to some containers so I can reach directly (without reverse-proxy) like if it's any physical server. I'm trying to achieve something similar with k8s.
I found out that:
I have to use Service
I can't use the LoadBalancer type, as it's only for compatible cloud providers (like GCE or AWS).
I should use ExternalIPs
Ingress Resources are some kind of reverse proxy ?
My yaml file is :
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: nginx-deployment
spec:
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.7.9
ports:
- containerPort: 80
nodeSelector:
kubernetes.io/hostname: k8s-slave-3
---
kind: Service
apiVersion: v1
metadata:
name: nginx-service
spec:
type: ClusterIP
selector:
app: nginx
ports:
- name: http
protocol: TCP
port: 80
targetPort: 80
externalIPs:
- A.B.C.D
I was hopping that my service would get the IP A.B.C.D (which is one of my company's network). My deployment is working as I can reach my nginx container from inside the k8s cluster using it's ClusterIP.
What am I missing ? Or at least, where can I find informations on my network traffic in order to see if packets are coming ?
EDIT :
$ kubectl get svc
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes 10.96.0.1 <none> 443/TCP 6d
nginx-service 10.102.64.83 A.B.C.D 80/TCP 23h
Thanks.
First of all run this command:
kubectl get -n namespace services
Above command will return output like this:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
backend NodePort 10.100.44.154 <none> 9400:3003/TCP 13h
frontend NodePort 10.107.53.39 <none> 3000:30017/TCP 13h
It is clear from the above output that External IPs are not assigned to the services yet. To assign External IPs to backend service run the following command.
kubectl patch svc backend -p '{"spec":{"externalIPs":["192.168.0.194"]}}'
and to assign external IP to frontend service run this command.
kubectl patch svc frontend -p '{"spec":{"externalIPs":["192.168.0.194"]}}'
Now get namespace service to check either external IPs assignment:
kubectl get -n namespace services
We get an output like this:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
backend NodePort 10.100.44.154 192.168.0.194 9400:3003/TCP 13h
frontend NodePort 10.107.53.39 192.168.0.194 3000:30017/TCP 13h
Cheers!!! Kubernetes External IPs are now assigned .
If this is just for testing, then try
kubectl port-forward service/nginx-service 80:80
Then you can
curl http://localhost:80
A solution that could work (and not only for testing, though it has its shortcomings) is to set your Pod to map the host network with the hostNetwork spec field set to true.
It means that you won't need a service to expose your Pod, as it will always be accessible on your host via a single port (the containerPort you specified in the manifest). No need to keep a DNS mapping record in that case.
This also means that you can only run a single instance of this Pod on a given node (talking about shortcomings...). As such, it makes it a good candidate for a DaemonSet object.
If your Pod still needs to access/resolve internal Kubernetes hostnames, you need to set the dnsPolicy spec field set to ClusterFirstWithNoHostNet. This setting will enable your pod to access the K8S DNS service.
Example:
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: nginx
spec:
template:
metadata:
labels:
app: nginx-reverse-proxy
spec:
hostNetwork: true
dnsPolicy: ClusterFirstWithHostNet
tolerations: # allow a Pod instance to run on Master - optional
- key: node-role.kubernetes.io/master
effect: NoSchedule
containers:
- image: nginx
name: nginx
ports:
- name: http
containerPort: 80
- name: https
containerPort: 443
EDIT: I was put on this track thanks to the the ingress-nginx documentation
You can just Patch an External IP
CMD: $ kubectl patch svc svc_name -p '{"spec":{"externalIPs":["your_external_ip"]}}'
Eg:- $ kubectl patch svc kubernetes -p '{"spec":{"externalIPs":["10.2.8.19"]}}'
you can try kube-keepalived-vip configurtion to route the traffic. https://github.com/kubernetes/contrib/tree/master/keepalived-vip
You can try to add "type: NodePort" in your yaml file for the service and then you'll have a port to access it via the web browser or from the outside. For my case, it helped.
I don't know if that helps in your particular case but what I did (and I'm on a Bare Metal cluster) was to use the LoadBalancer and set the loadBalancerIP as well as the externalIPs to my server IP as you did it.
After that the correct external IP showed up for the load balancer.
Always use the namespace flag either before or after the service name, because Namespace-based scoping is applicable for deployments and services and this points out to the service that is tagged to a specific namespace. kubectl patch svc service-name -n namespace -p '{"spec":{"externalIPs":["IP"]}}'
Just include additional option.
kubectl expose deployment hello-world --type=LoadBalancer --name=my-service --external-ip=1.1.1.1
I have tried some example to run a specific image on my kubernetes cluster, I did this command:
kubectl run my-nginx --image=nginx --replicas=2 --port=80
then:
kubectl expose rc my-nginx --port=80 --type=LoadBalancer
and when I go:
kubectl get service
I get:
NAME LABELS SELECTOR IP(S) PORT(S)
kubernetes component=apiserver,provider=kubernetes <none> 10.0.0.1 443/TCP
my-nginx run=my-nginx run=my-nginx 10.0.100.19 80/TCP
now, I have a cluster that I created with my kubernetes, and onw I want to put something in my browser and see the landing page of nginx...
I tried to put my master machine ip with port 80 in the end and it didnt work, what should I do?
thanks!!
describe svc:
Name: my-nginx
Namespace: default
Labels: run=my-nginx
Selector: run=my-nginx
Type: LoadBalancer
IP: x.x.xxx.xx
LoadBalancer Ingress: dasfasdgfgaasok23o4j34ij4ofa69da-1772099277.us-west-2.elb.amazonaws.com
Port: <unnamed> 80/TCP
NodePort: <unnamed> 31331/TCP
Endpoints: x.x.xxx.x:80,xx.xxx.x.x:80
Session Affinity: None
No events.