Having a cluster running two nodes, getting the following for one of the two kube-proxy pods :
kube-system kube-proxy-gke-app-... 0/1 Init:0/1 0 4m
A describe command more or less tells the same.
Not knowing enough to understand fully what's happening (the whoel cluster works quite fine), I was wondering if such state for one of the pod could have impact, what was the reason and/or the workaround to solve it
This all runs on gke with nginx-ingress-controller through the load balancer
Thx a lot for any help
Related
We're running a platform made out of 5 clusters. One of the clusters died. We're using Kibana because its cheaper than Cloudwatch (log router with fluentbit). The 14 hour span that the cluster was dead shows 0 logs on Kibana, and we have no idea what happened to the cluster. A simple restart of the cluster fixed our issue. So, to make sure it doesn't die again while we're away, we need to set it up so it automatically restarts. Dev did not implement a cluster health check. We're using Kibana, so I can't use Cloudwatch to implement metrics, alarms and actions. What do I do here? How do I make the cluster restart itself when Kibana detects no incoming logs from it? Thank you.
I ran into this issue the other day and I'm not sure if this the correct cause. Essentially I am spinning up 10 KubernetesPodOperators in parallel in airflow. When I request the 10 pods, the nodes will autoscale to meet the resource requirements of those 10 pods. However, once let's say 8/10 pods have completed their task, the autoscaler will scale down the nodes, which seemed to crash my currently running remaining 2 pods (as I assume they are being placed onto a new node). When I set autoscale to "off" in kubernetes and predefined the correct number of nodes, my 10 pods run fine. Does this logic make sense? If so has anyone faced a similar issue and if so is there any way around this? We are running airflow in an Azure AKS instance.
Thanks,
I would like to upgrade my weave network from version 2.5.0 to 2.5.2. I understand that it's "as simple" as updating the weave daemonset.... however, i was wondering if there is a way that this can be done with minimal disruption to running pods on the system.
An simple example in my mind would be to:
cordon node1
drain node1 of all pods
update weave on node1
uncordon node1
... then rinse and repeat for each k8s node until all done.
Basing on the weave net documentation
Upgrading the Daemon Sets
The DaemonSet definition specifies Rolling Updates, so when you apply a new version Kubernetes will automatically restart the Weave Net pods one by one.
With RollingUpdate update strategy, after you update a DaemonSet template, old DaemonSet pods will be killed, and new DaemonSet pods will be created automatically, in a controlled fashion.
As i could read in another stackoverflow answer
It is possible to perform rolling updates with no downtime using a DeamonSet as of Today! What you need is to have at least 2 nodes running on your cluster and set maxUnavailable to 1 in your DaemonSet configuration.
Assuming the previous configuration, when an update is pushed, a first node will start updating. The second will waiting until the first completes. Upon success, the second does the same.
The major drawback is that you need to keep 2 nodes runnings continuously or to take actions to spawn/kill a node before/after an update.
So i think the best option for you to upgrade your CNI plugin is using DaemonSet with rolling update and set maxUnavailable to 1 in your DaemonSet configuration.
I have a requirement to deploy an HTTP application in K8s with zero downtime. I also have a restriction of using a single pod (replica=1). But the problem is when I did that some of the HTTP requests get 502 Bad gateway when I did some changes to the K8s pod.
I refer the following two issues [1] [2], but those issues work fine when I have more than a single replica. For a single replica, NGINX ingress still has a slight downtime which is less than 1 millisecond.
The lifecycle spec and rolling updates spec of my deployment set as below according to the answers given by the above issues [1] [2].
spec:
strategy:
rollingUpdate:
maxSurge: 2
maxUnavailable: 0
type: RollingUpdate
...
spec:
....
lifecycle:
preStop:
exec:
command:
- sleep
- "30"
Note that I have config maps that mount to this deployment. I'm not sure that would affect this downtime or not.
Also, I refer to these two blogs [3] [4], but they did not solve my problem too. But when I refer this blog [4] it shows that K8s can achieve zero downtime even with a single replica. Unfortunately, in [4] he did not use an ingress-nginx controller.
In brief, I wanted to know that, is it possible to achieve zero-downtime in ingress-nginx with a single replication of pod?
References
1 https://github.com/kubernetes/ingress-nginx/issues/489
2 https://github.com/kubernetes/ingress-nginx/issues/322
3 https://blog.sebastian-daschner.com/entries/zero-downtime-updates-kubernetes
4 http://rahmonov.me/posts/zero-downtime-deployment-with-kubernetes/
I suppose that your single-pod restriction is at runtime and not during the upgrade, otherwise, you can't achieve your goal.
My opinion is your rolling upgrade strategy is good, you can add a PodDistruptionBudget to manage disruptions to be sure that at least 1 pod is available.
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
name: sample-pdb
spec:
minAvailable: 1
selector:
matchLabels:
<your_app_label>
Another very important thing is the probes, according to documentation:
The kubelet uses liveness probes to know when to restart a Container. For example, liveness probes could catch a deadlock, where an application is running, but unable to make progress. Restarting a Container in such a state can help to make the application more available despite bugs.
The kubelet uses readiness probes to know when a Container is ready to start accepting traffic. A Pod is considered ready when all of its Containers are ready. One use of this signal is to control which Pods are used as backends for Services. When a Pod is not ready, it is removed from Service load balancers.
You should set the liveness probe, but most of all the readiness probe, to return a success response only when your new pod is really ready to accept a new connection, otherwise k8s think that the new pod is up and the old pod will be destroyed before the new one can accept connections.
Nginx as a revers proxy is able to handle a 0 down time if the IP address of the backend didn't change, but in your case I think that the requirements of only 1 replica and the volumes mounted always makes the down process a little bit more slow, is not possible to achieve the 0 down time because if you are mounting the same volume on the new pod this need to wait the old pod to be destroyed and release the volume to start the wake up process.
In your referenced blog post where it explains how to achieve that, the example didn't use volumes and uses a very small image that makes the pull process and wake up very fast.
I recommend you to study your volume needs and try to not have this as a blocking thing on the weak up process.
Using nginx nginx-ingress-controller:0.9.0, below is the permanent state of the google cloud load balancer :
Basically, the single healthy node is the one running the nginx-ingress-controller pods. Besides not looking good on this screen, everything works super fine. Thing is, Im' wondering why such bad notice appears on the lb
Here's the service/deployment used
Am just getting a little lost over how thing works; hope to get some experienced feedback on how to do thing right (I mean, getting green lights on all nodes), or to double check if that's a drawback of not using the 'official' gcloud l7 thing
Your Service is using the service.beta.kubernetes.io/external-traffic: OnlyLocal annotation. This configures it so that traffic arriving at the NodePort for that service will never go a Pod on another node. Since your Deployment only has 1 replica, the only node that will receive traffic is the one where the 1 Pod is running.
If you scale your Deployment to 2 replicas, 2 nodes will be healthy, etc.
Using that annotation is a recommend configuration so that you are not introducing additional network hops.