I have a Kubernetes cluster i.e "Cluster1" with auto-scaling to a maximum of 2 nodes & a minimum of 1 node. I am trying to understand the digital ocean behavior for auto-downgrading the nodes in the following scenario.
The "Cluster1" cluster has an Nginx as the ingress-controller, which was added as part of the "1-click setup" during cluster provisioning.
This cluster has auto-scaling configured as 1 min node & 2 max nodes. Let's call them Node1 & Node2.
This cluster is behind a digital ocean load balancer i.e LB1 which talks to ingress controller i.e pod running Nginx.
Let's say there is a single replica (replica:1) deployment controller of "image1" which requires 80% of the CPU.
Initially, the image1 is deployed & since there is resource availability, the image1 starts running on Node1.
Consider the image1 is updated to image2, upstream. The deployment controller will see there's node unavailability & will provision Node2, & will create another pod running image2 on Node2, the pod running image1 will start to terminate once the image2 is up and running.
LB1 updates the routing to Node1, Node2.
Now after the pod (on Node1) for image1 is terminated because the replica:1 is set in the deployment controller, the Node1 is not running anything from the user perspective.
Ideally, there should be an automatic de-provisioning of the node i.e Node1.
I tried to manually remove the Node1 from the cluster using the DO dashboard.
LB1 updates & shows single node availability, but shows the status as down.
Upon investigating I found that "nginx-controller" was running only on Node1. When the Node1 is terminated, the "nginx-controller" takes a while to provision a new pod on then available Node2. There is however downtime all this while.
My question is how-to best use auto-scaling for downgrading. I have a few solutions I thought.
Is it possible to run "nginx-controller" on all nodes?
or
If I drain the node from the "kubectl", i.e kubectl drain , and then delete manually delete the node from the dashboard, there shouldn't any downtime?. or just doing kubectl drain, will make DigitalOcean auto downgrade.
if Nginx ingress controller is set as DaemonSet, it is possible
Please go through kubernetes official document about this topic.
https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/
Related
my natural thought is that if nginx is just a daemon process on the k8s node, but not a pod(container) in the k8s cluster, looks like it still can fullfill ingress controller jobs. because:
if it's a process, because it is on the k8s node, it still can talk to apiserver to fetch service backend pods information, like IP addresses, so it's still can be used as a http proxy server to direct traffic to different services.
so 2 questions,
why nginx ingress controller has to be a pod?
why nginx ingress controller only got 1 replica? and on which node? if nginx controller pod is dead, things will go unstable.
Thanks!
why Nginx ingress controller has to be a pod?
it is possible to run the Nginx controller as a daemon set in Kubernetes however I am not sure about the running on the node.
Manging the POD using daemon set and deployment of Kubernetes easy compare to process on Node.
By default Nginx daemon process is not part of any Kubernetes node, if you cluster autoscale will you install the Nginx process manually on Node?
If you thinking to create own AMI with Nginx process inside and use it inside the Node pool and scale that pool, it's possible but how about OS patching and maintenance ?
why nginx ingress controller only got 1 replica? and on which node? if
nginx controller pod is dead, things will go unstable.
Running replicas 1 is the default configuration but you can implement the HPA and increase the replicas as per need. Nginx is lightweight so handling a large volume of traffic not require more replicas.
Still, as per need, you can run multiple replicas with HPA or increase manually replicas to get high availability.
Because Pods are how you run daemon processes (or really, all processes) inside Kubernetes. That's just how you run stuff. I suppose there is nothing stopping you from running it outside the cluster, manually setting up API configuration and authentication, doing all the needed networking bits yourself. But ... why?
As for replicas, you should indeed generally have more than one across multiple physical nodes for redundancy. A lot of the tutorials show it with replicas: 1 because either it's for a single-node dev cluster like Minikube or it's only an example.
I have deployed the nginx pod and the service in the k-cluster via a yaml. All looks good (service, node, pods). This supposed to make nginx available from any node (http:/nodeA:port, http://nodeB:port, etc) but only one node replies (works).
All nodes have the firewall disabled.
All nodes identical OS.
Any ideas?
I am looking to map out various network connections between pods in a namespace to understand which pod is talking to which other pods. Is there a way to query the etcd to get this information?
There are many tools to visualize k8s topology.
In order of Github stars:
Cockpit:
Cockpit Project — Cockpit Project Cockpit makes GNU/Linux discoverable. See your server in a web browser and perform system tasks with a mouse. It’s easy to start containers, administer storage, configure networks, and inspect logs.
Weave Scope (Github: weaveworks/scope) is a troubleshooting and monitoring tool for Docker and Kubernetes clusters. It can automatically generate applications and infrastructure topologies which can help you to identify application performance bottlenecks easily. You can deploy Weave Scope as a standalone application on your local server/laptop, or you can choose the Weave Scope Software as a Service (SaaS) solution on Weave Cloud. With Weave Scope, you can easily group, filter or search containers using names, labels, and/or resource consumption. :
spekt8/spekt8: Visualize your Kubernetes cluster in real time :
SPEKT8 is a new visualization tool for your Kubernetes clusters. It automatically builds logical topologies of your application and infrastructure, which enable your SRE and Ops team to intuitively understand, monitor, and control your containerized, microservices based application. Simply deploy our containerized application directly into your Kubernetes cluster.
KubeView (Github: benc-uk/kubeview: Kubernetes cluster visualiser and graphical explorer )
KubeView displays what is happening inside a Kubernetes cluster, it maps out the API objects and how they are interconnected. Data is fetched real-time from the Kubernetes API. The status of some objects (Pods, ReplicaSets, Deployments) is colour coded red/green to represent their status and health.
Kubernetes Topology Graph:
Provides a simple force directed topology graph for kubernetes items.
You can try to use Weave Scope to make a graphical map of your Kubernetes cluster.
It will generates a map of your process, containers and hosts in real time. You can also get logs from containers and run some diagnostic commands via WEB-UI.
To install on Kubernetes you can run:
kubectl apply -f "https://cloud.weave.works/k8s/scope.yaml?k8s-version=$(kubectl version | base64 | tr -d '\n')"
After launch you don't need to configure anything, Scope will listen you pods and network and make a map of you network.
I'm trying to figure out a proper way to implement active/passive failover between replicas of service with Docker swarm mode.
The service will hold a valuable in-memory state that cannot be lost, that's why I need multiple replicas of it. The replicas will internally implement Raft so that only the replica which is active ("leader") at a given moment will accept requests from clients.
(If you're unfamiliar with Raft: simply put, it is a distributed consensus algorithm, which helps implement active/passive fault-tolerant cluster of replicas. According to Raft, the active replica - the leader - replicates changes in its data to passive replicas - the followers. The only leader accepts requests from clients. If the leader fails, a new leader is elected among the followers).
As far as I understand, Docker will guarantee that a specified number of replicas are up and running, but it will balance incoming requests among all of the replicas, in the active/active manner.
How can I tell Docker to route requests only to the active replica, but still guarantee that all replicas are up?
One option is routing all requests through an additional NGINX container, and updating its rules each time a new leader is elected. But that will be an additional hop, which I'd like to avoid.
I'm also trying to avoid external/overlapping tools such as consul or kubernetes, in order to keep the solution as simple as possible. (HAProxy is not an option because I need a Linux/Windows portable solution). So currently I'm trying to understand if this can be done with Docker swarm mode alone.
Another approach I came across is returning a failing health check from passive replicas. It does the trick with kubernetes according to this answer, but I'm not sure it will work with Docker. How does the swarm manager interpret failing health checks from task containers?
I'd appreciate any thoughts.
Active Passive replica can be achieved by having below deployment mode:
mode: global
With this port of the corresponding service is open, i.e., service is accessible via any the nodes in the swarm, but container will be running only on particular node.
Ref: https://docs.docker.com/compose/compose-file/#mode
Example:
VAULT-HA with Consul Backend docker stack file:
https://raw.githubusercontent.com/gtanand1994/VaultHA/master/docker-compose.yml
Here, Vault and Nginx containers will be seen only in one node in the swarm, but Consul containers (having mode: replicated) will be present on all the nodes of swarm.
But as I said before, VAULT, and NGINX services are available via 'any_node_ip:corresponding_port_number'
So Kubernetes has a pretty novel network model, that I believe is based on what it perceives to be a shortcoming with default Docker networking. While I'm still struggling to understand: (1) what it perceives the actual shortcoming(s) to be, and (2) what Kubernetes' general solution is, I'm now reaching a point where I'd like to just implement the solution and perhaps that will clue me in a little better.
Whereas the rest of the Kubernetes documentation is very mature and well-written, the instructions for configuring the network are sparse, largely incoherent, and span many disparate articles, instead of being located in one particular place.
I'm hoping someone who has set up a Kubernetes cluster before (from scratch) can help walk me through the basic procedures. I'm not interested in running on GCE or AWS, and for now I'm not interested in using any kind of overlay network like flannel.
My basic understanding is:
Carve out a /16 subnet for all your pods. This will limit you to some 65K pods, which should be sufficient for most normal applications. All IPs in this subnet must be "public" and not inside of some traditionally-private (classful) range.
Create a cbr0 bridge somewhere and make sure its persistent (but on what machine?)
Remove/disable the MASQUERADE rule installed by Docker.
Some how configure iptables routes (again, where?) so that each pod spun up by Kubernetes receives one of those public IPs.
Some other setup is required to make use of load balanced Services and dynamic DNS.
Provision 5 VMs: 1 master, 4 minions
Install/configure Docker on all 5 VMs
Install/configure kubectl, controller-manager, apiserver and etcd to the master, and run them as services/daemons
Install/configure kubelet and kube-proxy on each minion and run them as services/daemons
This is the best I can collect from 2 full days of research, and they are likely wrong (or misdirected), out of order, and utterly incomplete.
I have unbridled access to create VMs in an on-premise vCenter cluster. If changes need to be made to VLAN/Switches/etc. I can get infrastructure involved.
How many VMs should I set up for Kubernetes (for a small-to-medium sized cluster), and why? What exact corrections do I need to make to my vague instructions above, so as to get networking totally configured?
I'm good with installing/configuring all the binaries. Just totally choking on the network side of the setup.
For a general introduction into kubernetes networking, I found http://www.slideshare.net/enakai/architecture-overview-kubernetes-with-red-hat-enterprise-linux-71 pretty helpful.
On your items (1) and (2): IMHO they are nicely described in https://github.com/kubernetes/kubernetes/blob/master/docs/admin/networking.md#docker-model .
From my experience: What is the Problem with the Docker NAT type of approach? Sometimes you need to configure e.g. into the software all the endpoints of all nodes (172.168.10.1:8080, 172.168.10.2:8080, etc). in kubernetes you can simply configure the IP's of the pods into each others pod, Docker complicates it using NAT indirection.
See also Setting up the network for Kubernetes for a nice answer.
Comments on your other points:
1.
All IPs in this subnet must be "public" and not inside of some traditionally-private (classful) range.
The "internal network" of kubernetes normally uses private IP's, see also slides above, which uses 10.x.x.x as example. I guess confusion comes from some kubernetes texts that refer to "public" as "visible outside of the node", but they do not mean "Internet Public IP Address Range".
For anyone who is interested in doing the same, here is my current plan.
I found the kube-up.sh script which installs a production-ish quality Kubernetes cluster on your AWS account. Essentially it creates 1 Kubernetes master EC2 instance and 4 minion instances.
On the master it installs etcd, apiserver, controller manager, and the scheduler. On the minions it installs kubelet and kube-proxy. It also creates an auto-scaling group for the minions (nice), and creates a whole slew of security- and networking-centric things on AWS for you. If you run the script and it fails creating the AWS S3 bucket, create a bucket of the same exact name manually and then re-run the script.
When the script is finished you will have Kubernetes up and running and ready for near-production usage (I keep saying "near" and "production-ish" because I'm too new to Kubernetes to know what actually constitutes a real deal productionalized cluster). You will need the AWS CLI installed and configured with a user that has full admin access to your AWS account (it goes ahead and creates IAM roles, etc.).
My game plan will be to:
Get comfortable working with Kubernetes on AWS
Keep hounding the Kubernetes team on Slack to help me understand how Kubernetes works under the hood
Reverse engineer the kube-up.sh script so that I can get Kubernetes running on premise (vCenter)
Blog about this process
Update this answer with a link to said blog.
Give me some time and I'll follow through.