I used Confluent for Kubernetes solution to use a Kafka cluster on my bare metal server. Monitoring all pods I noticed that controlcenter-0 pod take more and more ram gradually. Why this behavior?
In a night it reached almost 6GB.
Related
A flask-api (using gunicorn) is used as an inference api of a deep learning model.
This specific inference process is very cpu intensive (not using gpu yet).
What is the best practice of deploying it to a kubernetes cluster, based on these aspects:
should I create multiple pods handling requests using single gunicorn worker or less pods enabling gunicorn multiple workers? (node memory footprint)
since google provides to expose your deployment as a service using an external loadbalancer,
do I need a nginx web server on my flask-gunicorn stack?
creating multiple identical pods on the same node, is it more memory intensive than handling all these requests using multithreading on a single pod?
More smaller pods is generally better, provided you're staying under "thousands". It is easier for the cluster to place a pod that requires 1 CPU and 1 GB of RAM 16 times than it is to place a single pod that requires 16 CPU and 16 GB RAM once. You usually want multiple replicas for redundancy, to tolerate node failure, and for zero-downtime upgrades in any case.
If the Istio Ingress system works for you, you may not need separate a URL-routing layer (Nginx) inside your cluster. If you're okay with having direct access to your Gunicorn servers with no routing or filtering in front of that, directly pointing a LoadBalancer Service at them is a valid choice.
Running 16 copies of 1 application will generally need more memory than 1 copy with 16 threads; how much more depends on the application.
In particular, if you load your model into memory and the model itself is large, but your multi-threaded setup can share a single copy of it, 1 large pod could use significantly less memory than 16 small pods. If the model is COPYed directly into the Docker image and the application code mmap()s it then you'd probably get to share memory at the kernel layer.
If the model itself is small and most of the memory is used in the processing, it will still use "more" memory to have multiple pods, but it would just be the cost of your runtime system and HTTP service; it shouldn't substantially change the memory required per thread/task/pod if that isn't otherwise shared.
I am looking for ball park numbers in production for following questions
1) How many flows(5 tuple: SRC-IP, DEST-IP, SRC-PORT, DEST-PORT, Protocol) does a pod open and how long these flow live ?.
2) When one moves from VM's to containers with K8, Is it typical to convert a VM in to a Pod ?
Are there any studies around this area ?. If so, can you provide pointers ?
1:
It depends on how many containers or connections your pod will make.
2:
It is definitely different thing:
Pod is a group of containers which is managed by K8s. VM is a Virtual Machine, you can't convert VM into a pod.
Usually, when you migrate from VMs to K8s, you change the architecture of your application.
What are the minimum hardware requirements for setting up an Apache Airflow cluster.
Eg. RAM, CPU, Disk etc for different types of nodes in the cluster.
I have had no issues using very small instances in pseudo-distributed mode (32 parallel workers; Postgres backend):
RAM 4096 MB
CPU 1000 MHz
VCPUs 2 VCPUs
Disk 40 GB
If you want distributed mode, you should be more than fine with that if you keep it homogenous. Airflow shouldn't really do heavy lifting anyways; push the workload out to other things (Spark, EMR, BigQuery, etc).
You will also have to run some kind of messaging queue, like RabbitMQ. I think they take Redis too. However, this doesn't really dramatically impact how you size.
We are running the airflow in AWS with below config
t2.small --> airflow scheduler and webserver
db.t2.small --> postgres for metastore
The parallelism parameter in airflow.cfg is set to 10 and there are around 10 users who access airflow UI
All we do from airflow is ssh to other instances and run the code from there
Using TOP to check the cpu and memory status of K8s cluster host, I found that the CPU cost of Nginx-ingress-controller is always high. If I set the cpu limit to 2, then the CPU time would be almost 200%. If without cpu limit, the CPU time would be almost 400% in a 4-core VM. The nginx ingress controller is 0.8.3.
Is that normal?
Thanks
Liu Peng
I'm looking at the documentation of deis and I'm not sure how the scheduler works.
Essentially I want to deploy small apps. My idea is to have different size of apps based on memory (64M, 128M, 256M and 512M).
Then I would have a cluster of small machines (1 CPU, ~3GB) and I want to deploy/undeploy any number of apps, where most of them will only have one instance.
So in this case I need a scheduler that looks at the free memory on each node and deploy the app to the node with more resources available (in this case memory based).
For example, if I have 2GB available for apps I could have the following balancing:
Node1: App1 (256M), App2 (256M), App3 (512M) => Total 1.5 GB
Node2: App4 (512M), App5 (128M), App6 (128M), App7 (256M), App8 (512M), App9 (256M) => Total 1.75 GB
Then if I need to deploy an app that will consume 512M, the scheduler should deploy the app in Node1.
So I wanted to understand if deis could be userful for this scenario.
Under the hood, Deis uses fleet as a scheduler. Currently fleet awards a job to whichever machine in the cluster responds first, and has no understanding of machine load. Smarter scheduling is a priority of the fleet project, and as it improves, Deis improves.