I'm working on a project to run a Kubernetes cluster on GCE. My goal is to run a cluster containing a WordPress site in multiple zones. I've been reading a lot of documentation, but I can't seem to find anything that is direct and to the point on persistent volumes and statefulsets in a multiple zone scenario. Is this not a supported configuration? I can get the cluster up and the statefulsets deployed, but I'm not getting the state replicated throughout the cluster. Any suggestions?
Thanks,
Darryl
Reading the docs, I see that the recommended configuration would be to create a MySQL cluster with replication: https://kubernetes.io/docs/tasks/run-application/run-replicated-stateful-application/. This way, you would have the data properly replicated between the instances of your cluster (if you are in a multi-zone deployment you may have to create an external endpoint).
Regarding the Wordpress data, my advice would be to go for an immutable deployment: https://engineering.bitnami.com/articles/why-your-next-web-service-should-be-immutable.html . This way, if you need to add a plugin or perform upgrades, you would create a new container image and re-deploy it. Regarding the media library assets and immutability, I think the best option would be to use an external storage service like S3 https://wordpress.org/plugins/amazon-s3-and-cloudfront/
So, to answer the original question: I think that statefulset synchronization is not available in K8s (at the moment). Maybe using a volume provider that allows ReadWriteMany access mode could fit your needs (https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes), though I am quite unsure about the stability of it.
Related
I have 2 nfs mounts of 100TB each i.e. 200TB in total. I have mounted these 2 on Kubernetes container. My file server is a typical log server that holds a mix of data types like JSON, HTML, images, logs and text files, etc. The size of files also varies a lot. I am kind of guessing what should be the ideal resource request for this kubernetes container? My assumption,
As this is file reads its i/o intensive operation, CPU should be high
Since we may have a large file size transferred over, Memory should also be high.
Just wanted to check if my assumptions are right?
Posting this community wiki answer to set a baseline and to show one possible set of actions that should led to solution.
Feel free to edit and expand.
As I stated previously, this setup will heavily depend on case to case basis and giving the approximate could be misleading. In my opinion the best course of actions to take would be:
Install monitoring tools
Deploy the application for testing
Simulate the load
Install monitoring tools
There are a lot of monitoring tools that can retrieve the data about the CPU and Memory usage of your Pods. You will need to choose the one that suits your workloads and infrastructure best.
Some of them are:
Prometheus.io
Elastic.co
Datadoghq.com
Deploy the application for testing
This can also be a quite wide topic considering the fact that the exact requirements and the infrastructure is not known. One of many questions is if the Deployment should have a steady replica amount or should use some kind of Horizontal Pod Autoscaling (basing on CPU and/or Memory). The access modes on the storage shouldn't matter as NFS supports RWX.
The basic implementation of the Deployment that could be used can be found in the official Kubernetes documentation:
Kubernetes.io: Docs: Concepts: Workloads: Controllers: Deployment: Creating a deployment
Kubernetes.io: Docs: Concepts: Storage: Volumes: NFS
Simulate the load
The simulation part could go either as a real life usage or by using a tool to simulate the load. You would need in this part to choose the option/tool that suits your requirements the most. This part will show you the approximate resources that should be allocated to your nginx file explorer.
A side note!
In my testing I've used ab to check if the load was divided equally by X amount of replicas.
Additional resources
I do recommend to check the official guide on official Kubernetes documentation regarding managing resources:
Kubernetes.io: Docs: Concepts: Configuration: Manage resources containers
I also think that the VPA could help you in the whole process as:
Vertical Pod Autoscaler (VPA) frees the users from necessity of setting up-to-date resource limits and requests for the containers in their pods. When configured, it will set the requests automatically based on usage and thus allow proper scheduling onto nodes so that appropriate resource amount is available for each pod. It will also maintain ratios between limits and requests that were specified in initial containers configuration.
It can both down-scale pods that are over-requesting resources, and also up-scale pods that are under-requesting resources based on their usage over time.
-- Github.com: Kubernetes: Autoscaler: Vertical Pod Autoscaler
I'd reckon you could also look on this answer:
Stackoverflow.com: Answers: PromQL query to find CPU and memory used for the last week
I'm learning Kubernetes, and trying to setup a cluster that could handle a single Wordpress site with high traffic. From reading multiple examples online from both Google Cloud and Kubernetes.io - they all set the "accessMode" - "readWriteOnce" when creating the PVCs.
Does this mean if I scaled the Wordpress Deployment to use multiple replicas, they all use the same single PVC to store persistent data - read/write data. (Just like they use the single DB instance?)
The google example here only uses a single-replica, single-db instance - https://cloud.google.com/kubernetes-engine/docs/tutorials/persistent-disk
My question is how do you handle persistent storage on a multiple-replica instance?
ReadWriteOnce means all replicas will use the same volume and therefore they will all run on one node. This can be suboptimal.
You can set up ReadWriteMany volume (NFS, GlusterFS, CephFS and others) storage class that will allow multiple nodes to mount one volume.
Alternatively you can run your application as StatefulSet with volumeClaimTemplate which ensures that each replica will mount its own ReadWriteOnce volume.
If on AWS (and therefore blocked by the limitation of EBS volumes only mountable on a single instance at a time), another option here is setting up the Pod Affinity to schedule on the same node. Not ideal from an HA standpoint, but it is an option.
If after setting that up you start running into any wonky issues (e.g. unable to login to the admin, redirect loops, disappearing media), I wrote a guide on some of the more common issues people run into when running Wordpress on Kubernetes, might be worth having a look!
Long story short - I need to use networking between projects to have separate billing for them.
I'd like to reach all the VMs in different projects from a single point that I will use for provisioning systems (let's call it coordinator node).
It looks like VPC network peering is a perfect solution to this. But unfortunately one of the existing networks is "legacy". Here's what google docs state about legacy networks.
About legacy networks
Note: Legacy networks are not recommended. Many newer GCP features are not supported in legacy networks.
OK, naturally the question arises: how do you migrate out of legacy network? Documentation does not address this topic. Is it not possible?
I have a bunch of VMs, and I'd be able to shutdown them one by one:
shutdown
change something
restart
unfortunately it does not seem possible to change network even when VM is down?
EDIT:
it has been suggested to recreate VMs keeping the same disks. I would still need a way to bridge legacy network with new VPC network to make migration fluent. Any thoughts on how to do that using GCE toolset?
One possible solution - for each VM in the legacy network:
Get VM parameters (API get method)
Delete VM without deleting PD (persistent disk)
Create VM in the new VPC network using parameters from step 1 (and existing persistent disk)
This way stop-change-start is not so different from delete-recreate-with-changes. It's possible to write a script to fully automate this (migration of a whole network). I wouldn't be surprised if someone already did that.
UDPATE
https://github.com/googleinterns/vm-network-migration tool automates the above process, plus it supports migration of a whole Instance Group or Load Balancer, etc. Check it out.
I am working on a new project with Kubernetes and I need three environments: DEV,QA and PROD.
What is most recommended, create Multiple Clusters or create one big cluster separating environments by namespace.
Are you just going to have a single prod cluster or multiple prod clusters? One thing to consider is that updating the cluster management software (to a new k8s release) can impact your application. If you only plan to have a single prod cluster, I'd recommend running qa and dev separately so that you can upgrade those clusters first to shake out any issues. If you are going to have multiple prod clusters, then you can upgrade them one at a time to ensure application availability and sharing the clusters between environments makes a lot more sense.
Namespaces will not bring you isolation, at the moment it's just a different subdomain in dns. It's better to have namespace per application.
I highly recommend you to have two clusters for prod (in case of updating k8s) and one-two for dev/qa.
Take a look at this blog post: Checklist: pros and cons of using multiple Kubernetes clusters, and how to distribute workloads between them.
I'd like to highlight some of the pros/cons:
Reasons to have multiple clusters
Separation of production/development/test: especially for testing a new version of Kubernetes, of a service mesh, of other cluster software
Compliance: according to some regulations some applications must run in separate clusters/separate VPNs
Better isolation for security
Cloud/on-prem: to split the load between on-premise services
Reasons to have a single cluster
Reduce setup, maintenance and administration overhead
Improve utilization
Cost reduction
Considering a not too expensive environment, with average maintenance, and yet still ensuring security isolation for production applications, I would recommend:
1 cluster for DEV/QA (separated by namespaces, maybe even isolated, using Network Policies, like in Calico)
1 cluster for PROD
Definitely concur that you want multiple clusters:
anything critical to k8s that may fail during an upgrade or because you screw up somewhere will affect the whole cluster.
for example, I had an issue with DNS which wrecked havoc in my cluster; all namespaces were affected.
Upgrades are usually not a big deal but one day you might hit a roadblock; if kubelet fails for too long your pods will get killed.
So it's best to upgrade your test/dev environments and iron things out there before upgrading in prod.
Sailsjs requires setup to handle scaling horizontally. There are multiple ways to do this. I'm not sure if I have done this correctly, due to poor performance during load testing. Please confirm if I understand and am doing the setup correctly.
I've created a load balancer on the Google platform for handling the distribution of requests across the instances. Much is spoken about of Nginx for distributing, but I understand Googles load balancer does all I need in this regard. Note, I use session affinity: Client IP.
I've set up config/session.js to use express-mysql-session, so MemoryStore is not used.
I haven't set up anything in config/sockets.js. My project doesn't use live chat etc with socket.io, all requests are to waterline for data from db. But if this is a issue, please refer me to a way to do this with Mysql db not redis (or memory).
I use pm2 as a way to keep it live and to distribute processing on a instance.
Those are the main factors I've found regarding horizontal scaling with sailsjs.