I have installed helm chart for jupyterhub bitnami over GKE.
When a singleuser pod is created, it gets "evicted" after some time because of ephermal storage with following error
"Pod ephemeral local storage usage exceeds the total limit 1Gi"
I did not find in documentation how to increase ephermal storage. Can you please tell how can I increase this ephermal storage or what is the alternative way to fix this issue?
After scratching my head for hours, I had to study the implementation of jupyter kubernetes spawner here and also kubernetes client library here. I've found the way to set ephermal-storage for jupyterhub user notebook. Unfortunately, there was not documentation for it.
I added extraResources in hub.configuration within values.yaml
extraResource:
limits:
ephemeral-storage: '5Gi'
guarantees:
ephemeral-storage: '5Gi'
Related
I have spring boot application with /health endpoint accessible deployed in AWS ECS Fargate. Sometimes the container is stopped with Task failed container health checks message. Sometimes happens once daily, sometimes once a week, maybe depends on the load. This is the healthcheck command specified in the Task Definition:
CMD-SHELL,curl -f http://localhost/actuator/health || exit 1
My question is how to troubleshoot what AWS receive when health-check is failed.
In case anyone else lands here because of failing container health checks (not the same as ELB health checks), AWS provides some basic advice:
Check that the command works from inside the container. In my case I had not installed curl in the container image, but when I tested it from outside the container it worked fine, which fooled me into thinking it was working.
Check the task logs in CloudWatch
If the checks are only failing sometimes (especially under load), you can try increasing the timeout, but also check the task metrics (memory and CPU usage). Garbage collection can cause the task to pause, and if all the vCPUs are busy handling other requests, the health check may be delayed, so you may need to allocate more memory and/or vCPUs to the task.
Thank #John Velonis,
I don't have enough reputation for commenting on your answer, so I post that in a different answer
For my case, the ecs container keeps getting UNKNOWN status from the ecs cluster. But I can access the healthcheck successfully. when I read this post, and check my base image which is node:14.19.1-alpine3.14, it doesn't have curl command.
so I have to install that in the Dockerfile
RUN apk --no-cache add curl
I'm attempting to encrypt our existing PVs in EKS. Basically been searching the net and haven't come up with a solid solution. Our K8s version is 1.11 on EKS. Our PVs are EBS volumes. We currently have account level ebs encryption enabled but didn't when these resources were made.
I attempted to stop the ASG and the node. Created a snapshot and new encrypted volume then ran:
kubectl patch pv pvc-xxxxxxxx-xxxxx-xxxxxxxx -p '{"spec":{"awsElasticBlockStore":{"volumeID":"aws://us-east-1b/vol-xxxxx"}}}'
but was met with:
The PersistentVolume is invalid: spec.persistentvolumesource: Forbidden: is immutable after creation
I'm looking for a solution to this issue or to validate that is in fact not possible. Potential relevant github issue:
https://github.com/kubernetes/kubernetes/issues/59642
Thanks in advance
I'm learning Kubernetes, and trying to setup a cluster that could handle a single Wordpress site with high traffic. From reading multiple examples online from both Google Cloud and Kubernetes.io - they all set the "accessMode" - "readWriteOnce" when creating the PVCs.
Does this mean if I scaled the Wordpress Deployment to use multiple replicas, they all use the same single PVC to store persistent data - read/write data. (Just like they use the single DB instance?)
The google example here only uses a single-replica, single-db instance - https://cloud.google.com/kubernetes-engine/docs/tutorials/persistent-disk
My question is how do you handle persistent storage on a multiple-replica instance?
ReadWriteOnce means all replicas will use the same volume and therefore they will all run on one node. This can be suboptimal.
You can set up ReadWriteMany volume (NFS, GlusterFS, CephFS and others) storage class that will allow multiple nodes to mount one volume.
Alternatively you can run your application as StatefulSet with volumeClaimTemplate which ensures that each replica will mount its own ReadWriteOnce volume.
If on AWS (and therefore blocked by the limitation of EBS volumes only mountable on a single instance at a time), another option here is setting up the Pod Affinity to schedule on the same node. Not ideal from an HA standpoint, but it is an option.
If after setting that up you start running into any wonky issues (e.g. unable to login to the admin, redirect loops, disappearing media), I wrote a guide on some of the more common issues people run into when running Wordpress on Kubernetes, might be worth having a look!
Ok so this might be a basic question, but i'm new to kubernetes and tried to install wordpress using helm unto it, using the stable/wordpress chart, but i keep getting an error "pod has unbound immediate PersistentVolumeClaims (repeated 2 times)" is this because of the requirement in here https://github.com/helm/charts/tree/master/stable/wordpress. "PV provisioner support in the underlying infrastructure" how do i enable this in my infrastructure, i have setup my cluster across three nodes on digitalocean, i've tried searching for tutorials on this, with no luck until now. Please let me know what i'm missing, thanks.
PersistentVolume types are implemented as plugins. Kubernetes currently supports the following plugins:
GCEPersistentDisk
AWSElasticBlockStore
AzureFile
AzureDisk
FC (Fibre Channel)
Flexvolume
Flocker
NFS
iSCSI
RBD (Ceph Block Device)
CephFS
Cinder (OpenStack block storage)
Glusterfs
VsphereVolume
Quobyte Volumes
HostPath (Single node testing only – local storage is not supported in any way and WILL NOT WORK in a multi-node cluster)
Portworx Volumes
ScaleIO Volumes
StorageOS
You can enable support for PVs or Dynamic PVs using thoese plugins.
detail reference
On Digital Ocean you can use block storage for volumes.
details
Kubernetes can be set-up for Dynamic Volume Provisioning. This would allow the Chart to run to completion using the default configuration as the PVs would be provisioned on-demand.
This is probably an inane question for those of you experienced in AWS but I've been googling a few hours and really need a straightforward guide. I have configured my site running bitnami wordpress on one T2 Micro EC2 instance.
I'm going to launch the site soon but would like it to elastically scale with demand. This might be an oversimplified question, but how do I set this up? Do I make a second instance from the same EBS volume and balance load between them? I'm just a little lost. Any guidance on where to start configuring scalability for a single EC2 instance would be very very helpful. Thank you.
To load balance your traffic you will need to configure ELB. To auto-scale your instance you will need to configure auto-scaling group. Once you are done with the wordpress configuration on your EC2 instance follow the below steps :
Create an AMI of your EC2 instance. Ref :
https://docs.aws.amazon.com/toolkit-for-visual-studio/latest/user-guide/tkv-create-ami-from-instance.html
Create a classic load balancer Ref : https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/elb-getting-started.html
Create launch configuration using the AMI created in step 1. Ref :
https://docs.aws.amazon.com/autoscaling/latest/userguide/create-launch-config.html
Create an auto-scaling group using launch configuration created in step 3. Ref :
https://docs.aws.amazon.com/autoscaling/latest/userguide/create-asg.html
(At step 6.f. in this link you will need to use ELB that you created
in step 2).
All the AWS docs/guides provide step by step explanation of how to configure things and hence I haven't provided more details in above steps but the sequence of steps that you need to perform.