I have a cluster of Elastic Beanstalk instances running Wordpress. RAM utilization is high, around 93% because of some plugins that we're running (a different issue altogether). However, every time one of the instances hits 90+% RAM utilization it kicks the instance into a "degraded" state and kicks it out of the cluster.
I can't for the life of me find a way to modify this RAM utilization check. Ideally I'd be able to bump the RAM utilization threshold up to 95% or so....how do I change this metric, disable it, or remove it altogether?
EDIT
Here is a screenshot of the elastic beanstalk Health screen. I may have jumped to conclusions assuming it's a Cloudwatch metric but the concept is still the same...how do I up the threshold?
Of particular note, the instance status and the message beneath it.
Related
I am experiencing an issue with my EC2 instance. I am scraping different websites using R programming and it works fine but after some hours, my EC2 instance is freezing.
After raising a ticket to AWS support, they noticed that this was caused by the rise of the "VolumeQueueLength" which then was decreasing the BurstBalance credits from 100 to 0.
See below when I tried around June 19th:
Would you know what is causing this VolumeQueueLength to go up?
Thanks a ton!
From I/O Characteristics and Monitoring - Amazon Elastic Compute Cloud:
If your I/O latency is higher than you require, check VolumeQueueLength to make sure your application is not trying to drive more IOPS than you have provisioned. If your application requires a greater number of IOPS than your volume can provide, you should consider using a larger gp2 volume with a higher base performance level or an io1 volume with more provisioned IOPS to achieve faster latencies.
For more information about Amazon EBS I/O characteristics, see the Amazon EBS: Designing for Performance re:Invent presentation on this topic.
This is basically saying that the IO allocated to an Amazon EBS 'General Purpose' volume is proportional to its size, so a larger volume might solve your IO problems. Alternatively, you could consider moving to a Provisioned IOPS volume (which is faster, but more expensive).
Your application seems to be using more IO than has been allocated for the volume.
For each time ECS Fargate service scales in or out, Cloudwatch show the CPU utilization with very low percent (about 2 -> 3%) (same with Memory) on graph and then it will be increased gradually although before that time, it's quite high (polices: 80% for scaling out, 40% for scaling in).
I just worry if there is any unavailable period (or break-time) when it's scaling?
I just worry if there is any unavailable period (or break-time) when it's scaling?
Technically, at the Fargate level, the answer is no as long as you have your service set with a minimum task count >= 1.
The wiggle room in saying "no" though, is if your app spikes from 70% CPUUtilization to 100%, the application itself may become unresponsive before Cloudwatch is able to trigger an alarm which in turn triggers the service to scale out.
although before that time, it's quite high
Keep in mind, scaling actions aren't instantaneous. If you're using a Cloudwatch metric for CPUUtilization with a period of 60 seconds, and a threshold of 2, that means your task would have to be > 80% utilization for over 2 minutes before autoscaling was triggered.
In addition to this, Fargate startup time is slower than ECS launch type startup time, because AWS has to do some magic behind the scenes - specifically download the image and also attach an ENI - to make it "serverless".
So, if your application goes over 80% utilization, you wouldn't see it autoscale immediately. This might explain what you're seeing in Cloudwatch with the utilization being high enough to trigger scaling, dropping off to 2% but after the scaling has already been triggered.
This is a very simple question for those with the knowledge, but I'm a newbie.
In essence, I just need to know if it would be considered okay to run a small, approx. 700 visitors/day bitnami wordpress blog on just one t2.medium EC2 instance (without any auto-scaling, beanstalk).
Am at risk of it crashing? What stats should I monitor or be aware of to be aware of potential dangers? Sorry for the basic nature of these questions, but this is new.
tl;dr: It might be "okay", but it's not ideal.
If your question is because of:
Initial setup time - Load-balancing and auto-scaling will be less expensive (more time-efficient) over time.
Cost - Auto-scaling spins down instances that aren't being used to reduce cost.
Minimal setup for a great user experience - The goal of a great AWS setup is to ensure that capacity matches demand
Am at risk of it crashing?
Possibly, yes. If you average 700 visitors, then the risk is traffic spikes if all visitors hit at the same. It also depends on what your maximum visitors are, which could vary widely from the average (or not)
What stats should I monitor or be aware of to be aware of potential dangers?
Monitor the usage on high traffic days (ie. public holiday sales)
Setup billing alerts
Setup the right metrics:
See John Rotenstein's SO answer:
CPU Utilization is not always the right measure to use -- your
application might only be able to handle a limited number of
connections, it might be squeezed on RAM and the types of requests
might vary too.
You can use normal monitoring tools, or you can write something that
pushes metrics to Amazon CloudWatch, so that you go beyond the basic
CPU and Network metrics that CloudWatch normally provides. You could
even use the Load Balancer's Latency metric to trigger scaling when
the application slows down (custom code required).
I'd start with:
Two or more instances - to deal with instance redundancy (an instance going down)
Several t2.small rather than one t2.medium can work out to be more cost-efficient, and more cost efficient than EC in some use cases.
Add auto-scaling - automatically spin up or down instances based on minimum and maximum counts
Load balancing - to re-route users from unhealthy to healthy instances. And also to keep all of the spun up instances all working as evenly as possible (rather than a single instance handling 80% of the workload while the others bludge).
You can always reduce your instances after time with monitoring.
In my opinion, with 700 visitors a day, the safer option would be to run a load balanced/auto-scaling environment on Elastic Beanstalk with at least 2 instances. The problem with running just one instance is that yes you are at a great risk of crashing in case you get an increase in traffic or when the instance goes down and with just one running you will not have a fallback. You can easily set up CloudWatch monitoring on NetworkIn, NetworkOut to get a sense of the number of requests your site is receiving and serving, and setup CPU Usage monitoring as well. The trade-off with running a load balanced environment over a single instance environment is that the cost might significantly increase as you introduce other things into your environment such as a load balancer. Also if you introduce a load balancer consider reducing the instance size to maybe a t2.small, could aid in reducing the cost.
It actually depends. This question range is wide. You have multiple options here.
You can use only ec2 instance for that much amount of visitors or even more if your application allows. You can also consider caching if your app need it.
You may add instance in an autoscaling group. So that if by any chance you need more resources you can increase them horizontally.
You can add load balancers lateron also. You just need to add user data in your launch configuration attached to autoscaling group. So when your instance get up it should automatically register itself in your load balancer.
For monitoring, you can check for the request metrics in cloudwarch for ELB. You have to keep an eye on your CPU and trigger the scale out policy once it reaches a particular threshold.
I have an application that spikes from 500 rpm to 5000 and stays there for 20-30min. I know that's not a ton of requests but its the magnitude of the jump that is killing me. AWS-EC2 takes 5 min to scale up so that's not helpful when things move so fast. Maybe multiple DB's that handle different pieces of the application.
How would you go about analyzing this and thinking about infrastructure if you will always go from 500 to 5000RPM or higher in one minute?
This is the graph from my AWS logs:
If you can predict that demand will increase at some point you can automate provisioning of new instances. If you can't determine this then you need to do proper capacity planning. For instance, how many servers/containers do you need running to sustain the load with an acceptable user experience? This will be key to determine.
You also should look at implement asynchronous messaging patterns that offload the spike although this may come with some performance degradation.
One additional consideration would be moving to a serverless architecture like AWS Lambda. This likely wouldn't fully solve the problem but would provide you more ability to quickly provision on demand infrastructure.
My cluster has one MASTER AGGREGATOR and one LEAF. After running two months, the CPU usage in LEAF is very high, almost at 100%. So, is this normal?
By the way, its size is 545 MB for table data.
This is not normal for MemSQL operation. Note that the Ops console is showing you all CPU use on that host, not just what MemSQL is using. I recommend running 'top' or similar to determine what process(es) are consuming resources.
You can also run 'SHOW PROCESSLIST' on any node to see if there is a long-running MemSQL process.