I am new to pcf and working on a task for drop all instances on a particular time.
I need drop all apps instances during non business hours and bring it back before start of business hours?
This can be achieved by using App Autoscaler where you can automatically increase and decrease instances based on either of following-
No of Http request (Http Throughput).
CPU utilization.
Http latency.
This will automatically drop the instances during off hours and increases when no of request start to increase i.e. in business hours.
You could also test this by doing stress testing using tools like jitter which helps you to increase and decrease the load artificially on your app.
Related
we have a reached a limit when it comes to build times within our CI/CD. We have pre-rendered 9k pages and we have switched to ISR that we can update and generate on the fly the less relevant pages.
During small load test we have seen a huge drop of the overall performance of the whole next.js app. Weird was: All servers never died. The load test also end up in timeouts, which the webserver somehow queued. We have seen build time on some pages up to 60 seconds and after then it were delivered and were available on cache.
has anyone faced a similar issue? Also someone an idea how we can increase the horse if we use as CI/CD bitbucket to decrease build time?
We have a backend API which runs in almost constant time (it does "sleep" for given period). When we run a managed API which proxies to it for a long time, we see that from time to time its execution time increases up to twice the average.
From analyzing the Amazon ALB data in production, it seems that the time the request spends inside Synapse remains the same, but the connection time (the time the request enters the queue for processing) is high.
In an isolated environment we noticed that those lags happen approximately every 10 minutes. In production, where we have multiple workers that gets request all the time, the picture is more obscured, as it happens more often (possibly the lag accumulates).
Does anyone aware of any periodic activity in the worker which result delays entering the queue every
few minutes? Any parameter that control this? Otherwise, any idea how to figure out what is the cause?
Attached is an image demonstrating it.
Could be due to gateway token cache invalidation. The default timeout is 15 minutes.
Once the key validation is done via the Key Manager, the key validation info is cached in the gateway. The subsequent API invocations, key validation will be done from this cache. During this time, the execution time will be lower.
After the cache invalidation, the token validation will be done via the key manager (using DB). This will cause an increase in the execution time.
Investigating further, we found out two causes for the spikes.
Our handler writes log to shared file system which was set as sync instead of async. This caused delays. This reduced most of the spikes.
Additional spikes seem to be related to registry updates. We did not investigate those, as they were more sporadic.
I am using the Google Calendar API to preprocess events that are being added (adjust their content depending on certain values they may contain). This means that theoretically I need to update any number of events at any given time, depending on how many are created.
The Google Calendar API has usage quotas, especially one stating a maximum of 500 operations per 100 seconds.
To tackle this I am using a time-based trigger (every 2 minutes) that does up to 500 operations (and only updates sync tokens when all events are processed). The downside of this approach is that I have to run a check every 2 minutes, whether or not anything has actually changed.
I would like to replace the time-based trigger with a watch. I'm not sure though if there is any way to limit the amount of watch calls so that I can ensure the 100 seconds quota is not exceeded.
My research so far shows me that it cannot be done. I'm hoping I'm wrong. Any ideas on how this can be solved?
AFAIK, that is one of the best practice suggested by Google. Using watch and push notification allows you to eliminate the extra network and compute costs involved with polling resources to determine if they have changed. Here are some tips to best manage working within the quota from this blog:
Use push notifications instead of polling.
If you cannot avoid polling, make sure you only poll when necessary (for example poll very seldomly at night).
Use incremental synchronization with sync tokens for all collections instead of repeatedly retrieving all the entries.
Increase page size to retrieve more data at once by using the maxResults parameter.
Update events when they change, avoid re-creating all the events on every sync.
Use exponential backoff for error retries.
Also, if you cannot avoid exceeding to your current limit. You can always request for additional quota.
In our application we need to make around 10k REST api calls to 10k different endpoints URLs per minute. Earlier I was using synchronous model, but quickly realized that I can not scale beyond ~2k+ limit, so I am working on switching to an async model. Using HttpCore-NIO lib I could scale upto 5k or so, but beyond that I randomly get an error 'I/O reactor has been shut down' and the entire app basically stops processing requests. Don't see any stack trace either, which makes it extremely hard to debug.
So I am trying to evaluate what could be the best strategy/library to achieve this scale with Java as the programming language. Any suggestions on which libraries out there should I look into ?
If your machine itself is not stressed out on CPU, Network etc, then perhaps one thing that can help is horizontal scaling - if you can achieve 5K with 1 process, try firing requests from two processes; see what you get; how far you can horizontally scale. The extra coding to achieve horizontal scale should not be that much.
OR try trapping exceptions and ignoring them, to prevent I/O reactor shutdown. Experiment with what can be ignored. See example here https://hc.apache.org/httpcomponents-core-4.4.x/tutorial/html/nio.html#d5e477, search for "I/O reactor exception handling"
Experiment with thread count - setIoThreadCount
Make sure you are not doing too much processing with the HTTP responses; that would hold up the reactor.
As the title implies, the first "cold" POST to our MVC3 app in the Azure cloud is very slow. Once it "spins up", the normal requests are blazing fast. The first spin-up after a brief period of rest takes a few seconds. Subsequent requests can be measured in milliseconds.
How can we keep this thing awake?
This is probably due to the appplication pool unloading after a period on inactivity. The next request has to take the overhead of starting it up again.
To confirm this, you need to turn on the performance counters and look at the numbers of app domain loads and unloads.
Either way, this blog post explains how to fix it.