Admission control is embedded within each impalad daemon and communicates through the statestore service. The impalad daemon determines if a query runs immediately or if the query is queued.
However If a sudden flow of requests causes more queries to run concurrently than expected, the overall Impala memory limit and the Linux cgroups mechanism at the cluster level serve as hard limits to prevent over allocation of memory. When queries hit these limits, Impala cancels the queries.
Does this mean Impala Resource Limits Enforced at individual Impala daemon level or at the cluster level?.
The answer is both. Each impalad daemon has its own MEM_LIMIT. Exceeding it will cause the query to be canceled. The admission control pool works at the cluster level, even though the gatekeep (decide whether the query should be run or queued) is at each impalad level, even though these impalad makes the admission decision based on the pool resource at the cluster level. That's why when there are a flood of queries sent to different impalad instances, the impalad daemons might admit more queries than they should because they cannot get the most current cluster resource usage information at the time. CGroup limit does not cause query to be canceled. It determines the percentage of CPU that the impalad should get when there is CPU contention.
Related
I use kusto ingest client kustoClient.IngestFromDataReader to ingest data. And it throws exception An error occurred for source: 'DataReader'. Error: 'Failed to ingest: State='Throttled', Status='The control command was aborted due to throttling. Retrying after some backoff might succeed. CommandType: 'DataIngestPull', Capacity: 18, Origin: 'CapacityPolicy/Ingestion'.'. I read the document here https://learn.microsoft.com/en-us/azure/data-explorer/kusto/management/capacitypolicy#ingestion-capacity, and guess it may because that there are too many requests run concurrently and the cluster capacity is limited, am I right?
I am still a bit confused about the document. What does the final number (Minimum(ClusterMaximumConcurrentOperations, Number of nodes in cluster * Maximum(1, Core count per node * CoreUtilizationCoefficient))) mean? Does it means the total concurrent operations number? And specifically does one kusto ingest client or one kusto ingest command only have one concurrent operation or it is configurable?
Thanks a lot!
Effectively the document means that ingest capacity (in terms of concurrent ingest operations) is 3/4 times the overall number of cores in the cluster, but not higher than 512.
You can view your cluster capacity and its utilization by running '.show cluster capacity' command.
If you do not want to handle the throttling by yourself, you should use the KustoQueuedIngestClient class, and pass to it the ingestion service endpoint (https://ingest-..kusto.windows.net).
The ingestion service will take care of managing the load on your cluster.
See Ingestion Overview article for more details.
I am getting {"code": "Too many requests", "message": "Request is denied due to throttling."} from ADX when I run some batch ADF pipelines. I have came across this document on workload group. I have a cluster where we did not configured work load groups. Now i assume all the queries will be managed by default workload group. I found that MaxConcurrentRequests property is 20. I have following doubts.
Does it mean that this is the maximum concurrent requests my cluster can handle?
If I create a rest API which provides data from ADX will it support only 20 requests at a given time?
How to find the maximum concurrent requests an ADX cluster can handle?
for understanding the reason your command is throttled, the key element in the error message is this: Capacity: 6, Origin: 'CapacityPolicy/Ingestion'.
this means - the number of concurrent ingestion operations your cluster can run is 6. this is calculated based on the cluster's ingestion capacity, which is part of the cluster's capacity policy.
it is impacted by the total number of cores/nodes the cluster has. Generally, you could:
scale up/out in order to reach greater capacity, and/or
reduce the parallelism of your ingestion commands, so that only up to 6 are being run concurrently, and/or
add logic to the client application to retry on such throttling errors, after some backoff.
additional reference: Control commands throttling
We are using Cosmos DB SDK whose version is 2.9.2. We perform Document CRUD operations. Usually, the end-to-end P95 latency is 20ms. But sometimes the latency is over 1000ms. The high latency period lasts for 10 hours to 1 day. The collection is not throttling.
We have get some background information from:
https://icm.ad.msft.net/imp/v3/incidents/details/171243015/home
https://icm.ad.msft.net/imp/v3/incidents/details/168242283/home
There are some diagnostics strings in the tickets.
We know that the client maintains a cache of the mapping of logical partition and physical replica address. This mapping may be outdated because of replicas movement or outage. So client tries to read from the second/third replica. However, this retry has significant impact on end to end latency. We also observe that the high latency/timeout can last for several hours, even days. I expect there’s some mechanism of refreshing mapping cache in the client. But it seems the client stops visiting more than one replica only after we redeploy our service.
Here are my questions:
How can the client tell whether it’s unable to connect to a certain replica? Will the client wait until timeout or server tells client that the replica is unavailable?
In which condition the mapping cache will be refreshed? We are using Session consistency and TCP mode.
Will restarting our service force the cache to be refreshed? Or refreshing only happens when the machine restarts?
When we find there’s replica outage, is there any way to quickly mitigate?
What operations are performed (Document CRUD or query)?
And what are the observed latencies & frequencies? Also please check if the collection is throttling (with custom throttling policy).
Client do manage the some metada and does handle its staleness efficiently with-in SLA bounds.
Can you please create a support ticket with account details and 'RequestDiagnostis' and we shall look into it.
We have a replicated cluster cache setup with two instances, everything runs well when both instances are on-line, and we are using Community Edition 4.8.
When we take an instance offline, cache management becomes very slow and even stopping and starting the cache from NCache Manager GUI takes a very long time and then shows a message stating that there is an instance that is un-reachable.
Also when trying to fetch data from cache or add data to it, it gives an exception of operation timeout, and there is no response from the single instance that is still running.
From what I understand, this scenario should be handled by the cache service it-self since it is replicated, and it should handle failure for an instance going offline.
Thank you,
I would like to explain the cause of slowness on your application when one of the server node gets removed from the cache cluster.
What happens is whenever a node gets removed from the cache cluster, the surviving node/nodes go into recovery process and try to re-establish connection with that downed server node. By default this Connection retry value is set to “2” which means that the surviving nodes would try to reconnect with the downed node two times and after the reconnection has failed, the cache cluster would consider the downed server and offline and the cluster would start handling requests like before. This reconnection process can take up to 90 seconds as this is the default TCP/IP timeout interval and if the connection retry is set to “2” the recovery process could take up to around 200 seconds. Your application(Or NCache Manager calls) could experience slowness or request timeouts during this 2-3 minutes window when the cluster is in the recovery mode but once the recovery process is finished, the application should start working without any issues. If the slowness or request timeouts last more than a few minutes
The Connection retry value can be changed from the NCache “Config.ncconf” file. Increasing the number of connection retries would mean that the cluster would spend more time in the recovery process. The purpose of this feature is that if there is a network glitch in the environment and the server nodes lose connection with eachother, the servers would get reconnected automatically due to this recovery process. This is the reason why it is recommended to keep the Connection Retry interval set to at least 1.
we're currently having issue in our production servers and would like to try to replicate the issue in our dev. I'm currently awaiting access to our Performance Monitoring Tool, and while waiting would like to play with it a little.
I'm thinking of, since I suspect a host throttling in prod, forcing hosts to throttle in dev and see if it will recreate the issue.
Is there a way to do this?
As others have mentioned, monitoring of the throttling counters and other counters like memory and WIP messages is a must to see what is going on in your production server. Also would recommend that set up a SCOM alert on throttling states of 3+ (publishing + delivery states), if you have SCOM.
Message throughput can grind to a halt on especially the memory (4, 5) and Queue Size (6) states. States 1+2 are generally short lived (e.g. arrival of a large batch of messages) and Biztalk recovers within a few seconds.
Simulating the memory state in your Dev environment should be straightforward by tweaking the throttling thresholds (obviously not something to be taken lightly in production!)
e.g. to trigger the Memory threshold states - AFAIK the lowest memory usage threshold you can set is 101MB. Running a load test in dev should then be able reproduce the throttle.
There is also apparantly a user-based throttling override to set states 10 and 11 although haven't actually tried this.
Some other experience on avoiding throttling:
(Caveat - I don't have an active BizTalk 2006/R2 setup - this is for 2009 / 2010)
If you do a lot of asynchronous processing (e.g. Queue receives), ensure that you have split functionality into separate Hosts for Receive, Processing and Send hosts. This way you can adjust the throttling for asynch Receive hosts to trigger much earlier than the processing and sending hosts - this should have the effect of constricting new incoming messages to the messagebox but allowing existing messages to complete processing.
On 64 bit hosts, the default 25% memory host usage throttling level is usually an unnecessary liability - we increased this using Yossi Dahan's recommendation of 50% on a 4GB server
Note that suspended messages count toward throttling state 6 - ensure that you have a strategy for dealing with suspended messages (and obviously ensure that the Sql Agent jobs are running!).