I have a route in camel to consumer from kafka. It is consuming and producing with a TPS of 2000 if the incoming message is 18000 TPS. so the consumer topic has consumer lag. If I keep max.poll.recors = 500 i'm able to achieve 2000 TPS. If I keep producer settings requestRequiredAcks=0 I can achieve 4000 TPS. but still with consumer lag.
We know that camel route is complete when from->to is complete. a consumer which is consuming from 2 partitions with a consumer count 2, is busy until the route is complete.
is there a way to make camel-kafka consumer asynchronous . any code example?
from("kafka:{{consumer.topic}}?brokers={{kafka_dev.host}}"
+ "&maxPollRecords={{consumer.maxPollRecords}}" + "&consumersCount=2"
+ "&seekTo=latest" + "&groupId={{consumer.group}}" + "&keyDeserializer="
+ KEYDESERIALIZER + "&valueDeserializer=" + VALUEDESERIALIZER + SSL).doTry()
.routeId("route1")
.process(new CamelProcessor())
.to("kafka:{{producer.topic}}?brokers={{kafka_dev.host}}" +"&requestRequiredAcks=1" )
.doCatch(Exception.class));
Also we have observed that introducing threads in this route re-reads the same processed and sent messages. is this link saying camel-kafka https://stackoverflow.com/questions/56716812/how-to-commit-offsets-thread-safe-using-camel-kafka
Related
I am stuck with an issue where lot of messages have reached several rabbitMQ dead-letter queues. I have never had to reprocess bulk messages from DLQs earlier. The requirement now is to reprocess all these messages but in the exact sequence how they were pushed, irrespective of which DLQ it was. Is there a way to group all these failed messages from across all the DLQs and then process them one by one in FIFO basis?
Example:
Event1.DLQ: Message1 at T[1], Message2 at T[3]
Event2.DLQ: Message1 at T[2], Message2 at T[5]
Event3.DLQ: Message1 at T[4], Message2 at T[6]
(T[n] -> Timestamp; T[n] is less than T[n+1])
I would like to combine the failed messages from all these 3 DLQs into a common queue and reprocess them in the following sequence one by one (in order of Time):
Event1.Message1, Event2.Message1, Event1.Message2, Event3.Message1, Event2.Message2, Event3.Message2
Not sure if this question is too naïve, any help is appreciated.
I am seeing the continuous rebalancing on my application. My application is developed in batch mode and here are configuration properties which have been added.
myapp.consumer.group.id= cg-id-local
myapp.changefeed.topic= test_topic
myapp.auto.offset.reset=latest
myapp.enable.auto.commit=false
myapp.max.poll.interval.ms=300000
myapp.max.poll.records= 20000
myapp.idle.time.between.polls=240000
myapp.concurrency = 10
container factory:
ConcurrentKafkaListenerContainerFactory<String, String> factory = new ConcurrentKafkaListenerContainerFactory<>();
factory.setConsumerFactory(consumerFactory(poSummaryCGID));
factory.setConcurrency(poSummNoOfConsumers);
factory.setBatchListener(true);
factory.setAckDiscarded(true);
factory.getContainerProperties().setAckMode(ContainerProperties.AckMode.MANUAL_IMMEDIATE);
factory.getContainerProperties().setIdleBetweenPolls(idleTimeBetweenPolls);
I have few Questions here:
I have setup the maximum record count per poll(4 min) is 20000 and we have 10 partitions in a TOPIC. Since i setup the concurrency as 10, so 10 consumers will up and running and each will listen to 1 partition. My question here is, does the record count will be split across all the consumers like each consumer can handle 2000 records ?
The max.poll.interval.ms has been setup with 5 min. I am sure that the consumer will process 2000(if my above understanding is correct) records in a given poll interval(4 min) which is less than max.poll.interval.ms which has upper bound limit. But not sure why rebalancing is happening? are there any other configuration properties i need to setup ?
Help would be greatly appreciated!!
Tried with these configurations:
myapp.max.poll.interval.ms=600000
myapp.max.poll.records= 2000
myapp.idle.time.between.polls=360000
myapp.max.poll.interval.ms=300000
myapp.max.poll.records= 2000
myapp.idle.time.between.polls=300000
myapp.max.poll.interval.ms=300000
myapp.max.poll.records= 2000
myapp.idle.time.between.polls=180000
EDIT FIX :
We should always
myapp.max.poll.interval.ms >
(myapp.idle.time.between.polls + myapp.max.poll.records processing time).
No. max.poll.records is per consumer, not per topic or container.
If you have concurrency=10 and 10 partitions you should reduce max.poll.records to 2000 so that each consumer gets a max of 2000 per poll.
The container will automatically reduce the idle between polls so that the max.poll.interval.ms won't be exceeded, but you should be conservative with these properties (max.poll.records and max.poll.interval.ms) such that it will never be possible to exceed the interval.
Current wrk configuration allows sending continuous requests for seconds (duration parameter).
Is there a way to use wrk to send requests and then exit.
My use case: I want to create large number of threads + connections (e.g. 1000 threads with 100 connections per thread) and send instantaneous bursts towards the server.
You can do it with LUA script:
local counter = 1
function response()
if counter == 100 then
wrk.thread:stop()
end
counter = counter + 1
end
Pass this script with -s command line parameter.
I make changes to wrk to introduce new knobs. Let me know if anyone is interested in the patch and I could post it.
I added a -r to send exactly requests and bail out.
Artem,
I have this code-change in my fork:
https://github.com/bhakta0007/wrk
On 2 occasions in the past month, we have managed to hit our daily limit on asynchronous apex executions. Salesforce temporarily increased our limit to 425000 but it will be scaled down to 250000 in a week's time. Once we reach the limit, a lot of the SF functions will fail and this has tremendously impacted both internal staff and external customers.
So to prevent this from happening in the future, we need to create some kind of alert in Salesforce to monitor our daily asynchronous apex method executions. Our maximum daily limit is 250000. The alert will need to create a P3 helpdesk ticket and notify couple of users say USER A and USER B once it reaches 70% threshold.
Kindly advise what is possible to achieve the same
Thanks & Regards,
Harjeet
There's a promising Limits method but it doesn't seem to work currently ("reserved for future use"): System.debug(Limits.getAsyncCalls() + ' / ' + Limits.getLimitAsyncCalls());
There's an idea you can upvote: https://success.salesforce.com/ideaView?id=0873A0000003VIFQA2 ;)
You could query SELECT COUNT() FROM AsyncApexJob WHERE ... but that sounds like a bad idea ;)
I think your best course of action is to use SF REST API. There's a "limits" resource you can fetch. You could do it from SF itself (bad idea because if you'd schedule it to run every hour then well, of course it will contribute to the limit consumption too ;)) or from some external app that'd connect to your SF...
https://developer.salesforce.com/docs/atlas.en-us.api_rest.meta/api_rest/dome_limits.htm
You can quickly try it out for example in workbench.developerforce.com before you decide you do want to deep dive into coding it.
Of course if you have control over your batch jobs, queuable, schedulable & #future calls you could implement some rough counter of executions in a helper object for example... won't help you much if most of the jobs are coming from managed packages though...
Got 1 more idea but it's pretty hardcore - you should be able to make a REST API call from javascript. so you could create a simple VF page (even without any apex controller), put JS callout on it, have it check every 5 mins and do something if threshold is hit... But that means IT person would have to have this page open all the time (perhaps as a home page component)... Messy :)
I was having the exact same issue so I created a simple JsForce script in NodeJS to monitor the call to the /limits endpoint.
You can connect a Free Monitoring service like UpTimerobot.com or PingDom.com and get an email when you find the Word "Warning" >50% or "Error" > 80%.
async function getSfLimits() {
try {
//Let's login into salesforce
const login = await conn.login(SF_USERNAME, SF_PASSWORD+SF_SECURITY_TOKEN);
//Call the API
const sfLimits = await conn.requestGet('/services/data/v51.0/limits');
return sfLimits;
} catch(err) {
console.log(err);
}
}
https://github.com/carlosdevia/salesforcelimits
I know client side _underscore.js can be used to throttle click rates, but how do you throttle calls server side? I thought of using the same pattern but unfortunately _throttle doesn't seem to allow for differentiating between Meteor.userId()'s.
Meteor.methods({
doSomething: function(arg1, arg2){
// how can you throttle this without affecting ALL users
}
);
Here's a package I've roughed up - but not yet submitted to Atmosphere (waiting until I familiarize myself with tinytest and write up unit tests for it).
https://github.com/zeroasterisk/Meteor-Throttle
Feel free to play with it, extend, fix and contribute (pull requests encouraged)
The concept is quite simple, and it only runs (should only be run) on the server.
You would first need to come up with a unique key for what you want to throttle...
eg: Meteor.userId() + 'my-function-name' + 'whatever'
This system uses a new Collection 'throttle' and some helper methods to:
check, set, and purge records. There is also a helper checkThenSet
method which is actually the most common pattern, check if we can do something,
and the set a record that we did.
Usage
(Use Case) If your app is sending emails, you wouldn't want to send the same email over
and over again, even if a user triggered it.
// on server
if (!Throttle.checkThenSet(key, allowedCount, expireInSec)) {
throw new Meteor.Error(500, 'You may only send ' + allowedCount + ' emails at a time, wait a while and try again');
}
....
On Throttle Methods
checkThenSet(key, allowedCount, expireInSec) checks a key, if passes it then sets the key for future checks
check(key, allowedCount) checks a key, if less than allowedCount of the (unexpired) records exist, it passes
set(key, expireInSec) sets a record for key, and it will expire after expireInSec seconds, eg: 60 = 1 min in the future
purge() expires all records which are no longer within timeframe (automatically called on every check)
Methods (call-able)
throttle(key, allowedCount, expireInSec) --> Throttle.checkThenSet()
throttle-check(key, allowedCount) --> Throttle.check()
throttle-set(key, expireInSec) --> Throttle.set()
there is not built in support for this currently in meteor, but its on the roadmap https://trello.com/c/SYcbkS3q/18-dos-hardening-rate-limiting
in theory you could use some of the options here Throttling method calls to M requests in N seconds but you would have to roll your own solution