I am using DefaultMessageListenerContainer as below :
private static final AnnotationConfigApplicationContext context = new AnnotationConfigApplicationContext(MessageConsumer.class);
public static final DefaultMessageListenerContainer container = context.getBean(DefaultMessageListenerContainer.class);
For a given queue that my listener listens to , the first time my program runs it starts the listener .... and this creates a Consumer for the queue that I can see in the Active MQ Console.
The problem I have is every time I checkin some new code, there is another new Consumer created for the queue and the old one is still hanging on creating some Out of Memory issue.
What am I doing wrong here? How do I make sure there is only 1 consumer and the old consumer is killed with every new code checkin? Hope I explained the issue clearly.
Related
An app I have been working on has started causing issues in our staging and production environment that are seemingly due to Kafka listeners no longer reading anything from their assigned topics after a few hours from the app starting.
The app is running in a cloud foundry environment and it has 13 #KafkaListener, reading from multiple topics based on their given pattern. The amount of topics is equal (each user on the app creates its own topic for each of the 13 listeners using the pattern). Topics have 3 partitions. Auto-scaling is also used, with a minimum of 2 instances of the app running at the same time. One of the topics is under heavier load than the others, receiving between 1 to 200 messages each second. The processing time for each message is short, as we receive batches and the processing part only proceeds to write the batch to a DB.
The current issue is, as stated, that it works for a while after starting and then suddenly the listeners are no longer picking up messages. With no apparent error or warning in the logs. A temporary endpoint was created where KafkaListenerEndpointRegistry is used to look at the Listener Containers, and all of them seem to be running and have proper partitions assigned. Doing a .stop() and .start() on the containers leads to one additional batch of messages being processed, and then nothing else.
The following are the configs used:
#Bean
public ConsumerFactory<String, String> consumerFactory(){
return new DefaultKafkaConsumerFactory<>(kafkaConfig.getConfiguration());
}
#Bean
public ConcurrentKafkaListenerContainerFactory<String, String> kafkaListenerContainerFactory(){
ConcurrentKafkaListenerContainerFactory<String, String> factory = new ConcurrentKafkaListenerContainerFactory<>();
factory.setConsumerFactory(consumerFactory());
factory.setBatchListener(true);
factory.setConcurrency(3);
factory.getContainerProperties().setPollTimeout(5000);
factory.getContainerProperties().setAckMode(ContainerProperties.AckMode.MANUAL_IMMEDIATE);
}
The kafkaConfig sets the following settings:
PARTITION_ASSIGNMENT_STRATEGY_CONFIG: RoundRobinAssignor
MAX_POLL_INTERVAL_MS_CONFIG: 60000
MAX_POLL_RECORDS_CONFIG: 10
MAX_PARTITION_FETCH_BYTES_CONFIG: Integer.MAX_VALUE
ENABLE_AUTO_COMMIT_CONFIG: false
METADATA_MAX_AGE_CONFIG: 15000
REQUEST_TIMEOUT_MS_CONFIG: 30000
HEARTBEAT_INTERVAL_MS_CONFIG: 15000
SESSION_TIMEOUT_MS_CONFIG: 60000
Additionally, each listener is in its own class and has the listen method as follows:
#KafkaListener(id="<patternName>-container", topicPattern = "<patternName>.*", groupId = "<patternName>Group")
public void listen(#Payload List<String> payloads,
#Header(KafkaHeaders.RECEIVED_TOPIC) String topics,
Acknowledgement acknowledgement){
//processPayload...
acknowledgement.acknowledge();
}
The spring-kakfa version is 2.7.4.
Is there an issue with this config that could solve the issue? I have recently tried multiple changes with no success, changing these config settings around, moving the #KafkaListener annotation at class level, restarting the Listener Containers when they stop reading, and even having all the processing on the messages be done asynchronously and acknowledging the messages the moment they are picked up by the listener method. There were no errors or warning logs, and I wasn't able to see anything helpful on debug logging due to the amount of messages sent each second. We also have another app running the same settings in the same environments, but only 3 listeners (different topic patterns), where this issue does not occur. It is under a similar load, as the messages received by those 3 listeners are being output to the topic causing the large load on the app with the problem.
I would very much appreciate any help or pointers to what else I can do, since this issue is blocking us heavily in our production. Let me know if I missed something that could help.
Thank you.
Most problems like this are due to the listener thread being stuck in user code somplace; take a thread dump when this happens to see what the threads are doing.
I've been given an old WinForms app to update and improve. I am trying to add DI using SimpleInjector. I'm used to .Net MVC but this is my first time working with WinForms.
The application uses a lot of BackGround workers. My understanding is that this is specific to WinForms and each Background worker creates a new thread.
I think my problem is that when I want to save data to the DB using EF6 the SaveChanges method isn't able to save because of the multiple threads.
My SimpleInjector container is set up as follows
_container = new Container();
_container.Options.DefaultScopedLifestyle = new ThreadScopedLifestyle();
// Register DbContext
_container.Register<DbContext, MyDbContext>(Lifestyle.Scoped);
When I call SaveChanges on my dbContext I get a result of 0, which indicates that no records were saved to the database. In my debugger I get an error saying that the DbContext has been disposed. If this is happening before the save it explains why a 0 is returned from SaveChanges. Unfortunately the previous developer caught every exception so the application is trying to handle every error and this is making troubleshooting difficult and leading to unexpected behavior.
I am expecting a new DbContext to be created for each thread and that SaveChanges will save the changes made in each thread, so that what happens in one context won't affect other DbContexts in other threads.
When I am reading from the database I manually create a new DbContext in each method. Is it possible that when the using block has completed it is disposing the DbContext ?
using (var newDbContext = new MyDbContext())
{
return newDbContext.Set<TableA>().First(x => x.Id == id);
}
I'm hoping that if I have SimpleInjector configured correctly I won't need to do this either.
I'm a little lost at this stage and think I might not be understanding the documentation correctly, any advise would be greatly appreciated. Thanks in advance.
I am expecting a new DbContext to be created for each thread
This is not how TheadScopedLifestyle works. With the ThreadScopedLifestyle, there will be one instance of your registration within the context of an explicitly started Scope and this scope is thread-specific. This means that one thread can have many instances of that service, since a thread can live for a long time, while a Scope will typically only live for a short amount of time.
A typical use for ThreadScopedLifestyle is the following:
void MethodThatRunsInABackGroundThread()
{
using (ThreadScopedLifestyle.BeginScope(container))
{
var service = container.GetInstance<ISomeService>();
service.DoWork();
}
}
When you resolve Scoped instances (or some object graph that contains scoped instances) outside an active scope, Simple Injector will throw an exception.
I need to use a Service which starts a Task more than once (= the same Service must run several parallelised Task). I read the JavaFX documentation, and they seem to say that a Service can run only one Task at once.
So if I call twice start with my Service object, the first Task returned by its createTask method would be stopped, as if I used restart after the first start.
However, that's not clear. As I told you, the documentation seems to tell that.
Indeed :
A Service creates and manages a Task that performs the work on the background thread.
Note that I could think they also say that a Service can have several Task started at the same time. Indeed :
a Service can be constructed declaratively and restarted on demand.
My question is : if I use N start in a row, will N Tasks be created AND KEEP EACH RUNNING ?
"If I use N start in a row, will N Tasks be created AND KEEP EACH RUNNING ?
In short, no.
"If I call start twice with my Service object..."
From the Javadocs:
public void start()
Starts this Service. The Service must be in the READY state to succeed in this call.
So if you call start() a second time without previously calling reset(), you will just get an exception. You can only call reset() if the Service is not in a RUNNING or SCHEDULED state. You can call restart(), which will have the effect of first canceling any current task, and then restarting the service. (This is what is meant by the documentation that says the "service can be restarted on demand".)
The net result of this is that a service cannot have two currently running tasks at the same time, since there is no sequence of calls that can get to that situation without throwing an IllegalStateException.
If you want multiple tasks running at once, simply create them yourself and submit them to an executor (or run each in its own thread, but an executor is preferred):
private final Executor exec = Executors.newCachedThreadPool(runnable -> {
Thread t = new Thread(runnable);
t.setDaemon(true);
return t ;
});
// ...
private void launchTask() {
Task<MyDataType> task = new Task<MyDataType>(){
#Override
protected Something call() {
// do work...
return new MyDataType(...);
}
};
task.setOnSucceeded(e -> { /* update UI ... */ });
task.setOnFailed(e -> { /* handle error ... */ });
exec.execute(task);
}
We have a Java class that listens to a database (Oracle) queue table and process it if there are records placed in that queue. It worked normally in UAT and development environments. Upon deployment in production, there are times when it cannot read a record from the queue. When a record is inserted, it cannot detect it and the records remain in the queue. This seldom happens but it happens. If I would give statistic, out of 30 records queued in a day, about 8 don't make it. We would need to restart the whole app for it to be able to read the records.
Here is a code snippet of my class..
public class SomeListener implements MessageListener{
public void onMessage(Message msg){
InputStream input = null;
try {
TextMessage txtMsg = (TextMessage) msg;
String text = txtMsg.getText();
input = new ByteArrayInputStream(text.getBytes());
} catch (Exception e1) {
// TODO Auto-generated catch block
logger.error("Parsing from the queue.... failed",e1);
e1.printStackTrace();
}
//process text message
}
}
Weird thing we cant find any traces of exceptions from the logs.
Can anyone help? by the way we set the receiveTimeout to 10 secs
We would need to restart the whole app for it to be able to read the records.
The most common reason for this is the listener thread is "stuck" in user code (//process text message). You can take a thread dump with jstack or jvisualvm or similar to see what the thread is doing.
Another possibility (with low volume apps like this) is the network (most likely a router someplace in the network) silently closes an idle socket because it has not been used for some time. If the container (actually the broker's JMS client library) doesn't know the socket is dead, it will never receive any more messages.
The solution to the first is to fix the code; the solution to the second is to enable some kind of heartbeat or keepalives on the connection so that the network/router does not close the socket when it has no "real" traffic on it.
You would need to consult your broker's documentation about configuring heartbeats/keepalives.
I am working on an asp.net mvc 5 web application , deployed inside IIS-8, and i have a method inside my application to perform a long running task which mainly scans our network for servers & VMs and update our database with the scan results. method execution might last between 30-40 minutes to complete on production environment. and i am using a schedule tool named Hangfire which will call this method 2 times a day.
here is the job definition inside the startup.cs file, which will call the method at 8:01 am & 8:01 pm:-
public void Configuration(IAppBuilder app)
{
var options = new SqlServerStorageOptions
{
PrepareSchemaIfNecessary = false
};
GlobalConfiguration.Configuration.UseSqlServerStorage("scanservice",options);
RecurringJob.AddOrUpdate(() => ss.Scan(), "01 8,20 ***");
}
and here is the method which is being called twice a day by the schedule tool:-
public void Scan()
{
Service ss = new Service();
ss.NetworkScan().Wait();
}
Finally the method which do the real scan is (i only provide a high level description of what the method will do):-
public async Task<ScanResult> NetworkScan()
{
// retrieve the server info from the DB
// loop over all servers & then execute some power shell commands to scan the network & retrieve the info for each server one by one...
// after the shell command completed for each server, i will update the related server info inside the DB
currently i did some tests on our test environment and every thing worked well ,, where the scan took around 25 seconds to scan 2 test servers.but now we are planning to move the application to production and we have around 120++ servers to scan. so i estimate the method execution to take around 30 -40 minutes to complete on the production environment. so my question is how i can make sure that this execution will never expire , and the ScanNetwork() method will complete till the end?
Instead of worrying about your task timing out, perhaps you could start a new task for each server. In this way each task will be very short lived, and any exceptions caused by scanning a single server will not effect all the others. Additionally, if your application is restarted in IIS any scans which were not yet completed will be resumed. With all scans happening in one sequential task this is not possible. You will likely also see the total time to complete a scan of your entire network plummet, as the majority of time would likely be spent waiting on remote servers.
public void Scan()
{
Service ss = new Service();
foreach (var server in ss.GetServers())
{
BackgroundJob.Enqueue<Service>(s => s.ServerScan(server));
}
}
Now your scheduled task will simply enqueue one new task for each server.