I have a spring-boot Kafka project which is a web-service exposing API to get Kafka message in response.
What i want is whenever i call the rest end point the Kafka should start searching from beginning it does it as i used earliest in auto-reset config but i have to start server again and again to make it listen to Kafka from starting.
#KafkaListener(topics = {"topic"})
public void storeMessagesMessages(ConsumerRecord record) {
if (record.value().toString().contains(uuid) {
this.messageToBeReturnedByApi = record.value()
}
}
Or i can say i want this listener part to be invoked only when i call web service endpoint
Your listener should extend AbstractConsumerSeekAware; you can then perform arbitrary seek operations. See https://docs.spring.io/spring-kafka/docs/2.6.2/reference/html/#seek
Related
I am working on a microservice based application in azure. My requirement is I had a service bus and I need to consume that service bus message in web api. Currently I implemented through azure functions, but my company asked to use api. Is it possible?, If possible please show me how to do it
You can create Background service to listen to message from service bus queue.
Below are few key points that needs to be noted:
Background task that runs on a timer.
Hosted service that activates a scoped service. The scoped service can use dependency injection (DI).
Queued background tasks that run sequentially.
App Settings:
1. {
2. "AppSettings": {
3. "QueueConnectionString": "<replace your RootManageSharedAccessKey here>",
4. "QueueName": "order-queue"
5. }
6. }
You can refer to c-sharpcorner blog for step by step process.
There is a simple way to get a simple message.
ServiceBusClient client = new ServiceBusClient("Endpoint=sb://yourservicesbusnamespace.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=Your_SharedAccess");
var receiver = client.CreateReceiver("Your Queue");
var message = await receiver.ReceiveMessagesAsync(1);
string ascii = Encoding.ASCII.GetString(message.FirstOrDefault().Body);
Console.WriteLine("Received Single Message: " + ascii);
await receiver.CompleteMessageAsync(message.FirstOrDefault());
I did some modifications from this post
https://ciaranodonnell.dev/posts/receiving-from-azure-servicebus/
Consider this extremely simple .NET Core 3.1 (and .NET 5) application with no special config or hosted services:
using System.Threading.Tasks;
using Microsoft.Extensions.Hosting;
internal class Program
{
public static async Task Main(string[] args)
{
var builder = Host.CreateDefaultBuilder(args);
builder.UseWindowsService();
var host = builder.Build();
var fireAndforget = Task.Run(async () => await host.RunAsync());
await Task.Delay(5000);
await host.StopAsync();
await Task.Delay(5000);
await host.RunAsync();
}
The first Run (sent as a background fire and forget task only for the purpose of this test) and Stop complete successfully. Upon calling Run a second time, I receive this exception:
System.AggregateException : 'Object name: 'EventLogInternal'.Cannot access a disposed object. Object name: 'EventLogInternal'.)'
If I do the same but using StartAsync instead of RunAsync (this time no need for a fireAndForget), I receive a System.OperationCanceledException upon called StartAsync the second time.
Am I right to deduce that .NET Generic Host aren't meant to be stopped and restarted?
Why do I need this?
My goal is to have a single application running as a Windows Service that would host two different .NET Generic Host. This is based on recommendation from here in order to have separate configuration and dependency injection rules and message queues.
One would stay active for all application lifetime (until the service is stopped in the Windows services) and would serve as a entry point to receive message events that would start/stop the other one which would be the main processing host with full services. This way the main services could be in "idle" state until they receive a message triggering their process, and another message could return them to idle state.
The host returned by CreateDefaultBuilder(...).Build() is meant to represent the whole application. From docs:
The main reason for including all of the app's interdependent resources in one object is lifetime management: control over app startup and graceful shutdown.
The default builder registers many services in singleton scope and when the host is stopped all of these services are disposed or switched to some "stopped" state. For example before calling StopAsync you can resolve IHostApplicationLifetime:
var appLifetime = host.Services.GetService<IHostApplicationLifetime>();
It has cancellation tokens representing application states. When you call StartAsync or RunAsync after stopping, all tokens still have IsCancellationRequested set to true. That's why the OperactionCancelledException is thrown in Host.StartAsync.
You can list other services during configuration:
For me it sounds like you just need some background jobs to process messages but I've never used NServiceBus so I don't know how it will work with something like Hangfire. You can also implement IHostedService and use it in the generic host builder.
I'm doing something like:
do
{
using IHost host = BuildHost();
await host.RunAsync();
} while (MainService.Restart);
with MainService constructor:
public MainService(IHostApplicationLifetime HostApplicationLifetime)
MainService.Restart is a static bool set by the MainService itself in response to some event which also calls HostApplicationLifetime.StopApplication().
In a spring-boot (version 2.1.4) application, there is requirement to migrate apache-kafka to spring-kafka.
Current kafka consumer does:
1) KafkaConsumer bean initialized at the time of application up
2) It has "0" topic partition set
3) poll the data using apache kafkaConsumer into ConsumerRecord
4) The application its own has Retry mechanism to wait and poll again till max_retry
The legacy code looks below:
while (!done.get()) {
ConsumerRecords<byte[], <byte[]> records = kafkaConsumer.poll(<MAX_VALUE>);
if (records.isEmpty()) {
retryCount++;
Thread.sleep(<some_time>);
} else {
// Process records;
}
if (retryCount > <max_retry_count>) {
done.set(true);
}
}
Tried below approaches:
1) Using spring kafka annotation (#KafkaListener), but it does not let us have control over polling.
2) Created "ConcurrentMessageListenerContainer" and setupMessageListener adds records into queue for polling. This gives us control on consumer.
I wanted to know, am I heading towards correct direction?
What would be better solution to achieve above requirement using spring-kafka?
It's not clear what you mean by "control on consumer". Creating a container is the same as using a #KafkaListener (a container is created under the covers).
Spring uses a "message-driven" approach.
You can set the idleEventInterval and the container will publish a ListenerContainerIdleEvent if no records are received during that time. You can listen for these events with an ApplicationListener bean, or an #EventListener method.
I am using 1.5.6.RELEASE to create REST Api that shall store user actions in MS SQLServer database. To make this user action service asynchronous, I have configured #EnableAsync in my main class.
#EnableAsync
public class WebApp extends SpringBootServletInitializer {
public static void main(String[] args) {
SpringApplication.run(WebApp.class, args);
}
}
#Async
public void saveAction(ActionInfo event) {
actionDataService.save(event);
}
I want to ensure that whatever happens(Server crashes, database goes down etc), user action posted by the client through Rest API service must be saved in database (may be through retry when database comes online or Application server starts again).
Proposed Solutions
Followings are the solutions:
1) Write AsyncUncaughtExceptionHandler to handle exceptions that retries again to save user action. But if Server crashes, the user action objects shall be lost.
2) Use JMS queues to store user action. If Server crashes, the JMS queue should not be lost. It must be able to retry it when server restores. (Just an idea, does not have much knowledge about queues)
Can you please suggest?
In RabbitMQ official website, they suggest to create a single callback queue per client to implement RPC pattern.
The code snippet for client(producer) looks like:
public RPCClient() throws Exception {
ConnectionFactory factory = new ConnectionFactory();
factory.setHost("localhost");
connection = factory.newConnection();
channel = connection.createChannel();
replyQueueName = channel.queueDeclare().getQueue();
consumer = new QueueingConsumer(channel);
channel.basicConsume(replyQueueName, true, consumer);
But I can't find any article to implement this by Spring Boot - AMQP, all the examples just send and receive sequentially (or say synchronously).
I wonder know how to achieve asynchronous message communication with just one request queue and reply queue.
p.s. I also tried AsyncRabbitTemplete but it didn't work.