The manual describes well how to consume simple messages using java.util.function.Consumer<T> and make a through processing in a single processor method with java.util.function.Function<T, T>, but it isn't clear the way to produce a message triggered by producer service itself. For instance, produce the message containing the value received by RestController.
EDIT
There is a solution, but it relies on a deprecated reactor.core.publisher.EmitterProcessor
public class MyProcessingService {
private final EmitterProcessor<Message<MyTask>> processor = EmitterProcessor.create(); // <- DEPRECATED
#Bean
public Consumer<Message<MyResult>> myResult() {
return input -> {
log.info("Received: {}", input);
};
}
#Bean
public Supplier<Flux<Message<MyTask>>> myTask() {
return () -> processor;
}
public void sendMyTask(String payload) {
processor.onNext(MessageBuilder.withPayload(MyTask.builder().payload(payload).build()).build());
}
}
Is there any recommended solution here? Thanks in advance.
Related
after using massTransit (8.0.8) I got following error :
'RoutingSlipCompleted' does not contain a definition for 'GetVariable'
and the best extension method overload
'RoutingSlipEventExtensions.GetVariable(ConsumeContext,
string, Guid)' requires a receiver of type
'ConsumeContext'
here is my code:
using MassTransit;
using MassTransit.Courier.Contracts;
using MassTransit.Courier;
public class CheckInventoriesConsumer: IConsumer<ICheckInventoryRequest>
, IConsumer<RoutingSlipCompleted>
, IConsumer<RoutingSlipFaulted>
{
private readonly IEndpointNameFormatter _formatter;
public CheckInventoriesConsumer(IEndpointNameFormatter formatter)
{
_formatter = formatter;
}
public async Task Consume(ConsumeContext<ICheckInventoryRequest> context)
{
var routingSlip = CreateRoutingSlip(context);
await context.Execute(routingSlip);
}
private RoutingSlip CreateRoutingSlip(ConsumeContext<ICheckInventoryRequest> context)
{ // lot of code here
}
public async Task Consume(ConsumeContext<RoutingSlipCompleted> context)
{
// error is here
context.Message.GetVariable<Guid>(nameof(ConsumeContext.RequestId));
throw new NotImplementedException();
}
}
It is not going to find GetVariable method from MassTransit.Courier and I encounter with this error.
As you've already found based upon your comments:
context.GetVariable<Guid>(nameof(ConsumeContext.RequestId));
Is the right solution.
MassTransit Version 8 has more extensive serialization support, and the SerializationContext (from ConsumeContext) is needed to properly deserialize the variable from the routing slip event.
I have a ServiceBusClient class that creates a QueueClient which is used to listen for messages on a bus. I have looked at the following articles to set this up:
Background tasks (Microsoft)
Hosted services (Microsoft)
Async and Await
My ServiceBusClient class that handles the QueueClient looks like this:
public class ServiceBusClient : IServiceBusClient
{
public ServiceBusClient(IEventService eventService, ServiceBusClientOptions options)
{
...
queueClient = new QueueClient(options.ConnectionString, options.QueueName);
}
public void Run()
{
RegisterOnMessageHandler();
}
private void RegisterOnMessageHandler()
{
...
queueClient.RegisterMessageHandler(ProcessMessagesAsync, messageHandlerOptions);
}
private async Task ProcessMessagesAsync(Message message, CancellationToken token)
{
var eventMessage = EventMessage.FromMessage(message);
await eventService.Write(eventMessage);
if (!token.IsCancellationRequested)
{
await queueClient.CompleteAsync(message.SystemProperties.LockToken);
}
}
private Task ExceptionReceivedHandler(ExceptionReceivedEventArgs exceptionReceivedEventArgs)
{
// log errors
...
return Task.CompletedTask;
}
}
I was hoping to launch from an IHostedService or even by extending the BackgroundService. In the examples I find, work is constantly being executed in a while loop which does not fit my scenario since I am only trying to run a single command.
So I created a super simple implementation like this:
protected override async Task ExecuteAsync(CancellationToken cancellationToken)
{
serviceBusClient.Run();
while (!cancellationToken.IsCancellationRequested)
{
// empty loop to keep running for lifetime of pod
}
}
If removing the async I obviously need to return something. I tried Task.CompletedTask but that required me to change the return type to Task<Task>.
If I have the async in place, I will need to await something, but I am not sure what.
This does not feel right. I would assume I would need to change something in the ServiceBusClient, but I am unsure what, since the ProcessMessagesAsync is async and does the heavy lifting in the background from my understanding.
All I want is for my web app to start listening for messages until it dies. How can I do that?
I gave up on using BackgroundService and implemented IHostedService instead.
public class MessageListenerService : IHostedService
{
private readonly IServiceBusClient client;
private readonly ITelemetryClient applicationInsights;
public MessageListenerService(IServiceProvider serviceProvider)
{
client = serviceProvider.GetService<IServiceBusClient>();
applicationInsights = serviceProvider.GetService<ITelemetryClient>();
}
public Task StartAsync(CancellationToken cancellationToken)
{
applicationInsights.TrackTrace(new TraceTelemetry("MessageListenerService is starting"));
client.Run();
return Task.CompletedTask;
}
public Task StopAsync(CancellationToken cancellationToken)
{
applicationInsights.TrackTrace(new TraceTelemetry("MessageListenerService is stopping"));
return client.Stop();
}
}
If you find issues with this code please let me know in the comments and I'll update as appropriate.
In the end we created a console app for it anyway.
Given the following unit test, which uses the Vertx Unit testing framework:
#RunWith(VertxUnitRunner.class)
public class VertxUnitTest {
private Vertx vertx;
#Rule
public RunTestOnContext rule = new RunTestOnContext(new VertxOptions().setClustered(false)
.setClusterManager(new HazelcastClusterManager()).setMaxEventLoopExecuteTime(2000000000000L)
.setMaxWorkerExecuteTime(60000000000000L).setBlockedThreadCheckInterval(1000000)
.setEventBusOptions(new EventBusOptions().setClustered(false).setIdleTimeout(0)));
#Before
public void setup() throws Exception {
io.vertx.core.Vertx v = rule.vertx();
vertx = Vertx.newInstance(v);
}
private class MyVerticle extends AbstractVerticle {}
#Test
public void runFlow_correctMessage_stepsCalledInCorrectOrder(TestContext context) {
Async async = context.async();
vertx.getDelegate().deployVerticle(new MyVerticle(), new DeploymentOptions().setWorker(true), c -> {
c.cause();
vertx.eventBus().<Object>send("", new JsonObject(), new DeliveryOptions(), rpl -> {
async.complete();
fail();
});
});
}
}
the call to fail() is throwing an exception to the console, but it is not actually failing the test itself, which finishes successfully and is green.
The same is true when working with Mockito. I can successfully verify the behavior of the verticle and its dependencies using mocks, but even when the Mockito assertions fail, the test itself will still pass. Calling fail on the vertx TestContext object - context.fail() - will also not fail the test.
The core issue is this: any call to fail() after async.complete() will not fail the test, only the console will show the error. But without the call to async.complete(), the code in the verticle (called upon consuming from the event bus), will not have run before the test assertions are called.
Without the call to async.complete(), the test will it appears never complete.
What is the correct approach to this?
Thanks
the correct approach is to call the TestContext.fail() method, like so:
#Test
public void runFlow_correctMessage_stepsCalledInCorrectOrder(TestContext context) {
Async async = context.async();
vertx.getDelegate().deployVerticle(new MyVerticle(), new DeploymentOptions().setWorker(true), c -> {
if(c.succeeded()) {
vertx.eventBus().<Object>send("", new JsonObject(), new DeliveryOptions(), rpl -> {
if(rpl.succeeded()) {
// make assertions based on reply contents, and then...
async.complete();
} else {
context.fail(rpl.cause());
}
});
} else {
context.fail(c.cause());
}
});
}
Here is my setup:
ConsumerSeekAware implementation:
public class ReplayJobKafkaConsumer implements ConsumerSeekAware, AcknowledgingMessageListener<String, String> {
#Override
public void onPartitionsAssigned(Map<TopicPartition, Long> map, ConsumerSeekCallback consumerSeekCallback) {
}
#Override
public void onIdleContainer(Map<TopicPartition, Long> map, ConsumerSeekCallback consumerSeekCallback) {
}
private static final ThreadLocal<ConsumerSeekCallback> seekCallBack = new ThreadLocal<>();
private static ConsumerSeekCallback consumerSeekCallback;;
#Override
public void registerSeekCallback(ConsumerSeekCallback callback) {
this.seekCallBack.set(callback);
consumerSeekCallback = callback;
}
public void onMessage(final ConsumerRecord<String, String> data, final Acknowledgment acknowledgment) {
}
public static ThreadLocal<ConsumerSeekCallback> getSeekCallback(){
return seekCallBack;
}
public static ConsumerSeekCallback getAnotherSeekCallback(){
return consumerSeekCallback;
}
}
My Spring Boot application approximates to:
#SpringBootApplication
public class ReplayJobApplication{
...
public void run(final String... args){
context = SpringApplication.run(ReplayJobApplication.class, args);
ReplayJobKafkaConsumer.getAnotherSeekCallback().seek("top", 0, 23);
}
...}
The above setup works. Now I can run this application using
java -jar -Dstart.offset=0....
But it only works if the seekcallback variable is not a ThreadLocal. I need this to be accessible at the Spring Boot application as that is how I intend running this consumer. TEMP-TOPIC's other consumers can still be processing, but I intend to run this consumer on a need basis with a start and end offset. While the command line parameters can be read in the consumer, the concerns I have are
callback variable is static (I cannot possibly create an instance of ReplayJobKafkaConsumer
it is a plain variable and not a ThreadLocal
Though the life time of this container is only going to be from start to end, I wonder if this setup is flawed and need some confirmation that this implementation is OK.
You appear to have some fundamental misunderstanding of what's going on.
The ThreadLocal is needed because the Kafka consumer object is not thread-safe. If you store the callback in a ThreadLocal, you can perform arbitrary seek operations at runtime - either from the onMessage method, or by listening for an ListenerContainerIdleEvent when there are no messages.
You can't perform arbitrary seeks ReplayJobKafkaConsumer.getAnotherSeekCallback().seek("top", 0, 23); from another thread.
You can't perform arbitrary seeks before partitions have been assigned.
So, as I have been telling you in other answers/comments, you must do the seek when the partition(s) are assigned.
#Override
public void onPartitionsAssigned(Map<TopicPartition, Long> map, ConsumerSeekCallback consumerSeekCallback) {
// Do the seeks here using the `consumerSeekCallback` parameter.
}
With modern versions of spring-kafka, you don't need to use ConsumerSeekAware unless you want to perform arbitrary seeks at runtime (after the initial seek). You can use a ConsumerAwareRebalanceListener instead.
Currently I'm beginnging with Spring + reactive programming. My aim is to return a result in a REST-endpoint from a long running method (polling on a database). I'm stuck on the api. I simply don't know how to return the result as Mono in my FooService.findFoo method.
#RestController
public class FooController {
#Autowired
private FooService fooService;
#GetMapping("/foo/{id}")
private Mono<ResponseEntity<Foo> findById(#PathVariable String id) {
return fooService.findFoo(id).map(foo -> ResponseEntity.ok(foo)) //
.defaultIfEmpty(ResponseEntity.notFound().build())
}
...
}
#Service
public class FooService {
public Mono<Foo> findFoo(String id) {
// this is the part where I'm stuck
// I want to return the results of the pollOnDatabase-method
}
private Foo pollOnDatabase(String id) {
// polling on database for a while
}
}
Use the Mono.fromSupplier method! :)
#Service
public class FooService {
public Mono<Foo> findFoo(String id) {
return Mono
.fromSupplier(() -> pollOnDatabase(id))
.subscribeOn(Schedulers.boundedElastic());
}
private Foo pollOnDatabase(String id) {
// polling on database for a while
}
}
With this method we return a Mono value ASAP, constant time with a supplier which will be evaluated on demand by the caller's subscribe. This is the non blocking way to call a long-running-blocking method.
BE AWARE that without subscription on boundedElastic the blocking pollOnDatabase method will block the original thread, which leads to thread starvation. You can find different schedules for every kind of tasks here.
DO NOT use Mono.just with long-running calculations as it will run the calculation before returning the Mono instance, thereby blocking the given thread.
+1: Watch this video to learn to avoid "reactor meltdown". Use some lib to detect blocking calls from non-blocking threads.
It's pretty simple. You could just do
#Service
public class FooService {
public Mono<Foo> findFoo(String id) {
return Mono.just(pollOnDatabase(id));
}
private Foo pollOnDatabase(String id) {
// polling on database for a while
}
}