Message Pact is non Http approach, see for more details:
https://docs.pact.io/getting_started/how_pact_works#non-http-testing-message-pact
ActiveSupport::Notification - is part of [Rails ActiveSupport][1], see for more details:
https://apidock.com/rails/ActiveSupport/Notifications
As I understand, ActiveSupport::Notification is using memory for a queue underneath, but not the external requests that are expected by Pact, so probably it can be done only via some other Message Queue, like Kafka, for example:
ActiveSupport::Notifications.subscribe("my_message") do |payload|
Kafka.produce(queue: 'test-queue', message: payload)
end
where Kafka.produce can be handled by Pact.
However, in this way, there is makes sense to remove ActiveSupport::Notifications and keep using only Kafka but this is the next step.
Related
I am working on Spring Kafka implementation and my use case is consume messages from Kafka topic as batch (using batch listener). when I consumer the list of messages, will iterate and call the REST endpoint for message enrichment. In case REST API fails for any runtime exception, I have implemented retry logic using spring retry. I want to stop the container, after the number of retries fails. So planning to use KafkaContainerStoppingErrorHandler to achieve this. Does the KafkaContainerStoppingErrorHandler commits the previous success messages - say if we receive 10 messages, and for message 1,2,3,4, enrichment call is success and for message 5 enrichment API call fails. so when we restart the container, will I get all 10 again or will I receive messages 5- 10?
or is there a way we can achieve above use case? I looked into all types of error handles of Spring kafka and need input on how to achieve above requirement.
You will get them all again.
You can use the DefaultErrorHandler (with a custom recoverer) and throw a BatchListenerFailedException to indicate which record in the batch failed.
The error handler will commit the offsets up to that record and call the recoverer with the failed record; in your custom recoverer you can stop the container (use the same logic as the container stopping error handler).
In versions before 2.8, this same functionality is provided by the RecoveringBatchErrorHandler.
I am thinking of a way to manage failed messages in Rebus.
In my second level retry strategy I want to save the message and exception details into the database so that I can later review the error details and decide whether to resend the message to the be reprocessed or ignore and delete.
In the handler I am capturing details as follows:
public async Task Handle(IFailed<StudentCreated> failedMessage)
{
//Logic to Defer Message with rebus_defer_count not shown
DictionarySerializer dictionarySerializer = new
DictionarySerializer();
ObjectSerializer objectSerializer = new ObjectSerializer();
string headers =
dictionarySerializer.SerializeToString(failedMessage.Headers);
string message =
objectSerializer.SerializeToString(failedMessage.Message);
Exception lastException= failedMessage.Exceptions.Last();
string exception = objectSerializer.SerializeToString(lastException);
//Logic to save the message and error details in the database not shown
}
This will enable me to save the message and error details into the database where I can create a dashboard to view the messages and resolve them as I wish rather than in the broker queue such as RabbitMQ.
Now my question is how can I return them to the handler where the error was raised using the information provided in the headers?
What is the best way to do it with REBUS provided I have all the details from the Failed Message as shown in my code snippet?
Regards
What you're trying to achieve will be much easier if you make a small change to your application. You see, Rebus already has a built-in service in place for handling failed messages called IErrorHandler.
You can register your own error handler like this:
Configure.With(...)
.(...)
.Options(o => o.Register<IErrorHandler>(c => new MyCustomErrorHandler()))
.Start();
thus replacing the default error handler (which btw. is PoisonQueueErrorHandler)
The error handler gets to handle the message in the form of the raw TransportMessage (i.e. simply headers and a byte[]) when all retries have failed, so this is the perfect place to save the message to your database.
If you then look here, you can see how Rebus' default error handler adds its own queue name as the rbs2-source-queue header, meaning that the message can later be sent back to that queue.
With this information, it should be fairly easy to write some code that inspects the message for its source queue and sends a RabbitMQ message to that queue.
This will only work if the re-delivery service has access to the RabbitMQ instance where all of your Rebus endpoints are running, of course. It's less straightforward, if you want to implement this in a general way: E.g. if you were using Fleet Manager, each Rebus instance would use a long-polling protocol to query the server for commands, which enables Fleet Manager to tell any Rebus instance to e.g. send a previously failed message to any queue it has access to.
Using MassTransit.RabbitMQ v5.3.2 and OpenTracing.Contrib.NetCore v0.5.0.
I'm able publish and consume events to RabbitMQ using MassTransit and I've got OpenTracing working with Jaeger, but I haven't managed to get my OpenTracing TraceIds propogated from my message publisher to my message consumer - The publisher and consumer traces have different TraceIds.
I've configured MassTransit with the following filter:
cfg.UseDiagnosticsActivity(new DiagnosticListener("test"));
I'm not actually sure what the listener name should be, hence "test". The documentation doesn't have an example for OpenTracing. Anyways, this adds a 'Publishing Message' span to the active trace on the publish side, and automatically sets up a 'Consuming Message' trace on the consumer side; however they're separate traces. How would I go about consolidating this into a single trace?
I could set a TraceId header using:
cfg.ConfigureSend(configurator => configurator.UseExecute(context => context.Headers.Set("TraceId", GlobalTracer.Instance.ActiveSpan.Context.TraceId)))
but then how would I configure my message consumer so that this is the root TraceId? Interested to see how I might do this, or if there's a different approach...
Thanks!
If anyone is interested; I ended up solving this by creating some publish and consume MassTransit middleware to do the trace propagation via trace injection and extraction respectively.
I've put the solution up on GitHub - https://github.com/yesmarket/MassTransit.OpenTracing
Still interested to hear if there's a better way of doing this...
My problematic seem to be simple, but I haven't find yet a way to solve it...
I have a legacy system which is working and a new system which will replace it. This is only rest webservices call, so I'm using simple bridge endpoint on http service.
To ensure the iso-functional run, I want to put them behind a camel route dispatching message to both system but returning only the response of the legacy one and log the response of both system to be sure there are running in same way...
I create this route :
from("servlet:proxy?matchOnUriPrefix=true")
.streamCaching()
.setHeader("CamelHttpMethod", header("CamelHttpMethod"))
.to("log:com.mylog?showAll=true&multiline=true&showStreams=true")
.multicast()
.to(urlServer1 + "?bridgeEndpoint=true")
.to(urlServer2 + "?bridgeEndpoint=true")
.to("log:com.mylog?showAll=true&multiline=true&showStreams=true")
;
It works to call each services and to log messages, but response are in a mess...
If the first server doesn't respond, the second is not call, if the second respond an error, only that error is send back to client...
Any Idea ?
You can check for some more details in multicast docs http://camel.apache.org/multicast.html
Default behaviour of multicast (your case) is:
parallelProcessing is false so routes are called one by one
To correctly implement your case you need probably:
add error handling for each external service call so exception will not stop correct processing
configure or implement some aggregator strategy and put it to the strategyRef so you can combine results from all calls to the single multicast result
I'll try provide as much information as possible:
No error message.
The instance stays in the "ready service instances".
The receive location has the same parameters (except URI, the three polling queries, user account/pw and receive pipeline) as another receive location that points to another database/table which works.
The pipeline is waiting for the correct schema.
The port surface and receive location are both waiting for the correct schema.
In my test example, there are only 10 lines being returned.
The message, which contains those 10 lines, validates against the schema.
I tried to let the instance alone to no avail - 30+ minutes - and no change in its condition.
I had also tried suspending and then resuming it which then places the instance in the "dehydrated orchestrations" list. Again, with no error message.
I'm able to get the message by looking at the body of the message that's in the "ready to run" service. (This is the message that validates versus the schema I use in Visual Studio.)
How might something like this arise?
Stupid question, but I have to ask... Is the corresponding host instance running?