I am working on Spring Kafka implementation and my use case is consume messages from Kafka topic as batch (using batch listener). when I consumer the list of messages, will iterate and call the REST endpoint for message enrichment. In case REST API fails for any runtime exception, I have implemented retry logic using spring retry. I want to stop the container, after the number of retries fails. So planning to use KafkaContainerStoppingErrorHandler to achieve this. Does the KafkaContainerStoppingErrorHandler commits the previous success messages - say if we receive 10 messages, and for message 1,2,3,4, enrichment call is success and for message 5 enrichment API call fails. so when we restart the container, will I get all 10 again or will I receive messages 5- 10?
or is there a way we can achieve above use case? I looked into all types of error handles of Spring kafka and need input on how to achieve above requirement.
You will get them all again.
You can use the DefaultErrorHandler (with a custom recoverer) and throw a BatchListenerFailedException to indicate which record in the batch failed.
The error handler will commit the offsets up to that record and call the recoverer with the failed record; in your custom recoverer you can stop the container (use the same logic as the container stopping error handler).
In versions before 2.8, this same functionality is provided by the RecoveringBatchErrorHandler.
Related
Message Pact is non Http approach, see for more details:
https://docs.pact.io/getting_started/how_pact_works#non-http-testing-message-pact
ActiveSupport::Notification - is part of [Rails ActiveSupport][1], see for more details:
https://apidock.com/rails/ActiveSupport/Notifications
As I understand, ActiveSupport::Notification is using memory for a queue underneath, but not the external requests that are expected by Pact, so probably it can be done only via some other Message Queue, like Kafka, for example:
ActiveSupport::Notifications.subscribe("my_message") do |payload|
Kafka.produce(queue: 'test-queue', message: payload)
end
where Kafka.produce can be handled by Pact.
However, in this way, there is makes sense to remove ActiveSupport::Notifications and keep using only Kafka but this is the next step.
while upgrading our spring kafka to 2.7.8, we are getting error in the setAckOnError(false) method as it has been deprecated now. Is there any way now to set the acknowledgement for the errors to false? Any other methods that can help me to set it as false for errors acknowledgement?
P.S: I am new to Kafka, any help appreciated!
That property was found to have a (very small) timing hole in that a record could be ack'd before the error handler handles it; if the app dies at that time, the record could be "lost".
It was replaced by a new feature in the error handlers ackAfterHandle; which is true by default - i.e. the record's offset is only committed if the error handler "handles" the error.
Records are now never ack'd if the error handler (such as the SeekToCurrentErrorHandler) throws an exception (after it repositions the partitions).
There is no extra configuration needed any more.
See Spring Kafka AckOnError for more details.
I am getting sometimes this error when calling load for DynamoDBMapper:
java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:735)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1593)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:82)
at com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.getToken(InstanceMetadataServiceResourceFetcher.java:91)
at com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.readResource(InstanceMetadataServiceResourceFetcher.java:69)
at com.amazonaws.internal.EC2ResourceFetcher.readResource(EC2ResourceFetcher.java:66)
at com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsEndpoint(InstanceMetadataServiceCredentialsFetcher.java:58)
at com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsResponse(InstanceMetadataServiceCredentialsFetcher.java:46)
at com.amazonaws.auth.BaseCredentialsFetcher.fetchCredentials(BaseCredentialsFetcher.java:112)
at com.amazonaws.auth.BaseCredentialsFetcher.getCredentials(BaseCredentialsFetcher.java:68)
at com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:166)
at com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper.getCredentials(EC2ContainerCredentialsProviderWrapper.java:75)
at com.amazonaws.auth.AWSCredentialsProviderChain.getCredentials(AWSCredentialsProviderChain.java:117)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1251)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:827)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:777)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:764)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:738)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:698)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:680)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:544)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:524)
at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.doInvoke(AmazonDynamoDBClient.java:5110)
at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:5077)
at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.executeGetItem(AmazonDynamoDBClient.java:2197)
at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.getItem(AmazonDynamoDBClient.java:2163)
at com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBMapper.load(DynamoDBMapper.java:431)
at com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBMapper.load(DynamoDBMapper.java:448)
at com.amazonaws.services.dynamodbv2.datamodeling.AbstractDynamoDBMapper.load(AbstractDynamoDBMapper.java:80)
I have 2 timeouts to PUT /latest/api/token, then I get a response. I am not sure what is wrong exactly or why do I have this behavior sometimes, but this leads to latency in my application.
Do I need to modify something in the settings? Is it related to DynamoMapper? Should I use low level Dynamo API?
These issues can occur when:
You call a remote API that takes too long to respond or that is unreachable.
Your API call doesn't get a response within the socket timeout.
Your API call doesn't get a response within the timeout period of your Lambda function.
If you make an API call using an AWS SDK and the call fails, the SDK automatically retries the call https://aws.amazon.com/premiumsupport/knowledge-center/lambda-function-retry-timeout-sdk/. How long and how many times the SDK retries is determined by settings that vary among each SDK. Here are the default values of these settings:
see the SDK client configuration documentation: https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/ClientConfiguration.html
I am thinking of a way to manage failed messages in Rebus.
In my second level retry strategy I want to save the message and exception details into the database so that I can later review the error details and decide whether to resend the message to the be reprocessed or ignore and delete.
In the handler I am capturing details as follows:
public async Task Handle(IFailed<StudentCreated> failedMessage)
{
//Logic to Defer Message with rebus_defer_count not shown
DictionarySerializer dictionarySerializer = new
DictionarySerializer();
ObjectSerializer objectSerializer = new ObjectSerializer();
string headers =
dictionarySerializer.SerializeToString(failedMessage.Headers);
string message =
objectSerializer.SerializeToString(failedMessage.Message);
Exception lastException= failedMessage.Exceptions.Last();
string exception = objectSerializer.SerializeToString(lastException);
//Logic to save the message and error details in the database not shown
}
This will enable me to save the message and error details into the database where I can create a dashboard to view the messages and resolve them as I wish rather than in the broker queue such as RabbitMQ.
Now my question is how can I return them to the handler where the error was raised using the information provided in the headers?
What is the best way to do it with REBUS provided I have all the details from the Failed Message as shown in my code snippet?
Regards
What you're trying to achieve will be much easier if you make a small change to your application. You see, Rebus already has a built-in service in place for handling failed messages called IErrorHandler.
You can register your own error handler like this:
Configure.With(...)
.(...)
.Options(o => o.Register<IErrorHandler>(c => new MyCustomErrorHandler()))
.Start();
thus replacing the default error handler (which btw. is PoisonQueueErrorHandler)
The error handler gets to handle the message in the form of the raw TransportMessage (i.e. simply headers and a byte[]) when all retries have failed, so this is the perfect place to save the message to your database.
If you then look here, you can see how Rebus' default error handler adds its own queue name as the rbs2-source-queue header, meaning that the message can later be sent back to that queue.
With this information, it should be fairly easy to write some code that inspects the message for its source queue and sends a RabbitMQ message to that queue.
This will only work if the re-delivery service has access to the RabbitMQ instance where all of your Rebus endpoints are running, of course. It's less straightforward, if you want to implement this in a general way: E.g. if you were using Fleet Manager, each Rebus instance would use a long-polling protocol to query the server for commands, which enables Fleet Manager to tell any Rebus instance to e.g. send a previously failed message to any queue it has access to.
Say I were to create a promoted property on an HL7 v2 schema. I want to use this promoted property in order to correlate two messages in a parallel convoy. I did a dummy application without using HL7 schemas and all worked just as expected. To sum up what I did, see this tutorial.
It's actually quite simple, which is why I'm wondering there is a catch when it comes to the HL7 schemas.
I created the PropertySchema, Promoted the field I'm after in the HL7 body schema, everything compiled and deployed fine, I ran a query for the subscriptions on my receive ports and they look OK
http://schemas.microsoft.com/BizTalk/2003/system-properties.ReceivePortID == {D2F99A76-E28A-4B3E-AC52-F4E2F92453C3}
And
http://schemas.microsoft.com/BizTalk/2003/system-properties.MessageType == http://microsoft.com/HealthCare/HL7/2X#ORU_ALL_25_GLO_DEF
And
https://myNS.ECGCorrelationPropertySchema.CommonAccessionID Exists
However, I get a
The published message could not be routed because no subscribers were found. This error occurs if the subscribing orchestration or send port has not been enlisted, or if some of the message properties necessary for subscription evaluation have not been promoted.
and looking in the Biztalk Group admin console, when I view the suspended messages, I don't see that the property has been promoted
Edit to add message context
Error Report Context
I have checked and double-checked, the schemas are there, there are duplicate schemas the Event log even shows A message was received of type http://microsoft.com/HealthCare/HL7/2X#ORU_ALL_25_GLO_DEF when I receive a message.
So now, I guess I have 2 questions:
If the promotions were happening successfully, would they show in the Message --> Context dialog (they did in my non-HL7 dummy app)
If they aren't supposed to show, is there a setting/property/anything somewhere I missed that is unique to using HL7v2 schemas?
Short answers:
1. Yes, provided this message came directly from the Receive Pipeline
2. No, once emitted from the disassembler, there is nothing special about an HL7 Message.
Make sure you're looking at the actual Suspended Message and not the Error Report. The best way to check this is to Stop, don't Unenlist, the Orchestration. The Message will then suspend Resumable.