I am using spring data 1.5.8 and manual ack mode of kafka. There is only a simple ErrorHandler interface, the handle method has parameter exception and consumerRecord, but how can I set the offset to next if exception threw when json can not be deserialized? I need omit the message or else the consumer will stuck there
spring data 2.0 introduce ConsumerAwareErrorHandler, the only way is update spring version?
Yes. You need to upgrade. The current version is 2 4.0 or 2.3.4.
Related
I am working on Spring Kafka implementation and my use case is consume messages from Kafka topic as batch (using batch listener). when I consumer the list of messages, will iterate and call the REST endpoint for message enrichment. In case REST API fails for any runtime exception, I have implemented retry logic using spring retry. I want to stop the container, after the number of retries fails. So planning to use KafkaContainerStoppingErrorHandler to achieve this. Does the KafkaContainerStoppingErrorHandler commits the previous success messages - say if we receive 10 messages, and for message 1,2,3,4, enrichment call is success and for message 5 enrichment API call fails. so when we restart the container, will I get all 10 again or will I receive messages 5- 10?
or is there a way we can achieve above use case? I looked into all types of error handles of Spring kafka and need input on how to achieve above requirement.
You will get them all again.
You can use the DefaultErrorHandler (with a custom recoverer) and throw a BatchListenerFailedException to indicate which record in the batch failed.
The error handler will commit the offsets up to that record and call the recoverer with the failed record; in your custom recoverer you can stop the container (use the same logic as the container stopping error handler).
In versions before 2.8, this same functionality is provided by the RecoveringBatchErrorHandler.
while upgrading our spring kafka to 2.7.8, we are getting error in the setAckOnError(false) method as it has been deprecated now. Is there any way now to set the acknowledgement for the errors to false? Any other methods that can help me to set it as false for errors acknowledgement?
P.S: I am new to Kafka, any help appreciated!
That property was found to have a (very small) timing hole in that a record could be ack'd before the error handler handles it; if the app dies at that time, the record could be "lost".
It was replaced by a new feature in the error handlers ackAfterHandle; which is true by default - i.e. the record's offset is only committed if the error handler "handles" the error.
Records are now never ack'd if the error handler (such as the SeekToCurrentErrorHandler) throws an exception (after it repositions the partitions).
There is no extra configuration needed any more.
See Spring Kafka AckOnError for more details.
I got the rebalance issue in Kafka Consumer and able to solve by setting correct value for max.poll.interval.ms.
But Currently I have seen some of the transaction missing.
So wanted to see the logs which offset value not processed.
So how I can see the logs in spring Kafka.
I am using spring Kafka version 2.2.2.RELEASE
If you are using #KafkaListener, you can add #Header(KafkaHeaders.OFFSET) long offset as a parameter to the listener method.
Also, you can set the commitLogLevel container property on the containerProperties to INFO and the container will log all commits at that level.
https://docs.spring.io/spring-kafka/docs/2.3.3.RELEASE/api/index.html?org/springframework/kafka/listener/ConsumerProperties.html
(ContainerProperties is s subclass of ConsumerProperties in 2.3).
Having recently upgraded to Corda 4.0 we are now seeing a serialization exception when trying to create transactions:
[ERROR] 10:09:49,500 [Mock network] amqp.SerializationOutput. -
Serialization failed direction="Serialize",
type="net.corda.core.transactions.TraversableTransaction",
msg="Defined getter for parameter commands returns type
java.util.List> yet underlying
type is java.util.List>",
ClassChain="net.corda.core.transactions.TraversableTransaction"
{actor_id=Only For Testing, actor_owning_identity=O=Supplier 1,
L=London, C=GB, actor_store_id=TEST, fiber-id=10000001,
flow-id=e61b7da7-6826-4410-9414-1c03bebbc3fe,
invocation_id=58c5f5ff-09f8-4bfa-bbe5-cf6b0ef47d06,
invocation_timestamp=2019-03-28T10:09:46.205Z, origin=Only For
Testing, session_id=58c5f5ff-09f8-4bfa-bbe5-cf6b0ef47d06,
session_timestamp=2019-03-28T10:09:46.205Z, thread-id=400}
Any ideas what causes this?
Anytime you see an error message of this form on Corda 4:
Defined getter for parameter [parameter name] returns type [type] yet
underlying type is [exactly the same type]
It's caused by a bug in certain versions of Guava.
Corda 4 uses Guava version 25.1-jre. However, if your node loads another version of Guava (e.g. 21.0) that overrides Corda's version of Java, you will see this error.
A fix is planned for Corda 4.1.
I faced this issue as well after upgrade to 4.0. My flows returned transaction as output and I've got TraversableTransaction serialization exception. I didn't solved it. I've created flow response DTO with all appropriated data, trxId etc...
Example:
SignedTransaction signedTransaction = subFlow(new FinalityFlow(signedTx, session));
response.setTrxId(signedTransaction.getId().toString());
return response;
I have many servlets in a web applicaton; for some stange reason, only and only one of them always fails in initialization with the following error trace:-
00000045 ServletWrappe E SRVE0100E: Did not realize init() exception thrown by servlet MyServletX: java.lang.NullPointerException
at com.ibm.ws.webcontainer.WebAppPmiListener.onServletStartInit(WebAppPmiListener.java:120)
at com.ibm.ws.webcontainer.webapp.FireOnServletStartInit.fireEvent(WebAppEventSource.java:237)
at com.ibm.ws.webcontainer.util.EventListeners.fireEvent(EventListeners.java:48)
at com.ibm.ws.webcontainer.webapp.WebAppEventSource.onServletStartInit(WebAppEventSource.java:105)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.init(ServletWrapper.java:261)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:444)
at com.ibm.ws.webcontainer.webapp.WebApp.handleRequest(WebApp.java:2841)
at com.ibm.ws.webcontainer.webapp.WebGroup.handleRequest(WebGroup.java:220)
at com.ibm.ws.webcontainer.VirtualHost.handleRequest(VirtualHost.java:204)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:1681)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:77)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:421)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewInformation(HttpInboundLink.java:367)
at com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java:94)
at com.ibm.ws.tcp.channel.impl.WorkQueueManager.requestComplete(WorkQueueManager.java:548)
at com.ibm.ws.tcp.channel.impl.WorkQueueManager.attemptIO(WorkQueueManager.java:601)
at com.ibm.ws.tcp.channel.impl.WorkQueueManager.workerRun(WorkQueueManager.java:934)
at com.ibm.ws.tcp.channel.impl.WorkQueueManager$Worker.run(WorkQueueManager.java:1021)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1332)
I could not figure out if there is anything extra ordinary with this servlet. There is no init() method in this servlet and it extends HTTPServlet. Any idea what could be reason? I am using websphere server 6.0.x. How to get more debugging information in this case?
Well I don't know still cause of above error, but this is how it started working strangely:- i) Re-applied recommended fixes by IBM for my WAS version (especially there are IBM JDK upgrade related fix patches) ii) created a new profile of server iii) Install web application to new profile and it started working.
I don't think this is a product issue.
To debug this problem what i would suggest is to place a simple servlet (kind of Hello World) and deploy it to the server and see what happens.
initialization does not necessarily mean init() method alone.
If you have a static block in your servlet, if you have any variables that are initialized they would all be part of the initialization activity.
Look at the FFDC logs that were generated when this error occurred and that should provide you with clues.
As bkail mentioned, also ensure that yo have the latest fixpacks just to eliminate known problems with the product.
if the hello world servlet works, suggest you place hte servlet code here along with the SystemOut and System Err logs that correspond to this issue along with the relevant FFDC logs and i am sure most of us will be able to help you out with this
HTH
Manglu