A simple example is currently being made through the spring kafka.
If an exception occurs at the service layer, I want to commit the original offset after trying to retry and loading it into the dead letter queue.
However, the dead letter queue is loaded properly, but the original message remains in the kafka because the commit is not processed.
To show you my code, it is as follows.
KafkaConfig.java
...
#Bean
public KafkaListenerContainerFactory<ConcurrentMessageListenerContainer<String, String>> kafkaListenerContainerFactory() {
ConcurrentKafkaListenerContainerFactory<String, String> factory = new ConcurrentKafkaListenerContainerFactory<>();
factory.setConsumerFactory(consumerFactory());
factory.setCommonErrorHandler(kafkaListenerErrorHandler());
factory.getContainerProperties().setAckMode(AckMode.MANUAL_IMMEDIATE);
return factory;
}
private CommonErrorHandler kafkaListenerErrorHandler() {
DefaultErrorHandler defaultErrorHandler = new DefaultErrorHandler(
new DeadLetterPublishingRecoverer(template, DEAD_TOPIC_DESTINATION_RESOLVER),
new FixedBackOff(1000, 3));
defaultErrorHandler.setCommitRecovered(true);
defaultErrorHandler.setAckAfterHandle(true);
defaultErrorHandler.setResetStateOnRecoveryFailure(false);
return defaultErrorHandler;
}
...
KafkaListener.java
...
#KafkaListener(topics = TOPIC_NAME, containerFactory = "kafkaListenerContainerFactory", groupId = "stock-adjustment-0")
public void subscribe(final String message, Acknowledgment ack) throws IOException {
log.info(String.format("Message Received : [%s]", message));
StockAdjustment stockAdjustment = StockAdjustment.deserializeJSON(message);
if(stockService.isAlreadyProcessedOrderId(stockAdjustment.getOrderId())) {
log.info(String.format("AlreadyProcessedOrderId : [%s]", stockAdjustment.getOrderId()));
} else {
if(stockAdjustment.getAdjustmentType().equals("REDUCE")) {
stockService.decreaseStock(stockAdjustment);
}
}
ack.acknowledge(); // <<< does not work!
}
...
Stockservice.java
...
if(stockAdjustment.getQty() > stock.getAvailableStockQty()) {
throw new RuntimeException(String.format("Stock decreased Request [decreasedQty: %s][availableQty : %s]", stockAdjustment.getQty(), stock.getAvailableStockQty()));
}
...
At this time, when RuntimeException occur in the service layer as above, the DLT is issued through an CommonErrorhandler according to the Kafka setting.
However, after issuing the DLT, the original message remains in Kafka, so there is a need for a solution.
I looked it up and found that the setting I wrote is processed through SeekUtils.seekOrRecover(), and if it is not processed even if the maximum number of attempts is not processed, an exception occurs and the original offset is rolled back without processing a commit.
According to the document, it seems that the AfterRollbackProcessor handles rollback if it fails with the default value, but I don't know how to write the code to commit even if it fails.
EDITED
The above code and settings work normally.
I thought the consumer lag would occur, but when I judged the actual logic code(SeekUtils.seekOrRecover()) and checked the offset commit and lag, I confirmed that it works normally.
I think it was caused by my mistake.
Records are never removed (until they expire), the consumer's committed offset is updated.
Use kafka-consumer-groups.sh to describe the group to see the committed offset for the failed record that was sent to the DLT.
Related
I have a gRPC server written in C++ and a client written in Java.
Everything was working fine using a blocking stub. Then I decided that I want to change one of the calls to be asynchronous, so I created an additional stub in my client, this one is created with newStub(channel) as opposed to newBlockingStub(channel). I didn't make any changes on the server side. This is a simple unary RPC call.
So I changed
Empty response = blockingStub.callMethod(request);
to
asyncStub.callMethod(request, new StreamObserver<Empty>() {
#Override
public void onNext(Empty response) {
logInfo("asyncStub.callMethod.onNext");
}
#Override
public void onError(Throwable throwable) {
logError("asyncStub.callMethod.onError " + throwable.getMessage());
}
#Override
public void onCompleted() {
logInfo("asyncStub.callMethod.onCompleted");
}
});
Ever since then, onError is called when I use this RPC (Most of the time) and the error it gives is "CANCELLED: io.grpc.Context was cancelled without error". I read about forking Context objects when making an RPC call from within an RPC call, but that's not the case here. Also, the Context seems to be a server side object, I don't see how it relates to the client. Is this a server side error propagating back to the client? On the server side everything seems to complete successfully, so I'm at a loss as to why this is happening. Inserting a 1ms sleep after calling asyncStub.callMethod seems to make this issue go away, but defeats the purpose. Any and all help in understanding this would be greatly appreciated.
Some notes:
The processing time on the server side is around 1 microsecond
Until now, the round trip time for the blocking call was several hundred microseconds (This is the time I'm trying to cut down, as this is essentially a void function, so I don't need to wait for a response)
This method is called multiple times in a row, so before it used to wait until the previous one finished, now they just fire off one after the other.
Some snippets from the proto file:
service EventHandler {
rpc callMethod(Msg) returns (Empty) {}
}
message Msg {
uint64 fieldA = 1;
int32 fieldB = 2;
string fieldC = 3;
string fieldD = 4;
}
message Empty {
}
So it turns out that I was wrong. The context object is used by the client too.
The solution was to do the following:
Context newContext = Context.current().fork();
Context origContext = newContext.attach();
try {
// Call async RPC here
} finally {
newContext.detach(origContext);
}
Hopefully this can help someone else in the future.
We are using spring-kafka-2.2.8.RELEASE. I have an specific situation where I need help. I have 4 topics topic, retryTopic, successTopic and errorTopic. If topic fails, should be redirected to retryTopic where the 3 attempts to retry will be made. If those attempts fails, must redirect to errorTopic. In case of sucess on both topic and retryTopic, should be redirected to the sucessTopic. This situation is already implemented based on the question How to retry with spring kafka version 2..2.
But now, I have a new situation where I need to call the retryTopic listener from inside the topic listener based on a business logic error without an Exception been thrown(it already calls the retryTopic when an exception is thrown and it must remain with this behavior). And I also need to know on which retry attempt number the retryTopic is been called as a paramater of the listener bellow.
#KafkaListener(id = "so60172304.2", topics = "retryTopic")
public void listen2(String in) {
RetryTemplate retryTemplate = new RetryTemplate();
retryTemplate.execute(new RetryCallback<Void, RuntimeException>() {
#Override
public Void doWithRetry(RetryContext retryContext) throws RuntimeException {
// Can I get the retry count here? It didn't work
Integer count =RetrySynchronizationManager.getContext().getRetryCount());
return this.doWithRetry(retryContext);
}
});
}
There is no reason you can't call one listener from another (but you won't get retries unless you call it using a RetryTemplate in the first method).
If you use a RetryTemplate configured on the container factory to do the retries (rather than adding a BackOff to the SeektoCurrentErrorHandler in versions 2.3.x and higher), you can obtain the retry count (starting at zero) like this...
#KafkaListener(id = "so60172304.2", topics = "retryTopic")
public void listen2(String in) {
int retryCount = RetrySynchronizationManager.getContext().getRetryCount();
...
}
getContext() will return null if you call this directly from the first method (unless you wrap the call in a RetryTemplate.execute()).
In 2.5.x a delivery attempt header will be available (optionally) even if using the SeektoCurrentErrorHandler with a BackOff instead of using a RetryTemplate in the container factory.
we are using Spring KafkaListener which acknowledges each records after it is processed to DB. If we have problems writing to DB we don't acknowledge the record so that offsets are not committed for the consumer. this works fine. Now we want to get the failed messages in next poll to retry them. we added errorhandler to our listener and invoked ConsumerAwareListenerErrorHandler and tried to do consumer.seek() for the failed message offset. Expectation is during next poll, we should received the failed messages. This is not happening. Next poll fetches only the new messages and not the failed messages Code snippet is given below.
#Service
public class KafkaConsumer {
#KafkaListener(topics = ("${kafka.input.stream.topic}"), containerFactory = "kafkaManualAckListenerContainerFactory", errorHandler = "listen3ErrorHandler")
public void onMessage(ConsumerRecord<Integer, String> record,
Acknowledgment acknowledgment ) throws Exception {
try {
msg = JaxbUtil.convertJsonStringToMsg(record.value());
onHandList = DCMUtil.convertMsgToOnHandDTO(msg);
TeradataDAO.updateData(onHandList);
acknowledgment.acknowledge();
recordSuccess = true;
LOGGER.info("Message Saved in Teradata DB");
} catch (Exception e) {
LOGGER.error("Error Processing On Hand Data ", e);
recordSuccess = false;
}
}
#Bean
public ConsumerAwareListenerErrorHandler listen3ErrorHandler() throws InterruptedException {
return (message, exception, consumer) -> {
this.listen3Exception = exception;
MessageHeaders headers = message.getHeaders();
consumer.seek(new org.apache.kafka.common.TopicPartition(
headers.get(KafkaHeaders.RECEIVED_TOPIC, String.class),
headers.get(KafkaHeaders.RECEIVED_PARTITION_ID, Integer.class)),
headers.get(KafkaHeaders.OFFSET, Long.class));
return null;
};
}
}
Container Class
#Bean
public Map<Object,Object> consumerConfigs() {
Map<Object,Object> props = new HashMap<Object,Object> ();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG,
localhost:9092);
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG,
StringDeserializer.class);
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG,
StringDeserializer.class);
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
props.put(ConsumerConfig.GROUP_ID_CONFIG, "example-1");
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false);
return props;
}
#SuppressWarnings({ "rawtypes", "unchecked" })
#Bean
public ConsumerFactory consumerFactory() {
return new DefaultKafkaConsumerFactory(consumerConfigs());
}
#SuppressWarnings("unchecked")
#Bean
KafkaListenerContainerFactory<ConcurrentMessageListenerContainer<Integer, String>>
kafkaManualAckListenerContainerFactory() {
ConcurrentKafkaListenerContainerFactory<Integer, String> factory =
new ConcurrentKafkaListenerContainerFactory<>();
factory.setConsumerFactory(consumerFactory());
factory.getContainerProperties().setAckMode(AckMode.MANUAL);
return factory;
}
It's supposed to work like this:
The error handler needs to throw an exception if you want to discard additional records from the previous poll.
Since you are "handling" the error, the container knows nothing and will continue to call the listener with the remaining records from the poll.
That said, I see that the container is also ignoring an exception thrown by the error handler (it will discard if the error handler throws an Error not an exception). I will open an issue for this.
Another work around would be to add the Consumer to the listener method signature and do the seek there (and throw an exception). If there is no error handler, the rest of the batch is discarded.
Correction
If the container has no ErrorHandler, any Throwable thrown by a ListenerErrorHandler will cause the remaining records to be discarded.
Please try using SeekToCurrentErrorHandler. The doc says "This allows implementations to seek all unprocessed topic/partitions so the current record (and the others remaining) will be retrieved by the next poll. The SeekToCurrentErrorHandler does exactly this.
The container will commit any pending offset commits before calling the error handler."
https://docs.spring.io/autorepo/docs/spring-kafka-dist/2.1.0.BUILD-SNAPSHOT/reference/htmlsingle/#_seek_to_current_container_error_handlers
I'm quite new to the Microservice world and particularly vertX. I want my verticle to start anyway even there is no database connection available (e.g. database URL missing in configuration). I already managed to do this and my verticle is starting.
The issue now is that I want my verticle to notice when the database connection is available again and connect to it. How can I do this ?
I thought about creating another Verticle "DatabaseVerticle.java" which would send the current DB config on the event bus and my initial verticle would consume this message and check whether the config info is consistent (reply with success) or still missing some data (reply with fail and make the DatabaseVerticle check again).
This might work (and might not) but does not seem to be the optimal solution for me.
I'd be very glad if someone could suggest a better solution. Thank you !
For your use case, I'd recommend to use the vertx-config. In particular, have a look at the Listening to configuration changes section of the Vert.x Config documentation.
You could create a config retriever and set a handler for changes:
ConfigRetrieverOptions options = new ConfigRetrieverOptions()
.setScanPeriod(2000)
.addStore(myConfigStore);
ConfigRetriever retriever = ConfigRetriever.create(vertx, options);
retriever.getConfig(json -> {
// If DB config available, start the DB client
// Otherwise set a "dbStarted" variable to false
});
retriever.listen(change -> {
// If "dbStarted" is still set to false
// Check the config and start the DB client if possible
// Set "dbStarted" to true when done
});
The ideal way would be some other service telling your service about database connection. Either through event bus or HTTP, what you can do is when someone tries to access your database when connection is not made just try to make some DB call and handle the exception, return a boolean as false. Now when you get a message on event bus, consume it and save it in some config pojo. Now when someone tries to access your database, look for config and if available make a connection.
Your consumer:
public void start(){
EventBus eb = vertx.eventBus();
eb.consumer("database", message -> {
config.setConfig(message.body());
});
}
Your db client(Mongo for this eg):
public class MongoService{
private MongoClient client;
public boolean isAvailable = false;
MongoService(Vertx vertx){
if(config().getString("connection")){
client = MongoClient.createShared(vertx, config().getString("connection"));
isAvailable = true;
}
}
}
Not everything in Vertx should be solved by another verticle.
In this case, you can use .periodic()
http://vertx.io/docs/vertx-core/java/#_don_t_call_us_we_ll_call_you
I assume you have some function that checks the DB for the first time.
Let's call it checkDB()
class PeriodicVerticle extends AbstractVerticle {
private Long timerId;
#Override
public void start() {
System.out.println("Started");
// Should be called each time DB goes offline
final Long timerId = this.vertx.setPeriodic(1000, (l) -> {
final boolean result = checkDB();
// Set some variable telling verticle that DB is back online
if (result) {
cancelTimer();
}
});
setTimerId(timerId);
}
private void cancelTimer() {
System.out.println("Cancelling");
getVertx().cancelTimer(this.timerId);
}
private void setTimerId(final Long timerId) {
this.timerId = timerId;
}
}
Here I play a bit with timerId, since we cannot pass it to cancelTimer() right away. But otherwise, it's quite simple.
I am building a HTTP proxy with netty, which supports HTTP pipelining. Therefore I receive multiple HttpRequest Objects on a single Channel and got the matching HttpResponse Objects. The order of the HttpResponse writes is the same than I got the HttpRequest. If a HttpResponse was written, the next one will be written when the HttpProxyHandler receives a writeComplete event.
The Pipeline should be convenient:
final ChannelPipeline pipeline = Channels.pipeline();
pipeline.addLast("decoder", new HttpRequestDecoder());
pipeline.addLast("encoder", new HttpResponseEncoder());
pipeline.addLast("writer", new HttpResponseWriteDelayHandler());
pipeline.addLast("deflater", new HttpContentCompressor(9));
pipeline.addLast("handler", new HttpProxyHandler());
Regarding this question only the order of the write calls should be important, but to be sure I build another Handler (HttpResponseWriteDelayHandler) which suppresses the writeComplete event until the whole response was written.
To test this I enabled network.http.proxy.pipelining in Firefox and visited a page with many images and connections (a news page). The problem is, that the browser does not receive some responses in spite of the logs of the proxy consider them as sent successfully.
I have some findings:
The problem only occurs if the connection from proxy to server is faster than the connection from proxy to browser.
The problem occurs more often after sending a larger image on that connection, e.g. 20kB
The problem does not occur if only 304 - Not Modified responses were sent (refreshing the page considering browser cache)
Setting bootstrap.setOption("sendBufferSize", 1048576); or above does not help
Sleeping a timeframe dependent on the responses body size in before sending the writeComplete event in HttpResponseWriteDelayHandler solves the problem, but is a very bad solution.
I found the solution and want to share it, if anyone else has a similar problem:
The content of the HttpResponse is too big. To analyze the content the whole HTML document was in the buffer. This must be splitted in Chunks again to send it properly. If the HttpResponse is not chunked I wrote a simple solution to do it. One needs to put a ChunkedWriteHandler next to the logic handler and write this class instead of the response itself:
public class ChunkedHttpResponse implements ChunkedInput {
private final static int CHUNK_SIZE = 8196;
private final HttpResponse response;
private final Queue<HttpChunk> chunks;
private boolean isResponseWritten;
public ChunkedHttpResponse(final HttpResponse response) {
if (response.isChunked())
throw new IllegalArgumentException("response must not be chunked");
this.chunks = new LinkedList<HttpChunk>();
this.response = response;
this.isResponseWritten = false;
if (response.getContent().readableBytes() > CHUNK_SIZE) {
while (CHUNK_SIZE < response.getContent().readableBytes()) {
chunks.add(new DefaultHttpChunk(response.getContent().readSlice(CHUNK_SIZE)));
}
chunks.add(new DefaultHttpChunk(response.getContent().readSlice(response.getContent().readableBytes())));
chunks.add(HttpChunk.LAST_CHUNK);
response.setContent(ChannelBuffers.EMPTY_BUFFER);
response.setChunked(true);
response.setHeader(HttpHeaders.Names.TRANSFER_ENCODING, HttpHeaders.Values.CHUNKED);
}
}
#Override
public boolean hasNextChunk() throws Exception {
return !isResponseWritten || !chunks.isEmpty();
}
#Override
public Object nextChunk() throws Exception {
if (!isResponseWritten) {
isResponseWritten = true;
return response;
} else {
HttpChunk chunk = chunks.poll();
return chunk;
}
}
#Override
public boolean isEndOfInput() throws Exception {
return isResponseWritten && chunks.isEmpty();
}
#Override
public void close() {}
}
Then one can call just channel.write(new ChunkedHttpResponse(response) and the chunking is done automatically if needed.