I am using spring-kafka 2.8.9 and kafka-clients 2.8.1 . I want to skip a message which is failed to de-serialize . Since setErrorHandler is deprecated , I tried using CommonErrorHandler . But I am not sure how to skip current error message and move to next record . The only option I can see is using pattern matching by extracting relevant details from below line like offset and partition .
org.apache.kafka.common.errors.SerializationException: Error deserializing key/value for partition test-0 at offset 1. If needed, please seek past the record
Is there any other way like RecordDeserializationException to get necessary information from the exception or any other means without pattern matching . I can not upgrade to kafka 3.X.X .
My config
#Bean
public ConsumerFactory<String, Farewell> farewellConsumerFactory()
{
groupId = LocalTime.now().toString();
Map<String, Object> props = new HashMap<>();
props.put( ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapAddress);
props.put( ConsumerConfig.GROUP_ID_CONFIG, groupId);
props.put(JsonDeserializer.TRUSTED_PACKAGES, "*");
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG,"earliest");
return new DefaultKafkaConsumerFactory<>(props,new StringDeserializer(),new JsonDeserializer<>(Farewell.class));
}
#Bean
public ConcurrentKafkaListenerContainerFactory<String, Farewell> farewellKafkaListenerContainerFactory() {
ConcurrentKafkaListenerContainerFactory<String, Farewell> factory = new ConcurrentKafkaListenerContainerFactory<>();
factory.setCommonErrorHandler(new CommonErrorHandler()
{
#Override
public void handleOtherException(Exception thrownException, Consumer<?, ?> consumer, MessageListenerContainer container, boolean batchListener)
{
CommonErrorHandler.super.handleOtherException(thrownException, consumer, container, batchListener);
}
});
factory.setConsumerFactory(farewellConsumerFactory());
return factory;
}
My listener class
#KafkaListener(topics = "${topicId}",
containerFactory = "farewellKafkaListenerContainerFactory")
public void farewellListener(Farewell message) {
System.out.println("Received Message in group " + groupId + "| " + message);
}
Domain class
public class Farewell {
private String message;
private Integer remainingMinutes;
public Farewell(String message, Integer remainingMinutes)
{
this.message = message;
this.remainingMinutes = remainingMinutes;
}
// standard getters, setters and constructor
}
I have checked these links
How to skip a msg that have error in kafka when i use ConcurrentMessageListenerContainer?
Better way of error handling in Kafka Consumer
Use an ErrorHandlingDeserializer as a wrapper around your real deserializer.
Serialization exceptions will be sent directly to the DefaultErrorHandler, which treats such exceptions as fatal (by default) and sends them directly to the recoverer.
I am using spring-kafka 2.2.8 and trying to understand if there is an option to deploy a kafka consumer being in pause mode until i signal to start consume the messages. Please suggest.
I see in the below post, we can pause and start the consumer but I need the consumer to be in pause mode when it's deployed.
how to pause and resume #KafkaListener using spring-kafka
#KafkaListener(id = "foo", ..., autoStartup = "false")
Then start it using the KafkaListenerEndpointRegistry when you are ready
registry.getListenerContainer("foo").start();
There is not much point in starting it in paused mode, but you can do that...
#SpringBootApplication
public class So62329274Application {
public static void main(String[] args) {
SpringApplication.run(So62329274Application.class, args);
}
#KafkaListener(id = "so62329274", topics = "so62329274", autoStartup = "false")
public void listen(String in) {
System.out.println(in);
}
#Bean
public NewTopic topic() {
return TopicBuilder.name("so62329274").partitions(1).replicas(1).build();
}
#Bean
public ApplicationRunner runner(KafkaListenerEndpointRegistry registry, KafkaTemplate<String, String> template) {
return args -> {
template.send("so62329274", "foo");
registry.getListenerContainer("so62329274").pause();
registry.getListenerContainer("so62329274").start();
System.in.read();
registry.getListenerContainer("so62329274").resume();
};
}
}
You will see a log message like this when the partitions are assigned:
Paused consumer resumed by Kafka due to rebalance; consumer paused again, so the initial poll() will never return any records
I have a gRPC client in a kafka application. This means the client will constantly open and close channels.
public class UserAgentClient {
protected final Logger logger = LoggerFactory.getLogger(getClass());
private static final Config uaparserConfig = ConfigFactory.load().getConfig(ua);
private final ManagedChannel channel;
private final UserAgentServiceGrpc.UserAgentServiceBlockingStub userAgentBlockingStub;
public UserAgentParserClient() {
this(ManagedChannelBuilder.forAddress(uaConfig.getString("host"), uaConfig.getInt("port")).usePlaintext());
}
public UserAgentClient(ManagedChannelBuilder<?> usePlaintext) {
channel = usePlaintext.build();
userAgentBlockingStub = UserAgentServiceGrpc.newBlockingStub(channel);
}
public UserAgentParseResponse getUserAgent(String userAgent ) {
UserAgentRequest request = UserAgentRequest.newBuilder().setUserAgent(userAgent).build();
UserAgentParseResponse response = null;
try {
response = userAgentBlockingStub.parseUserAgent(request);
} catch(Exception e) {
logger.warn("An exception has occurred during gRPC call to the user agent.", e.getMessage());
}
shutdown();
return response;
}
public void shutdown() {
try {
channel.shutdown();
} catch (InterruptedException ie) {
logger.warn("Interrupted exception during gRPC channel close", ie);
}
}
}
I was wondering if I can keep the channel open the whole time? Or do I have to open a channel every time I make a new call? I was wondering because I was testing the performance and it seems to improve drastically if I just keep the channel open. On the other hand is there something that I'm missing?
creating a new channel has huge overhead, you should keep the channel open as long as possible.
Since the opening and closing of channel is expensive I removed the channel = usePlaintext.build(); completely from my client
Instead I'm opening and closing it in my kafka Transformer. In my class UserAgentDataEnricher that implements Transformer.
public class UserAgentDataEnricher implements Transformer<byte[], EnrichedData, KeyValue<byte[], EnrichedData>> {
private UserAgentParserClient userAgentParserClient;
#Override
public void init(ProcessorContext context) {
this.context = context;
open();
// schedule a punctuate() method every 15 minutes
this.context.schedule(900000, PunctuationType.WALL_CLOCK_TIME, (timestamp) -> {
close();
open();
logger.info("Re-opening of user agent channel is initialized");
});
}
#Override
public void close() {
userAgentParserClient.shutdown();
}
private void open() {
channel = ManagedChannelBuilder.forAddress("localhost", 50051).usePlaintext().build();
userAgentClient = new UserAgentClient(channel);
}
...
}
and now I initialize my client like that:
public UserAgentClient(ManagedChannel channel) {
this.channel = channel;
userAgentBlockingStub = UserAgentServiceGrpc.newBlockingStub(channel);
}
we are using Spring KafkaListener which acknowledges each records after it is processed to DB. If we have problems writing to DB we don't acknowledge the record so that offsets are not committed for the consumer. this works fine. Now we want to get the failed messages in next poll to retry them. we added errorhandler to our listener and invoked ConsumerAwareListenerErrorHandler and tried to do consumer.seek() for the failed message offset. Expectation is during next poll, we should received the failed messages. This is not happening. Next poll fetches only the new messages and not the failed messages Code snippet is given below.
#Service
public class KafkaConsumer {
#KafkaListener(topics = ("${kafka.input.stream.topic}"), containerFactory = "kafkaManualAckListenerContainerFactory", errorHandler = "listen3ErrorHandler")
public void onMessage(ConsumerRecord<Integer, String> record,
Acknowledgment acknowledgment ) throws Exception {
try {
msg = JaxbUtil.convertJsonStringToMsg(record.value());
onHandList = DCMUtil.convertMsgToOnHandDTO(msg);
TeradataDAO.updateData(onHandList);
acknowledgment.acknowledge();
recordSuccess = true;
LOGGER.info("Message Saved in Teradata DB");
} catch (Exception e) {
LOGGER.error("Error Processing On Hand Data ", e);
recordSuccess = false;
}
}
#Bean
public ConsumerAwareListenerErrorHandler listen3ErrorHandler() throws InterruptedException {
return (message, exception, consumer) -> {
this.listen3Exception = exception;
MessageHeaders headers = message.getHeaders();
consumer.seek(new org.apache.kafka.common.TopicPartition(
headers.get(KafkaHeaders.RECEIVED_TOPIC, String.class),
headers.get(KafkaHeaders.RECEIVED_PARTITION_ID, Integer.class)),
headers.get(KafkaHeaders.OFFSET, Long.class));
return null;
};
}
}
Container Class
#Bean
public Map<Object,Object> consumerConfigs() {
Map<Object,Object> props = new HashMap<Object,Object> ();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG,
localhost:9092);
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG,
StringDeserializer.class);
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG,
StringDeserializer.class);
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
props.put(ConsumerConfig.GROUP_ID_CONFIG, "example-1");
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false);
return props;
}
#SuppressWarnings({ "rawtypes", "unchecked" })
#Bean
public ConsumerFactory consumerFactory() {
return new DefaultKafkaConsumerFactory(consumerConfigs());
}
#SuppressWarnings("unchecked")
#Bean
KafkaListenerContainerFactory<ConcurrentMessageListenerContainer<Integer, String>>
kafkaManualAckListenerContainerFactory() {
ConcurrentKafkaListenerContainerFactory<Integer, String> factory =
new ConcurrentKafkaListenerContainerFactory<>();
factory.setConsumerFactory(consumerFactory());
factory.getContainerProperties().setAckMode(AckMode.MANUAL);
return factory;
}
It's supposed to work like this:
The error handler needs to throw an exception if you want to discard additional records from the previous poll.
Since you are "handling" the error, the container knows nothing and will continue to call the listener with the remaining records from the poll.
That said, I see that the container is also ignoring an exception thrown by the error handler (it will discard if the error handler throws an Error not an exception). I will open an issue for this.
Another work around would be to add the Consumer to the listener method signature and do the seek there (and throw an exception). If there is no error handler, the rest of the batch is discarded.
Correction
If the container has no ErrorHandler, any Throwable thrown by a ListenerErrorHandler will cause the remaining records to be discarded.
Please try using SeekToCurrentErrorHandler. The doc says "This allows implementations to seek all unprocessed topic/partitions so the current record (and the others remaining) will be retrieved by the next poll. The SeekToCurrentErrorHandler does exactly this.
The container will commit any pending offset commits before calling the error handler."
https://docs.spring.io/autorepo/docs/spring-kafka-dist/2.1.0.BUILD-SNAPSHOT/reference/htmlsingle/#_seek_to_current_container_error_handlers
We have rest api application. We use redis for API response caching and internal method caching. If redis connection then it is making our API down. We want to bypass the redis caching if that redis connection fails or any exception instead of making our API down.
There is a interface CacheErrorHandler but it handles the redis get set operation failures not redis connection problems. We are using Spring 4.1.2.
Let's boil this down a bit. Your application uses caching (implemented with Redis). If the Redis connection is stale/closed or otherwise, then you want the application to bypass caching and (presumably) go directly to an underlying data store (e.g. RDBMS). The application Service logic might look similar to...
#Service
class CustomerService ... {
#Autowired
private CustomerRepository customerRepo;
protected CustomerRepository getCustomerRepo() {
Assert.notNull(customerRepo, "The CustomerRepository was not initialized!");
return customerRepo;
}
#Cacheable(value = "Customers")
public Customer getCustomer(Long customerId) {
return getCustomerRepo().load(customerId);
}
...
}
All that matters in Spring core's Caching Abstraction to ascertain a Cache "miss" is that the value returned is null. As such, Spring Caching Infrastructure will then proceed in calling the actual Service method (i.e. getCustomer). Keep in mind on the return of the getCustomerRepo().load(customerId) call, you also need to handle the case where Spring's Caching Infrastructure attempts to now cache the value.
In the spirit of keeping it simple, we will do without AOP, but you should be able to achieve this using AOP as well (your choice).
All you (should) need is a "custom" RedisCacheManager extending the SDR CacheManager implementation, something like...
package example;
import org.springframework.cache.Cache;
import org.springframework.data.redis.cache.RedisCacheManager;
...
class MyCustomRedisCacheManager extends RedisCacheManager {
public MyCustomerRedisCacheManager(RedisTemplate redisTemplate) {
super(redisTemplate);
}
#Override
public Cache getCache(String name) {
return new RedisCacheWrapper(super.getCache(name));
}
protected static class RedisCacheWrapper implements Cache {
private final Cache delegate;
public RedisCacheWrapper(Cache redisCache) {
Assert.notNull(redisCache, "'delegate' must not be null");
this.delegate = redisCache;
}
#Override
public Cache.ValueWrapper get(Object key) {
try {
delegate.get(key);
}
catch (Exception e) {
return handleErrors(e);
}
}
#Override
public void put(Object key, Object value) {
try {
delegate.put(key, value);
}
catch (Exception e) {
handleErrors(e);
}
}
// implement clear(), evict(key), get(key, type), getName(), getNativeCache(), putIfAbsent(key, value) accordingly (delegating to the delegate).
protected <T> T handleErrors(Exception e) throws Exception {
if (e instanceof <some RedisConnection Exception type>) {
// log the connection problem
return null;
}
else if (<something different>) { // act appropriately }
...
else {
throw e;
}
}
}
}
So, if Redis is unavailable, perhaps the best you can do is log the problem and proceed to let the Service invocation happen. Clearly, this will hamper performance but at least it will raise awareness that a problem exists. Clearly, this could be tied into a more robust notification system, but it is a crude example of the possibilities. The important thing is, your Service remains available while the other services (e.g. Redis) that the application service depends on, may have failed.
In this implementation (vs. my previous explanation) I chose to delegate to the underlying, actual RedisCache implementation to let the Exception occur, then knowing full well a problem with Redis exists, and so that you can deal with the Exception appropriately. However, if you are a certain that the Exception is related to a connection problem upon inspection, you can return "null" to let Spring Caching Infrastructure proceed as if it were a Cache "miss" (i.e. bad Redis Connection == Cache miss, in this case).
I know something like this should help your problem as I built a similar prototype of a "custom" CacheManager implementation for GemFire and one of Pivotal's customers. In that particular UC, the Cache "miss" had to be triggered by an "out-of-date version" of the application domain object where production had a mix of newer and older application clients connecting to GemFire through Spring's Caching Abstraction. The application domain object fields would change in newer versions of the app for instance.
Anyway, hope this helps or gives you more ideas.
Cheers!
So, I was digging through the core Spring Framework Caching Abstraction source today addressing another question and it seems if a CacheErrorHandler is implemented properly, then perhaps a problematic Redis Connection could still result in the desired behavior, e.g. cache "miss" (triggered with the return of a null value).
See the AbstractCacheInvoker source for more details.
The cache.get(key) should result in an exception due to a faulty Redis Connection and thus Exception handler would be invoked...
catch (RuntimeException e) {
getErrorHandler().handleCacheGetError(e, cache, key);
return null; // If the exception is handled, return a cache miss
}
If the CacheErrorHandler properly handles the Cache "get" error (and does not re-throw the/an Exception), then a null value will be returned indicating a cache "miss".
Thank you #John Blum. My solution in Spring Boot is as follows.
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.cache.Cache;
import org.springframework.data.redis.cache.RedisCacheManager;
import org.springframework.data.redis.core.RedisOperations;
import org.springframework.util.Assert;
import java.util.concurrent.Callable;
class CustomRedisCacheManager extends RedisCacheManager {
private static Logger logger = LoggerFactory.getLogger(CustomRedisCacheManager.class);
public CustomRedisCacheManager(RedisOperations redisOperations) {
super(redisOperations);
}
#Override
public Cache getCache(String name) {
return new RedisCacheWrapper(super.getCache(name));
}
protected static class RedisCacheWrapper implements Cache {
private final Cache delegate;
public RedisCacheWrapper(Cache redisCache) {
Assert.notNull(redisCache, "delegate cache must not be null");
this.delegate = redisCache;
}
#Override
public String getName() {
try {
return delegate.getName();
} catch (Exception e) {
return handleException(e);
}
}
#Override
public Object getNativeCache() {
try {
return delegate.getNativeCache();
} catch (Exception e) {
return handleException(e);
}
}
#Override
public Cache.ValueWrapper get(Object key) {
try {
return delegate.get(key);
} catch (Exception e) {
return handleException(e);
}
}
#Override
public <T> T get(Object o, Class<T> aClass) {
try {
return delegate.get(o, aClass);
} catch (Exception e) {
return handleException(e);
}
}
#Override
public <T> T get(Object o, Callable<T> callable) {
try {
return delegate.get(o, callable);
} catch (Exception e) {
return handleException(e);
}
}
#Override
public void put(Object key, Object value) {
try {
delegate.put(key, value);
} catch (Exception e) {
handleException(e);
}
}
#Override
public ValueWrapper putIfAbsent(Object o, Object o1) {
try {
return delegate.putIfAbsent(o, o1);
} catch (Exception e) {
return handleException(e);
}
}
#Override
public void evict(Object o) {
try {
delegate.evict(o);
} catch (Exception e) {
handleException(e);
}
}
#Override
public void clear() {
try {
delegate.clear();
} catch (Exception e) {
handleException(e);
}
}
private <T> T handleException(Exception e) {
logger.error("handleException", e);
return null;
}
}
}
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.data.redis.cache.RedisCacheManager;
import org.springframework.data.redis.core.RedisTemplate;
#Configuration
public class RedisConfig {
#Bean
public RedisCacheManager redisCacheManager(RedisTemplate redisTemplate) {
CustomRedisCacheManager redisCacheManager = new CustomRedisCacheManager(redisTemplate);
redisCacheManager.setUsePrefix(true);
return redisCacheManager;
}
}
actually my response is directed to Mr. #Vivek Aditya - I faced the same problem: new spring-data-redis api and not constructing RedisCacheManager per RedisTemplate. The only option - based on #John Blum suggestions - was to use aspects. And below is my code.
#Aspect
#Component
public class FailoverRedisCacheAspect {
private static class FailoverRedisCache extends RedisCache {
protected FailoverRedisCache(RedisCache redisCache) {
super(redisCache.getName(), redisCache.getNativeCache(), redisCache.getCacheConfiguration());
}
#Override
public <T> T get(Object key, Callable<T> valueLoader) {
try {
return super.get(key, valueLoader);
} catch (RuntimeException ex) {
return valueFromLoader(key, valueLoader);
}
}
private <T> T valueFromLoader(Object key, Callable<T> valueLoader) {
try {
return valueLoader.call();
} catch (Exception e) {
throw new ValueRetrievalException(key, valueLoader, e);
}
}
}
#Around("execution(* org.springframework.cache.support.AbstractCacheManager.getCache (..))")
public Cache beforeSampleCreation(ProceedingJoinPoint proceedingJoinPoint) {
try {
Cache cache = (Cache) proceedingJoinPoint.proceed(proceedingJoinPoint.getArgs());
if (cache instanceof RedisCache) {
return new FailoverRedisCache((RedisCache) cache);
} else {
return cache;
}
} catch (Throwable ex) {
return null;
}
}
}
works fine for all reasonable scenarios:
app starts fine with redis down
app (still) works during (sudden) redis outage
when redis starts working again, app sees it
Edit: the code is more like a poc - only for "get", and I don't like reinstantiating FailoverRedisCache every single cache hit - there should be a map.
None of the above worked for us when using Spring Boot 2.3.9.release with Redis. We ended up creating and registering our own customized CacheErrorHandler named CustomCacheErrorHandler to override the default SimpleCacheErrorHandler provided by Spring Framework. This will work perfectly.
#Configuration
public class CachingConfiguration extends CachingConfigurerSupport {
#Override
public CacheErrorHandler errorHandler() {
return new CustomCacheErrorHandler();
}
}
class CustomCacheErrorHandler implements CacheErrorHandler {
Logger log = Logger.get(CustomCacheErrorHandler.class);
#Override
public void handleCacheGetError(RuntimeException e, Cache cache, Object o) {
log.error(e.getMessage(), e);
}
#Override
public void handleCachePutError(RuntimeException e, Cache cache, Object o, Object o1) {
log.error(e.getMessage(), e);
}
#Override
public void handleCacheEvictError(RuntimeException e, Cache cache, Object o) {
log.error(e.getMessage(), e);
}
#Override
public void handleCacheClearError(RuntimeException e, Cache cache) {
log.error(e.getMessage(), e);
}
}
I had same problem, but, unfortunately, none of the above solutions work for me. I checked for the problem and found out that the executed command never timed out if there was no connection to Redis. So I start to study lettuce library for a solution. I solve the problem by rejecting the command when there is no connection:
#Bean
public LettuceConnectionFactory lettuceConnectionFactory()
{
final SocketOptions socketOptions = SocketOptions.builder().connectTimeout(Duration.ofSeconds(10)).build();
ClientOptions clientOptions = ClientOptions.builder()
.socketOptions(socketOptions)
.autoReconnect(true)
.disconnectedBehavior(ClientOptions.DisconnectedBehavior.REJECT_COMMANDS)
.build();
LettuceClientConfiguration clientConfig = LettuceClientConfiguration.builder()
.commandTimeout(Duration.ofSeconds(10))
.clientOptions(clientOptions).build();
RedisStandaloneConfiguration redisStandaloneConfiguration = new RedisStandaloneConfiguration(this.host, this.port);
return new LettuceConnectionFactory(redisStandaloneConfiguration, clientConfig);
}
All the core Spring Framework Cache abstraction annotations (e.g. #Cacheable) along with the JSR-107 JCache annotations supported by the core SF delegate to the underlying CacheManager under-the-hood, and for Redis, that is the RedisCacheManager.
You would configure the RedisCacheManager in Spring XML configuration meta-data similar to here.
One approach would be to write an AOP Proxy for the (Redis)CacheManager that uses the RedisConnection (indirectly from the RedisTemplate) to ascertain the state of the connection on each (Redis)CacheManger operation.
If the connection has failed, or is closed, for standard cache ops, the (Redis)CacheManager could return an instance of RedisCache for getCache(String name) that always returns null (indicating a Cache miss on an entry), thus passing through to the underlying data store.
There maybe better ways to handle this as I am not an expert on all things Redis (or SDR), but this should work and perhaps give you a few ides of your own.
Cheers.
You can use CacheErrorHandler. But you should make sure to make
RedisCacheManager transactionAware to false in your Redis Cache Config(to make sure the transaction is committed early when executing the caching part and the error is caught by CacheErrorHandler and don't wait until the end of the execution which skips CacheErrorHandler part). The function to set transactionAware to false looks like this:
#Bean
public RedisCacheManager redisCacheManager(LettuceConnectionFactory lettuceConnectionFactory) {
JdkSerializationRedisSerializer redisSerializer = new JdkSerializationRedisSerializer(getClass().getClassLoader());
RedisCacheConfiguration redisCacheConfiguration = RedisCacheConfiguration.defaultCacheConfig()
.entryTtl(Duration.ofHours(redisDataTTL))
.serializeValuesWith(RedisSerializationContext.SerializationPair.fromSerializer(redisSerializer));
redisCacheConfiguration.usePrefix();
RedisCacheManager redisCacheManager = RedisCacheManager.RedisCacheManagerBuilder.fromConnectionFactory(lettuceConnectionFactory)
.cacheDefaults(redisCacheConfiguration)
.build();
redisCacheManager.setTransactionAware(false);
return redisCacheManager;
}