I'd like to use a Kafka state store of type KeyValueStore in a sample application using the Kafka Binder of Spring Cloud Stream.
According to the documentation, it should be pretty simple.
This is my main class:
#SpringBootApplication
public class KafkaStreamTestApplication {
public static void main(String[] args) {
SpringApplication.run(KafkaStreamTestApplication.class, args);
}
#Bean
public BiFunction<KStream<String, String>, KeyValueStore<String,String>, KStream<String, String>> process(){
return (input,store) -> input.mapValues(v -> v.toUpperCase());
}
#Bean
public StoreBuilder myStore() {
return Stores.keyValueStoreBuilder(
Stores.persistentKeyValueStore("my-store"), Serdes.String(),
Serdes.String());
}
}
I suppose that the KeyValueStore should be passed as the second parameter of the "process" method, but the application fails to start with the message below:
Caused by: java.lang.IllegalStateException: No factory found for binding target type: org.apache.kafka.streams.state.KeyValueStore among registered factories: channelFactory,messageSourceFactory,kStreamBoundElementFactory,kTableBoundElementFactory,globalKTableBoundElementFactory
at org.springframework.cloud.stream.binding.AbstractBindableProxyFactory.getBindingTargetFactory(AbstractBindableProxyFactory.java:82) ~[spring-cloud-stream-3.0.3.RELEASE.jar:3.0.3.RELEASE]
at org.springframework.cloud.stream.binder.kafka.streams.function.KafkaStreamsBindableProxyFactory.bindInput(KafkaStreamsBindableProxyFactory.java:191) ~[spring-cloud-stream-binder-kafka-streams-3.0.3.RELEASE.jar:3.0.3.RELEASE]
at org.springframework.cloud.stream.binder.kafka.streams.function.KafkaStreamsBindableProxyFactory.afterPropertiesSet(KafkaStreamsBindableProxyFactory.java:103) ~[spring-cloud-stream-binder-kafka-streams-3.0.3.RELEASE.jar:3.0.3.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1855) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1792) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
I found the solution about how to use a store reading an unit test in Spring Cloud Stream.
The code below is how I applied that solution to my code.
The transformer uses the Store provided by Spring bean method "myStore"
#SpringBootApplication
public class KafkaStreamTestApplication {
public static final String MY_STORE_NAME = "my-store";
public static void main(String[] args) {
SpringApplication.run(KafkaStreamTestApplication.class, args);
}
#Bean
public Function<KStream<String, String>, KStream<String, String>> process2(){
return (input) -> input.
transformValues(() -> new MyValueTransformer(), MY_STORE_NAME);
}
#Bean
public StoreBuilder<?> myStore() {
return Stores.keyValueStoreBuilder(
Stores.persistentKeyValueStore(MY_STORE_NAME), Serdes.String(),
Serdes.String());
}
}
public class MyValueTransformer implements ValueTransformer<String, String> {
private KeyValueStore<String,String> store;
private ProcessorContext context;
#Override
public void init(ProcessorContext context) {
this.context = context;
store = (KeyValueStore<String, String>) this.context.getStateStore(KafkaStreamTestApplication.MY_STORE_NAME);
}
#Override
public String transform(String value) {
String tValue = store.get(value);
if(tValue==null) {
store.put(value, value.toUpperCase());
}
return tValue;
}
#Override
public void close() {
if(store!=null) {
store.close();
}
}
}
Related
Just wanted to make sure that whether messages are processed in correct way or not. When the message gets received at listener, it will be always processed by a new thread( defined the processor bean as prototype). is this implementation correct ? (i have Considered the listener is not thread safe, so for this reason the prototype scope of bean to process the message has been used)
(Input : TestTopic- 5 partitions - 1 consumer) or (Input : TestTopic- 5 partitions - 5 consumers)
public class EventListener {
#Autowired
private EventProcessor eventProcessor;
#KafkaListener(topics = "TestTopic", containerFactory = "kafkaListenerContainerFactory",
autoStartup = "true")
public void onMessage(
#Payload List<ConsumerRecord<String, String>> consumerRecords, Acknowledgment acknowledgment) {
eventProcessor.processAndAcknowledgeBatchMessages(consumerRecords, acknowledgment);
}
}
//event processor
#Slf4j
#Component
#Scope(ConfigurableBeanFactory.SCOPE_PROTOTYPE)
#NoArgsConstructor
#SuppressWarnings("unused")
public class EventProcessorImpl implements EventProcessor {
#Autowired
private KafkaProducerTemplate kafkaProducerTemplate;
#Autowired
private ObjectMapper localObjectMapper;
#Autowired
private Dao dao;
public void processAndAcknowledgeBatchMessages(
List<ConsumerRecord<String, String>> consumerRecords, Acknowledgment acknowledgment) {
long start = System.currentTimeMillis();
consumerRecords.forEach( consumerRecord -> {
Event event = localObjectMapper.readValue(consumerRecord.value(), Event.class);
dao.save(process(event));
});
acknowledgment.acknowledge();
}
}
No it is not correct; you should not execute on another thread; it will cause problems with committing offsets and error handling.
Also, making the EventProcessorImpl a prototype bean won't help. That just means a new instance is used each time the bean is referenced.
Since it is #Autowired it is only referenced once, during initialization. To get a new instance for each request, you would need to call getBean() on the application context each time.
It is better to make your code thread-safe.
EDIT
There are (at least) a couple of ways to deal with a not thread-safe service defined in prototype scope.
Use a ThreadLocal:
#SpringBootApplication
public class So68447863Application {
public static void main(String[] args) {
SpringApplication.run(So68447863Application.class, args);
}
private static final ThreadLocal<NotThreadSafeService> SERVICES = new ThreadLocal<>();
#Autowired
ApplicationContext context;
#KafkaListener(id = "so68447863", topics = "so68447863", concurrency = "5")
void listen(String in) {
NotThreadSafeService service = SERVICES.get();
if (service == null) {
service = this.context.getBean(NotThreadSafeService.class);
SERVICES.set(service);
}
service.process(in);
}
#EventListener
void removeService(ConsumerStoppedEvent event) {
System.out.println("Consumer stopped; removing TL");
SERVICES.remove();
}
#Bean
NewTopic topic() {
return TopicBuilder.name("so68447863").partitions(10).replicas(1).build();
}
#Bean
#Scope(ConfigurableBeanFactory.SCOPE_PROTOTYPE)
NotThreadSafeService service() {
return new NotThreadSafeService();
}
}
class NotThreadSafeService {
void process(String msg) {
System.out.println(msg + " processed by " + this);
}
}
Use a pool of instances.
#SpringBootApplication
public class So68447863Application {
public static void main(String[] args) {
SpringApplication.run(So68447863Application.class, args);
}
private static final BlockingQueue<NotThreadSafeService> SERVICES = new LinkedBlockingQueue<>();
#Autowired
ApplicationContext context;
#KafkaListener(id = "so68447863", topics = "so68447863", concurrency = "5")
void listen(String in) {
NotThreadSafeService service = SERVICES.poll();
if (service == null) {
service = this.context.getBean(NotThreadSafeService.class);
}
try {
service.process(in);
}
finally {
SERVICES.add(service);
}
}
#Bean
NewTopic topic() {
return TopicBuilder.name("so68447863").partitions(10).replicas(1).build();
}
#Bean
#Scope(ConfigurableBeanFactory.SCOPE_PROTOTYPE)
NotThreadSafeService service() {
return new NotThreadSafeService();
}
}
class NotThreadSafeService {
void process(String msg) {
System.out.println(msg + " processed by " + this);
}
}
I am trying to write kafka consumer using spring-kafka version 2.3.0.M2 library.
To handle run time errors I am using SeekToCurrentErrorHandler.class with DeadLetterPublishingRecoverer as my recoverer. This works fine only when my consumer code throws exception, but fails when unable to deserialize the message.
I tried implementing ErrorHandler myself and I was successful but with this approach I myself end up writing DLT code to handle error messages which I do not want to do.
Below are my kafka properties
spring:
kafka:
consumer:
bootstrap-servers: localhost:9092
group-id: group_id
auto-offset-reset: latest
key-deserializer: org.springframework.kafka.support.serializer.ErrorHandlingDeserializer2
value-deserializer: org.springframework.kafka.support.serializer.ErrorHandlingDeserializer2
properties:
spring.json.trusted.packages: com.mypackage
spring.deserializer.key.delegate.class: org.apache.kafka.common.serialization.StringDeserializer
spring.deserializer.value.delegate.class: org.apache.kafka.common.serialization.StringDeserializer
public ConcurrentKafkaListenerContainerFactory kafkaListenerContainerFactory(
ConcurrentKafkaListenerContainerFactoryConfigurer configurer,
ConsumerFactory<Object, Object> kafkaConsumerFactory,
KafkaTemplate<Object, Object> template) {
ConcurrentKafkaListenerContainerFactory<Object, Object> factory = new ConcurrentKafkaListenerContainerFactory<>();
configurer.configure(factory, kafkaConsumerFactory);
factory.setErrorHandler(new SeekToCurrentErrorHandler(new DeadLetterPublishingRecoverer(template), maxFailures));}
It works fine for me (note that Boot will auto-configure the error handler)...
#SpringBootApplication
public class So56728833Application {
public static void main(String[] args) {
SpringApplication.run(So56728833Application.class, args);
}
#Bean
public SeekToCurrentErrorHandler errorHandler(KafkaTemplate<String, String> template) {
SeekToCurrentErrorHandler eh = new SeekToCurrentErrorHandler(new DeadLetterPublishingRecoverer(template), 3);
eh.setClassifier( // retry for all except deserialization exceptions
new BinaryExceptionClassifier(Collections.singletonList(DeserializationException.class), false));
return eh;
}
#KafkaListener(id = "so56728833"
+ "", topics = "so56728833")
public void listen(Foo in) {
System.out.println(in);
if (in.getBar().equals("baz")) {
throw new IllegalStateException("Test retries");
}
}
#KafkaListener(id = "so56728833dlt", topics = "so56728833.DLT")
public void listenDLT(Object in) {
System.out.println("Received from DLT: " + (in instanceof byte[] ? new String((byte[]) in) : in));
}
#Bean
public NewTopic topic() {
return TopicBuilder.name("so56728833").partitions(1).replicas(1).build();
}
#Bean
public NewTopic dlt() {
return TopicBuilder.name("so56728833.DLT").partitions(1).replicas(1).build();
}
public static class Foo {
private String bar;
public Foo() {
super();
}
public Foo(String bar) {
this.bar = bar;
}
public String getBar() {
return this.bar;
}
public void setBar(String bar) {
this.bar = bar;
}
#Override
public String toString() {
return "Foo [bar=" + this.bar + "]";
}
}
}
spring:
kafka:
consumer:
auto-offset-reset: earliest
enable-auto-commit: false
key-deserializer: org.springframework.kafka.support.serializer.ErrorHandlingDeserializer2
value-deserializer: org.springframework.kafka.support.serializer.ErrorHandlingDeserializer2
properties:
spring.json.trusted.packages: com.example
spring.deserializer.key.delegate.class: org.springframework.kafka.support.serializer.JsonDeserializer
spring.deserializer.value.delegate.class: org.springframework.kafka.support.serializer.JsonDeserializer
spring.json.value.default.type: com.example.So56728833Application$Foo
producer:
key-serializer: org.springframework.kafka.support.serializer.JsonSerializer
value-serializer: org.springframework.kafka.support.serializer.JsonSerializer
logging:
level:
org.springframework.kafka: trace
I have 3 records in the topic:
"badJSON"
"{\"bar\":\"baz\"}"
"{\"bar\":\"qux\"}"
I see the first one going directly to the DLT, and the second one goes there after 3 attempts.
This is the Application.java class
#Configuration
#SpringBootApplication
//#EnableGlobalMethodSecurity(prePostEnabled=true)
#EnableTransactionManagement
#EnableCaching
#EnableJpaRepositories(basePackages="om.gov.moh.irs.dao.repos",repositoryImplementationPostfix="CustomImpl")
public class Application extends SpringBootServletInitializer {
#Autowired
Environment env;
#Bean
#ConfigurationProperties("spring.datasource")
public ComboPooledDataSource dataSource() {
return new ComboPooledDataSource();
}
#Override
protected SpringApplicationBuilder configure(SpringApplicationBuilder application) {
return application.sources(Application.class);
}
public static void main(String[] args) {
SpringApplication.run(Application.class, args);
}
#Bean
public ResourceBundleMessageSource messageSource() {
ResourceBundleMessageSource source = new ResourceBundleMessageSource();
source.setBasenames("messages"); // name of the resource bundle
source.setUseCodeAsDefaultMessage(true);
return source;
}
#Bean
MultipartConfigElement multipartConfigElement() {
MultipartConfigFactory factory = new MultipartConfigFactory();
factory.setMaxFileSize(env.getProperty("multipart.maxFileSize"));
factory.setMaxRequestSize(env.getProperty("multipart.maxRequestSize"));
return factory.createMultipartConfig();
}
}
This is the dto class which Im trying to cache.
public class PaginationDto implements Serializable {
private static final long serialVersionUID = 1L;
public Integer totalRecords;
public List<?> paginatedList;
//constructor and getter setter here
}
Controller class
#RequestMapping(value="categories", method=RequestMethod.POST, produces = { "application/json" })
public #ResponseBody ResultDecorator searchCategories(#RequestBody CategoryDto categoryDto) int pageSize, #RequestParam("sort") String sortOrder) throws BusinessException{
return handler.resolveResult(categoryService.searchCategory(categoryDto), OperationEnum.SEARCH);
}
Service class where #cacheable is defined.
#Override
#Cacheable("category")
public PaginationDto searchCategory(CategoryDto categoryDto) throws CategoryException {
System.out.println("#######category");
PaginationDto paginationDtoResponse = null;
paginationDtoResponse = categoryRepoCustom.fetchCategories(categoryDto);
return paginationDtoResponse;
}
Defined ehcache.xml file
<?xml version="1.0" encoding="UTF-8"?>
<ehcache xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="http://www.ehcache.org/ehcache.xsd"
updateCheck="true" monitoring="autodetect" dynamicConfig="true">
<cache name="category"
maxElementsInMemory="100" eternal="false"
overflowToDisk="false"
timeToLiveSeconds="30000" timeToIdleSeconds="0"
memoryStoreEvictionPolicy="LFU" transactionalMode="off">
</cache>
</ehcache>
On hitting this http://localhost:9190/isa/categories each time console log is printing, which means data is not fetching from the cache.
I have a simple resource like:
#Path("/")
public class RootResource {
#Context WebConfig wc;
#PostConstruct
public void init() {
assertNotNull(wc);
}
#GET
public void String method() {
return "Hello\n";
}
}
Which I am trying to use with JerseyTest (2.x, not 1.x) and the GrizzlyTestContainerFactory.
I can't work out what I need to do in terms of config to get the WebConfig object injected.
I solved this issue by creating a subclass of GrizzlyTestContainerFactory and explicitly loading the Jersey servlet. This triggers the injection of the WebConfig object. The code looks like this:
public class ExtendedGrizzlyTestContainerFactory implements TestContainerFactory {
private static class GrizzlyTestContainer implements TestContainer {
private final URI uri;
private final ApplicationHandler appHandler;
private HttpServer server;
private static final Logger LOGGER = Logger.getLogger(GrizzlyTestContainer.class.getName());
private GrizzlyTestContainer(URI uri, ApplicationHandler appHandler) {
this.appHandler = appHandler;
this.uri = uri;
}
#Override
public ClientConfig getClientConfig() {
return null;
}
#Override
public URI getBaseUri() {
return uri;
}
#Override
public void start() {
if (LOGGER.isLoggable(Level.INFO)) {
LOGGER.log(Level.INFO, "Starting GrizzlyTestContainer...");
}
try {
this.server = GrizzlyHttpServerFactory.createHttpServer(uri, appHandler);
// Initialize and register Jersey Servlet
WebappContext context = new WebappContext("WebappContext", "");
ServletRegistration registration = context.addServlet("ServletContainer", ServletContainer.class);
registration.setInitParameter("javax.ws.rs.Application",
appHandler.getConfiguration().getApplication().getClass().getName());
// Add an init parameter - this could be loaded from a parameter in the constructor
registration.setInitParameter("myparam", "myvalue");
registration.addMapping("/*");
context.deploy(server);
} catch (ProcessingException e) {
throw new TestContainerException(e);
}
}
#Override
public void stop() {
if (LOGGER.isLoggable(Level.INFO)) {
LOGGER.log(Level.INFO, "Stopping GrizzlyTestContainer...");
}
this.server.stop();
}
}
#Override
public TestContainer create(URI baseUri, ApplicationHandler application) throws IllegalArgumentException {
return new GrizzlyTestContainer(baseUri, application);
}
Notice that the Jersey Servlet configuration is being loaded from the ApplicationHandler that is passed in as a parameter using the inner Application object's class name (ResourceConfig is a subclass of Application). Therefore, you also need to create a subclass of ResourceConfig for this approach to work. The code for this is very simple:
package com.example;
import org.glassfish.jersey.server.ResourceConfig;
public class MyResourceConfig extends ResourceConfig {
public MyResourceConfig() {
super(MyResource.class);
}
}
This assumes the resource you are testing is MyResource. You also need to override a couple of methods in your test like this:
public class MyResourceTest extends JerseyTest {
public MyResourceTest() throws TestContainerException {
}
#Override
protected Application configure() {
return new MyResourceConfig();
}
#Override
protected TestContainerFactory getTestContainerFactory() throws TestContainerException {
return new ExtendedGrizzlyTestContainerFactory();
}
#Test
public void testCreateSimpleBean() {
final String beanList = target("test").request().get(String.class);
Assert.assertNotNull(beanList);
}
}
Finally, for completeness, here is the code for MyResource:
#Path("test")
public class MyResource {
#Context WebConfig wc;
#PostConstruct
public void init() {
System.out.println("WebConfig: " + wc);
String url = wc.getInitParameter("myparam");
System.out.println("myparam = "+url);
}
#GET
#Produces(MediaType.APPLICATION_JSON)
public Collection<TestBean> createSimpleBean() {
Collection<TestBean> res = new ArrayList<TestBean>();
res.add(new TestBean("a", 1, 1L));
res.add(new TestBean("b", 2, 2L));
return res;
}
#POST
#Produces(MediaType.APPLICATION_JSON)
#Consumes(MediaType.APPLICATION_JSON)
public TestBean roundTrip(TestBean s) {
return s;
}
}
The output of running the test shows that the WebConfig is loaded and the init param is now available:
WebConfig: org.glassfish.jersey.servlet.WebServletConfig#107d0f44
myparam = myvalue
The solution from #ametke worked well but wasn't picking up my ExceptionMapper classes. To solve this I simplified the start() method to:
#Override
public void start() {
try {
initParams.put("jersey.config.server.provider.packages", "my.resources;my.config");
this.server = GrizzlyWebContainerFactory.create(uri, initParams);
} catch (ProcessingException | IOException e) {
throw new TestContainerException(e);
}
}
This was based on Problems running JerseyTest when dealing with HttpServletResponse
I create form and controller this form have some validation constrains using Hibernate validator. I face problem when starting test the validation constrains but I got Blue Exception page with the attributemodel with the rejected.
This the configuration
#Configuration
#ComponentScan(basePackages = {"com.whatever.core.web"})
#EnableWebMvc
public class WebMvcConfig extends WebMvcConfigurationSupport {
private static final String MESSAGE_SOURCE = "/WEB-INF/classes/messages";
private static final String TILES = "/WEB-INF/tiles/tiles.xml";
private static final String VIEWS = "/WEB-INF/views/**/views.xml";
private static final String RESOURCES_HANDLER = "/resources/";
private static final String RESOURCES_LOCATION = RESOURCES_HANDLER + "**";
#Override
public RequestMappingHandlerMapping requestMappingHandlerMapping() {
RequestMappingHandlerMapping requestMappingHandlerMapping = super.requestMappingHandlerMapping();
requestMappingHandlerMapping.setUseSuffixPatternMatch(false);
requestMappingHandlerMapping.setUseTrailingSlashMatch(false);
return requestMappingHandlerMapping;
}
public void configureMessageConverters(List<HttpMessageConverter<?>> converters) {
converters.add(new MappingJacksonHttpMessageConverter());
}
#Bean(name = "messageSource")
public MessageSource configureMessageSource() {
ReloadableResourceBundleMessageSource messageSource = new ReloadableResourceBundleMessageSource();
messageSource.setBasename(MESSAGE_SOURCE);
messageSource.setCacheSeconds(5);
return messageSource;
}
#Bean
public TilesViewResolver configureTilesViewResolver() {
return new TilesViewResolver();
}
#Bean
public TilesConfigurer configureTilesConfigurer() {
TilesConfigurer configurer = new TilesConfigurer();
configurer.setDefinitions(new String[] {TILES, VIEWS});
return configurer;
}
#Override
public Validator getValidator() {
LocalValidatorFactoryBean validator = new LocalValidatorFactoryBean();
validator.setValidationMessageSource(configureMessageSource());
return validator;
}
#Override
public void addResourceHandlers(ResourceHandlerRegistry registry) {
registry.addResourceHandler(RESOURCES_HANDLER).addResourceLocations(RESOURCES_LOCATION);
}
#Override
public void configureDefaultServletHandling(DefaultServletHandlerConfigurer configurer) {
configurer.enable();
}
}
and the controller here
if(result.hasErrors()){
return null; OR "view name"
}
User user = new User();
user.setUsername(userModel.getUsername());
user.setFirstName(userModel.getFirstName());
user.setLastName(userModel.getLastName());
user.setGender(userModel.getGender());
user.setLocation(userModel.getLocation());
user.setPassword(passwordEncoder.encodePassword(userModel.getPassword(),null));
userRepository.save(user);
doAutoLogin(userModel.getUsername(),userModel.getPassword(),request);
return "redirect:/home";
NOTE: I use springMVC, spring security, tiles, and hibernate validator
I used SpringMVC with hibernate validator with XML configuration and portal environment and work fine I don't know what the wrong here!!
I Found the issue! the signature of the method controller should be like this
public String signup(#ModelAttribute("userModel") #Valid SignupForm userModel,BindingResult result,HttpServletRequest request,HttpServletResponse response,ModelMap model)
as what I read in sprinsource forum, the BindingResult should follow the modelAttribute and work find. I didn't find any official documentation for this but its work now.
to see the thread of springsource forum check this link http://forum.springsource.org/showthread.php?85815-BindException-Thrown-on-bind-errors-(instead-of-returning-errors-to-controller-method