I'm setting up contract tests for Kafka messaging with Test Containers in a way described in spring-cloud-contract-samples/producer_kafka_middleware/. Works good with Embedded Kafka but not with TestContainers.
When I try to run the generated ContractVerifierTest:
public void validate_shouldProduceKafkaMessage() throws Exception {
// when:
triggerMessageSent();
// then:
ContractVerifierMessage response = contractVerifierMessaging.receive("kafka-messages",
contract(this, "shouldProduceKafkaMessage.yml"));
Cannot invoke "org.springframework.messaging.Message.getPayload()" because "receive" is null
is thrown
Kafka container is running, the topic is created. When debugging receive method I see the message is null in the message(destination);
Contract itself:
label("triggerMessage")
input {
triggeredBy("triggerMessageSent()")
}
outputMessage {
sentTo "kafka-messages"
body(file("kafkaMessage.json"))
Base test configuration:
#SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.NONE, classes = {TestConfig.class, ServiceApplication.class})
#Testcontainers
#AutoConfigureMessageVerifier
#ActiveProfiles("test")
public abstract class BaseClass {
What am I missing? Maybe a point of communication between the container and ContractVerifierMessage methods?
Resolved the issue by adding a specific topic name to listen() method in KafkaMessageVerifier implementation class.
So instead of #KafkaListener(id = "listener", topicPattern = ".*"), it works with:
#KafkaListener(topics = {"my-messages-topic"})
public void listen(ConsumerRecord payload, #Header(KafkaHeaders.RECEIVED_TOPIC)
Related
I am developing spring kafka consumer. Due to message volume, I need use concurrency to make sure throughput. Due to used concurrency, I used threadlocal object to save thread based data. Now I need remove this threadlocal object after use it.
Spring document with below links suggested to implement a EventListener which listen to event ConsumerStoppedEvent . But did not mention any sample eventlistener code to get threadlocal object and remove the value. May you please let me know how to get the threadlocal instance in this case?
Code samples will be appreciated.
https://docs.spring.io/spring-kafka/docs/current/reference/html/#thread-safety
Something like this:
#SpringBootApplication
public class So71884752Application {
public static void main(String[] args) {
SpringApplication.run(So71884752Application.class, args);
}
#Bean
public NewTopic topic2() {
return TopicBuilder.name("topic1").partitions(2).build();
}
#Component
static class MyListener implements ApplicationListener<ConsumerStoppedEvent> {
private static final ThreadLocal<Long> threadLocalState = new ThreadLocal<>();
#KafkaListener(topics = "topic1", groupId = "my-consumer", concurrency = "2")
public void listen() {
long id = Thread.currentThread().getId();
System.out.println("set thread id to ThreadLocal: " + id);
threadLocalState.set(id);
}
#Override
public void onApplicationEvent(ConsumerStoppedEvent event) {
System.out.println("Remove from ThreadLocal: " + threadLocalState.get());
threadLocalState.remove();
}
}
}
So, I have two concurrent listener containers for those two partitions in the topic. Each of them is going to call this my #KafkaListener method anyway. I store the thread id into the ThreadLocal. For simple use-case and testing the feature.
The I implement ApplicationListener<ConsumerStoppedEvent> which is emitted in the appropriate consumer thread. And that one helps me to extract ThreadLocal value and clean it up in the end of consumer life.
The test against embedded Kafka looks like this:
#SpringBootTest
#EmbeddedKafka(bootstrapServersProperty = "spring.kafka.bootstrap-servers")
#DirtiesContext
class So71884752ApplicationTests {
#Autowired
KafkaTemplate<String, String> kafkaTemplate;
#Autowired
KafkaListenerEndpointRegistry kafkaListenerEndpointRegistry;
#Test
void contextLoads() throws InterruptedException {
this.kafkaTemplate.send("topic1", "1", "foo");
this.kafkaTemplate.send("topic1", "2", "bar");
this.kafkaTemplate.flush();
Thread.sleep(1000); // Give it a chance to consume data
this.kafkaListenerEndpointRegistry.stop();
}
}
Right. It doesn't verify anything, but it demonstrate how that event can happen.
I see something like this in log output:
set thread id to ThreadLocal: 125
set thread id to ThreadLocal: 127
...
Remove from ThreadLocal: 125
Remove from ThreadLocal: 127
So, whatever that doc says is correct.
Is it possible to get a hold of underlying KafkaConsumer bean for a defined PollableMessageSource?
I have Binding defined as:
public interface TestBindings {
String TEST_SOURCE = "test";
#Input(TEST_SOURCE)
PollableMessageSource testTopic();
}
and config class:
#EnableBinding(TestBindings.class)
public class TestBindingsPoller {
#Bean
public ApplicationRunner testPoller(PollableMessageSource testTopic) {
// Get kafka consumer for PollableMessageSource
KafkaConsumer kafkaConsumer = getConsumer(testTopic);
return args -> {
while (true) {
if (!testTopic.poll(...) {
Thread.sleep(500);
}
}
};
}
}
The question is, how can I get KafkaConsumer that corresponds to testTopic? Is there any way to get it from beans that are wired in spring cloud stream?
The KafkaMessageSource populates a KafkaConsumer into headers, so it is available in the place you receive messages: https://github.com/spring-projects/spring-kafka/blob/master/spring-kafka/src/main/java/org/springframework/kafka/support/converter/MessageConverter.java#L57.
If you are going to do stuff like poll yourself, I would suggest to inject a ConsumerFactory and use a consumer from there already.
I have a Flink Job reading events from a Kafka queue then calling another service if certain conditions are met.
I wanted to use Retrofit2 to call the REST endpoint of that service but I get a is not Serializable Exception. I have several Flat Maps connected to each other (in series) then calling the service happens in the last FlatMap. The exception I get:
Exception in thread "main"
org.apache.flink.api.common.InvalidProgramException: The
implementation of the RichFlatMapFunction is not serializable. The
object probably contains or references non serializable fields.
...
Caused by: java.io.NotSerializableException: retrofit2.Retrofit$1
...
The way I am initializing retrofit:
RetrofitClient.getClient(BASE_URL).create(NotificationService.class);
And the NotificationService interface
public interface NotificationService {
#PUT("/test")
Call<String> putNotification(#Body Notification notification);
}
The RetrofitClient class
public class RetrofitClient {
private static Retrofit retrofit = null;
public static Retrofit getClient(String baseUrl) {
if (retrofit == null) {
retrofit = new Retrofit.Builder().baseUrl(baseUrl).addConverterFactory(GsonConverterFactory.create())
.build();
}
return retrofit;
}
Put your Notification class code for more details, but looks like this answer helps
java.io.NotSerializableException with "$1" after class
I was trying to verify whether my log warning message is written via NUnit mocking. I am getting this error message :
An exception of type 'System.NotSupportedException' occurred in Moq.dll but was not handled in user code
Additional information: Invalid verify on a non-virtual (overridable in VB) member: m => m.LogWarning(String.Format("comments not found for part number :{0}", (Object)0), new[] { "111" })
code:
mockLogger.Verify(m => m.LogWarning($"comments not found for part number :{0}", "111"), Times.Exactly(1));
This is happening because NUnit mocking framework does not support extension methods. A few people on stack overflow have suggested to use Log method instead of level wise methods.
What am I missing?
Firstly, you don't need the $ at the start of the string. That's for string interpolation. The LogWarning message is doing a string.format, hence the {0}
Mock frameworks cannot directly mock static methods. The problem in your case is the LogWarning method - that is the static (extension) method.
The simplest way of overcoming this issue is by using a wrapper class. Here's how I got it, in your case.
Firstly I created an interface
public interface IMyLogWarning
{
void LogWarning(string msg, params object[] args);
}
Then I created a class which implements that interface
public class MyLogWarning<T> : IMyLogWarning where T : class
{
private readonly ILogger _logger;
public MyLogWarning(ILogger<T> logger)
{
// Using constructor for DI
_logger = logger;
}
public void LogWarning(string msg, params object[] args)
{
_logger.LogWarning(msg, args);
}
}
The reason for these two is that I'll use these in my code as well as the unit test.
The constructor in the class is setup so it can be populated using dependency injection, something like this in your ConfigureServices method. Feel free to change this; was a quick stab at it on my part.
services.AddTransient<IMyLogWarning, MyLogWarning<MyViewModel>>();
You can then create a unit test that's roughly like this
[Test]
public void LoggingTest_LogAMessage_ConfirmedLogWasRun()
{
// TODO - add the rest of your test code
// Arrange
var warningMsg = "comments not found for part number :{0}";
var partNumber = "111";
var mockLogger = new Mock<IMyLogWarning>();
// Act
mockLogger.Object.LogWarning(warningMsg, partNumber);
// Assert
mockLogger.Verify(m => m.LogWarning(warningMsg, partNumber), Times.Exactly(1));
}
Sample application for this question is here: https://github.com/olemerdy-fa/webmvctest
I am bootstrapping a new project using Spring Boot 1.4.1. I try to leverage the new features from this great framework, especially the ability to (quite-)unit test 'slices' of my application.
I am now struggled with the #WebMvcTest feature when used on a #Controller declaring a #PathVariable annotated method.
Indeed, a #WebMvcTest is supposed to bootstrap a single controller and the MockMvc testing facility, without providing anything else. Using #MockBean, it's still quite easy to provide mocks as dependencies to inject inside this controller.
But what about a #PathVariable annotated parameter whose type is, say, a JPA Entity whose converter is usually registered by Spring Data?
The sample project joined to this question contains a few samples:
MyEntity is a simple JPA entity and MyEntityRepository its Spring Data associated repository
Webmvctest1Controller has a load method retrieving the id from the path and calls itself the MyEntityRepository.findOne(id) method
Webmvctest1ControllerUnitTest tests this controller by mocking MyEntityRepository and everything just goes well
Webmvctest2Controller has a load method with a #PathVariable annotated MyEntity which is looked up by Spring Data registered converter
#RestController
public class Webmvctest2Controller {
#RequestMapping("load2/{id}")
public MyEntity load2(#PathVariable("id") MyEntity myEntity) {
return myEntity;
}
}
Webmvctest2ControllerUnitTest is where I'm stuck, as I do not know how to provide a mock entity as the parameter while still using MockMvc
#RunWith(SpringRunner.class)
#WebMvcTest(Webmvctest2Controller.class)
public class Webmvctest2ControllerUnitTest {
#Autowired
private MockMvc mvc;
#Test
public void load2() throws Exception {
// How do I mock converter to PathVariable here?
mvc.perform(get("/load2/123").accept(MediaType.APPLICATION_JSON))
.andExpect(status().isOk())
.andExpect(content().json("{id:123,name:'My Entity 123'}"));
}
}
This fails with a org.springframework.web.method.annotation.MethodArgumentConversionNotSupportedException exception
2016-10-25 14:27:55.699 WARN 20753 --- [ main] .w.s.m.s.DefaultHandlerExceptionResolver : Failed to convert request element: org.springframework.web.method.annotation.MethodArgumentConversionNotSupportedException: Failed to convert value of type [java.lang.String] to required type [com.stackoverflow.MyEntity]; nested exception is java.lang.IllegalStateException: Cannot convert value of type [java.lang.String] to required type [com.stackoverflow.MyEntity]: no matching editors or conversion strategy found
MockHttpServletRequest:
HTTP Method = GET
Request URI = /load2/123
Parameters = {}
Headers = {Accept=[application/json]}
Handler:
Type = com.stackoverflow.Webmvctest2Controller
Method = public com.stackoverflow.MyEntity com.stackoverflow.Webmvctest2Controller.load2(com.stackoverflow.MyEntity)
Async:
Async started = false
Async result = null
Resolved Exception:
Type = org.springframework.web.method.annotation.MethodArgumentConversionNotSupportedException
ModelAndView:
View name = null
View = null
Model = null
FlashMap:
Attributes = null
MockHttpServletResponse:
Status = 500
Error message = null
Headers = {}
Content type = null
Body =
Forwarded URL = null
Redirected URL = null
Cookies = []
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.375 sec <<< FAILURE! - in com.stackoverflow.Webmvctest2ControllerUnitTest
load2(com.stackoverflow.Webmvctest2ControllerUnitTest) Time elapsed: 0.015 sec <<< FAILURE!
java.lang.AssertionError: Status expected:<200> but was:<500>
at org.springframework.test.util.AssertionErrors.fail(AssertionErrors.java:54)
at org.springframework.test.util.AssertionErrors.assertEquals(AssertionErrors.java:81)
at org.springframework.test.web.servlet.result.StatusResultMatchers$10.match(StatusResultMatchers.java:664)
at org.springframework.test.web.servlet.MockMvc$1.andExpect(MockMvc.java:171)
at com.stackoverflow.Webmvctest2ControllerUnitTest.load2(Webmvctest2ControllerUnitTest.java:28)
WebmvctestApplicationTests shows that everything is fine in both cases when the application is fully bootstrapped
Any idea of how I could keep my #PathVariable entity parameters while still only testing my web slice with #WebMvcTest?
Thanks :)
Thanks to #zeroflagL hinting about possible contributions to HandlerMethodArgumentResolver, I've come up with a solution that seems to fit my need. Contributing a custom Converter handling the mock entity injection in the controller seems to do the job
#RunWith(SpringRunner.class)
#WebMvcTest(Webmvctest2Controller.class)
public class Webmvctest2ControllerUnitTest {
#Autowired
private MockMvc mvc;
#Test
public void load2() throws Exception {
mvc.perform(get("/load2/123").accept(MediaType.APPLICATION_JSON))
.andExpect(status().isOk())
.andExpect(content().json("{id:123,name:'My Entity'}"));
}
#TestConfiguration
static class InternalConfig {
#Bean
WebMvcConfigurer configurer() {
return new WebMvcConfigurerAdapter() {
#Override
public void addFormatters(FormatterRegistry registry) {
registry.addConverter(String.class, MyEntity.class, id -> {
if (id.equals("123")) {
MyEntity myEntity = new MyEntity(123);
myEntity.setName("My Entity");
return myEntity;
}
throw new IllegalArgumentException();
});
}
};
}
}
}
It is not perfect, as the mock Entity cannot be provided by the test method itself but it is still allows the web-unit-test bootstrap and run I wanted to keep.