I have difficulty understanding some Kafka concepts in Java Spring Boot. I’d like to test a consumer against a real Kafka broker running on a server, which has some producers that write / have already written data to various topics. I would like to establish a connection with the server, consume the data, and verify or process its content in a test.
An enormous majority of examples (actually all I have seen so far) in the internet refer to embedded kafka, EmbeddedKafkaBroker, and show both a producer and a consumer implemented on one machine, locally. I haven’t found any example that would explain how to make a connection with a remote kafka server and read data from a particular topic.
I've written some code and I've printed the broker address with:
System.out.println(embeddedKafkaBroker.getBrokerAddress(0));
What I got is 127.0.0.1:9092, which means that it is local, so the connection with the remote server has not been established.
On the other hand, when I run the SpringBootApplication I get the payload from the remote broker.
Receiver:
#Component
public class Receiver {
private static final String TOPIC_NAME = "X";
private static final Logger LOGGER = LoggerFactory.getLogger(Receiver.class);
private CountDownLatch latch = new CountDownLatch(1);
public CountDownLatch getLatch() {
return latch;
}
#KafkaListener(topics = TOPIC_NAME)
public void receive(final byte[] payload) {
LOGGER.info("received the following payload: '{}'", payload);
latch.countDown();
}
}
Config:
#EnableKafka
#Configuration
public class ByteReceiverConfig {
#Autowired
EmbeddedKafkaBroker kafkaEmbeded;
#Value("${spring.kafka.bootstrap-servers}")
private String bootstrapServers;
#Value("${spring.kafka.consumer.group-id}")
private String groupIdConfig;
#Bean
public KafkaListenerContainerFactory<?> kafkaListenerContainerFactory() {
final ConcurrentKafkaListenerContainerFactory<Object, Object> factory =
new ConcurrentKafkaListenerContainerFactory<>();
factory.setConsumerFactory(consumerFactory());
return factory;
}
#Bean
ConsumerFactory<Object, Object> consumerFactory() {
return new DefaultKafkaConsumerFactory<>(consumerProperties());
}
#Bean
Map<String, Object> consumerProperties() {
final Map<String, Object> properties =
KafkaTestUtils.consumerProps("junit-test", "true", this.kafkaEmbeded);
properties.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
properties.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, ByteArrayDeserializer.class);
properties.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, ByteArrayDeserializer.class);
properties.put(ConsumerConfig.GROUP_ID_CONFIG, groupIdConfig);
return properties;
}
Test:
#EnableAutoConfiguration
#EnableKafka
#SpringBootTest(classes = {ByteReceiverConfig.class, Receiver.class})
#EmbeddedKafka
#ContextConfiguration(classes = ByteReceiverConfig.class)
#TestPropertySource(properties = { "spring.kafka.bootstrap-servers=${spring.embedded.kafka.brokers}",
"spring.kafka.consumer.group-id=EmbeddedKafkaTest"})
public class KafkaTest {
#Autowired
private KafkaListenerEndpointRegistry kafkaListenerEndpointRegistry;
#Autowired
EmbeddedKafkaBroker embeddedKafkaBroker;
#Autowired
Receiver receiver;
#BeforeEach
void waitForAssignment() {
for (MessageListenerContainer messageListenerContainer : kafkaListenerEndpointRegistry.getListenerContainers()) {
System.out.println(messageListenerContainer.getAssignedPartitions().isEmpty());
System.out.println(messageListenerContainer.toString());
System.out.println(embeddedKafkaBroker.getTopics().size());
System.out.println(embeddedKafkaBroker.getPartitionsPerTopic());
System.out.println(embeddedKafkaBroker.getBrokerAddress(0));
System.out.println(embeddedKafkaBroker.getBrokersAsString());
ContainerTestUtils.waitForAssignment(messageListenerContainer,
embeddedKafkaBroker.getPartitionsPerTopic());
}
#Test
public void testReceive() {
}
}
I would like somebody to shed some light on the following issues:
1.Can an instance of the class EmbeddedKafkaBroker be used to test data that comes from a remote broker, or is it only used for local tests, in which I would procude i.e send data to a topic that I created and consume data myself?
2.Is it possible to write a test class for a real kafka server? For instance to verify if a connection has been establish, or if a data has been read from a specific topic. What annotations, configurations, and classes would be needed in such case?
3.If I only want to consume data, do I have to provide the producer configuration in a config file (it would be strange, but all examples I have encountered so far did it)?
4.Do you know any resources (books, websites etc.) that show real examples of using kafka i.e. with a remote kafka server, with a procuder or a consumer only?
You don't need an embedded broker at all if you want to talk to an external broker only.
Yes, just set the bootstrap servers property appropriately.
No, you don't need producer configuration.
EDIT
#SpringBootApplication
public class So56044105Application {
public static void main(String[] args) {
SpringApplication.run(So56044105Application.class, args);
}
#Bean
public NewTopic topic() {
return new NewTopic("so56044105", 1, (short) 1);
}
}
spring.kafka.bootstrap-servers=10.0.0.8:9092
spring.kafka.consumer.enable-auto-commit=false
#RunWith(SpringRunner.class)
#SpringBootTest(classes = { So56044105Application.class, So56044105ApplicationTests.Config.class })
public class So56044105ApplicationTests {
#Autowired
public Config config;
#Test
public void test() throws InterruptedException {
assertThat(config.latch.await(10, TimeUnit.SECONDS)).isTrue();
assertThat(config.received.get(0)).isEqualTo("foo");
}
#Configuration
public static class Config implements ConsumerSeekAware {
List<String> received = new ArrayList<>();
CountDownLatch latch = new CountDownLatch(3);
#KafkaListener(id = "so56044105", topics = "so56044105")
public void listen(String in) {
System.out.println(in);
this.received.add(in);
this.latch.countDown();
}
#Override
public void registerSeekCallback(ConsumerSeekCallback callback) {
}
#Override
public void onPartitionsAssigned(Map<TopicPartition, Long> assignments, ConsumerSeekCallback callback) {
System.out.println("Seeking to beginning");
assignments.keySet().forEach(tp -> callback.seekToBeginning(tp.topic(), tp.partition()));
}
#Override
public void onIdleContainer(Map<TopicPartition, Long> assignments, ConsumerSeekCallback callback) {
}
}
}
There are some examples in this repository for bootstrapping real Kafka producers and consumers across a variety of configurations — plaintext, SSL, with and without authentication, etc.
Note: the repo above contains examples for the Effective Kafka book, which I am the author of. However, they can be used freely without the book and hopefully they make just as much sense on their own.
More to the point, here are a pair of examples for a basic producer and a consumer.
/** A sample Kafka producer. */
import static java.lang.System.*;
import java.util.*;
import org.apache.kafka.clients.producer.*;
import org.apache.kafka.common.serialization.*;
public final class BasicProducerSample {
public static void main(String[] args) throws InterruptedException {
final var topic = "getting-started";
final Map<String, Object> config =
Map.of(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092",
ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName(),
ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName(),
ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);
try (var producer = new KafkaProducer<String, String>(config)) {
while (true) {
final var key = "myKey";
final var value = new Date().toString();
out.format("Publishing record with value %s%n",
value);
final Callback callback = (metadata, exception) -> {
out.format("Published with metadata: %s, error: %s%n",
metadata, exception);
};
// publish the record, handling the metadata in the callback
producer.send(new ProducerRecord<>(topic, key, value), callback);
// wait a second before publishing another
Thread.sleep(1000);
}
}
}
}
/** A sample Kafka consumer. */
import static java.lang.System.*;
import java.time.*;
import java.util.*;
import org.apache.kafka.clients.consumer.*;
import org.apache.kafka.common.serialization.*;
public final class BasicConsumerSample {
public static void main(String[] args) {
final var topic = "getting-started";
final Map<String, Object> config =
Map.of(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092",
ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName(),
ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName(),
ConsumerConfig.GROUP_ID_CONFIG, "basic-consumer-sample",
ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest",
ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false);
try (var consumer = new KafkaConsumer<String, String>(config)) {
consumer.subscribe(Set.of(topic));
while (true) {
final var records = consumer.poll(Duration.ofMillis(100));
for (var record : records) {
out.format("Got record with value %s%n", record.value());
}
consumer.commitAsync();
}
}
}
}
Now, these are obviously not unit tests. But with very little rework they could be turned into one. The next step would be to remove Thread.sleep() and add assertions. Note, since Kafka is inherently asynchronous, naively asserting a published message in a consumer immediately after publishing will fail. For a robust, repeatable test, you may want to use something like Timesert.
Related
I have a problem that I can`t create HttpClient for integration tests.
I have ready carefully this article:
https://learn.microsoft.com/en-us/aspnet/core/test/integration-tests?view=aspnetcore-6.0
Microsoft.AspNetCore.Mvc.Testing installed
<Project Sdk="Microsoft.NET.Sdk.Web"> in csproj
CustomWebApplicationFactory:
public class CustomWebApplicationFactory<TStartup>
: WebApplicationFactory<TStartup> where TStartup : class
{
protected override void ConfigureWebHost(IWebHostBuilder builder)
{
Environment.SetEnvironmentVariable("HANGFIRE_DASHBOARD_USERNAME", "test");
Environment.SetEnvironmentVariable("HANGFIRE_DASHBOARD_PASSWORD", "test");
Environment.SetEnvironmentVariable("ASPNETCORE_URLS", "https://+:1229");
base.ConfigureWebHost(builder);
}
}
ControllerIntegrationTests:
public class ControllerIntegrationTests : IClassFixture<CustomWebApplicationFactory<Startup>>
{
private readonly HttpClient client;
private readonly CustomWebApplicationFactory<Startup> factory;
public ControllerIntegrationTests(CustomWebApplicationFactory<Startup> factory)
{
this.factory = factory;
client = this.factory.CreateClient();
}
[Fact]
public async Task TestGetEndpointReturnSuccess()
{
// Arrange
// Act
var response = await client.GetAsync("/Information");
// Assert
response.EnsureSuccessStatusCode();
}
}
Nothing special in my code, just a simple example. As a result in Output I have :
Hosting environment:Development
Now listening on : https://[::]:1229
Application started. Press Ctrl+C to shut down.
It seems like it started the main application. And I cant move to Act in test, because its hanging in CreateClient(). So I can`t finish my test. Whats wrong?
You forget in your CustomWebApplicationFactory to override the CreateHostBuilder.
protected override IHostBuilder CreateHostBuilder()
{
return Host.CreateDefaultBuilder().ConfigureWebHostDefaults(webBuilder =>
{
webBuilder.UseStartup<TStartup>();
});
}
Try to add this method in your Factory class using the TStartup.
I have a stateful service that stores a bunch of data about my users that is stored in a reliable dictionary and obviously also retrieves it from there too.
However, I also have a SQL database that used to store this info. On initialization of a new stateful service instance, I need to migrate that info from my SQL database into the new reliable storage mechanism. From that point on, the stateful service is the source of truth. Ideally, I'd like to delay availability of my stateful service until this initialization process is completed.
Are there any suggestions on an approach for how to do this?
Something like does will do the trick:
public interface IStateful1 : IService
{
Task MyMethod();
}
internal sealed class Stateful1 : StatefulService, IStateful1
{
private bool isReady = false;
public Stateful1(StatefulServiceContext context)
: base(context)
{ }
public Task MyMethod()
{
if(!isReady)
throw new NotImplementedException(); // Probably throw or return something more meaningful :-)
return Task.CompletedTask; // Do your thing here
}
protected override IEnumerable<ServiceReplicaListener> CreateServiceReplicaListeners()
{
return new ServiceReplicaListener[0];
}
protected override async Task RunAsync(CancellationToken cancellationToken)
{
await Task.Run(() => {
// Simulation of some work
Thread.Sleep((int)TimeSpan.FromMinutes(5).TotalMilliseconds);
});
isReady = true;
}
}
In this setup the import from the DB into the reliable collection is done in the RunAsync method.
Unfortunately, AFAIK, there is not way to plug in the communication listeners at a later time. That would make things way easier.
If CreateServiceReplicaListeners would be an async operation we could await the initialization task here, but we can't right now. Using .Wait() is not going to work as it will report that the instance is taking to long to get running and will mark the instance as unhealthy.
A complete overview of the lifecycle of a service can be found in the docs
I am not sure if I got you right. But based on your comment I would suggest the following solution for returning the 'Not ready' response during the migration.
public interface IMigrationService
{
bool IsDone();
}
public class MigrationService : IMigrationService
{
private bool migrating = tu;
public bool BeginMigration()
{
this.migrating = true;
}
public bool EndMigration()
{
this.migrating = false;
}
public bool IsDone()
{
return this.migrating;
}
}
// WebHost startup class
public class Startup
{
public void Configure(IApplicationBuilder app, IHostingEnvironment env)
{
// Register a middle-ware that would short circuit responses
// while migration is in progress.
app.Use(
async (context, next) =>
{
var migrationService =
context.RequestServices.GetService<IMigrationService>();
if (!migrationService.IsDone())
{
/* short circuit the response with approriate error message */
}
await next();
});
app.UseMvc();
}
}
public class Stateful : StatefulService
{
private readonly IMigrationService migrationService;
public Stateful(StatefulServiceContext context)
: base(context)
{
this.migrationService = new MigrationService();
}
protected override IEnumerable<ServiceReplicaListener>
CreateServiceReplicaListeners()
{
/*
Create a listener here with WebHostBuilder
Use Startup class with the middle-ware defined and
add configure services -> .ConfigureServices()
with services.AddSingleton<IMigrationService>(this.migrationService)
*/
}
protected override async Task
RunAsync(CancellationToken cancellationToken)
{
this.migrationService.StartMigration();
/* Migration code */
this.migrationService.EndMigration();
}
}
This would allow you to roll-out a new version of the service that would short circuit all requests with appropriate error message while the migration is in progress.
Hope this helps.
I am trying to mock a responder flow inside my unit tests, my responder flow does several validations that deals with configurations and off ledger services. I would like to mock the value to always return a true so that the unit test does not have any dependencies on the other components in the network.
The purpose is only for unit tests, is there any way I could mock the response using API as I am aware that we have to register the responder classes during the mock network setup?
Simply define a dummy responder flow, and register that instead of the real responder flow when setting up the mock network:
public class FlowTests {
private MockNetwork network;
private StartedMockNode a;
private StartedMockNode b;
#InitiatedBy(ExampleFlow.Initiator.class)
public static class DummyResponder extends FlowLogic<Void> {
private final FlowSession otherPartySession;
public DummyResponder(FlowSession otherPartySession) {
this.otherPartySession = otherPartySession;
}
#Suspendable
#Override
public Void call() throws FlowException {
otherPartySession.send(true);
return null;
}
}
#Before
public void setup() {
network = new MockNetwork(ImmutableList.of("com.example.contract"));
a = network.createPartyNode(null);
b = network.createPartyNode(null);
// For real nodes this happens automatically, but we have to manually register the flow for tests.
for (StartedMockNode node : ImmutableList.of(a, b)) {
node.registerInitiatedFlow(DummyResponder.class);
}
network.runNetwork();
}
#After
public void tearDown() {
network.stopNodes();
}
#Rule
public final ExpectedException exception = ExpectedException.none();
#Test
public void flowUsesDummyResponder() throws ExecutionException, InterruptedException {
ExampleFlow.Initiator flow = new ExampleFlow.Initiator(-1, b.getInfo().getLegalIdentities().get(0));
CordaFuture<Boolean> future = a.startFlow(flow);
network.runNetwork();
Boolean bool = future.get();
assertEquals(true, bool);
}
}
I am using spring-amqp to consume messages from RabbitMQ in my web application.
Web application consists of multiple components in it such as (Redis, OracleDB)
Now i have a scenario, if any exception occurs due to infrastructure like Oracle server is down, Redis connection issue, i want to push message back to the same queue and after certain specified delay i want to consume the message back.
After certain delay then also the message is leading to same exception, probably i want to use maximum attempts option or do the same as above push the message back to queue and send a mail to administrator stating "Infrastructure Issue".
Does Spring AMQP supports above scenario.? If yes please provide me how to come up with such or similar solutions.
I tried below piece of code. Message is not going for dead letter queue instead it is re-queuing to same queue causing infinite loop. Please correct me where am i going wrong
Configuration class
#Configuration
public class MQConfig {
public static final String OUTGOING_QUEUE = "my.outgoing.example";
public static final String INCOMING_QUEUE = "my.incoming.example";
public static final String DEAD_LETTER_QUEUE = "my.deadletter.queue.example";
#Autowired
private ConnectionFactory cachingConnectionFactory;
// Setting the annotation listeners to use the jackson2JsonMessageConverter
#Bean
public SimpleRabbitListenerContainerFactory rabbitListenerContainerFactory() {
SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory();
factory.setConnectionFactory(cachingConnectionFactory);
factory.setMessageConverter(jackson2JsonMessageConverter());
factory.setDefaultRequeueRejected(false);
return factory;
}
// Standardize on a single objectMapper for all message queue items
#Bean
public Jackson2JsonMessageConverter jackson2JsonMessageConverter() {
return new Jackson2JsonMessageConverter();
}
#Bean
public Queue outgoingQueue() {
Map<String, Object> args = new HashMap<String, Object>();
args.put("x-dead-letter-exchange", "dlx");
args.put("x-dead-letter-routing-key", DEAD_LETTER_QUEUE);
args.put("x-message-ttl", 50000);
return new Queue(OUTGOING_QUEUE, false, false, false, args);
}
#Bean
public RabbitTemplate outgoingSender() {
RabbitTemplate rabbitTemplate = new RabbitTemplate(cachingConnectionFactory);
rabbitTemplate.setQueue(outgoingQueue().getName());
// rabbitTemplate.setRoutingKey(outgoingQueue().getName());
rabbitTemplate.setMessageConverter(jackson2JsonMessageConverter());
return rabbitTemplate;
}
#Bean
public Queue incomingQueue() {
return new Queue(INCOMING_QUEUE);
}
#Bean
public Queue deadLetterQueue() {
return new Queue(DEAD_LETTER_QUEUE);
}
#Bean
public DirectExchange dlx() {
return new DirectExchange(DEAD_LETTER_QUEUE);
}
#Bean
public Binding dlqBinding() {
return BindingBuilder.bind(deadLetterQueue()).to(dlx()).with(DEAD_LETTER_QUEUE);
}
}
Core logic
#Component
public class DeadLetterSendReceive {
private static final Logger LOGGER = LoggerFactory.getLogger(DeadLetterSendReceive.class);
#Autowired
private RabbitTemplate outgoingSender;
// Scheduled task to send an object every 5 seconds
#Scheduled(fixedDelay = 5000)
public void sender() {
Integer int1[] = new Integer[]{10,20,30,40,50};
for (int i = 0; i <= int1.length; i++){
System.out.println(int1[i]);
if(int1[i]/10 == 1){
throw new AmqpRejectAndDontRequeueException("to deadletter queue");
}
else{
ExampleObject ex = new ExampleObject();
ex.setValue(int1[i]);
LOGGER.info("Sending example object at " + ex.getValue());
outgoingSender.convertAndSend(ex);
}
}
}
// Annotation to listen for an ExampleObject
#RabbitListener(queues = MQConfig.INCOMING_QUEUE)
public void handleMessage(ExampleObject exampleObject) {
LOGGER.info("Received incoming object at " + exampleObject.getValue());
}
}
Pojo Class
import java.util.Date;
public class ExampleObject {
private Date date = new Date();
private int value;
public int getValue() {
return value;
}
public void setValue(int value) {
this.value = value;
}
public ExampleObject() {
}
#Override
public String toString() {
return "ExampleObject{" +
"date= " + date +
'}';
}
public Date getDate() {
return date;
}
public void setDate(Date date) {
this.date = date;
}
}
There are a couple of ways to do it; use the delayed message exchange plugin and publish the failed message to it. You can set a header to track how many attempts have been made.
Or you can do it with a dead letter queue with a TTL where the dead-letter queue is configured with dead-lettering to send the expired message back to the original queue. See my answer to this question and its link to another answer.
You can use the x-death header to track retries; it has been changed in recent brokers to now keep a count instead of keep adding new entries to the header.
To force the message to go to the DLQ, set defaultRequeueRejected to false or throw an AmqpRejectAndDontRequeueException.
I am working in a Spring-MVC application and I would like to include personalized chat as a feature in it. After some research I found out Cometd to be a suitable option. After going through the documentation and forever repeating samples, I have a little bit of setup which I have done. I need some help to integrate a personalized chat service in the spring-mvc app, and enabling private chat when user pushes chat button.
So basically, I found out, "/service/chat" can be used for private chat, so I have a class for that, and to use private chat, I must have a mapping of userid<-->sessionId, but I cannot find examples anywhere how to do it. I am posting some of the code I have, kindly let me know what is remaining to do, and if possible, some resources, samples for that.
Controller code:
#Controller
#Singleton
public class MessageController {
private MessageService messageService;
#Autowired(required = true)
#Qualifier(value ="messageService")
public void setMessageService(MessageService messageService){this.messageService=messageService;}
#RequestMapping(value = "/startchatting", produces = "application/text")
#ResponseBody
public String startChattingService(){
return "OK";
}
#RequestMapping(value = "/stopchatting",produces = "application/text")
#ResponseBody
public String stopChatting(){
return "OK";
}
}
Private Message Service :
#Service
public class PrivateMessageService {
#Session
private ServerSession session;
#Listener("/service/private")
public void handlePrivateMessage(ServerSession sender, ServerMessage message){
String userId = (String) message.get("targetUserId");
//Mapping code necessary to map userids to session-id's.
//ServerSession recipient = findServerSessionFromUserId(userId);
//recipient.deliver(session,message.getChannel(),message.getData(),null);
}
}
CometConfigurer :
#Component
#Singleton
public class CometConfigurer {
private BayeuxServer bayeuxServer;
private ServerAnnotationProcessor processor;
#Inject
public void setBayeuxServer(BayeuxServer bayeuxServer){this.bayeuxServer = bayeuxServer;}
#PostConstruct
public void init() {this.processor= new ServerAnnotationProcessor(bayeuxServer);}
public Object postProcessBeforeInitialization(Object bean, String name) throws BeansException {
System.out.println("Configuring service " + name);
processor.processDependencies(bean);
processor.processConfigurations(bean);
processor.processCallbacks(bean);
return bean;
}
public Object postProcessAfterInitialization(Object bean, String name) throws BeansException {
return bean;
}
public void postProcessBeforeDestruction(Object bean, String name) throws BeansException {
processor.deprocessCallbacks(bean);
}
#Bean(initMethod = "start", destroyMethod = "stop")
public BayeuxServer bayeuxServer() {
BayeuxServerImpl bean = new BayeuxServerImpl();
// bean.setOption(BayeuxServerImpl.LOG_LEVEL, "3");
return bean;
}
public void setServletContext(ServletContext servletContext) {
servletContext.setAttribute(BayeuxServer.ATTRIBUTE, bayeuxServer);
}
}
Cometd beans :
<beans:bean id="bayeuxServer" class="org.cometd.server.BayeuxServerImpl" init-method="start" destroy-method="stop"/>
I have directly included the JSP files which have cometd configuration and setup from https://github.com/fredang/cometd-spring-example, and modified them to serve my needs. Kindly let me know what else is remaining, all suggestions are welcome, I am unable to find any examples for same task on net, which are detailed, and have more code then explanation. Thank you.
Using Spring 4.x's new WebSocket feature would definitely work; moreover, this new module ships with lots of very interesting features for your use case:
STOMP protocol support
messaging abstractions
session management
pub/sub mechanisms
etc
You can check this nice chat application that demonstrates all those features.