Sending events transactionally in logstash - tcp

Question

I am trying to use logstash to receive events from TCP socket, and output them to a Kafka topic. My current configuration is able to do that perfectly, but I want to be able to conduct events to Kafka in a transactional manner. I mean, the system should not send the events to kafka, until commit message is received:
START TXN 123 --No message sent to Kafka
123 - Event1 Message --No message sent to Kafka
123 - Event2 Message --No message sent to Kafka
123 - Event3 Message --No message sent to Kafka
COMMIT TXN 123 --Event1, Event2, Event3 messages sent to Kafka
Is there any possibility to achieve this using logstash only or should I introduce any other transaction coordinator in between source and logstash? Here is my current config:
input {
tcp {
port => 9000
}
}
output {
kafka {
bootstrap_servers => "localhost:9092"
topic_id => "alpayk"
}
}
I tried to use use logstash' s aggregate filter for this purpose, but I couldn' t end up with something works.
Thank you very much in advance

Answer 1

I finally decided to use Apache Flume for this purpose. I modified its netcat source so that uncommited messages reside in flume' s heap, and as soon as a commit message is received for a transaction, all messages are sent to kafka sink.
I am going to change the message storing location from flume heap to an external cache, so that I will be able to expire the stored messages if transaction abends or rolls back.
Below is my piece of code for that transaction logic:
String eventMessage = new String(body);
int indexOfTrxIdSeparator = eventMessage.indexOf("-");
if (indexOfTrxIdSeparator != -1) {
String txnId = eventMessage.substring(0, indexOfTrxIdSeparator).trim();
String message = eventMessage.substring(indexOfTrxIdSeparator + 1).trim();
ArrayList<Event> events = cachedEvents.get(txnId);
if (message.equals("COMMIT")) {
System.out.println("##### COMMIT RECEIVED");
if (events != null) {
for (Event eventItem : events) {
ChannelException ex = null;
try {
source.getChannelProcessor().processEvent(eventItem);
} catch (ChannelException chEx) {
ex = chEx;
}
if (ex == null) {
counterGroup.incrementAndGet("events.processed");
} else {
counterGroup.incrementAndGet("events.failed");
logger.warn("Error processing event. Exception follows.", ex);
}
}
cachedEvents.remove(txnId);
}
} else {
System.out.println("##### MESSAGE RECEIVED: " + message);
if (events == null) {
events = new ArrayList<Event>();
}
events.add(EventBuilder.withBody(message.getBytes()));
cachedEvents.put(txnId, events);
}
}
I added this code into processEvents method of the netcat source of Flume. I didn' t want to work with Ruby code, that' s why I decided to switch to Flume. However the same thing could also be done in logstash.
Thank you