I'm trying to making a Consumer Kafka using NET CORE 2.1, this consumer should read one message compare timestamp and commit or not, so this consumer can stay on same message until this validation is true. See my code:
while(true)
{
try
{
var cr = consumer.Consume(TimeSpan.FromMilliseconds(4000));
if (cr == null)
{
Console.WriteLine("Exiting ... no messages to process");
break;
}
double totalSeconds = (DateTime.Now - cr.Timestamp.UtcDateTime).TotalSeconds;
Console.WriteLine($"TotalSeconds = {totalSeconds} , Resume = {resumeTimeSeconds}");
if (totalSeconds > resumeTimeSeconds)
{
Console.WriteLine($"Message = {cr.Value}");
consumer.Commit();
}else
{
Console.WriteLine($"Skipping... {cr.Value}");
continue;
}
}
catch (ConsumeException e)
{
Console.WriteLine($"Error occured: {e.Error.Reason}");
}
}
So, I have 10 messages in my topic and LAG is 2. I want to the next message is called only if i Commit() the previous message, but the consumer.Consume() method always call the next message.
The consumer commit comes into play only when your consumer start ( or recover from a crash). Your consumer will internally keep track of the last received offset for each partition.
What you can do is maybe use seek() to get back to the previous offset you just tried to process, and then retry.
Yannick
Related
I am trying to consume a max of 1000 messages from kafka at a time. (I am doing this because i need to batch insert into MSSQL.) I was under the impression that kafka keeps an internal queue which fetches messages from the brokers and when i use the consumer.consume() method it just checks if there are any messages in the internal queue and returns if it finds something. otherwise it just blocks until the internal queue is updated or until timeout.
I tried to use the solution suggested here: https://github.com/confluentinc/confluent-kafka-dotnet/issues/1164#issuecomment-610308425
but when i specify TimeSpan.Zero (or any other timespan up to 1000ms) the consumer never consumes any messages. but if i remove the timeout it does consume messages but then i am unable to exit the loop if there are no more messages left to be read.
I also saw an other question on stackoverflow which suggested to read the offset of the last message sent to kafka and then read messages until i reach that offset and then break from the loop. but currently i only have one consumer and 6 partitions for a topic. I haven't tried it yet but i think managing offsets for each of the partition might make the code messy.
Can someone please tell me what to do?
static List<RealTime> getBatch()
{
var config = new ConsumerConfig
{
BootstrapServers = ConfigurationManager.AppSettings["BootstrapServers"],
GroupId = ConfigurationManager.AppSettings["ConsumerGroupID"],
AutoOffsetReset = AutoOffsetReset.Earliest,
};
List<RealTime> results = new List<RealTime>();
List<string> malformedJson = new List<string>();
using (var consumer = new ConsumerBuilder<Ignore, string>(config).Build())
{
consumer.Subscribe("RealTimeTopic");
int count = 0;
while (count < batchSize)
{
var consumerResult = consumer.Consume(1000);
if (consumerResult?.Message is null)
{
break;
}
Console.WriteLine("read");
try
{
RealTime item = JsonSerializer.Deserialize<RealTime>(consumerResult.Message.Value);
results.Add(item);
count += 1;
}
catch(Exception e)
{
Console.WriteLine("malformed");
malformedJson.Add(consumerResult.Message.Value);
}
}
consumer.Close();
};
Console.WriteLine(malformedJson.Count);
return results;
}
I found a workaround.
For some reason the consumer first needs to be called without a timeout. That means it will wait for a message until it gets at least one. after that using consume with timeout zero fetches all the rest of the messages one by one from the internal queue. this seems to work out for the best.
I had a similar problem, updating the Confluent.Kafka and lidrdkafka libraries from version 1.8.2 to 2.0.2 helped
We have integration setup that creates purchase orders on the batch server. For example the batch job may run and pick up 5 invoices coming from an external source and attempt to post them.
If 4 are successful and 1 fails, we catch the error using the code below:
errEnumerator = SysInfologEnumerator::newData(infolog.cut());
while (errEnumerator.moveNext())
{
msgStruct = new SysInfologMessageStruct(errEnumerator.currentMessage());
errException = errEnumerator.currentException();
messageBody += msgStruct.message() + "\n";
}
Which works great in catching the error and then we return it into a log. The issue is the entire message will be shown. "Number of vouchers posted to the journal 1." 4 times and then the error message.
After each successful post we do clear the infolog by doing infolog.clear();.
If you debug this code in X++ it does clear it each time and the error will only show the actual error without the previous successful posts. But the batch job running on the batch server for some reason does not clear the infolog after each successful post. After CILs, restarting services etc. nothing seems to work.
Is there another way to clear the infolog on the batch server? thanks
If your goal is to store only the error lines in messageBody and not the 'success' lines, you don't have to clear the Infolog. You only need to add the following check at the beginning of your while cycle:
if (errEnumerator.currentException() == Exception::Info ||
errEnumerator.currentException() == Exception::Warning)
{
continue;
}
Do not mess with the infolog!
This will hide information, warnings and errors that you will need for example for batch problem solving.
So please do not clear() or cut().
Instead copy what you want:
numLine = infologLine();
try
{
// Do something useful
}
catch (Exception::Error)
{
doTheLog(infolog.copy(numLine + 1, infologLine()));
throw error("That did not work!");
}
First store the current infolog number. On error process the relevant infologs.
If the infolog is long consider transferring the numbers rather than call by value the container:
doTheLog(numLine + 1, infologLine());
Then infolog.copyin the method.
Is it possible to execute a query asynchronously in hive server?
For eg, How can I /Is it possible to do something like this from the client-
QueryHandle handle = executeAsyncQuery(hiveQuery);
Status status = handle.checkStatus();
if(status.isCompleted()) {
QueryResult result = handle.fetchResult();
}
I also had a look at How do I make an async call to Hive in Java?. But did not help. The answers were mostly around the thrift clients taking a callback argument.
Any help would be appreciated. Thanks!
[EDIT 1]
I went through the HiveConnection.java in hive-jdbc. hive-jdbc by default uses the async thrift APIs. Hence it submits a query and polls for result sets (look at HiveStatement.java). Now i am able to write a piece of code which is purely non blocking. But the problem is as soon as the client disconnect the foot print about the query is lost.
Client 1
final TCLIService.Client client = new TCLIService.Client(createBinaryTransport(host, port, loginTimeout, sessConf, false)); // from HiveConnection.java
TSessionHandle sessionHandle = openSession(client) // from HiveConnection.java
TExecuteStatementReq execReq = new TExecuteStatementReq(sessionHandle, sql);
execReq.setRunAsync(true);
execReq.setConfOverlay(sessConf);
final TGetOperationStatusReq handle = client.ExecuteStatement(execReq)
writeHandleToFile("~/handle", handle)
Client 2
final TGetOperationStatusReq handle = readHandleFromFile("~/handle")
final TCLIService.Client client = new TCLIService.Client(createBinaryTransport(host, port, loginTimeout, sessConf, false));
while (true) {
System.out.println(client.GetOperationStatus(handle).getOperationState());
Thread.sleep(1000);
}
Client 2 keeps printing FINISHED_STATE as long as Client 1 is alive. But if client 1 process completes or gets killed, client 2 starts printing null which means hiveserver2 is cleaning up the resources as soon as a client disconnects.
Is it possible to configure hiveserver2 to configure this clean up process based on time or something?
Thanks!
Did some research and figured out that this happens only with binary transport (tcp)
#Override
public void deleteContext(ServerContext serverContext,
TProtocol input, TProtocol output) {
Metrics metrics = MetricsFactory.getInstance();
if (metrics != null) {
try {
metrics.decrementCounter(MetricsConstant.OPEN_CONNECTIONS);
} catch (Exception e) {
LOG.warn("Error Reporting JDO operation to Metrics system", e);
}
}
ThriftCLIServerContext context = (ThriftCLIServerContext) serverContext;
SessionHandle sessionHandle = context.getSessionHandle();
if (sessionHandle != null) {
LOG.info("Session disconnected without closing properly, close it now");
try {
cliService.closeSession(sessionHandle);
} catch (HiveSQLException e) {
LOG.warn("Failed to close session: " + e, e);
}
}
}
The above stub (from ThriftBinaryCLIService) gets executed through this piece of code from TThreadPoolServer which is used by ThriftBinaryCLIService.
eventHandler.deleteContext(connectionContext, inputProtocol,
outputProtocol);
Apparently http transport (ThriftHttpCLIService) has a different strategy of cleaning up operation handles (not greedy like tcp)
Will check with hive community on this to understand a bit more and see if there is an issue addressing this already.
We are trying to load and resume workflows which have a delay. I have seen the Microsoft sample of Absolute Delay for this using store.WaitForEvents and LoadRunnableInstance to load the workflow. However here the workflow is already known.
In our case we want to have an event waiting for the store.WaitForEvents after every say 5 seconds to check if there is a runnable instance and if so only load and run that /those particular instances. Is there a way I could know which workflow instance is ready.
We are maintaing the workflow id and the xaml associated to it in our database, so if we could know the workflow instance id we could get the xaml mapped to it, create the workflow and then do a LOadRunnableInstance on it.
Any help would be greatly appreciated.
Microsoft sample (Absolute Delay)
public void Run(){
wfHostTypeName = XName.Get("Version" + Guid.NewGuid().ToString(),
typeof(WorkflowWithDelay).FullName);
this.instanceStore = SetupSqlpersistenceStore();
this.instanceHandle =
CreateInstanceStoreOwnerHandle(instanceStore, wfHostTypeName);
WorkflowApplication wfApp = CreateWorkflowApp();
wfApp.Run();
while (true)
{
this.waitHandler.WaitOne();
if (completed)
{
break;
}
WaitForRunnableInstance(this.instanceHandle);
wfApp = CreateWorkflowApp();
try
{
wfApp.LoadRunnableInstance();
waitHandler.Reset();
wfApp.Run();
}
catch (InstanceNotReadyException)
{
Console.WriteLine("Handled expected InstanceNotReadyException, retrying...");
}
}
Console.WriteLine("workflow completed.");
}
public void WaitForRunnableInstance(InstanceHandle handle)
{
var events=instanceStore.WaitForEvents(handle, TimeSpan.MaxValue);
bool foundRunnable = false;
foreach (var persistenceEvent in events)
{
if (persistenceEvent.Equals(HasRunnableWorkflowEvent.Value))
{
foundRunnable = true;
break;
}
}
if (!foundRunnable) {
Console.WriteLine("no runnable instance");
}
}
Thanks
Anamika
I had a similar problem with durable delay activities and WorkflowApplicationHost. Ended up creating my own 'Delay' activity that worked essentially the same way as the one out of the box, (takes an arg that describes when to resume the workflow, and then bookmarks itself). Instead of saving delay info in the SqlInstanceStore though, my Delay Activity created a record in a seperate db. (similar to the one you are using to track the Workflow Ids and Xaml). I then wrote a simple service that polled that DB for expired delays and initiated a resume of the necessary workflow.
Oh, and the Delay activity deleted it's record from that DB on bookmark resume.
HTH
I'd suggest having a separate SqlPersistenceStore for each workflow definition you're hosting.
I just tried some code from the internet and ran it, but it blocked my emulator. The code is:
public void getcontents()
{
HttpConnection c = null;
InputStream is = null;
StringBuffer sb = new StringBuffer();
try
{
c = (HttpConnection)Connector.open("http://www.java-samples.com",Connector.READ_WRITE, true);
c.setRequestMethod(HttpConnection.GET); //default
is = c.openInputStream(); // transition to connected!
int ch = 0;
for(int ccnt=0; ccnt < 150; ccnt++) { // get the title.
ch = is.read();
if (ch == -1){
break;
}
sb.append((char)ch);
}
}
catch (IOException x){
x.printStackTrace();
}
finally{
try{
is.close();
c.close();
} catch (IOException x){
x.printStackTrace();
}
}
System.out.println(sb.toString());
}
I called the function with an OK command.
The emulator got blocked until I killed the process.
How do I solve this?
Try stepping through the code in the debugger. Or at the very least add some log statements. My guess is that the stream is waiting on data from the HTTP connection and isn't getting flushed but I haven't ran the code to verify that assertion.
The only loop I can see in your code is the for loop, which is finite (no more that 150 iterations), so that would not make the code execute indefinitely.
What I would suggest is place a number of debug output statements (output to a console or even dialog box alerts) at various points through the code. This will help you work out which line of code is causing the problem. For instance, if you put a line before and after the for loop and, when executing, only the first one is displayed, you know your problem is somewhere within the loop. You can then narrow it down by putting debug lines within the loop (including the loop number) to find out which line exactly is causing your problem.
Try checking the response code before attempting to read the response body from the server. This will either confirm the connection succeeds or print out the error response. Place the following after the call to Connector.open() :
if (c.getResponseCode() != HttpConnection.HTTP_OK) {
throw new IOException("HTTP response code: " + c.getResponseCode());
} else {
System.out.println("**Debug** : HTTP_OK received, connection established");
}
If running the code then gives no output of either the exception or the HTTP confirmation then you are likely blocking on the connection attempt (check your emulator's connectivity to the internet). If you do get the HTTP_OK then you are likely blocking on the server's HTTP response, or lack thereof. Posting a comment with your results would be a good idea.