I am using spring-kafka RetryableTopic for non-blocking retries with a fixed BackOff and a single retry topic.
I noticed that if I use more than 127 attempts the retry never stops and also if I use this header:
#Header(name = RetryTopicHeaders.DEFAULT_HEADER_ATTEMPTS, required = false) int attempt
It overflows to 0 after 127.
I couldn't find any limit for max_retries in spring-kafka documentation, but in the source code I see that it takes only first byte.
Is it a bug or feature? Are there plans to support max_attempts more than 127?
Please, raise a GH issue against spring-kafka project.
The bug is here in the DeadLetterPublishingRecovererFactory:
private int getAttempts(ConsumerRecord<?, ?> consumerRecord) {
Header header = consumerRecord.headers().lastHeader(RetryTopicHeaders.DEFAULT_HEADER_ATTEMPTS);
return header != null
? header.value()[0]
: 1;
}
The header.value() is a byte[] array from the:
headers.add(RetryTopicHeaders.DEFAULT_HEADER_ATTEMPTS,
BigInteger.valueOf(attempts + 1).toByteArray());
But that [0] takes really only one byte from the array and cast it to int.
Related
MyApp.exe is installed using an InstallScript 2014 project. The folks in manufacturing recently tried to upgrade to a more recent development version but did not close the existing instance of MyApp. This resulted in many permission denied errors since the various dll's used by the app were locked and in use.
I expected the InstallScript executable to do the usual "stage and reboot" thing that all Windows folks are intimate with. It did not do that, and I can't see anything in the InstallShield project editor that obviously lets me force that behavior.
I also expected InstallScript to allow me to somehow detect that my app was already running - if I can do that, I can display a dialog to give the user a chance to close the app and continue. The only solution for this is InstallSite.org List and Shutdown Running Processes. (Note this is unanswered on another S/O question.)
That does not properly detect all of the running tasks, including my own.
Before I spend a couple of days trying to fix what seems to be an obviously missing feature of InstallScript, I thought I'd ask if there's a better approach.
Here's what I came up with. Hope this helps.
// defines/protos for finding a process
#define TH32CS_SNAPPROCESS 0x00000002
// in Kernel32.dll
prototype NUMBER Kernel32.CreateToolhelp32Snapshot(NUMBER , NUMBER);
prototype BOOL Kernel32.Process32First(HWND , POINTER );
prototype BOOL Kernel32.Process32Next(HWND , POINTER );
// from minwindef.h, windows api
typedef PROCESSENTRY32
begin
number dwSize;
number cntUsage;
number th32ProcessID; // this process
number th32DefaultHeapID;
number th32ModuleID; // associated exe
number cntThreads;
number th32ParentProcessID; // this process's parent process
number pcPriClassBase; // Base priority of process's threads
number dwFlags;
STRING szExeFile[MAX_PATH]; // Path
end;
// ========================================================================================
// list all of the running processes, see if Flex is running
// based on https://msdn.microsoft.com/en-us/library/windows/desktop/ms686701(v=vs.85).aspx
// ========================================================================================
function BOOL IsProcessRunning(sProcessName)
HWND hProcessSnap;
PROCESSENTRY32 pe;
POINTER ppe;
NUMBER ret;
begin
hProcessSnap = CreateToolhelp32Snapshot(TH32CS_SNAPPROCESS, 0);
pe.dwSize = SizeOf(pe);
ppe = &pe;
ret = Process32First(hProcessSnap, ppe);
if (ret == 0) then
//printError(TEXT("Process32First")); // show cause of failure
CloseHandle(hProcessSnap); // clean the snapshot object
return(FALSE);
endif;
repeat
if (StrCompare(sProcessName, pe.szExeFile) == 0) then
CloseHandle(hProcessSnap); // clean the snapshot object
return(TRUE);
endif;
ret = Process32Next(hProcessSnap, ppe);
until (ret == 0);
return(FALSE);
end;
You then call it
if (IsProcessRunning("WORD.EXE")) then
blah blah
endif;
I have just downloaded the latest Arduino Library code from Github, and it's broken my MQTT client program. I'm using PubSubClient 1.91 on Arduino, and Mosquitto 1.1.2 (Build 2013-03-07) on Mac OSX. (I also tested against Mosquitto on Windows 7, same problem.)
The supplied Mosquitto clients work fine, (Mac over to Windows, Windows over to Mac) so it's some problem with what's coming from the Arduino end. A wireshark trace shows the Arduino client sending the following data packet:
10:15:ff:ff:4d:51:49:73:64:70:03:02:00:0f:00:07:41:72:64:75:69:6e:6f
And the Mosquitto broker shows:
New connection from 10.0.0.115
Socket read error on client (null), disconnecting.
Before I start to crawl through the MQTT spec, can anyone see anything wrong with the data packet being sent? It's got to be something to do with new Arduino library code...
* Update
Upon further investigation, it appears to be a code generation problem with avr-g++, although life experience tells me it will turn out not to be so. Here is a snippet of code from PubSubClient.cpp
boolean PubSubClient::connect(char *id, char *user, char *pass, char* willTopic, uint8_t willQos, uint8_t willRetain, char* willMessage) {
if (!connected()) {
int result = 0;
if (domain != NULL) {
result = _client->connect(this->domain, this->port);
} else {
result = _client->connect(this->ip, this->port);
}
if (result) {
nextMsgId = 1;
uint8_t d[9] = { 0x00, 0x06, 'M','Q','I','s','d','p',MQTTPROTOCOLVERSION};
// d[0] = 0;
// d[1] = 6;
Serial.print("d[0]="); Serial.println(d[0],HEX);
Now, the result of the Serial.print just above turns out to be 0xFF !!! So, the uint8_t array is not being initialised correctly. #knoleary Your pointer to the bad FF bytes lead me to this.
If I now uncomment the two lines above, and manually initialise the first 2 bytes to 0 and 6, all works fine, and my program communicates happily with Mosquitto.
I've looked at the generated code, but I'm not an Atmel expert.
Does anyone have any clue why this might be?
I'm compiling using the AVR-G++ toolset from Arduino 1.05, in Eclipse.
I'm going for a beer!
OK, I found it. It's a relatively subtle bug. Essentially, when the following line of source code is compiled;
uint8_t d[9] = { 0x00, 0x06, 'M','Q','I','s','d','p',MQTTPROTOCOLVERSION};
the 9 bytes get stored as a constant in the data section of the image. At runtime, a small loop copies the 9 bytes into the array (d[]) By looking at a combined Assembler / source listing, I could see where in the data section the 9 bytes were stored, and then print them out at regular intervals, until I found what was over-writing them. (A bit primitive, I know!)
It turns out the there's a bug in WiFi.cpp , the Arduino WiFi code. Here's the code:
uint8_t WiFiClient::connected() {
if (_sock == 255) {
return 0;
} else {
uint8_t s = status();
return !(s == LISTEN || s == CLOSED || s == FIN_WAIT_1 ||
s == FIN_WAIT_2 || s == TIME_WAIT ||
s == SYN_SENT || s== SYN_RCVD ||
(s == CLOSE_WAIT));
}
}
It turns out the the _sock variable is actually initialised like this:
WiFiClient::WiFiClient() : _sock(MAX_SOCK_NUM) {
}
and MAX_SOCK_NUM is 4, not 255. So, WiFiClient::status returned true, instead of false for an unused Socket.
This method was called by the MQTT Client like this:
boolean PubSubClient::connected() {
boolean rc;
if (_client == NULL ) {
rc = false;
} else {
rc = (int)_client->connected();
if (!rc) _client->stop();
}
return rc;
}
And, since the _client->connected() method erroneously returned true, the _client_stop() method was called. This resulted in a write to a non-existent socket array element, and so overwrote my string data.
#knolleary, I was wondering, is there any specific reason that your PubSubClient::connected() method does a disconnect? I use the ::connected method in a loop, to check that I'm still connected, and, of course it results in my getting a disconnect / reconnect each time round the loop. Any chance we could just make connected return true / false , and handle the disconnect in PuBSubClient::connect?
Nearly one and a half year later I ran into the same problem. Removing the
boolean PubSubClient::connected() {
int rc = (int)_client->connected();
if (!rc) _client->stop();
return rc;
}
the _client->stop() from the connected method of PubSubClient forehand fixed this problem for me. However, I'm not sure whether this is actually a solution or just a very dirty quick hack to localize the problem.
What have you done to fix this problem - your explanation of the problem above is fine however, I was not able to extract the solution easily ;-)
I'm a newbie in QT and C++, I'm trying to create a QTcpserver using QThreadpools so it can handle multiple clients. Multiple clients are able to connect without any issues. But I'm trying to send an image from an android phone, with a footer "IMGPNG", indicating the end of image data. Now the issue when the readyRead signal is emitted I'm tring to read all the data available data and then perform some string operation later and reconstruct the image. I'm not sure how to receive the complete image for each client and then process it accordingly.
void VireClients::readyRead()//read ready
{
int nsize = socket->bytesAvailable();//trying to check the available bytes
qDebug()<< "Bytes Available" << nsize;
while(socket->bytesAvailable() < nsize){
QByteArray data = socket->readAll();//how to receive all the data and then process it
}
/*!These lines call the threadpool instance and reimplement run*/
imageAnalysis = new VireImageAnalysis(); //creating a new instance of the QRunnable
imageAnalysis->setAutoDelete(true);
connect(imageAnalysis,SIGNAL(ImageAnalysisResult(int)),this,SLOT(TaskResult(int)),Qt::QueuedConnection);
QThreadPool::globalInstance()->start(imageAnalysis);
}
Now i'm not sure how to get the data completely or save the received data in an image format. i want to know how to completely receive the image data. Please help.
A call to readAll() will not always read the complete image as it obviously cannot know the size of the image. It will only read all currently available bytes which might be less than your whole file, or more if the sender is really fast and you cannot catch up reading. The same way readyRead() only informs you that there are bytes available but not that a whole file has been received. It could be a single byte or hundreds of bytes.
Either you know the size of your image in the first place because it is always fixed or the sender has to tell the receiver the number of bytes he wants to sent.
Then you can either just ignore all readyRead() signals until bytesAvailable() matches your image size and call readAll() to read the whole image at once. Or you read whenever there are available bytes and fill up your buffer until the number of bytes read matches the bytes the receiver told you he will send.
Solved saving image issue by collecting, the string in temp variable and finally, used opencv imwrite to save the image, this solved this issue:
while(iBytesAvailable > 0 )
{
if(socket->isValid())
{
char* pzBuff = new char[iBytesAvailable];
int iReadBytes = socket->read(pzBuff, iBytesAvailable);
if( iReadBytes > 0 )
{
result1 += iReadBytes;
str += std::string(reinterpret_cast<char const *>(pzBuff), iReadBytes);
if(str.size() > 0){
search = str.find("IMGPNG");
if(search == result1-6){
finalID = QString::fromStdString(str);
Singleton_Global *strPtr = Singleton_Global::instance();
strPtr->setResult(finalID);
/*!Process the received image here*/
SaveImage= new VSaveImage();
SaveImage->setAutoDelete(false);
connect(SaveImage,SIGNAL(SaveImageResult(QString)),this,SLOT(TaskResult(QString)),Qt::QueuedConnection);
threadPool->start(SaveImage);
}
}
}
Finally did the image saving on the run method -->SaveImage, #DavidSchwartz you were a great help thanks. Thanks all for your help.
Hi All
I am using a non blocking Socket for sending messages.We were getting EGAIN error occassioanally .So I have decided to use Flush(socket) to flush the buffer and make space for new space so that i can avoid EGAIN error .But the problem is Flush(socket) is stuck for indefinite time .
Here is the code
int res = send(socket, buffer, size+lengthSize,0);
delete buffer;
if ( res== -1 )
{
int error = errno;
cout("ERROR on SendOnPortString, errno = " << error);
return 0 ;
}
else
{
cout<<"Send SucessFul = " << res << "Total Message size"<< size+lengthSize;
if(res==size+lengthSize)
flush((ostream&)socket);
//flush(socket);
return 1 ;
}
This code printing
Send SucessFul = 11Total Message size 11
But after that its getting stuck in flush(socket) method .Any Idea why its behaving like that
You cast a socket handle of type int to a reference to a std::ostream in order to avoid compiler warnings/errors when you tried to hand it to flush. I'm surprised it's not crashing.
You can't make more space. The problem isn't you: the problem is that the system's internal buffers are full, and they will drain at their own pace in their own time. You can either poll by trying to send over and over till it works (in which case, why are you using non-blocking sockets?), or you use select, poll, kqueue, epoll, libevent, etc. to sleep till the socket is able to accept more data.
I have an application where every now and then I'm getting a strange error.
This is the piece of code:
Dim XMLWriter As New System.Xml.XmlTextWriter(Me.Context.Response.OutputStream, Encoding.UTF8)
XMLWriter.WriteStartDocument()
XMLWriter.WriteStartElement("Status")
Message.SerializeToXML(XMLWriter)
XMLWriter.WriteEndElement()
XMLWriter.WriteEndDocument()
XMLWriter.Flush()
XMLWriter.Close()
The error i'm getting is:
Message: Object reference not set to an instance of an object.
on line XMLWriter.Flush();
To make things more fun, this is absolutely non-reproducible. It just happens every now and then....
Since it's happening when flushing the XML i'm guessing the Object that is now null has to be the Response.OutputStream.
This is the relevant part of the stack trace:
Description:
An unhandled exception occurred and the process was terminated.
Exception: System.NullReferenceException
Message: Object reference not set to an instance of an object.
StackTrace: at System.Web.HttpWriter.BufferData(Byte[] data, Int32 offset, Int32 size, Boolean needToCopyData)
at System.Web.HttpWriter.WriteFromStream(Byte[] data, Int32 offset, Int32 size)
at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
at System.Xml.XmlTextWriter.Flush()
at RequestData.CompleteRequest(MessageBase Message) in RequestData.vb:line 142
My question is, in what cases could this be happening?
This server is a long-polling server, so the client asks for something, and I may not answer for 30 seconds...
Is it possible that this Stream will become Null if the client disconnects (ie. closes the browser window)?
Any other ideas? (any pointers appreciated)
Reflector gives this:
private void BufferData(byte[] data, int offset, int size, bool needToCopyData)
{
int num;
if (this._lastBuffer != null)
{
num = this._lastBuffer.Append(data, offset, size);
size -= num;
offset += num;
}
else if ((!needToCopyData && (offset == 0)) && !this._responseBufferingOn)
{
this._buffers.Add(new HttpResponseBufferElement(data, size));
return;
}
while (size > 0)
{
this._lastBuffer = this.CreateNewMemoryBufferElement();
this._buffers.Add(this._lastBuffer);
num = this._lastBuffer.Append(data, offset, size);
offset += num;
size -= num;
}
}
The only object that is not null checked, initialized or referenced through another method(which would show in the stack trace) is this._buffers. The only place it is set to null in that class is in RecycleBufferElements() which if you dig deeper can occur when the client disconnects.
The call to Flush is what will cause anything cached in memory to be written out to the stream and ultimately the client so yes, it may be the problem.
You mentioned that the request is expected to take a long time to execute so it may be possible that ASP.Net or IIS are timing you out too early. I would suggest having a look at the executionTimeout property in the web.config and similar settings.
No, if it happens when you call Flush, that's way later than the only time Context.Response.OutputStream is actually referenced. The value is fetched in the call to the XmlTextWriter constructor, and then not looked at again.
Do you have any more information from the stack trace?