I'm using Arduino IDE 1.6.8 and ESP8266 Arduino community ver 2.2.0
Every page request the free heap size goes down, until eventually it crashes.
I've seen String class usage on other web serve code so I assume it isn't that..
#include <ESP8266WiFi.h>
String freeHeap;
WiFiServer webServer(80);
const char* html1 = "<!DOCTYPE html><html><head><meta name=\"viewport\" content=\"width=device-width,initial-scale=1,maximum-scale=1,minimum-scale=1\"/></head><body style=\"background-color:#EEE;font-family:Arial,Tahoma,Verdana;\"><h1>Title</h1>";
String html2 = "";
String req;
void setup() {
WiFi.mode(WIFI_STA);
WiFi.begin("name","pwd");
WiFi.config(IPAddress (192,168,1,201), IPAddress (192,168,1,1), IPAddress (255,255,255,0));
WiFi.softAPdisconnect(true);
webServer.begin();
}
void loop() {
WiFiClient webClient = webServer.available();
if (!webClient) {
delay(1);
return;
}
req = webClient.readStringUntil('\r');
webClient.flush();
setHTML();
webClient.print(html1);
webClient.print(html2);
delay(1);
}
void setHTML() {
long fh = ESP.getFreeHeap();
char fhc[20];
ltoa(fh, fhc, 10);
freeHeap = String(fhc);
html2 = "Heap " + freeHeap + "</body></html>";
}
Although WiFiClient is destroyed at the end of loop, the actual TCP buffer living behind it doesn't get free'd until either the client disconnects, or the TCP MSL limit is reached. Either the browser is keeping the connection alive, or more likely the buffer is kept alive after loop because you (implicitly) destroy the WiFiClient before the client can acknowledge that they received the data correctly.
Per the spec, the ESP has to keep this data around for the entire MSL duration, or until the client says they have received everything correctly. Since this can't happen in your loop, it'll consume RAM until this 2 minute time period is up.
This is easily fixed by waiting until the client has actively closed the connection with some sort of while loop using client.connected(), and additionally instructing the browser/HTTP client to close the connection when they are done with the Connection: close header. Once it has disconnected, then you should call client.stop();
Related
So I've been struggling to have this working.
I have an ESP32, but I can't use the wifi on the module (I simply can't, due to the implementation of the ESP).
Now, I have an W5500 shield for arduino, but I figure, that this should work just fine on an ESP, since it communicates via SPI.
I have had code that works perfectly with the Wifi of the ESP, but now it needs to run on the Ethernet thing.
HTTPClient http;
void sendHeartBeat(void *pvParameters) {
// Infinite loop (while true is more risky for some reason)
unsigned long previousMillis = 0;
const long interval = 300000; // 5 minutes
for (;;) {
unsigned long currentMillis = millis();
if (currentMillis - previousMillis >= interval) {
previousMillis = currentMillis; // reset timer
int httpResponseCode;
http.begin(DIGITAL_HQ_HEARTBEAT_ENDPOINT); // test environment
http.addHeader("Content-Type", "application/json");
http.setUserAgent(DIGITAL_HQ_USER_AGENT);
StaticJsonDocument<128> doc;
doc["mac"] = deviceMacAddress;
doc["key"] = DEVICE_SECRET;
doc["type"] = DIGITAL_HQ_SOFTWARE_TYPE;
String output;
serializeJson(doc, output);
httpResponseCode = http.POST(output);
// If the device was deleted from HQ, re-register it.
if (httpResponseCode == 202) {
registerDevice();
}
http.end();
}
}
}
The above code worked perfectly on the ESP when using wifi. Now it doesn't anymore and just crashes the whole device.
This is the stacktrace;
16:20:52.874 -> assertion "Invalid mbox" failed: file "/home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/api/tcpip.c", line 374, function: tcpip_send_msg_wait_sem
16:20:53.110 -> abort() was called at PC 0x400dced7 on core 1
16:20:53.143 ->
16:20:53.143 -> ELF file SHA256: 0000000000000000
16:20:53.176 ->
16:20:53.176 -> Backtrace: 0x400891f4:0x3ffb1a00 0x4008946d:0x3ffb1a20 0x400dced7:0x3ffb1a40 0x401122bb:0x3ffb1a70 0x40111d55:0x3ffb1aa0 0x40111f08:0x3ffb1ac0 0x40108f10:0x3ffb1b00 0x400d4fbd:0x3ffb1b20 0x400d4cee:0x3ffb1da0 0x400d4e09:0x3ffb1dd0 0x40126337:0x3ffb1df0 0x400d783e:0x3ffb1e10 0x400d78d1:0x3ffb1e40 0x400d7a63:0x3ffb1e90 0x400d7a83:0x3ffb1eb0 0x400d284e:0x3ffb1ed0 0x400d9fd1:0x3ffb1fb0 0x4008a472:0x3ffb1fd0
What am I doiçng wrong? How can I resolve this issue without too much hassle? I also have this on any other used library (e.g. MQTTClient.h, WiFiClientSecure.h). Those I use and I really don't want to re-write all of the code just for the sake of 'I use an rj45 connector instead of wifi'....
Edit:
The issue was with https, but I need to be able to use https urls also... When I use http, the request just doesn't hapen either.
I've created a TCP server application using BSD sockets and NUCLEO-H743ZI2 development board with STM32CubeMX 5.6.0 & LwIP 2.0.3 in Keil-MDKARM.
I noticed that:
If a client connects and sends 11 bytes or more at first, server
receives the data correctly and read() responds displaying the data.
However, if client sends the first data lower than 11
bytes, read() function blocks even next received data is higher than 11 bytes, until client disconnects. After the disconnection, all the data queued is displayed.
Namely, if first data sent from a client to my server is lower than 11 bytes, event_callback for a rcvevent is not triggered until disconnection.
My aim is to make the server available to one byte reception.
I've pasted my Server task/thread below. Let me have your kind response at your earliest convenience and feel free to request other related files/libraries(lwip.h, lwipopts.h..).
Kind Regards
void StartTask01(void const * argument)
{
/* USER CODE BEGIN StartTask01 */
MX_LWIP_Init();
/*start a listening tcp server*/
int iServerSocket;
struct sockaddr_in address;
if ((iServerSocket = socket(AF_INET, SOCK_STREAM, 0)) < 0)
{
printf("Socket could not be created\n");
}
else
{
address.sin_family = AF_INET;
address.sin_port = htons(80);
address.sin_addr.s_addr = INADDR_ANY;
if (bind(iServerSocket, (struct sockaddr *)&address, sizeof (address)) < 0)
{
printf("socket could not be bound\n");
}
else
{
listen(iServerSocket, MEMP_NUM_NETCONN);
}
}
/*server started listening*/
struct sockaddr_in remoteHost;
int newconn;
char caReadBuffer[1500];
memset(caReadBuffer, 0, 1500);
for(;;)
{
/*block until accepting an incoming connection*/
newconn = accept(iServerSocket, (struct sockaddr *)&remoteHost, (socklen_t *)(sizeof(remoteHost)));
if (newconn != -1)/*if accepted well*/
{
/*block until data arrives*/
read(newconn, caReadBuffer, sizeof(caReadBuffer));
printf("data read: %s\n", caReadBuffer);
memset(caReadBuffer, 0, 1500);
}
}
/* USER CODE END StartTask01 */
}
The problem that's causing this issue is that you only call read once on each connection. If you don't happen to receive all the data from that single call to read (which is entirely unpredictable), you will never call read on that connection again.
When you call read on a blocking TCP connection, it will only block if there is no data available. Otherwise, it will give you whatever data is available up to the maximum number of bytes you ask for. It will not wait for more data if only some is available. It's up to you to call read again if you didn't receive all the data you expected.
One your second iteration of the for loop, you overwrite newconn with a new connection. You don't close the old connection. So you have a socket leak.
SOLVED:
The problem is, my server was listening port 80. I changed it to port 7 and thankfully bug is resolved, now read() works as expected.
This bug let me think that LwIP had problems on listening that web(80) port instead of others. There should be a some kind of discrimination between listening some spectacular ports even for unimplemented protocols.
I want to read the data through socket in Qt. I am using QBytearray to store the data. Actually server sends 4095 bytes in a single stretch, but in the QT client side I am receiving in different chunks because of my application design.
void Dialog::on_pushButton_clicked()
{
socket=new QTcpSocket(this);
socket->connectToHost("172.17.0.1",5000);
if(socket->waitForConnected(-1))
qDebug()<<"Connected";
Read_data();
}
void Dialog::Read_data()
{
QString filename(QString("%1/%2.bin").arg(path,device));
qDebug()<<"filename"<<filename;
QFile fileobj(filename);
int cmd,file_size,percentage_completed;
if(!fileobj.open(QFile::WriteOnly | QFile::Text))
{
qDebug()<<"Cannot open file for writting";
return;
}
QTextStream out(&fileobj);
while(1)
{
socket->waitForReadyRead(-1);
byteArray=socket->read(4);
qDebug()<<"size of bytearray"<<byteArray.size();
length=0xffff & ((byteArray[3]<<8)|(0x00ff & byteArray[2]));
int rem;
byteArray=socket->read(length);
while(byteArray.size()!=length)
{
rem=length-byteArray.size();
byteArray.append( socket->read(rem));
}
fileobj.write(byteArray);
fileobj.flush();
byteArray.clear();
}
}
server code:
#include<stdio.h>
#include<stdlib.h>
#include<fcntl.h>
#include<sys/ioctl.h>
#include<mtd/mtd-user.h>
#include <sys/socket.h>
#include <sys/types.h>
#include <netinet/in.h>
#include <netdb.h>
#include <string.h>
#include <unistd.h>
#include <errno.h>
#include <arpa/inet.h>
#include<math.h>
#include <netinet/tcp.h>
static int msb,lsb,size,listenfd = 0, connfd = 0,len;
main()
{
struct sockaddr_in serv_addr;
serverlen=sizeof(serv_addr);
listenfd = socket(AF_INET, SOCK_STREAM, 0);
memset(&serv_addr, '0', sizeof(serv_addr));
serv_addr.sin_family = AF_INET;
serv_addr.sin_addr.s_addr = htonl(INADDR_ANY);
serv_addr.sin_port = htons(5000);
if(bind(listenfd,(struct sockaddr*)&serv_addr, sizeof(serv_addr))<0)
{
perror("\n Error in binding");
exit(1);
}
size=100000;
listen(listenfd, 1);
fd=fopen(new.bin,"r");
len=4089;
while(1)
{
buff[0]=25;
buff[1]=2;
buff[2]=60;
buff[3]=47;
n=fread(buff+4,1,length, fd);
buff[len+4]=5;
buff[len+5]='\n';
if(n>0)
sent_bytes=send(connfd,buff,n+6,0);
size =size-len;
if(size==0)
break;
}
}
If I execute the code in localhost(127.0.0.1) I can receive the data fully. The problem arises only when I connect to different host IP. Kindly help me in this regard
EDIT 1:
The problem is when bytesAvailable() returns the maximum bytes I am waiting for waitForReadyRead() times out. It works fine if the bytesAvailable() is less than as expected. Does bytesAvailable() allocate any buffer annoyed by this behaviour.
while(1)
{
while(socket->bytesAvailable()<4)
{
if (!socket->waitForReadyRead())
{
qDebug() << "waitForReadyRead() timed out";
return;
}
}
byteArray=socket->read(4);
length=0xffff & ((byteArray[3]<<8)|(0x00ff & byteArray[2]));
int rem_bytes=length+2;
qDebug()<<"bytes available"<<socket->bytesAvailable();
while(socket->bytesAvailable()<=rem_bytes)
{
qDebug()<<"reading";
if (!socket->waitForReadyRead(10000))//times out here if bytesAvailable() == rem_bytes but executes well in other cases
{
qDebug() << "waitForReadyRead() timed out";
return;
}
qDebug()<<"ready";
byteArray.append(socket->read(rem_bytes));
qDebug()<<"size of bytearray"<<byteArray.size();
if(byteArray.size()==length+2)
{
for(int j=0;j<length;j++)
newarray.append(byteArray[j]);
fileobj.write(newarray);
fileobj.flush();
newarray.clear();
byteArray.clear();
break;
}
else
{
rem_bytes -=byteArray.size();
}
}
Send();
}
I have tried by sending different data sizes cannot figure it out why?. Please provide me a solution pointing where I have gone wrong
Your problem stems from your misunderstanding of how TCP works.
When data is transmitted from a sender, it is broken into packets and then each packet is transmitted one by one until all the data has finished sending. If packets go missing, they are re-transmitted until either they reach their destination, or a timeout is reached.
As an added complication, each packet might follow various routes before arriving at the destination. The receiver has the task of acknowledging to the sender that packets have been received and then making sure that the packets are joined back together in the correct order.
For this reason, the longer the network route, the greater the chance of getting a delay in getting the data re-assembled. This is what you've been experiencing with your localhost versus networked-computer tests.
The IP stack on your computer does not wait for the complete data to arrive before passing it to your application but it will pause if it's missing a packet in sequence.
e.g. If you have 10 packets and packet 4 arrives last, the IP stack will pass the data to your application in two sets: 1-2-3, [[wait for 4 to arrive]], 4-5-6-7-8-9-10.
For this reason, when waitForReadyRead() returns true, you cannot expect that all your data has arrived, you must always check how many bytes have been actually received.
There are two places in your code where you wait for data. The first thing you wait for is a four-byte number to tell you how much data has been sent. Even though it's highly likely that you will have received all four bytes, it's good practice to check.
while(socket.bytesAvailable() < 4){
if (!socket.waitForReadyRead()) { // timeout after 30 second, by default
qDebug() << "waitForReadyRead() timed out";
return;
}
}
byteArray=socket->read(4);
qDebug()<<"size of bytearray"<<byteArray.size();
length=0xffff & ((byteArray[3]<<8)|(0x00ff & byteArray[2]));
The next thing you need to do is keep cycling through a wait-read-wait-read loop until all your data has arrived, each time keeping track of how many bytes you still expect to receive.
int bytesRemaining = length;
while(socket->bytesAvailable() < bytesRemaining){
if (!socket->waitForReadyRead()){
qDebug() "waitForReadyRead() timed out";
return;
}
// calling read() with the bytesRemaining argument will not guarantee
// that you will receive all the data. It only means that you will
// receive AT MOST bytesRemaining bytes.
byteArray = socket->read(bytesRemaining);
bytesRemaining -= byteArray.size();
fileobj.write(byteArray);
fileobj.flush();
}
All this said, you should not use the blocking API in your main thread or your GUI could freeze up. I suggest either using the asynchronous API, or create a worker thread to handle the downloading (and use the blocking API in the worker thread).
To see examples of how to use the two different APIs, looking in the documentation for the Fortune Client Example and the Blocking Fortune Client Example.
EDIT:
My apologies, there's a bug in the code above that doesn't take an number of possibilities into account, most importantly, if all data has already been received, and the end game once all data has finally arrived.
The following one-line change should clear that up:
Change
while(socket->bytesAvailable() < bytesRemaining){
To
while (bytesRemaining > 0) {
So you are saying that waitForReadyRead() returns false regardless of the time given once your the buffer has all 3000 expected bytes. What other behavior would you want? Perhaps you need rethink the trigger logic here. Many TCP/IP app protocols have some sort of frame start detection logic they combine with the required message size to then trigger processing. This lets them cope with widely different package sizes that the intermediate networks will impose, as well as truncated/partial messages. Once you have it working, connect to it by way of your cell phone and you will get different set of packet fragmentation examples to test with.
I'm currently working with an Arduino trying to build an ad hoc network to which a device can connect to and send web requests to. The problem I am currently having is that I can only set up one connection and then when that connection is terminated (with client.stop()), all subsequent connections are not picked up by the server, even a cURL command just sits there spinning. The first connection I start when I reset the server works fine, and I am able to talk to the server; but after that, the Arduino can no longer find new clients (even though it's trying with the library given).
I`m using the SparkFun library for the WiFly shield cloned from GitHub, along with an Arduino Uno.
My current code is based off their default example 'WiFly_AdHoc_Example', but I had to remove a few things to get the network to start up which might be the cause of this problem.
Here is the .ino file that I am running.
#include <SPI.h>
#include <WiFly.h>
//#include <SoftwareSerial.h>
//SoftwareSerial mySerial( 5, 4); //Part from example not used (see below)
WiFlyServer server(80); //Use telnet port instead, if debugging with telnet
void setup()
{
Serial.begin(9600);
//The code below is from the example, but when I run it the WiFly will hang
// on Wifly.begin(). Without it, the WiFly starts up fine.
//mySerial.begin(9600);
//WiFly.setUart(&mySerial); // Tell the WiFly library that we are not
// using the SPIUart
Serial.println("**************Starting WiFly**************");
// Enable Adhoc mod
WiFly.begin(true);
Serial.println("WiFly started, creating network.");
if (!WiFly.createAdHocNetwork("wifly"))
{
Serial.print("Failed to create ad hoc network.");
while (1)
{
// Hang on failure.
}
}
Serial.println("Network created");
Serial.print("IP: ");
Serial.println(WiFly.ip());
Serial.println("Starting Server...");
server.begin();
Serial.print("Server started, waiting for client.");
}
void loop()
{
delay(200);
WiFlyClient client = server.available();
if (client)
{
Serial.println("Client Found.");
// A string to store received commands
String current_command = "";
while (client.connected())
{
if (client.available())
{
//Gets a character from the sent request.
char c = client.read();
if (c=='#' || c=='\n') //End of extraneous output
{
current_command = "";
}
else if(c!= '\n')
{
current_command+=c;
}
if (current_command== "get")
{
// output the value of each analog input pin
for (int i = 0; i < 6; i++)
{
client.print("analog input ");
client.print(i);
client.print(" is ");
client.print(analogRead(i));
client.println("<br />");
}
}
else if(current_command== "hello")
{
client.println("Hello there, I'm still here.");
}
else if (current_command== "quit")
{
client.println("Goodbye...");
client.stop();
current_command == "";
break;
}
else if (current_command == "*OPEN*")
{
current_command == "";
}
}
}
// Give the web browser time to receive the data
delay(200);
// close the connection
client.stop();
}
}
This script is just a mini protocol I set up to test. Once connected with the wifly module you can send text such as "get" "hello" or "quit" and the wifly module should respond back.
Using Telnet I can successfully connect (the first time) and send commands to the Arduino including "quit" to terminate the connection (calls the client.stop() method). But when I try to reconnect though Telnet, it says the connection was successful, but on the Arduino it's still looping thinking the client is still false. What??
I know right, I'm getting mixed messages from Telnet vs Arduino. None of the commands work obviously since the Ardunio is still looping waiting for a client that evaluates to true. I'm going to take a look at WiFlyServer from the library I imported and see if I can dig up the problem, because somehow that server.available() method isn't finding new clients.
I am noticing a lot of TODO's in the library code....
So I found the reason for the problem. It was in the WiFlyServer.cpp file from the SparkFun library. The code that was causing the reconnect issue was in fact the server.availible() method. Right at the top of the method, there is a check:
// TODO: Ensure no active non-server client connection.
if (!WiFly.serverConnectionActive) {
activeClient._port = 0;
}
For some reason when I comment this out, I can connect and reconnect perfectly fine and everything works as it should. I will now dive into the library and see if I can fix this, I'm not exactly sure what this is doing, but it gets called when the server connection is not active and is somehow blocking subsequent connections. The problem with this solution is that the Arduino always thinks it has found a client since client and client.connected() evaluate to true even if one doesn't exist. Even client.available() evaluates to true right when the connection is terminated and the ghost "client" is found, but after that first run through the if-statement the ghost "client" is no longer available(). Even with this flaw it still picks up a new client when it comes along which is why it works.
How might I get to the root of this problem without using this commenting hack?
Are their any risks or future problems I might run into doing it this way?
What is the purpose of the block that I commented out in the first place?
Well, when you're calling client.stop(); how does the Arduino know whether the client has to start again?
Remember setup() executes only once.
Have you tried to include the following code in your loop to tell the Arduino to create the WiFly AdHoc network again? This may or may not work. I don't have one myself and haven't played with the Wifly shield but it's worth a try.
Remember to only ever execute the code once every time you need to connect again since it's sitting inside a loop that's always going to be running.
WiFly.begin(true);
Serial.println("WiFly started, creating network.");
if (!WiFly.createAdHocNetwork("wifly"))
{
Serial.print("Failed to create ad hoc network.");
while (1)
{
// Hang on failure.
}
}
I'm developing an application that uses IPC between a local server and a client application. There is nothing particular to it, as it's structured like the Qt documentation and examples.
The problem is that the client sends packets frequently and connecting/disconnecting from the server local socket (named pipe on NT) is very slow. So what I'm trying to achieve is a "persistent" connection between the two applications.
The client application connects to the local server (QLocalServer) without any problem:
void IRtsClientImpl::ConnectToServer(const QString& name)
{
connect(_socket, SIGNAL(connected()), this, SIGNAL(connected()));
_blockSize = 0;
_socket->abort();
_socket->connectToServer(name, QIODevice::ReadWrite);
}
And sends requests also in the traditional Qt manner:
void IRtsClientImpl::SendRequest( quint8 cmd, const QVariant* const param_array,
unsigned int cParams )
{
// Send data through socket
QByteArray hdr(PROTO_BLK_HEADER_PROJ);
QByteArray dataBlock;
QDataStream out(&dataBlock, QIODevice::WriteOnly);
out.setVersion(QDataStream::Qt_4_5);
quint8 command = cmd;
out << blocksize_t(0) // block size
<< hdr // header
<< quint32(PROTO_VERSION_PROJ) // protocol version
<< command // command
<< cParams; // number of valid parameters
for (unsigned int i = 0; i < cParams; ++i)
out << param_array[i];
// Write the current block size
out.device()->seek(0);
out << dataBlock.size() - sizeof(blocksize_t);
_socket->write(dataBlock);
}
No problem. But the trick resides on the readyRead() signal in the server-side. Here's the current implementation of the readyRead() handling slot:
void IRtsServerImpl::onReadyRead()
{
QDataStream in(_lsock);
in.setVersion(QDataStream::Qt_4_5);
if (_blocksize == 0)
{
qDebug("Bytes Available on socket: %d", _lsock->bytesAvailable());
if (_lsock->bytesAvailable() < sizeof(blocksize_t))
return;
in >> _blocksize;
}
// We need more data?
if (_lsock->bytesAvailable() < _blocksize)
return;
ReadRequest(in);
// Reset
_blocksize = 0;
}
Without setting _blocksize to zero I could not receive more data, only the first block group (I would expect an entire block to arrive without segmentation since this is through a pipe, but it does not, go figure). I expect that behavior, sure, since the _blocksize does not represent the current stream flow anymore. All right, resetting _blocksize does the trick, but I can't resend another packet from the client without getting an increasing array of bytes on the socket. What I want is to process the request in ReadRequest and receive the next data blocks without resorting to connecting/reconnecting the applications involved.
Maybe I should 'regulate' the rate of the incoming data?
Thank you very much.