I am running a TCP HTTP server on my NodeMCU. I am serving files, such as HTML, CSS and JavaScript. This was fine until the code started to get long.
What's happening is that the response is just getting cut off. It's happening at about exactly 1024 characters (it seems to be the magic number).
A solution would be to host the files on a server such as Google Drive, Dropbox or Github. However, internet access is not available, because the server is being run through a hotspot created by the NodeMCU.
Any way to override this limit?
The ESP cannot stream data across multiple IP packets, which is why it gets cropped after 1024 characters.
Here is how I am serving larger files on my ESP running NodeMCU:
responseQueue = {}
function processQueue(socket)
if #responseQueue > 0 then
socket:send( table.remove(responseQueue, 1) )
else
socket:close()
collectgarbage()
end
end
function sendFile(conn, filename)
if file.open(filename, "r") then
table.insert(responseQueue, 'HTTP/1.1 200 OK\r\n')
table.insert(responseQueue, 'Content-Type: text/html\r\n\r\n')
local lastPos = 0
repeat
file.seek("set", lastPos)
local line = file.read(512)
file.close()
if line then
table.insert(responseQueue, line)
lastPos = lastPos + 512
else
lastPos = nil
end
until(line == nil)
end
processQueue(conn)
end
Side note: The original answer that was given by #TerryE here pointed me to the above posted implementation.
Related
I have built a prototype using a raspberry and a sim5320E module. The goal is to send a large amount of data (~100Kb) through HTTP using this 3G module.
I have followed the instructions specified in section 16.5 (HTTPS) of the AT Command set for the SIM5320:
https://cdn-shop.adafruit.com/datasheets/SIMCOM_SIM5320_ATC_EN_V2.02.pdf
And it worked fine, except that it is slow.
From what I understand from the documentation (and seen from my tests), the data to be sent must be divided in chunks of max 4096 bytes.
Every chunk must be sent to what is called the "sending buffer" using the command AT+CHTTPSSEND.
Every now and then, we must check that the sending buffer does not have too much data in cache using the AT+CHTTPSSEND? command.
The last AT+CHTTPSSEND command commits all sending data.
My problem is that every AT+CHTTPSSEND takes around 10 seconds to complete, which means that my HTTP request will take around 250 seconds to complete.
Anybody knows what might cause this slowness?
Here is some code to illustrate the issue:
def send_chunk(self, chunk):
# Send chunk
self._send('CHTTPSSEND={}'.format(len(chunk)), wait_for=">")
self._send_raw(chunk.encode())
# Check how much data is left in the sending buffer
# Wait for this data to be under 3Kb
data_left = 3001
while data_left > 3000:
response = self._send('CHTTPSSEND?', wait_for="+CHTTPSSEND:")
data_left = int(response.strip().split(" ")[1])
time.sleep(2)
And here are the logs I get:
>> AT+CHTTPSSEND=4096 -> This commands takes ~10 seconds
<< >
>> Sending chunk of data
<< OK
>> AT+CHTTPSSEND?
<< +CHTTPSSEND: 0
I am trying to download files to disk from squeak.
My method worked fine for small text/html files,
but due to lack of buffering,
it was very slow for the large binary file
https://mirror.racket-lang.org/installers/6.12/racket-6.12-x86_64-win32.exe.
Also, after it finished, the file was much larger (113 MB)
than shown on download page (75MB).
My code looks like this:
download: anURL
"download a file over HTTP and save it to disk under a name extracted from url."
| ios name |
name := ((anURL findTokens: '/') removeLast findTokens: '?') removeFirst.
ios := FileStream oldFileNamed: name.
ios nextPutAll: ((HTTPClient httpGetDocument: anURL) content).
ios close.
Transcript show: 'done'; cr.
I have tried [bytes = stream next bufSize. bytes printTo: ios] for fixed size blocks in HTTP response's contentStream using a [stream atEnd] whileFalse: loop, but that garbled the output file with single quotes around each block, and also extra content after the blocks, which looked like all characters of the stream, each single quoted.
How can I implement buffered writing of an HTTP response to a disk file?
Also, is there a way to do this in squeak while showing download progress?
As Leandro already wrote the issue is with #binary.
Your code is nearly correct, I have taken the liberty to run it - now it downloads the whole file correctly:
| ios name anURL |
anURL := ' https://mirror.racket-lang.org/installers/6.12/racket-6.12-x86_64-win32.exe'.
name := ((anURL findTokens: '/') removeLast findTokens: '?') removeFirst.
ios := FileStream newFileNamed: 'C:\Users\user\Downloads\_squeak\', name.
ios binary.
ios nextPutAll: ((HTTPClient httpGetDocument: anURL) content).
ios close.
Transcript show: 'done'; cr.
As for the freezing, I think the issue is with the one thread for the whole environment while you are downloading. That means that means till you download the whole file you won't be able to use Squeak.
Just tested in Pharo (easier install) and the following code works as you want:
ZnClient new
url: 'https://mirror.racket-lang.org/installers/6.12/racket-6.12-x86_64-win32.exe';
downloadTo: 'C:\Users\user\Downloads\_squeak'.
The WebResponse class, when building the response content, creates a buffer large enough to hold the entire response, even for huge responses! I think this happens due to code in WebMessage>>#getContentWithProgress:.
I tried to copy data from the input SocketStream of WebResponse directly to an output FileStream.
I had to subclass WebClient and WebResponse, and write a two methods.
Now the following code works as required.
| client link |
client := PkWebClient new.
link := 'http://localhost:8000/racket-6.12-x86_64-linux.sh'.
client download: link toFile: '/home/yo/test'.
I have verified block by block update and integrity of the downloaded file.
I include source below. The method streamContentDirectToFile: aFilePathString is the one that does things differently and solves the problem.
WebClient subclass: #PkWebClient
instanceVariableNames: ''
classVariableNames: ''
poolDictionaries: ''
category: 'PK'!
!PkWebClient commentStamp: 'pk 3/28/2018 20:16' prior: 0!
Trying to download http directly to file.!
!PkWebClient methodsFor: 'as yet unclassified' stamp: 'pk 3/29/2018 13:29'!
download: urlString toFile: aFilePathString
"Try to download large files sensibly"
| res |
res := self httpGet: urlString.
res := PkWebResponse new copySameFrom: res.
res streamContentDirectToFile: aFilePathString! !
WebResponse subclass: #PkWebResponse
instanceVariableNames: ''
classVariableNames: ''
poolDictionaries: ''
category: 'PK'!
!PkWebResponse commentStamp: 'pk 3/28/2018 20:49' prior: 0!
To make getContentwithProgress better.!
]style[(38)f1!
!PkWebResponse methodsFor: 'as yet unclassified' stamp: 'pk 3/29/2018 13:20'!
streamContentDirectToFile: aFilePathString
"stream response's content directly to file."
| buffer ostream |
stream binary.
buffer := ByteArray new: 4096.
ostream := FileStream oldFileNamed: aFilePathString.
ostream binary.
[stream atEnd]
whileFalse: [buffer := stream nextInBuffer: 4096.
stream receiveAvailableData.
ostream nextPutAll: buffer].
stream close.
ostream close! !
I am using the following python script to write AT+CSQ on serial port ttyUSB1.
But I cannot read anything.
However, when I fire AT+CSQ on minicom, I get the required results.
What may be the issue with this script?
Logs:
Manual Script
root#imx6slzbha:~# python se.py
Serial is open
Serial is open in try block also
write data: AT+CSQ
read data:
read data:
read data:
read data:
Logs:
Minicom console
1. ate
OK
2. at+csq
+CSQ: 20,99
3. at+csq=?
OKSQ: (0-31,99),(99)
How can I receive these results in the following python script?
import serial, time
#initialization and open the port
#possible timeout values:
# 1. None: wait forever, block call
# 2. 0: non-blocking mode, return immediately
# 3. x, x is bigger than 0, float allowed, timeout block call
ser = serial.Serial()
ser.port = "/dev/ttyUSB1"
ser.baudrate = 115200
ser.bytesize = serial.EIGHTBITS #number of bits per bytes
ser.parity = serial.PARITY_NONE #set parity check: no parity
ser.stopbits = serial.STOPBITS_ONE #number of stop bits
ser.timeout = None #block read
#ser.timeout = 0 #non-block read
ser.timeout = 3 #timeout block read
ser.xonxoff = False #disable software flow control
ser.rtscts = False #disable hardware (RTS/CTS) flow control
ser.dsrdtr = False #disable hardware (DSR/DTR) flow control
ser.writeTimeout = 2 #timeout for write
try:
ser.open()
print("Serial is open")
except Exception, e:
print "error open serial port: " + str(e)
exit()
if ser.isOpen():
try:
print("Serial is open in try block also")
ser.flushInput() #flush input buffer, discarding all its contents
ser.flushOutput()#flush output buffer, aborting current output
#and discard all that is in buffer
#write data
ser.write("AT+CSQ")
time.sleep(1)
# ser.write("AT+CSQ=?x0D")
print("write data: AT+CSQ")
# print("write data: AT+CSQ=?x0D")
time.sleep(2) #give the serial port sometime to receive the data
numOfLines = 1
while True:
response = ser.readline()
print("read data: " + response)
numOfLines = numOfLines + 1
if (numOfLines >= 5):
break
ser.close()
except Exception, e1:
print "error communicating...: " + str(e1)
else:
print "cannot open serial port "
You have two very fundamental flaws in your AT command handling:
time.sleep(1)
and
if (numOfLines >= 5):
How bad are they? Nothing will ever work until you fix those, and by that I mean completely change the way you send and receive command and responses.
Sending AT commands to a modem is a communication protocol like any other protocols, where certain parts and behaviours are required and not optional. Just like you would not write a HTTP client that completely ignores the responses it gets back from the HTTP server, you must never write a program that sends AT commands to a modem and completely ignores the responses the modem sends back.
AT commands are a link layer protocol, with with a window size of 1 - one. Therefore after sending a command line, the sender MUST wait until has received a response from the modem that it is finished with processing the command line, and that kind of response is called Final result code.
If the modem uses 70ms before it responds with a final result code you have to wait at least 70ms before continuing, if it uses 4 seconds you have to wait at least 4 seconds before continuing, if it uses several minutes (and yes, there exists AT commands that can take minutes to complete) you have to wait for several minutes. If the modem has not responded in an hour, your only options are 1) continue waiting, 2) just give up or 3) disconnect, reconnect and start all over again.
This is why sleep is such a horrible approach that in the very best case is a time wasting ticking bomb. It is as useful as kicking dogs that stand in your way in order to get them to move. Yes it might actually work some times, but at some point you will be sorry for taking that approach...
And regarding numOfLines there is no way anyone in advance can know exactly how many lines a modem will respond with. What if your modem just responds with a single line with the ERROR final result code? The code will deadlock.
So this line number counting has to go completely away, and instead your code should be sending a command line and then wait for the final result code by reading and parsing the response lines from the modem.
But before diving too deep into that answer, start by reading the V.250 specification, at least all of chapter 5. This is the standard that defines the basics of AT command, and will for instance teach you the difference between a command and a command line. And how to correctly terminate a command line which you are not doing, so the modem will never start processing the commands you send.
I program an ESP8266 NodeMCU with Lua script. As I was debugging the problem that strings just were cut off at the beginning and extended further. I send from ESP8266 to an Android Phone.
I looked more in testing the esp via UART Interface, following problem:
the maximum String size when I declare a string container is 247 characters. After I exceed the 247th there is an error:
stdin:1: unexpected symbol near '='
The String is obviously too long but I need to send at least 2048 bytes per String for maximum efficiency. Is it possible to extend the input limit of a string variable?
(I build a 2048 bytes Packet and 86 bytes Overhead for the HTTP Get Response)
The TCP Tx Buffer of ESP8266 is 2920 bytes.
str_resp0 = "HTTP/1.1 200 OK\r\nContent-Type: text/html; charset=UTF-8\r\n";
str_resp1 = "Connection: close\r\n\r\n";
send_buf = "";
uart.on("data", "$",
function(data)
t = {send_buf,data}
send_buf = table.concat(t);
if data=="quit$" then
uart.on("data") -- quit the function
end
end, 0)
srv=net.createServer(net.TCP)
srv:listen(80,function(conn)
conn:on("receive",function(conn,payload)
--print(payload)
conn:send(str_resp0)
conn:send(str_resp1)
conn:send(send_buf)
send_buf = "";
end)
conn:on("sent",function(conn) conn:close()
end)
end)
stdin:1: unexpected symbol near '='
sounds very much like a problem with your "IDE" (ESPlorer?).
Furthermore, you should send long payloads in batches. The SDK limits the packet size to some 1500 bytes which is the standard Ethernet MTU.
http://nodemcu.readthedocs.io/en/latest/en/modules/net/#netsocketsend has some explanations and a nice example.
srv = net.createServer(net.TCP)
srv:listen(80, function(conn)
conn:on("receive", function(sck, req)
local response = {}
-- if you're sending back HTML over HTTP you'll want something like this instead
-- local response = {"HTTP/1.0 200 OK\r\nServer: NodeMCU on ESP8266\r\nContent-Type: text/html\r\n\r\n"}
response[#response + 1] = "lots of data"
response[#response + 1] = "even more data"
response[#response + 1] = "e.g. content read from a file"
-- sends and removes the first element from the 'response' table
local function send(sk)
if #response > 0
then sk:send(table.remove(response, 1))
else
sk:close()
response = nil
end
end
-- triggers the send() function again once the first chunk of data was sent
sck:on("sent", send)
send(sck)
end)
end)
Update 2016-07-18
Results of some experimenting in ESPlorer. The test uses a single initialized variable:
buffer = "A very long string with more than 250 chars..."
Hitting 'Send to ESP' (sending char-by-char over UART) will fail with an error similar to the one reported here.
Saving this into a file works fine.
Hit 'Save', will save the line to a file on your file system e.g. paul.lua
Hit 'Save to ESP', will send paul.lua to the device.
If it's not done automatically as part of 'Save to ESP' you can send dofile("paul.lua") to the device. This will make the variable buffer available in the global space.
Sending print(buffer) will print the entire string to the terminal window.
The Problem is as it seems unavoidable. Referring to the espressif forums:
"the delay added by the lower level code is 20ms it is documented!"
so the event frame could'nt get handled faster than this. The only way to tweak this might be buffering the data in microcontroller and sending it at once every 20ms or installing the so called "FreeRTOS SDK for ESP8266", whose transfer speed is only limited by uart speed.
I have a simple script which transfers an image file from one machine to another,
does image processing and returns a result as "dice count".
The problem is that randomly the image received is partially grey. There seems no
reason as to when the transfer is incomplete and I see no other issues in the code.
The client gives no error or any indication that sendall failed or was incomplete.
Thanks guys.
server code:
def reciveImage():
#create local buffer file
fo = open("C:/foo.jpg", "wb")
print "inside reciveImage"
while True:
print "inside loop"
data = client.recv(4096)
print "data length: ", len(data)
fo.write(data)
print "data written"
if (len(data) < 4096):
break
print "break"
fo.close()
print "Image received"
And the (simplifyed) client code:
data = open("/home/nao/recordings/cameras/image1.jpg", "rb")
# prepare to send data
binary_data = data.read()
data.close()
sock.sendall(binary_data)
Normal server output:
Client Command: findBlob. Requesting image...
inside reciveImage
inside loop
data length: 4096
data written
#... This happens a bunch of times....
inside loop
data length: 4096
data written
inside loop
data length: 1861
data written
Image received
dice count: 0
Waiting for a connection
But randomly it will only loop a few times or less, like:
Client Command: findBlob. Requesting image...
inside reciveImage
inside loop
data length: 1448
data written
Image received
dice count: 0
Waiting for a connection
recv does not block until all data are received, it only blocks until some data are received and returns them. E.g. if the client send first 512 bytes and then a second later another 512 byte your recv will return the first 512 bytes, even if you asked for 4096. So you should only break if recv returns, that no more data are available (connection closed).