I'm using ESP32-CAM for video streaming using ESP-IDF framework. I was able to create server on ESP32-CAM and stream video sucessfully. For streaming I'm using HTTP protocol with Content-Type: multipart/x-mixed-replace; boundary=" PART_BOUNDARY "\r\n headers and it's working great :)
Now, I want to add a servo motor to the camera to adjust pan remotely. I think of 2 ways to do it.
Create another endpoint for servo, but for that I need to disconnect video streaming untill servo request is completed. This gives significant lag in video.
send data in between video streaming connection and get minimum lag in video.
I was able to use 2nd option in Arduino IDE for WebServer, there we have an option of reading binary data from client in an ongoing request. Example below
// only showing the relevent code ...
// create server.
WebServer server(80);
// then register different endpoint handlers ...
// ...
// ...
void video_stream_handler(){
// initilize camera stuff, nothing to worry here
camera_fb_t * fb = NULL;
esp_err_t res = ESP_OK;
size_t _jpg_buf_len = 0;
uint8_t * _jpg_buf = NULL;
char buf[32];
sensor_t* sensor_settings = esp_camera_sensor_get();
sensor_settings->set_framesize(sensor_settings, FRAMESIZE_VGA);
sensor_settings->set_quality(sensor_settings, 20);
// get client handle
WiFiClient client = server.client();
// now we can write headers as well as data to client. this works in ESP-IDF as well :)
client.write(<some-headers>, <header-length>);
// now this is interesting.
// we can read from client as well
client.read(); // gives bytes read from client
My question is, is it possible with esp-idf to do something like this?
Or if there is any other better alternative than all of it? I want minimum lag in video streaming while still performing servo action in between.
Hardware: ESP32-CAM(single core) with 4MB PSRAM
ESP-IDF http_server reference
P.S. I'm using Python socket to read/process the video stream and to send binary data in ongoing connection.
Related
I am making a system where I have a Pi that will send a string over UDP to an Arduino that has Wifi. The Arduino will then send some data back to the Pi. This part works flawlessly. Then the Pi will relay this same data over Serial to an Arduino Mega that has no Wifi. This is my problem. I am able to receive all the data over UDP but when I try to receive data and then send this over Serial, it glitches and nothing works. It just receives null values. I know this because on the Arduino Mega, I have setup a system where it will simply take the data it received and just send it back with the letters "ACK" signed on them so that I know that it was successfully received. All I get is null things. But when I just send the string "HI" over the Serial port and disable the UDP stuff, it magically works... sometimes. What could be going wrong?
Code on the Pi
#!/usr/bin/env python3
import serial
import time
import socket
serialData = 'null \n'
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
message = b'pi'
addr = ("192.168.71.146", 2390)
def wifirecv():
global serialData
s.sendto(message, addr)
data, address = s.recvfrom(10240)
#print(data.decode())
angle = data.decode()
serialData = angle + ' \n'
while(True):
ser = serial.Serial('/dev/ttyACM0', 115200, timeout=1)
ser.flush()
ser.close()
ser.open()
while True:
try:
wifirecv()
print(serialData)
ser.write(serialData.encode('utf-8'))
#ser.write(b'hi')
line = ser.readline().decode('utf-8').rstrip()
print(line)
except(KeyboardInterrupt):
ser.close()
print("Serial Closed")
exit()
Code on the Mega
void setup() {
Serial.begin(115200);
}
void loop() {
while(Serial.available() != 0){
String data = Serial.readStringUntil('\n');
Serial.println(data + ": ACK");
}
}
Despite you mention that the response from the Arduino works fine, I believe that you should use the SerialEvent interrupt callback rather than polling for the .available method. It is a much more reliable way to detect when data is present in the input buffer. I would form the input string char by char, as done in the official tutorial.
I think that the approach you opted for might be causing timing issues on the serial port since you said:
I am able to receive all the data over UDP but when I try to receive data and then send this over Serial, it glitches and nothing works
those glitches might also be a consequence of the selected baudrate or (again) the polling approach. This is reinforced by your statement:
Serial port and disable the UDP stuff, it magically works... sometimes
Have you tried using read_until(LF, None) instead of readline()? I've had better results with the first one in some acquisition routines. Check this for reference
Are you using any other resources from the Pi or the Arduino? I am not sure if the UDP messes up with the serial communication from the RPi. I would seriously doubt it, but just check if there are no reported issues with the socket library.
Finally, a tiny time.sleep(ms) between the write() and the read() in your RPi might not hurt the communication too much. For instance, in my recent experience with the I2C protocol, a tiny delay makes a huge difference for reliable communication.
The bottom line is: you have timing issues.
I am developing code for ESP32 (16MB) using PlatformIO with Arduino framework, On ESP32, I am using both Bluetooth as well as Wi-Fi connectivity. When Mobile App is connected with Bluetooth of ESP32 data will be sent on mobile using Bluetooth on BLE characteristics (working perfectly fine), and when Bluetooth is not connected data will be logged in .json file (have used SPIFFS library for this and file creation is working fine). When user wants to have log data, ESP32 will create Wi-Fi Access Point (AP) (used WiFi.h) and by connecting to that AP user should be able to download .json file by accessing link "192.168.4.1/download" on browser. Once the file will be downloaded on Mobile App user will access link "192.168.4.1/finisheddownload" and file will be deleted from ESP32.
after successfully connecting to AP of ESP32, while I access link "192.168.4.1/download" from browser, the code gets stuck at that point or sometimes it reboots the ESP32 and performs nothing and if I access "192.168.4.1/finisheddownload", it successfully deletes the file. My file size is around 10MB but even I perform this action when file size is of 1KB it behaves same. I have tested the same code on ESP32 (4MB) module and was able to download file for around 50KB.
I am not sure whether it is the issue of memory or library which I am using or limitation of ESP32. I am attaching code snippet along with memory partition I have used, please guide.
Error when ESP32 gets reboot after accessing link for file download:
abort() was called at PC 0x401d844b on core 1
Backtrace: 0x40092b60:0x3ffcf8e0 0x40092d91:0x3ffcf900 0x401d844b:0x3ffcf920 0x401d8492:0x3ffcf940 0x401bf359:0x3ffcf960 0x401bf0dc:0x3ffcf980 0x400fa67e:0x3ffcf9a0 0x400fa75a:0x3ffcf9d0 0x400e62b9:0x3ffcf9f0 0x400d23e4:0x3ffcfa20 0x400d27e3:0x3ffcfa40 0x400d2ed7:0x3ffcfa70 0x400d35f1:0x3ffcfa90 0x400e9151:0x3ffcfb10 0x4008f275:0x3ffcfb30
Rebooting...
rst:0xc (SW_CPU_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0018,len:4
load:0x3fff001c,len:1044
load:0x40078000,len:8896
load:0x40080400,len:5828
entry 0x400806ac
Code snippet:
void downloadfile(){ //function to download file on mobile App
delay(500);
do{
BLEAdvertising *pAdvertising = pServer->getAdvertising();
pAdvertising->stop();
if(!SPIFFS.begin()){
Serial.println("An Error has occurred while mounting SPIFFS");
return;
}
WiFi.mode(WIFI_AP);
delay(100);
Serial.print("Setting soft-AP ... ");
Serial.println(WiFi.softAP("ESP32","123456789") ? "Ready" : "Failed!");
Serial.print("Soft-AP IP address = ");
Serial.println(WiFi.softAPIP());
delay(100);
// send a file when /download is requested
server.on("/download", HTTP_ANY, [](AsyncWebServerRequest *request){
request->send(SPIFFS, "/file1.json", "text/plain", true);
});
server.on("/finisheddownload", HTTP_GET, [](AsyncWebServerRequest *request){
logfile.del_file();
request->send(200, "text/html", "Download complete response");
delay(500);
Actual_filexist = 0;
WiFi.softAPdisconnect(true);
BLEAdvertising *pAdvertising = pServer->getAdvertising();
pAdvertising->start();
delay(500);
});
server.begin();
delay(500);
}while(Actual_filexist == 1);
}
memory partition:
#name Type Subtype Offset Size Flags
nvs data nvs 0x9000 0x7000
app0 app ota_0 0x10000 0x2A0000
eeprom data 0x99 0x2F0000 0x1A8CE0
spiffs data spiffs 0x498CE0 0xA7D8C0
Memory usage:
RAM: [== ] 19.9% (used 65292 bytes from 327680 bytes)
Flash: [====== ] 60.1% (used 1654361 bytes from 2752512 bytes)
Thank you....
You're using ESPAsyncWebServer. Its documentation explicitly states that you should not call delay() or yield() or any function that calls them in callbacks from the web server (Important things to remember. You're calling delay() twice in the /finisheddownload handler.
You're also calling WiFi.softAPdisconnect() in the handler before the handler can return, which means you're turning off Wifi before the callback returns and potentially before the response is sent to the browser. It's not safe to do this in the callback.
You need to do as little as possible in the callback from the web server. Rewrite that handler to set a flag which loop() will inspect so that all of the work it does except for the call to request->send() is actually done in loop().
I'm working on a project based on the ESP32 platform. The aim is to count the number of MAC addresses in the area, and transmit this information over WiFi (using an http POST request).
The first task is achieved by sniffing WIFI packets and collecting the contained addresses, following this example: https://blog.podkalicki.com/esp32-wifi-sniffer/
I believe that the code which "sniffs" the packets sets the ESP to run in promiscuous mode, and therefore I cannot connect to any AP anymore.
I've tried several solutions, first starting with timer interrupts. However this approach always led to a Core Panic and reset of the chip.
I also learnt I could use RTOS to run different tasks in parallel on the two cores of the CPU, but that didn't help to solve the problem.
void wifi_sniffer_packet_handler(void* buff, wifi_promiscuous_pkt_type_t type)
{
if (type != WIFI_PKT_MGMT)//aggiungere filtro su RSSI a questa altezza.
return;
const wifi_promiscuous_pkt_t *ppkt = (wifi_promiscuous_pkt_t *)buff;
const wifi_ieee80211_packet_t *ipkt = (wifi_ieee80211_packet_t *)ppkt->payload;
const wifi_ieee80211_mac_hdr_t *hdr = &ipkt->hdr;
//some analysis and then print the MAC address
}
void setup() {
Serial.begin(115200);
timer = timerBegin(0, 80, true);
timerAttachInterrupt(timer, &chUpdate, true);
timerAlarmWrite(timer, 1000000, true);//timer, arr_val, reload=true
delay(4000);
wifi_sniffer_init();
timerAlarmEnable(timer);
}
// the loop function runs over and over again forever
void loop() {
//Serial.print("inside loop");
WiFi.begin(ssid, password);
while (WiFi.status() != WL_CONNECTED) {
delay(1000);
Serial.println("Establishing connection to WiFi..");
}
Serial.println("Connected to network");
}
I also noticed that the code in the loop gets stuck into the while, and is restarted every time the packet handler is run (I nevere get to see "Connected to network", but i see "Establishing connection to WiFi.." several times.
Anyone can explain me what's going on? Is there a different approach to achieve this result?
Thank you.
You may have two tasks and two cores, but the ESP32 still has only one wifi chip. The way your code is written (at least, the code you shared), you'll be trying to connect to a wifi network at the same time as you're trying to run promiscuous mode. You can do only one of those things at a time.
You'll need to stop promiscuous mode before you attempt to connect to an access point. Right now your code constantly attempt to connect to a wifi access point. Use a volatile variable to store the current mode - promiscuous or connected. Change it when you need to change states. Only attempt to connect to wifi when the variable says you want to be in connected mode.
There may be some code you need to run to turn off promiscuous mode when you change states, before you connect to a wifi access point.
If you're using wifi_sniffer_init() from the example you linked to, that code isn't meant to be run in an Arduino Core application. It does some network initialization that the Arduino Core will also do. It may not be safe to do that twice (it might work, it might not... but it's definitely not intended to be done that way).
You're setting an interrupt handle chUpdate() which you didn't share. I'd bet that's the cause of your Core Panics. You can do very little in an interrupt handler. You definitely can't call most Arduino Core functions or most ESP-IDF functions. Most code isn't protected against interrupts, so the timer interrupt can occur while data structures are in an inconsistent state. Re-entering code can corrupt the data structures and cause the kind of crash you described. You're best off setting a volatile variable and waking up a task that will do the work you need done while not in the interrupt handler.
Finally, you should call WiFi.mode(WIFI_STA); before you call WiFi.begin().
For anyone confused about why you can't connect to a wifi network while in promiscuous mode - what ESP8266 and ESP32 call "promiscuous mode" is really "wifi monitor mode", which lets you monitor a wifi radio channel and see all wifi frames sent on it. They use the term "promiscuous mode" differently from the rest of the industry. Usually "promiscuous mode" means seeing all the packets being sent on the network (wifi or hardwired) that you're connected to.
I know that it's been just over a year but THANKS to the info provided by romkey I think I solved this problem within my app by calling this routine before connecting to WiFi to upload MAC data.
void end_Scan_WiFi() {
esp_wifi_set_promiscuous(false);
esp_wifi_stop();
}
Followed by this ...
WiFi.mode(WIFI_STA);
WiFi.begin(ssid,pass);
I would like to know if there is anything like GCM for arduino.
I'm working on a project where I need to send push notifications to the arduino connected with WiFi shield. Arduino then will do some action based on the notification it gets or sync up with the server data.
Any advice on how I need to proceed would be helpful.
Thank you.
Yes you can send push notification to Arduino. There are multiple ways from where you can send push notification to Arduino.
I am using push notification from Parse.com.To send push notification from Parse just register with Parse.com and create account there.
Once you create account,you can send push notification from Parse dashboard.
To receive notification on Arduino you need to write sketch on Arduino, following is sketch for help.
Include Parse library to make sure following code will work.
/***************************************************************************************************************************
setup function
****************************************************************************************************************************/
void setup() {
Bridge.begin();
Serial.begin(9600);
while (!Serial);
Parse.begin("***E0uUjQkMa7nj5D5BALvzegzfyVNSG22BD2FJ", "umPSsggp5JgMFmSHfloewW5oixlM5ibt9LBS***");
// In this example, we associate this device with a pre-generated installation
Parse.getInstallationId();
Parse.startPushService();
}//setup function block
void loop() {
if (Parse.pushAvailable()) {
Serial.println("Start push");
ParsePush push = Parse.nextPush();
// Print whole JSON body
String message = push.getJSONBody();
// Serial.print("New push message size: ");
// Serial.println(message.length());
// Serial.print("New push message content: ");
Serial.println(message);
// Do something with the push
// IMPORTANT, close your push message
push.close();
checkStatus(message);
Serial.println("End Push");
}//if push notification block
} //loop
You can read documentation here https://parse.com/docs/arduino/guide#push-notifications.
I hope this will help.
You would have to run an HTTP server on the wifi shield which accepts requests, then send such a request from your outside system (command line on your desktop, IFTTT recipe, etc.)
I'm assuming this is one of the older 8bit Arduinos (since you mention the wifishield rather than something like the Yun with a Linux partition), so you will need to write a http server sketch on the Arduino.
Here is tutorial for a simple HTTP server with the wifi shield from arduino.cc
https://www.arduino.cc/en/Tutorial/WiFiWebServer
I'm trying to figure out how to stream audio in realtime over TCP sockets in Qt. What I'm using is a QAudioInput on the client and QAudioOutput on the server. Both are using the following format:
QAudioFormat format;
format.setChannelCount(1);
format.setSampleRate(8000);
format.setSampleSize(8);
format.setCodec("audio/pcm");
format.setByteOrder(QAudioFormat::LittleEndian);
format.setSampleType(QAudioFormat::UnSignedInt);
I have a simple socket server setup already and managed to stream the audio from the client to the server using:
//client
QAudioInput *audio = new QAudioInput(format, this);
audio->setBufferSize(1024);
audio->start(socket);
//server
QAudioOutput *audio = new QAudioOutput(format, this);
audio->setBufferSize(1024);
Then on the server I'm receiving the data and appending it to a QByteArray
On the server, I create a QBuffer and give it the QByteArray once the client closes, and then play it like this:
QByteArray *data = new QByteArray();
while(1)
{
if(socket->waitForReadyRead(30000))
data->append(socket->readAll());
else
break;
}
QBuffer *buffer = new QBuffer(data);
QEventLoop *loop = new QEventLoop(this);
buffer->open(QIODevice::ReadOnly);
audio->start(buffer);
loop->exec();
This will play the entire stream AFTER the client closes. What I'm looking for is to modify the server to play it in realtime, but I can't figure out how. I've gotten close to realtime but it had loud clicks between packets and was delayed several seconds.
I've tried playing the stream like how I used to send it:
audio->start(socket);
but this doesn't do anything. Maybe if I use QDataStream instead of directly using the sockets?
In order to solve that problem I simply increased the setBufferSize from your 1024 to a higher value, for example i've used 8192, that way you give the chance to the devices read more data at a time, and I was able to stream audio over my wireless network.
My class design is a little bit different than yours, I have different classes for sockets and audio, and you can see it here.