I'm trying to figure out how to stream audio in realtime over TCP sockets in Qt. What I'm using is a QAudioInput on the client and QAudioOutput on the server. Both are using the following format:
QAudioFormat format;
format.setChannelCount(1);
format.setSampleRate(8000);
format.setSampleSize(8);
format.setCodec("audio/pcm");
format.setByteOrder(QAudioFormat::LittleEndian);
format.setSampleType(QAudioFormat::UnSignedInt);
I have a simple socket server setup already and managed to stream the audio from the client to the server using:
//client
QAudioInput *audio = new QAudioInput(format, this);
audio->setBufferSize(1024);
audio->start(socket);
//server
QAudioOutput *audio = new QAudioOutput(format, this);
audio->setBufferSize(1024);
Then on the server I'm receiving the data and appending it to a QByteArray
On the server, I create a QBuffer and give it the QByteArray once the client closes, and then play it like this:
QByteArray *data = new QByteArray();
while(1)
{
if(socket->waitForReadyRead(30000))
data->append(socket->readAll());
else
break;
}
QBuffer *buffer = new QBuffer(data);
QEventLoop *loop = new QEventLoop(this);
buffer->open(QIODevice::ReadOnly);
audio->start(buffer);
loop->exec();
This will play the entire stream AFTER the client closes. What I'm looking for is to modify the server to play it in realtime, but I can't figure out how. I've gotten close to realtime but it had loud clicks between packets and was delayed several seconds.
I've tried playing the stream like how I used to send it:
audio->start(socket);
but this doesn't do anything. Maybe if I use QDataStream instead of directly using the sockets?
In order to solve that problem I simply increased the setBufferSize from your 1024 to a higher value, for example i've used 8192, that way you give the chance to the devices read more data at a time, and I was able to stream audio over my wireless network.
My class design is a little bit different than yours, I have different classes for sockets and audio, and you can see it here.
Related
I'm using ESP32-CAM for video streaming using ESP-IDF framework. I was able to create server on ESP32-CAM and stream video sucessfully. For streaming I'm using HTTP protocol with Content-Type: multipart/x-mixed-replace; boundary=" PART_BOUNDARY "\r\n headers and it's working great :)
Now, I want to add a servo motor to the camera to adjust pan remotely. I think of 2 ways to do it.
Create another endpoint for servo, but for that I need to disconnect video streaming untill servo request is completed. This gives significant lag in video.
send data in between video streaming connection and get minimum lag in video.
I was able to use 2nd option in Arduino IDE for WebServer, there we have an option of reading binary data from client in an ongoing request. Example below
// only showing the relevent code ...
// create server.
WebServer server(80);
// then register different endpoint handlers ...
// ...
// ...
void video_stream_handler(){
// initilize camera stuff, nothing to worry here
camera_fb_t * fb = NULL;
esp_err_t res = ESP_OK;
size_t _jpg_buf_len = 0;
uint8_t * _jpg_buf = NULL;
char buf[32];
sensor_t* sensor_settings = esp_camera_sensor_get();
sensor_settings->set_framesize(sensor_settings, FRAMESIZE_VGA);
sensor_settings->set_quality(sensor_settings, 20);
// get client handle
WiFiClient client = server.client();
// now we can write headers as well as data to client. this works in ESP-IDF as well :)
client.write(<some-headers>, <header-length>);
// now this is interesting.
// we can read from client as well
client.read(); // gives bytes read from client
My question is, is it possible with esp-idf to do something like this?
Or if there is any other better alternative than all of it? I want minimum lag in video streaming while still performing servo action in between.
Hardware: ESP32-CAM(single core) with 4MB PSRAM
ESP-IDF http_server reference
P.S. I'm using Python socket to read/process the video stream and to send binary data in ongoing connection.
I am making a system where I have a Pi that will send a string over UDP to an Arduino that has Wifi. The Arduino will then send some data back to the Pi. This part works flawlessly. Then the Pi will relay this same data over Serial to an Arduino Mega that has no Wifi. This is my problem. I am able to receive all the data over UDP but when I try to receive data and then send this over Serial, it glitches and nothing works. It just receives null values. I know this because on the Arduino Mega, I have setup a system where it will simply take the data it received and just send it back with the letters "ACK" signed on them so that I know that it was successfully received. All I get is null things. But when I just send the string "HI" over the Serial port and disable the UDP stuff, it magically works... sometimes. What could be going wrong?
Code on the Pi
#!/usr/bin/env python3
import serial
import time
import socket
serialData = 'null \n'
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
message = b'pi'
addr = ("192.168.71.146", 2390)
def wifirecv():
global serialData
s.sendto(message, addr)
data, address = s.recvfrom(10240)
#print(data.decode())
angle = data.decode()
serialData = angle + ' \n'
while(True):
ser = serial.Serial('/dev/ttyACM0', 115200, timeout=1)
ser.flush()
ser.close()
ser.open()
while True:
try:
wifirecv()
print(serialData)
ser.write(serialData.encode('utf-8'))
#ser.write(b'hi')
line = ser.readline().decode('utf-8').rstrip()
print(line)
except(KeyboardInterrupt):
ser.close()
print("Serial Closed")
exit()
Code on the Mega
void setup() {
Serial.begin(115200);
}
void loop() {
while(Serial.available() != 0){
String data = Serial.readStringUntil('\n');
Serial.println(data + ": ACK");
}
}
Despite you mention that the response from the Arduino works fine, I believe that you should use the SerialEvent interrupt callback rather than polling for the .available method. It is a much more reliable way to detect when data is present in the input buffer. I would form the input string char by char, as done in the official tutorial.
I think that the approach you opted for might be causing timing issues on the serial port since you said:
I am able to receive all the data over UDP but when I try to receive data and then send this over Serial, it glitches and nothing works
those glitches might also be a consequence of the selected baudrate or (again) the polling approach. This is reinforced by your statement:
Serial port and disable the UDP stuff, it magically works... sometimes
Have you tried using read_until(LF, None) instead of readline()? I've had better results with the first one in some acquisition routines. Check this for reference
Are you using any other resources from the Pi or the Arduino? I am not sure if the UDP messes up with the serial communication from the RPi. I would seriously doubt it, but just check if there are no reported issues with the socket library.
Finally, a tiny time.sleep(ms) between the write() and the read() in your RPi might not hurt the communication too much. For instance, in my recent experience with the I2C protocol, a tiny delay makes a huge difference for reliable communication.
The bottom line is: you have timing issues.
I am trying to send and receive some serial data with QSerialPort but it take a little time. So I open a Serial Terminal. (Terminal v1.93b - 20141030B) Then with it terminal I read and send data and when I tried again with my application, it starts correctly.
So why it happens after do I open the serial terminal and some data are sended the QSerialPort works fine. It looks like if I need to reset or refresh the serial buffer on my port.
Try to call the function clear() after opening a serial port, or to call the function flush() after each sending data.
bool QSerialPort::clear(Directions directions = AllDirections)
bool QSerialPort::flush()
I am writing a small application in QT that sends a UDP packet broadcast over the local network and waits for a UDP responce packet from one or more devices over the network.
Creating the sockets and sending the broadcast packet.
udpSocketSend = new QUdpSocket(this);
udpSocketGet = new QUdpSocket(this);
bcast = new QHostAddress("192.168.1.255");
udpSocketSend->connectToHost(*bcast,65001,QIODevice::ReadWrite);
udpSocketGet->bind(udpSocketSend->localPort());
connect(udpSocketGet,SIGNAL(readyRead()),this,SLOT(readPendingDatagrams()));
QByteArray *datagram = makeNewDatagram(); // data from external function
udpSocketSend->write(*datagram);
The application sends the packet properly and the response packet arrives but the readPendingDatagrams() function is never called. I have verified the packets are sent and received using Wireshark and that the application is listening on the port indicated in wireshark using Process Explorer.
I solved the problem. Here is the solution.
udpSocketSend = new QUdpSocket(this);
udpSocketGet = new QUdpSocket(this);
host = new QHostAddress("192.168.1.101");
bcast = new QHostAddress("192.168.1.255");
udpSocketSend->connectToHost(*bcast,65001);
udpSocketGet->bind(*host, udpSocketSend->localPort());
connect(udpSocketGet,SIGNAL(readyRead()),this,SLOT(readPendingDatagrams()));
QByteArray *datagram = makeNewDatagram(); // data from external function
udpSocketSend->write(*datagram);
The device on the network listens on port 65001 and responds to packets on the source port of the received packet. It is necessary to use connectToHost(...) in order to know what port to bind for the response packet.
It is also necessary to bind to the correct address and port to receive the packets. This was the problem.
You're binding your udpSocketSend in QIODevice::ReadWrite mode. So that's the object that's going to be receiving the datagrams.
Try one of:
binding the send socket in write only mode, and the receive one in receive only mode
using the same socket for both purposes (remove udpSocketGet entirely).
depending on your constraints.
For me, changing the bind from
udpSocket->bind(QHostAddress::LocalHost, 45454);
to simple
udpSocket->bind(45454);
does the trick!
I have a network application which uses UDP broadcasts for device discovery, but only accepts one connection at a time. So, when a new TCP connection is made, I delete the QUdpSocket that was used for discovery.
However, when the remote device is disconnected, I want to create a new QUdpSocket and start listening again:
// Set up a UDP server to respond to any "discovery" messages:
udpServer = new QUdpSocket(this);
if (udpServer -> bind(QHostAddress::Any, DISCOVERY_PORT))
connect(udpServer, SIGNAL(readyRead()),
this, SLOT(beDiscovered()));
else
{
fprintf(stderr, "UDP port not bound successfully: %d, ", udpServer ->error());
fprintf(stderr, udpServer ->errorString().toLocal8Bit());
fprintf(stderr, "\r\n");
fflush(stderr);
#ifdef WIN32
_commit(_fileno(stderr));
#else
fsync(_fileno(stderr));
#endif
}
The re-bind fails, however, with code 8, "The bound address is already in use".
So, how can I make sure that when the 'old' QUdpSocket was deleted, it fully releases the address(es) it was bound to?
Alternatievly, should I be binding with QUdpSocket::ShareAddress or QUdpSocket::ReuseAddressHint? This doesn't seem right, as neither really describe the behaviour I want, namely an exclusive binding for my QUdpSocket during its lifetime, and in any case QUdpSocket::ShareAddress is supposed to be the default on Windows.
Thanks,
Stephen.
...so in other words the question has answered itself!