Cannot receive large amount of serial data from Arduino - arduino

Question

I am trying to receive sensor data from Arduino and write the readings into a file using Processing IDE (using serial communication / USB).
After doing a large amount of tests, I am pretty sure it's the Processing side unable to process the data. Only the first few (< 100) "samples" are written successfully, after that Serial.available() always returns false.
I am sending two-byte chunks from Arduino, 57600 baud, default settings (8 bit, no parity, 1 stop bit).
Arduino code:
unsigned int data = 0;
unsigned char buf[2];
void setup() {
Serial.begin(57600);
}
void loop() {
data = analogRead(A0);
buf[0] = data & 0xFF; // low byte
buf[1] = data >> 8; // high byte
Serial.write(buf, 2);
Serial.flush();
delayMicroseconds(300);
}
Processing code:
import processing.serial.*;
Serial serialPort;
String serialData;
PrintWriter output;
int recordingTime = 1000; // how many miliseconds of data stream to record
byte[] dataBuffer = new byte[2]; // reserve memory for 2 bytes and initialize to 0 (java stuff)
int receivedBytes = 0;
void setup()
{
serialPort = new Serial(this, "/dev/ttyUSB0", 57600);
output = createWriter("serialData.txt");
}
void draw()
{
while(millis() < recordingTime) {
if (serialPort.available() > 0) {
receivedBytes = serialPort.readBytes(dataBuffer);
output.print("\n");
output.print("Received bytes: ");
output.print(receivedBytes);
output.print("\n");
output.println(binary(dataBuffer[0])); // low byte
output.println(binary(dataBuffer[1])); // high byte
output.println("");
}
else {
output.print("\n");
output.println("No data available");
}
}
output.flush();
output.close();
exit();
}
Output:
Received bytes: 2
11101001
00000011
Received bytes: 2
11101001
00000011
Received bytes: 2
11101001
00000011
...after some lines...
No data available
No data available
No data available
No data available
No data available
No data available
Why is this happening? Why is there "no data available" after few samples? If I watch the serial monitor output in Arduino IDE, it works fine.

Answer 1

I can read the serial data from Arduino using screen and using Python. Still cannot get it to work in Processing - only receiving few samples (17 exactly).
Screen command: $ screen <port name> <baud rate> (press ctrl+a and then shift+k to stop the program; add -L flag for logging to a file). Works as expected.
I managed to achieve the same results using Pyserial library in Python:
#
# log_serial.py
# Writes incoming serial data to file.
import time
import serial
# Edit this parameters =========================================================
serialPort = "/dev/ttyUSB0"
baudrate = 57600
recordTime = 1000 # milliseconds
# ==============================================================================
def millis():
"""
Returns current (wall-)time in milliseconds
"""
return int(round(time.time() * 1000))
ser = serial.Serial(serialPort, baudrate)
with open("output.txt", "w") as f:
startTime = millis()
f.write("Recording started at: ")
f.write(str(startTime))
f.write("\n")
while (millis() - startTime) <= recordTime:
inData = ser.read(2) # reads two bytes
inInt = int.from_bytes(inData, byteorder='little') # merges them into an integer
f.write(str(inInt))
f.write("\n")
f.write("Recording finished at: ")
f.write(str(millis()))
f.write("\n")
Still don't know why the Processing version can't handle it...
There was a bug (wrong while loop condition) in my first Processing code. Here is updated version, using Serial.read() rather than Serial.readBytes(buffer). Still doesn't solve my problem, only getting 17 samples:
import processing.serial.*;
Serial serialPort;
String serialData;
PrintWriter output;
int recordingTime = 5000; // how many miliseconds of data stream to record
void setup()
{
serialPort = new Serial(this, "/dev/ttyUSB0", 57600);
output = createWriter("serialData.txt");
}
void draw()
{
int startTime = millis();
while((millis() - startTime) <= recordingTime) {
if (serialPort.available() > 0) {
int b1 = serialPort.read();
int b2 = serialPort.read();
int value = (b2 << 8) + (b1 & 0xFF);
output.println(value);
}
}
output.flush();
output.close();
exit();
}

Answer 2

Please try to run this code and post the content of the written file afterwards:
import processing.serial.*;
Serial serialPort;
String serialData;
PrintWriter output;
int recordingTime = 5000; // how many miliseconds of data stream to record
void setup()
{
serialPort = new Serial(this, "/dev/ttyUSB0", 57600);
output = createWriter("serialData.txt");
}
void draw()
{
int startTime = millis();
while((millis() - startTime) <= recordingTime) {
if (serialPort.available() > 0) {
int b1 = serialPort.read();
int b2 = serialPort.read();
int value = (b2 << 8) + (b1 & 0xFF);
output.print(millis()); //to each received value there is written the actual millis()-value
output.print(",");
output.println(value);
}
}
output.flush();
output.close();
exit();
}

Answer 3

I have no problems with the Serial communication between Processing and my Arduino Uno/Mega using serialPort.read() instead of serialPort.readBytes(dataBuffer). Maybe this makes a difference...
Have you already tried to increase the variable recordingTime?