Reading RAW data from an R socket - r

To exchange data very fast between Python and R I programmed a rather dirty solution, which works. On linux and OSX. May it not be that I now have to get this working in windows.
The below code runs a python script that builds a raw vector which can be serialised by RApiSerialise to an R object.
COMMAND = "python"
PATH_TO_SCRIPT='/GetCassandraData.py'
QueryCassandra <- function(query){
allArgs = c(PATH_TO_SCRIPT, query)
output.connection <- rawConnection(raw(length = 0), "r+")
exec_wait(COMMAND, args = allArgs, std_out = output.connection)
output <- rawConnectionValue(output.connection)
close(output.connection)
final <- unserializeFromRaw(output)
return(final)
}
This works as intended on OSX & linux however, windows has the tendancy to put a 0x0d (Carriage return) byte before a 0x0a (line feed) byte which makes RApiSerialise unable to deserialise it.
I am now attempting to solve the problem by communicating through sockets but I do not seem to be able to find a way to read data from a make.socket() object to a raw vector.
I have tried:
data <- read.socket(datasocket)
Which resulted in:
Error in read.socket(datasocket) :
embedded nul in string: 'X\n\0\0\0\002\0\003\004\002\0\002\003\0'
The function read.socket() tries to read a string and doesn't accept null bytes.
Is there a way to read socket data to a raw vector in R?

R server-side:
library(sys)
COMMAND = "python"
PATH_TO_SCRIPT='/lengthCheck.py'
allArgs = c(PATH_TO_SCRIPT)
sys::exec_background(COMMAND, args = allArgs, std_out = TRUE, std_err = TRUE)
datasocket <- socketConnection(port = 1205, server = TRUE, open = "w+b", blocking = TRUE)
on.exit(close(datasocket))
datasize <- readBin(datasocket, what = "double")
data <- readBin(datasocket, what = "raw", n = datasize)
Python client-side:
import struct
import socket
import time
your_raw_array_to_send = bytearray([0x58, 0x0a, 0x00, 0x00, 0x00, 0x02, 0x00, 0x03, 0x04, 0x02, 0x00, 0x02, 0x03, 0x00])
arrayLength = len(your_raw_array_to_send)
datasize = struct.pack('d', arrayLength)
# Wait 100ms for R to set up a listening socket
time.sleep(.100)
client_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client_socket.connect(('localhost', 1205))
client_socket.send(datasize)
client_socket.send(your_raw_array_to_send)
client_socket.close()

Related

R How to convert a byte in a raw vector into a ascii space

I am reading some very old files created by C code that consist of a header (ASCII) and then data. I use readBin() to get the header data. When I try to convert the header to a string it fails because there are 3 'bad' bytes. Two of them are binary 0 and the other binary 17 (IIRC).
How do I convert the bad bytes to ASCII SPACE?
I've tried some versions of the below code but it fails.
hd[hd == as.raw(0) | hd == as.raw(0x17)] <- as.raw(32)
I'd like to replace each bad value with a space so I don't have to recompute all the fixed data locations in parsing the string derived from hd.
I normally just go through a conversion to integer.
Suppose we have this raw vector:
raw_with_null <- as.raw(c(0x48, 0x65, 0x6c, 0x6c, 0x6f, 0x00,
0x57, 0x6f, 0x72, 0x6c, 0x64, 0x21))
We get an error if we try to convert it to character because of the null byte:
rawToChar(raw_with_null)
#> Error in rawToChar(raw_with_null): embedded nul in string: 'Hello\0World!'
It's easy to convert to numeric and replace any 0s or 23s with 32s (ascii space)
nums <- as.integer(raw_with_null)
nums[nums == 0 | nums == 23] <- 32
We can then convert nums back to raw and then to character:
rawToChar(as.raw(nums))
#> [1] "Hello World!"
Created on 2022-03-05 by the reprex package (v2.0.1)

How can I cast byte arrays to struct in Julia?

I'm new to Julia. I'm trying to parse a structured binary file. I read n bytes from the file and I want to cast the byte array to an object of type X.
struct X
messageType::UInt8
second::UInt32
end
f = open("myfile.bin")
bytes = read(f, 5)
And now I want to cast bytes to an object of X. How can I do this?
You can use StructIO here is how.
Setup:
using StructIO
#io struct XX
messageType::UInt8
second::UInt32
end align_packed
bytes = UInt8[0x72, 0xa3, 0x97, 0xcf, 0x64]
buf = IOBuffer(bytes)
And now running the code:
julia> seekstart(buf); unpack(buf, XX)
XX(0x72, 0x64cf97a3)
julia> seekstart(buf); unpack(buf, XX, :BigEndian)
XX(0x72, 0xa397cf64)

TypeError: initializer for ctype 'unsigned int *' must be a cdata pointer, not bytes

I try to convert PIL image to leptonica PIX. Here is my code python 3.6:
import os, cffi
from PIL import Image
# initialize leptonica
ffi = cffi.FFI()
ffi.cdef("""
typedef int l_int32;
typedef unsigned int l_uint32;
struct Pix;
typedef struct Pix PIX;
PIX * pixCreate (int width, int height, int depth);
l_int32 pixSetData (PIX *pix, l_uint32 *data);
""")
leptonica = ffi.dlopen(os.path.join(os.getcwd(), "leptonica-1.78.0.dll"))
# convert PIL to PIX
im = Image.open("test.png").convert("RGBA")
depth = 32
width, height = im.size
data = im.tobytes("raw", "RGBA")
pixs = leptonica.pixCreate(width, height, depth)
leptonica.pixSetData(pixs, data)
pixSetData failes with message: TypeError: initializer for ctype 'unsigned int *' must be a cdata pointer, not bytes.
How to convert bytes object (data) to cdata pointer?
I got answer from Armin Rigo at python-cffi forum:
Assuming you have the recent cffi 1.12, you can do:
leptonica.pixSetData(pixs, ffi.from_buffer("l_uint32[]", data))
The backward-compatible way is more complicated because we need to
make sure an intermediate object stays alive:
p = ffi.from_buffer(data)
leptonica.pixSetData(pixs, ffi.cast("l_uint32 *", p))
# 'p' must still be alive here after the call, so put it in a variable above!
PIL and Leptonica seem not to share exactly the same raw format. At last RGBA vs. ABGR. What worked for me was to use uncompressed TIFF as a fast and dependable data exchange format.
# Add these to ffi.cdef():
#
# typedef unsigned char l_uint8;
# PIX * pixReadMem(const l_uint8 *data, size_t size);
# l_ok pixWriteMem(l_uint8 **pdata, size_t *psize, PIX *pix, l_int32 format);
from io import BytesIO
import PIL.Image
IFF_TIFF = 4
def img_pil_to_lepto(pilimage):
with BytesIO() as bytesio:
pilimage.save(bytesio, 'TIFF')
tiff_bytes = bytesio.getvalue()
cdata = ffi.from_buffer('l_uint8[]', tiff_bytes)
pix = leptonica.pixReadMem(cdata, len(tiff_bytes))
return pix
def img_lepto_to_pil(pix):
cdata_ptr = ffi.new('l_uint8**')
size_ptr = ffi.new('size_t*')
leptonica.pixWriteMem(cdata_ptr, size_ptr, pix, IFF_TIFF)
cdata = cdata_ptr[0]
size = size_ptr[0]
tiff_bytes = bytes(ffi.buffer(cdata, size))
with BytesIO(tiff_bytes) as bytesio:
pilimage = PIL.Image.open(bytesio).copy()
return pilimage

igraph.to.gexf encoding issue

I'm trying to export a network I've built in igraph to a gexf format using rgexf so I can use it in Gephi. My basic code is below, plus then I have a few formatting things (removing labels and such) which I haven't included below.
df <- read.csv ("<file>", header = TRUE, sep = ",")
df.network <- graph.data.frame(df, directed=F)
V(df.network)$type <-bipartite.mapping(df.network)$type
plot(df.network, vertex.label.cex = 0.8, vertex.label.color = "black", layout=layout_with_kk, asp=0)
I then use the following
g1.gexf <- igraph.to.gexf(df.network)
but I get the following error message:
Input is not proper UTF-8, indicate encoding !
Bytes: 0xED 0x6E 0x20 0x46
Error: 1: Input is not proper UTF-8, indicate encoding !
Bytes: 0xED 0x6E 0x20 0x46
and g1.gexf isn't written. Can anyone help with what might have gone wrong?

Arduino and Raspberry Pi Serial communication + multiple variables

I have a raspberry pi and an arduino. So far I have been able to have the Pi send data to the arduino using serial communication, however it only send one variable and I have multiple variables(2) that I want to send to the arduino (x,y coordinates). Does anyone know if this is possible. I want the first number that is sent from the pi to be the x and the second one the y and the next one the x of the next coord ect.
I have tried editing the code that I use to send one variable but it doesn't work.
Any help would be awesome
Consider the following method to send 2 variable at the same time:
int xpos, ypos;
char x_tx_buffer[20], y_tx_buffer[20];
char x_dummy_buffer[20];
char y_dummy_buffer[20];
char *p_x_tx_buffer, *p_y_tx_buffer;
sprintf(x_dummy_buffer,"%d", xposs);
sprintf(y_dummy_buffer,"%d", yposs);
p_x_tx_buffer = &x_tx_buffer[0];
*p_x_tx_buffer++ = x_dummy_buffer[0];
*p_x_tx_buffer++ = x_dummy_buffer[1];
*p_x_tx_buffer++ = x_dummy_buffer[2];
*p_x_tx_buffer++ = x_dummy_buffer[3];
p_y_tx_buffer = &y_tx_buffer[0];
*p_y_tx_buffer++ = y_dummy_buffer[0];
*p_y_tx_buffer++ = y_dummy_buffer[1];
*p_y_tx_buffer++ = y_dummy_buffer[2];
*p_y_tx_buffer++ = y_dummy_buffer[3];
uart0_filestream = open("/dev/ttyAMA0", O_RDWR | O_NOCTTY | O_NDELAY); //Open in non blocking read/write mode
if (uart0_filestream == -1)
{
//ERROR - CAN'T OPEN SERIAL PORT
printf("Error - Unable to open UART. Ensure it is not in use by another application\n");
}
if (uart0_filestream != -1)
{
int countx = write(uart0_filestream, &x_tx_buffer[0], (p_x_tx_buffer - &x_tx_buffer[0])); //Filestream, bytes to write, number of bytes to write
int county = write(uart0_filestream, &y_tx_buffer[0], (p_y_tx_buffer - &y_tx_buffer[0])); //Filestream, bytes to write, number of bytes to write
if (countx < 0 || county < 0)
{
printf("UART TX error\n");
}
}
close(uart0_filestream);
You can send a max of 8 bytes at a time. Keep that in mind and with that you can modify the about code to send your x and y values in the same uart0_filestream.
Good luck.

Resources