How can I cast byte arrays to struct in Julia? - julia

I'm new to Julia. I'm trying to parse a structured binary file. I read n bytes from the file and I want to cast the byte array to an object of type X.
struct X
messageType::UInt8
second::UInt32
end
f = open("myfile.bin")
bytes = read(f, 5)
And now I want to cast bytes to an object of X. How can I do this?

You can use StructIO here is how.
Setup:
using StructIO
#io struct XX
messageType::UInt8
second::UInt32
end align_packed
bytes = UInt8[0x72, 0xa3, 0x97, 0xcf, 0x64]
buf = IOBuffer(bytes)
And now running the code:
julia> seekstart(buf); unpack(buf, XX)
XX(0x72, 0x64cf97a3)
julia> seekstart(buf); unpack(buf, XX, :BigEndian)
XX(0x72, 0xa397cf64)

Related

getelementptr in LLVM IR

I don't understand the following code written for LLVM IR. I hope, you can give me a hint.
%struct.foo_struct = type {[3 x i32], i16*, i32}
;struct foo_struct {
; [3 x i32] f0;
; i16* f1;
; i32 f2;
; };
define i32 #foo(%struct.foo_struct* %P) {
entry:
; &P[0].f1
%tmp0 = getelementptr inbounds %struct.foo_struct, %struct.foo_struct* %P, i64 0, i32 1
; P[0].f1
%tmp1 = load i16*, i16** %tmp0
; &P[0].f1[0]
%tmp2 = getelementptr inbounds i16, i16* %tmp1, i64 0
Specifically, in the first code line in entry, we have at the end i32 1. Why i32? Since we want to jump to the next field, namely f1, we have to jump over an array (f0), which 3xi32. So what is this i32? What would be there if we want to have e.g. &P[0].f2?
Thank you for any help
Since we want to jump to the next field, namely f1, we have to jump over an array (f0), which 3xi32. So what is this i32?
1 is the index of the member in the struct and i32 is the type of this index.
The fact that we're jumping over a 3xi32 to get to f1 or that we're thus jumping by 12 bytes is not directly encoded in the instruction. All we're specifying is that we want the second member (i.e. the member at index 1) and how that translates to an offset in bytes is calculated by LLVM based on the types involved. We don't spell this out in the instruction.
What would be there if we want to have e.g. &P[0].f2?
i32 2
Why i32?
Struct indices always have the type i32. Since struct indices must always be constants, there wasn't really a need to allow different types and i32 is large enough for all practical purposes (i.e. you won't have a struct with more than 2^32 members).

Unpack Binary Data with specific format in Julia

I am trying to convert a Binary file parser to Julia from Python. I am struggling to figure out how to unpack the binary stream with a specific format. I found this discourse thread that is exactly what I am trying to do, but its from 2017 and doesn't seem to have a working solution. Does anyone have a solution?
In Python it looks like this:
In [22]: struct.unpack('>idi', b'\x00\x00\x00\x17#\t\x1e\xb8Q\xeb\x85\x1f\x00\x00\x00*')
Out[22]: (23, 3.14, 42)
In Julia I am here:
data = open(filename, "r")
seek(data, 0)
# now I want to get the first 12 bytes of the file and convert to a string.. and am stumped..
I'm not very familiar with struct.unpack in Python, but maybe you could do something like this:
julia> data = IOBuffer("\x00\x00\x00\x17#\t\x1e\xb8Q\xeb\x85\x1f\x00\x00\x00*")
IOBuffer(data=UInt8[...], readable=true, writable=false, seekable=true, append=false, size=16, maxsize=Inf, ptr=1, mark=-1)
julia> seekstart(data)
IOBuffer(data=UInt8[...], readable=true, writable=false, seekable=true, append=false, size=16, maxsize=Inf, ptr=1, mark=-1)
julia> i = bswap(read(data, Int32))
23
julia> pi = bswap(read(data, Float64))
3.14
julia> i = bswap(read(data, Int32))
42
bswaps are there because there seems to be a difference in endianness between what Julia internally uses and what's encoded in your binary stream. Apart from that, this is just a plain use of read, specifying the type of data to be read.
By the way, here is how you would read the first 12 bytes of the file, and convert them to a string (which is not really necessary in this case, but could be useful in others):
julia> seekstart(data)
IOBuffer(data=UInt8[...], readable=true, writable=false, seekable=true, append=false, size=16, maxsize=Inf, ptr=1, mark=-1)
julia> bytes = read(data, 12)
12-element Array{UInt8,1}:
0x00
0x00
0x00
0x17
0x40
0x09
0x1e
0xb8
0x51
0xeb
0x85
0x1f
# note the capital "S" in "String"
julia> String(bytes)
"\0\0\0\x17#\t\x1e\xb8Q\xeb\x85\x1f"

Julia convert UInt32 to UInt8 vector

I have the following code and I need to covert several UInt32 variables to UInt8 vectors so then combine them into a single UInt8 vector.
The goal is to take the record I have decoded from a Pcap file and put it into a format that I can append to the end of an existing Pcap file.
The code below takes output from a previous function and returns a hex output of 4 UInt 32's and a vector of UInt8's for the payload.
function pcap_get_record(s::PcapOffline)
rec = PcapRec()
if (!eof(s.file))
rec.ts_sec = s.is_big ? read(s.file, UInt32) : ntoh(read(s.file, UInt32))
rec.ts_usec = s.is_big ? read(s.file, UInt32) : ntoh(read(s.file, UInt32))
rec.incl_len = s.is_big ? read(s.file, UInt32) : ntoh(read(s.file, UInt32))
rec.orig_len = s.is_big ? read(s.file, UInt32) : ntoh(read(s.file, UInt32))
rec.payload = read(s.file, rec.incl_len)
return rec
end
nothing
end
Thanks
Here you are
julia> reinterpret(UInt8, rand(UInt32, 1)) |> Vector
4-element Array{UInt8,1}:
0x4d
0x54
0x34
0xd3
remember to check the byte order.
Update: So I have solved this and I was overthinking what needed to be done.
I just wrote the UInt variable in their raw form and that did the trick.
write(pcap, rec.orig_len) #this is a UInt32
write(pcap, rec.payload) #this is a UInt8 vector
Original:
I was having a hard time making my previous comment readable.
Thanks for the response. I am not however able to get the reinterpret to work with my UInt32 variable.
a = reinterpret(UInt8, rec.ts_usec) |> Vector
ERROR: bitcast: argument size does not match size of target type
Stacktrace:
[1] reinterpret(::Type{UInt8}, ::UInt32) at .\essentials.jl:370
[2] top-level scope at none:0
typeof(rec.ts_usec)
UInt32
after messing around some more I was able to get this to work but this doesn't seem very efficient.
"Edit" I just found that this wont work since it cuts off any leading zeros in the UInt32. example rec.incl_len = 0x00000516 would come out as "516" instead of "00000516" which is needed.
julia> hex(n) = string(n, base = 16, pad = 2)
julia> a = hex2bytes(hex(rec.ts_sec))
4-element Array{UInt8,1}:
0x5b
0x60
0xa3
0xa1

Parsing an 8 bit char array to integer

I am new to Arduino and all I want to do is parse a String of binary numbers to an exact integer representation.
char* byte1 = "11111111"
int binary1 = atoi(byte1);
Serial.print(binary1);
However this prints out: -19961
Can anyone explain why? I am coming from a Java and JavaScript perspective.
atoi converts a decimal (base 10) string to int. If you want to convert a binary string to int, you can use strtol:
char *byte1 = "11111111";
int val1 = strtol(byte1, 0, 2);
std::cout << val1 << std::endl;
strtol can convert any base -- the 3rd argument is the base to use.
You get -19961 because on Arduino int is 16 bit wide and cannot hold any number bigger than 32767. To hold an integer representation of 11111111 you have to use long (which on Arduino is 32 bit) and strtol.
long val = strtol(byte1, NULL, 10);

Golang: map to struct

For some reason (fixed-length data file parsing), I've got a map and I want the elements of the map being saved in a struct.
Let's say:
type Point struct {X, Y int}
point := make(map[string]int)
point["X"] = 15
point["Y"] = 13
p := Point{point} // doesn't work
How do I do that? Or have I taken the wrong path?
As far as I know you cant have automatic mapping like this unless you're using the encoding package, but you can use the following way:
p := Point{X: point["X"], Y: point["Y"]}
If the efficiency is not so important, you can marshal the map into JSON bytes and unmarshal it back to a struct.
import "encoding/json"
type Point struct {X, Y int}
point := make(map[string]int)
point["X"] = 15
point["Y"] = 13
bytes, err := json.Marshal(point)
var p Point
err = json.Unmarshal(bytes, &p)
This maks the code easier to be modified when the struct contains a lot of fields.

Resources