How to get a `Ptr` to an element of an `NTuple`? - julia

Say I have a tuple of Cchar like
str = ('f', 'o', 'o', '\0', '\0')
and I want to convert it to a more traditional string. If str were a Vector, I could create Ptr and do all sorts of things with that. I've tried various ways of passing str to methods of pointer, Ptr, Ref, and unsafe_string without success since those normally work on arrays rather than tuples. Any suggestions?
Note: what I really have is a C struct that looks like
typedef struct foo {
char str[FOO_STR_MAX_SZ];
...
} foo_t;
which Clang.jl wrapped as
struct foo_t
str :: NTuple{FOO_STR_MAX_SZ, UInt8}
...
end
I also played around with NTuple of Cchar (ie, Int8) instead of UInt8, and I tried to use SVector instead of NTuple as well. But I still couldn't find a way to generate a Ptr from the str field. Am I missing something?

Since you asked the question, I think collecting it to an array a = collect(x.str) is not the answer you are expecting...
You could use ccall(:jl_value_ptr, Ptr{Cvoid}, (Any,), a) to get the pointer of a even if a is immutable. However, blindly using it will produce some confusing results:
julia> struct foo_t
str::NTuple{2, UInt8}
end
julia> a = foo_t((2, 3))
foo_t((0x02, 0x03))
julia> ccall(:jl_value_ptr, Ptr{Cvoid}, (Any,), a.str)
Ptr{Nothing} #0x00007f4302c4f670
julia> ccall(:jl_value_ptr, Ptr{Cvoid}, (Any,), a.str)
Ptr{Nothing} #0x00007f4302cc47e0
We got two different pointers from the same object! The reason is that since NTuple is immutable, the compiler will do many "optimizations" for it, for example, coping it every time you use it. This is why getting pointers from immutable objects is explicitly forbidden in the source code:
function pointer_from_objref(#nospecialize(x))
#_inline_meta
typeof(x).mutable || error("pointer_from_objref cannot be used on immutable objects")
ccall(:jl_value_ptr, Ptr{Cvoid}, (Any,), x)
end
However, there are several workarounds for it. First, since the expression a.str copies the tuple, you can avoid this expressoin and calculate the address of it directly using the address of a and fieldoffset(typeof(a), 1). (1 means str is the first field of foo_t)
julia> p = Ptr{UInt8}(ccall(:jl_value_ptr, Ptr{UInt8}, (Any,), a)) + fieldoffset(typeof(a), 1)
Ptr{UInt8} #0x00007f4304901df0
julia> p2 = Ptr{UInt8}(ccall(:jl_value_ptr, Ptr{UInt8}, (Any,), a)) + fieldoffset(typeof(a), 1)
Ptr{UInt8} #0x00007f4304901df0
julia> p === p2
true
julia> unsafe_store!(p, 5)
Ptr{UInt8} #0x00007f4304901df0
julia> a
foo_t((0x05, 0x03))
It now works. However, there are still caveats: when you try to wrap the code in a function, it became wrong again:
julia> mut!(a) = unsafe_store!(Ptr{UInt8}(ccall(:jl_value_ptr, Ptr{UInt8}, (Any,), a)) + fieldoffset(typeof(a), 1), 8)
mut! (generic function with 1 method)
julia> mut!(a)
Ptr{UInt8} #0x00007f42ec560294
julia> a
foo_t((0x05, 0x03))
a is not changed because, well, foo_t itself is also immutable and will be copied to mut!, so the change made within the function will not be visible outside. To solve this, we need to wrap a in a mutable object to give it a stable address in the heap. Base.RefValue can be used for this purpose:
julia> b = Base.RefValue(a)
Base.RefValue{foo_t}(foo_t((0x05, 0x03)))
julia> mut!(b) = unsafe_store!(Ptr{UInt8}(ccall(:jl_value_ptr, Ptr{UInt8}, (Any,), b)) + fieldoffset(typeof(b), 1) + fieldoffset(typeof(a), 1), 8)
mut! (generic function with 1 method)
julia> mut!(b)
Ptr{UInt8} #0x00007f43057b3820
julia> b
Base.RefValue{foo_t}(foo_t((0x08, 0x03)))
julia> b[]
foo_t((0x08, 0x03))

As explained by #张实唯, str is a constant array which is stack-allocated, so you need to use pointer arithmetics to access the field. There is a package called Blobs.jl for this kinda purpose. As for the mutability, you could also use Setfield.jl for convenience.
BTW, Clang.jl do support generating mutable structs via ctx.options["is_struct_mutable"] = true.

Related

Shorthand for Int32 literals in Julia

I use lots of Int32s in my code because I have some large arrays of those. But for some x::Int32 we have typeof(x+1) == Int64 since numeric literals are Int64 by default (I have to use 64bit Julia to handle my arrays). The problem is, if I have some function f(x::Int32) then f(x+1) will method error. I don't want to implement a f(x::Int64) = f(convert(Int32, x)) for almost every function and want to use concrete types for type stability. Currently, I simply have expressions like x + Int32(1) all over my code which looks really cluttered. For other types we have shorthands, i.e., 1.f0 gives me a Float32 and big"1" a BigInt. Is there something similar for Int32?
Since you explicitly mention the big_str macro (big"") you can easily define a similar macro for Int32 (the same way the uint128_str and int128_str is defined):
macro i32_str(s)
parse(Int32, s)
end
julia> typeof(i32"1")
Int32
this might still clutter your code too much so alternatively you could exploit that a number followed by a name is multiplication:
struct i32 end
(*)(n, ::Type{i32}) = Int32(n)
julia> typeof(1i32)
Int32
You can make a macro to replace every literal integer with an Int32, a bit like what ChangePrecision.jl does for floats. A very quick first attempt is:
julia> macro literal32(ex)
esc(literal32(ex))
end;
julia> literal32(ex::Expr) = Expr(ex.head, literal32.(ex.args)...);
julia> literal32(i::Int) = Int32(i);
julia> literal32(z) = z; # ignore Symbol, literal floats, etc.
julia> #literal32 [1,2] .+ 3
2-element Vector{Int32}:
4
5
julia> #literal32 function fun(x::AbstractVector)
x[1] + 2 # both 1 and 2 are changed
end
fun (generic function with 1 method)
julia> fun(Int32[3,4]) |> typeof
Int32
One place this may have unexpected consequences is literal type parameters:
julia> #literal32([1,2,3]) isa Array{Int32,1}
true
julia> #literal32 [1,2,3] isa Array{Int32,1}
false
Another is that x^2 will not use Base.literal_pow, e.g. #literal32 Meta.#lower pi^2.
What if you say:
# Or, a::Int32 = 1
julia> a = Int32(1)
1
julia> b::Int32 = a+2
3
julia> typeof(b)
Int32
julia> f(b)
...

How to create a non-standard string literal with parameters in Julia?

I am creating a non-standard string literal with a macro with something like that:
macro R13_str(p)
rotate(13, p)
end
and it works. I can call it as:
R13"abc"
But I would like to declare the macro to work with any integer, like:
R1"abc"
or
R244"abc"
Let's say the function rotate() is:
function rotate(shift_amount::Int64, s::String)
# Ensure the shift is no bigger than the string
shift = shift_amount ≤ length(s) ? shift_amount : shift_amount % length(s)
# Circular shift
return s[end-shift+1:end] * s[1:end-shift]
end
How can I do that? I have checked all the docs, but it's not clear to me.
Can't see how to achieve exactly what is required. But the following might be good enough:
julia> macro R_str(p,flag)
rotate(flag, p)
end
#R_str (macro with 1 method)
julia> R"hello"3
"llohe"
julia> R"abc"1
"cab"
julia> R"abc"244
"cab"
See https://docs.julialang.org/en/v1/manual/metaprogramming/#meta-non-standard-string-literals
Trying to conform to OP call format:
julia> macro rework(expr)
if expr.head != :macrocall return expr ; end
r = String(expr.args[1])
rr = parse(Int, r[3:findfirst('_',r)-1])
:(rotate($rr, $(expr.args[3])))
end
#rework (macro with 1 method)
julia> #rework R13"hello"
"llohe"
This macro could help to read the prepared test cases??
I have found a solution:
for n in 0:244
#eval macro $(Symbol(:R, n, :_str))(s)
rotate($n, s)
end
end
While I believe that using flags is the better approach, with this for loop I can generate all the macros that I need.
Julia> R1"abc"
Julia> R24"acb"
Julia> R56"abc"
simply work.

How to dispatch based on the type of any of the splatted args?

Consider an existing function in Base, which takes in a variable number of arguments of some abstract type T. I have defined a subtype S<:T and would like to write a method which dispatches if any of the arguments is my subtype S.
As an example, consider function Base.cat, with T being an AbstractArray and S being some MyCustomArray <: AbstractArray.
Desired behaviour:
julia> v = [1, 2, 3];
julia> cat(v, v, v, dims=2)
3×3 Array{Int64,2}:
1 1 1
2 2 2
3 3 3
julia> w = MyCustomArray([1,2,3])
julia> cat(v, v, w, dims=2)
"do something fancy"
Attempt:
function Base.cat(w::MyCustomArray, a::AbstractArray...; dims)
pritnln("do something fancy")
end
But this only works if the first argument is MyCustomArray.
What is an elegant way of achieving this?
I would say that it is not possible to do it cleanly without type piracy (but if it is possible I would also like to learn how).
For example consider cat that you asked about. It has one very general signature in Base (actually not requiring A to be AbstractArray as you write):
julia> methods(cat)
# 1 method for generic function "cat":
[1] cat(A...; dims) in Base at abstractarray.jl:1654
You could write a specific method:
Base.cat(A::AbstractArray...; dims) = ...
and check if any of elements of A is your special array, but this would be type piracy.
Now the problem is that you cannot even write Union{S, T} as since S <: T it will be resolved as just T.
This would mean that you would have to use S explicitly in the signature, but then even:
f(::S, ::T) = ...
f(::T, ::S) = ...
is problematic and a compiler will ask you to define f(::S, ::S) as the above definitions lead to dispatch ambiguity. So, even if you wanted to limit the number of varargs to some maximum number you would have to annotate types for all divisions of A into subsets to avoid dispatch ambiguity (which is doable using macros, but grows the number of required methods exponentially).
For general usage, I concur with Bogumił, but let me make an additional comment. If you have control over how cat is called, you can at least write some kind of trait-dispatch code:
struct MyCustomArray{T, N} <: AbstractArray{T, N}
x::Array{T, N}
end
HasCustom() = Val(false)
HasCustom(::MyCustomArray, rest...) = Val(true)
HasCustom(::AbstractArray, rest...) = HasCustom(rest...)
# `IsCustom` or something would be more elegant, but `Val` is quicker for now
Base.cat(::Val{true}, args...; dims) = println("something fancy")
Base.cat(::Val{false}, args...; dims) = cat(args...; dims=dims)
And the compiler is cool enough to optimize that away:
julia> args = (v, v, w);
julia> #code_warntype cat(HasCustom(args...), args...; dims=2);
Variables
#self#::Core.Compiler.Const(cat, false)
#unused#::Core.Compiler.Const(Val{true}(), false)
args::Tuple{Array{Int64,1},Array{Int64,1},MyCustomArray{Int64,1}}
Body::Nothing
1 ─ %1 = Main.println("something fancy")::Core.Compiler.Const(nothing, false)
└── return %1
If you don't have control over calls to cat, the only resort I can think of to make the above technique work is to overdub methods containing such call, to replace matching calls by the custom implementation. In which case you don't even need to overload cat, but can directly replace it by some mycat doing your fancy stuff.

Check if an Object is an Array or a Dict

I'd like to check if var is an Array or a Dict.
typeof(var) == Dict
typeof(var) == Array
But it doesn't work because typeof is too precise: Dict{ASCIIString,Int64}.
What's the best way ?
If you need a "less precise" check, you may want to consider using the isa() function, like this:
julia> d = Dict([("A", 1), ("B", 2)])
julia> isa(d, Dict)
true
julia> isa(d, Array)
false
julia> a = rand(1,2,3);
julia> isa(a, Dict)
false
julia> isa(a, Array)
true
The isa() function could then be used in control flow constructs, like this:
julia> if isa(d, Dict)
println("I'm a dictionary!")
end
I'm a dictionary!
julia> if isa(a, Array)
println("I'm an array!")
end
I'm an array!
Note: Tested with Julia 0.4.3
Instead of checking for a particular concrete type, such as Array, or Dict, you might do better by checking for the abstract types, and gain a lot of flexibility.
For example:
julia> x = [1,2,3]
3-element Array{Int64,1}:
1
2
3
julia> d = Dict(:a=>1,:b=>2)
Dict(:a=>1,:b=>2)
julia> isa(d, Associative)
true
julia> isa(x, AbstractArray)
true
There are many different types of arrays in Julia, so checking for Array is likely to be too restrictive, you won't get sparse matrices, for example.
There are also a number of different types of associative structures, Dict, ObjectIdDict, SortedDict, OrderedDict.

How do I create a method that takes any iterable collection of strings?

I have a function, f. I want to add a method that takes any container of Strings. For example, I want to write a method that generates the following when needed:
f(xs::Array{String, 1}) = ...
f(xs::DataArray{String, 1}) = ...
f(xs::ITERABLE{String}) = ...
Is this possible to do in Julia's type system? Right now, I'm using a macro to write a specialized method when I need it.
#make_f(Array{String, 1})
#make_f(DataArray{String, 1})
This keeps things DRY, but it feels...wrong.
Can't you just use duck typing? I.e., just assume that you're feeding the function an object of the right type and throw an error if at some point e.g. you don't have a string in your iterable.
This should improve once you can really talk about iterables using traits; currently there is no iterable type. Scott's answer, for example, will not work with a tuple of strings, even though that is iterable.
E.g.
julia> f(x) = string(x...) # just concatenate the strings
f (generic function with 1 method)
julia> f(("a", "á"))
"aá"
julia> f(["a", "á"])
"aá"
julia> f(["a" "b"; "c" "d"]) # a matrix of strings!
"acbd"
At least in Julia 0.4, the following should work:
julia> abstract Iterable{T} <: AbstractVector{T}
julia> f{T<:Union{Vector{String},Iterable{String}}}(xs::T) = 1
f (generic function with 1 method)
julia> x = String["a", "é"]
2-element Array{AbstractString,1}:
"a"
"é"
julia> f(x)
1

Resources