i am new in Julia. i read a doc about julia static-analysis. it gives a function.
function foo(x,y)
z = x + y
return 2 * z
end
and use the julia introspection function code_typed get output:
code_typed(foo,(Int64,Int64))
1-element Array{Any,1}:
:($(Expr(:lambda, {:x,:y}, {{:z},{{:x,Int64,0},{:y,Int64,0},{:z,Int64,18}},{}},
:(begin # none, line 2:
z = (top(box))(Int64,(top(add_int))(x::Int64,y::Int64))::Int64 # line 3:
return (top(box))(Int64,(top(mul_int))(2,z::Int64))::Int64
end::Int64))))
it has an Expr . but when i call code_typed, the output is :
code_typed(foo, (Int64,Int64))
1-element Vector{Any}:
CodeInfo(
1 ─ %1 = Base.add_int(x, y)::Int64
│ %2 = Base.mul_int(2, %1)::Int64
└── return %2
) => Int64
it has a CodeInfo.it is different with the output in doc.
does julia have some change? and how can i get the Exprs according to my function and argtypes?
That code snippet appears to be taken from https://www.aosabook.org/en/500L/static-analysis.html, which was published in 2016 (about two years before the release of Julia 1.0) and references Julia version 0.3 from 2015.
Julia 1.0 gives
julia> code_typed(foo,(Int64,Int64))
1-element Array{Any,1}:
CodeInfo(
2 1 ─ %1 = (Base.add_int)(x, y)::Int64 │╻ +
3 │ %2 = (Base.mul_int)(2, %1)::Int64 │╻ *
└── return %2 │
) => Int64
while more current versions such as 1.6 and 1.7 give
julia> code_typed(foo,(Int64,Int64))
1-element Vector{Any}:
CodeInfo(
1 ─ %1 = Base.add_int(x, y)::Int64
│ %2 = Base.mul_int(2, %1)::Int64
└── return %2
) => Int64
(a much more minor change, but nonetheless)
If for any reason, you want this result in the form of an array or vector of Exprs, you seem to be able to get this using (e.g.)
julia> t = code_typed(foo,(Int64,Int64));
julia> t[1].first.code
3-element Vector{Any}:
:(Base.add_int(_2, _3))
:(Base.mul_int(2, %1))
:(return %2)
though this is likely considered an implementation detail and liable to change between minor versions of Julia.
I have a number of very large CSV files which I would like to parse into custom data structures for subsequent processing. My current approach involves CSV.File and then converting each CSV.Row into the custom data structure. It works well for small test cases but gets really inefficient for the large files (GC very high). The problem is in the second step and I suspect is due to type instability. I'm providing a mock example below.
(I'm new to Julia, so apologies if I misunderstood something)
Define data structure and conversion logic:
using CSV
struct Foo
a::Int32
b::Float32
end
Foo(csv_row::CSV.Row) = Foo(csv_row.a, csv_row.b)
Using the default constructor causes 0 allocations:
julia> #allocated foo1 = Foo(1, 2.5)
0
However, when creating the object from CSV.Row all of a sudden 80 bytes are allocated:
julia> data = CSV.File(Vector{UInt8}("a,b\n1,2.5"); threaded = false)
1-element CSV.File{false}:
CSV.Row: (a = 1, b = 2.5f0)
julia> #allocated foo2 = Foo(data[1])
80
In the first case all types are stable:
julia> #code_warntype Foo(1, 2)
Variables
#self#::Core.Compiler.Const(Foo, false)
a::Int64
b::Int64
Body::Foo
1 ─ %1 = Main.Foo::Core.Compiler.Const(Foo, false)
│ %2 = Core.fieldtype(%1, 1)::Core.Compiler.Const(Int32, false)
│ %3 = Base.convert(%2, a)::Int32
│ %4 = Core.fieldtype(%1, 2)::Core.Compiler.Const(Float32, false)
│ %5 = Base.convert(%4, b)::Float32
│ %6 = %new(%1, %3, %5)::Foo
└── return %6
Whereas in the second case they are not:
julia> #code_warntype Foo(data[1])
Variables
#self#::Core.Compiler.Const(Foo, false)
csv_row::CSV.Row
Body::Foo
1 ─ %1 = Base.getproperty(csv_row, :a)::Any
│ %2 = Base.getproperty(csv_row, :b)::Any
│ %3 = Main.Foo(%1, %2)::Foo
└── return %3
So I guess my question is: How can I make the second case type-stable and avoid the allocations?
Providing the types explicitly in CSV.File does not make a difference by the way.
While this does not focus on type stability, I would expect the highest performance combined with flexibility from the following code:
d = DataFrame!(CSV.File(Vector{UInt8}("a,b\n1,2.5\n3,4.0"); threaded = false))
The above efficiently transforms a CSV.File into a type stable structure, additionally avoiding data copying in this process. This should work for your case of huge CSV files.
And now:
julia> Foo.(d.a, d.b)
2-element Array{Foo,1}:
Foo(1, 2.5f0)
Foo(3, 4.0f0)
I have read that Julia has Macros but I am unsure if the Macros Julia provide are the ones I am thinking about.
I have the following expression:
Global.data[
Dates.value(Dates.Year(dtCursor)) - 2000,
Dates.value(Dates.Month(dtCursor)),
Dates.value(Dates.Day(dtCursor)),
Dates.value(Dates.Hour(dtCursor)) + 1,
Dates.value(Dates.Minute(dtCursor)) + 1,
1
]
And I repeat this a lot. I am wondering if I could have a macro that with dtCursor as parameter (it might be other variables in other cases) types all that for me. I am therefore looking for the Macro expansion functionality which was traditionally found in Macro assemblers.
I definitively do not want to include this as a function as this code is executed tens of thousands of times and therefore I do not want to add the overhead of a function call.
I have tried:
macro readData(_dtCursor, value)
return :(
Global.data[
Dates.value(Dates.Year(_dtCursor)) - 2000,
Dates.value(Dates.Month(_dtCursor)),
Dates.value(Dates.Day(_dtCursor)),
Dates.value(Dates.Hour(_dtCursor)) + 1,
Dates.value(Dates.Minute(_dtCursor)) + 1,
value
]
)
end
And later be invoked by:
println(#readData(dtCursor, 1))
Where dtCursor is a DateTime variable.
But I am getting:
ERROR: LoadError: UndefVarError: _dtCursor not defined
I have read https://docs.julialang.org/en/v1/manual/metaprogramming/index.html#man-macros-1 but a bit of help understanding what to do in this case is really welcomed.
Use a function
I definitively do not want to include this as a function as this code is executed tens of thousands of times and therefore I do not want to add the overhead of a function call.
You are definitively wrong.
You might be right in some languages, but not in JuliaLang.
(I do think this is a very useful questiom though because can highlight for others not to do this 😀)
That function-call in-lines away, and even if it didn't we have other tools (#inline) we would want to use before using a macro.
Macro's are for syntactic transformations.
If you are not doing a syntactic tranformation think again before using macros.
Here is a link to a good point made during by Steven G. Johnson at his keynote in juliacon:
"Functions are mostly good enough for Jeff Bezanson. Don't try to outsmart Jeff Bezason"
How to write it as a Macro and as a Function
The following answers your original question question
using Dates
using BenchmarkTools
macro readData(_dtCursor, value)
return :(
Global.data[
Dates.value(Dates.Year($(esc(_dtCursor)))) - 2000,
Dates.value(Dates.Month($(esc(_dtCursor)))),
Dates.value(Dates.Day($(esc(_dtCursor)))),
Dates.value(Dates.Hour($(esc(_dtCursor)))) + 1,
Dates.value(Dates.Minute($(esc(_dtCursor)))) + 1,
$value
]
)
end
function readData(_dtCursor, value)
Global.data[
Dates.value(Dates.Year(_dtCursor)) - 2000,
Dates.value(Dates.Month(_dtCursor)),
Dates.value(Dates.Day(_dtCursor)),
Dates.value(Dates.Hour(_dtCursor)) + 1,
Dates.value(Dates.Minute(_dtCursor)) + 1,
value
]
end
Benchmark it.
You say this is going to be run on 10,000s of times.
So I will benchmark on 100_000 uses, just to be safe.
const Global = (; data=[join((y, m, d, h, M, s)," ") for y in 2000:2010, m in 1:3, d in 1:20, h in 1:10, M in 1:30, s in 1:30]);
size(Global.data)
length(Global.data)
const sample_dts = map(1:100_000) do _
y, m, d, h, M, s = rand.(axes(Global.data))
dt = DateTime(y+2000, m, d, h-1, M-1)
end;
func_demo() = [readData(dt, 3) for dt in sample_dts];
macro_demo() = [#readData(dt, 3) for dt in sample_dts];
#btime func_demo()
#btime macro_demo()
They benchmark as identical
julia> #btime macro_demo();
5.409 ms (3 allocations: 781.34 KiB)
julia> #btime func_demo();
5.393 ms (3 allocations: 781.34 KiB)
Infact they specialize into (basically) the same code.
julia> #code_typed macro_demo()
CodeInfo(
1 ─ %1 = Main.sample_dts::Core.Compiler.Const(DateTime[2002-01-18T04:19:00, 2001-01-19T08:22:00, 2006-02-08T04:07:00, 2011-01-08T09:03:00, 2006-02-10T06:18:00, 2002-03-12T00:05:00, 2011-02-20T08:29:00, 2011-02-20T07:12:00, 2005-01-13T03:22:00, 2006-01-01T00:29:00 …
2005-03-10T04:29:00, 2002-03-12T09:11:00, 2002-03-11T00:28:00, 2007-02-12T02:26:00, 2003-02-15T07:29:00, 2009-01-01T02:02:00, 2009-
01-03T02:11:00, 2001-02-16T03:16:00, 2004-01-17T05:12:00, 2010-02-02T05:10:00], false)
│ %2 = %new(Base.Generator{Array{DateTime,1},getfield(Main, Symbol("##50#51"))}, getfield(Main, Symbol("##50#51"))(), %1)::Base.Gen
erator{Array{DateTime,1},getfield(Main, Symbol("##50#51"))}
│ %3 = invoke Base.collect(%2::Base.Generator{Array{DateTime,1},getfield(Main, Symbol("##50#51"))})::Array{String,1}
└── return %3
) => Array{String,1}
julia> #code_typed getfield(Main, Symbol("##50#51")).instance(1) # check the internals
│ %1 = %1 = Main.Global::Core.Compiler.Const((#==GIANT Inlined Const ==#)
│ %2 = Base.getfield(%1, :data)::Array{String,6}
│ %3 = Base.sub_int(dt, 2000)::Int64
│ %4 = Base.add_int(dt, 1)::Int64
│ %5 = Base.add_int(dt, 1)::Int64
│ %6 = Base.arrayref(true, %2, %3, dt, dt, %4, %5, 3)::String
└── return %6
) => String
julia> #code_typed func_demo()
CodeInfo(
1 ─ %1 = Main.sample_dts::Core.Compiler.Const(DateTime[2002-01-18T04:19:00, 2001-01-19T08:22:00, 2006-02-08T04:07:00, 2011-01-08T09:03:00, 2006-02-10T06:18:00, 2002-03-12T00:05:00, 2011-02-20T08:29:00, 2011-02-20T07:12:00, 2005-01-13T03:22:00, 2006-01-01T00:29:00 … 2005-03-10T04:29:00, 2002-03-12T09:11:00, 2002-03-11T00:28:00, 2007-02-12T02:26:00, 2003-02-15T07:29:00, 2009-01-01T02:02:00, 2009-
01-03T02:11:00, 2001-02-16T03:16:00, 2004-01-17T05:12:00, 2010-02-02T05:10:00], false)
│ %2 = %new(Base.Generator{Array{DateTime,1},getfield(Main, Symbol("##43#44"))}, getfield(Main, Symbol("##43#44"))(), %1)::Base.Gen
erator{Array{DateTime,1},getfield(Main, Symbol("##43#44"))}
│ %3 = invoke Base.collect(%2::Base.Generator{Array{DateTime,1},getfield(Main, Symbol("##43#44"))})::Array{String,1}
└── return %3
) => Array{String,1}
julia> #code_typed getfield(Main, Symbol("##43#44")).instance(1)
CodeInfo(
1 ─ %1 = Main.Global::NamedTuple{(:data,),Tuple{Array{String,6}}}
│ %2 = Base.getfield(%1, :data)::Array{String,6}
│ %3 = Base.sub_int(dt, 2000)::Int64
│ %4 = Base.add_int(dt, 1)::Int64
│ %5 = Base.add_int(dt, 1)::Int64
│ %6 = Base.arrayref(true, %2, %3, dt, dt, %4, %5, 3)::String
└── return %6
) => String
There is a very minor difference in the generors function between the two.
Wheree the value became a Compliler.Const or a NamedTuple when inlining,
but after that goes the LLVM that difference goes way too I think
(Check #code_llvm if your really interested. But we are already super deap into the weeds.)
This is probably the wrong code to be optimizing in the first place.
A long with the guidance to benchmark any optimization you do.
One should also profile the code to decide what is worth optimizing.
A function that is only called 10,000s of times and not allocating giant arrays etc, probably not worth worrying too much about.
Especially if you are just worrying about function call overhead,
which is only a handful of CPU cycles.
You have to splice in the variables you pass as macro arguments:
julia> macro readData(dtCursor, value)
return :(
Global.data[
Dates.value(Dates.Year($dtCursor)) - 2000,
Dates.value(Dates.Month($dtCursor)),
Dates.value(Dates.Day($dtCursor)),
Dates.value(Dates.Hour($dtCursor)) + 1,
Dates.value(Dates.Minute($dtCursor)) + 1,
$value
]
)
end
#readData (macro with 1 method)
julia> #macroexpand #readData(dtCursor, 1)
:((Main.Global).data[(Main.Dates).value((Main.Dates).Year(Main.dtCursor)) - 2000, (Main.Dates).value((Main.Dates).Month(Main.dtCursor)), (Main.Dates).value((Main.Dates).Day(Main.dtCursor)), (Main.Dates).value((Main.Dates).Hour(Main.dtCursor)) + 1, (Main.Dates).value((Main.Dates).Minute(Main.dtCursor)) + 1, 1])
Furthermore, Julia macros are hygenic; that means that there will be no confusion about the name _dtCursor in the macro definition, and the name dtCursor at call site. One thing you might need to do is to escape the inputs, though.
Also, this might be an overkill. You should benchmark the macro version against the function version; maybe, there's enough inlining happening that the macro doesn't actually matter.
Consider the following functions whose definitions contain re-definitions of functions.
function foo1()
x = 1
if x != 1
error("Wrong")
end
x = 2
end
function foo()
function p(t) return t + 1 end
if p(1) != 2
error("Wrong")
end
function p(t) return 1 end
end
foo1() runs without error, but foo() gives the error Wrong. I thought it may have something to do with Julia not supporting redefining functions in general, but I'm not sure. Why is this happening?
I would say that this falls into a more general known problem https://github.com/JuliaLang/julia/issues/15602.
In your case consider a simpler function:
function f()
p() = "before"
println(p())
p() = "after"
nothing
end
Calling f() will print "after".
In your case you can inspect what is going on with foo in the following way:
julia> #code_typed foo()
CodeInfo(
4 1 ─ invoke Main.error("Wrong"::String)::Union{} │
│ $(Expr(:unreachable))::Union{} │
└── $(Expr(:unreachable))::Union{} │
) => Union{}
and you see that Julia optimizes out all internal logic and just calls error.
If you inspect it one step earlier you can see:
julia> #code_lowered foo()
CodeInfo(
2 1 ─ p = %new(Main.:(#p#7)) │
3 │ %2 = (p)(1) │
│ %3 = %2 != 2 │
└── goto #3 if not %3 │
4 2 ─ (Main.error)("Wrong") │
6 3 ─ return p │
)
any you see that p in top line is assigned only once. Actually the second definition is used (which is not visible here, but could be seen above).
To solve your problem use anonymous functions like this:
function foo2()
p = t -> t + 1
if p(1) != 2
error("Wrong")
end
p = t -> 1
end
and all will work as expected. The limitation of this approach is that you do not get multiple dispatch on name p (it is bound to a concrete anonymous function, but I guess you do not need multiple dispatch in your example).
I would like to see devectorized code of some expression say here
obj.mask = exp.(1.0im*lambda*obj.dt/2.);
How one could print a generic expression in devectorized form in Julia?
I don't think that what you ask for exists (please proof me wrong if I'm mistaken!).
The best you can do is use #code_lowered, #code_typed, #code_llvm, #code_native macros (in particular #code_lowered) to see what happens to your Julia code snippet. However, as Julia isn't translating all dots to explicit for loops internally, non of these snippets will show you a for-loop version of your code.
Example:
julia> a,b = rand(3), rand(3);
julia> f(a,b) = a.*b
f (generic function with 1 method)
julia> #code_lowered f(a,b)
CodeInfo(
1 1 ─ %1 = Base.Broadcast.materialize │
│ %2 = Base.Broadcast.broadcasted │
│ %3 = (%2)(Main.:*, a, b) │
│ %4 = (%1)(%3) │
└── return %4 │
)
So, Julia translates the .* into a Base.Broadcast.broadcasted call. Of course, we can go further and do
julia> #which Base.Broadcast.broadcasted(Main.:*, a, b)
broadcasted(f, arg1, arg2, args...) in Base.Broadcast at broadcast.jl:1139
and check broadcast.jl in line 1139 and so on to trace the actual broadcasted method that will be called (maybe Tim Holy's Rebugger is useful here :D). But as I said before, there won't be a for-loop in it. Instead you will find something like this:
broadcasted(::DefaultArrayStyle{1}, ::typeof(*), r::AbstractRange, x::Number) = range(first(r)*x, step=step(r)*x, length=length(r))
broadcasted(::DefaultArrayStyle{1}, ::typeof(*), r::StepRangeLen{T}, x::Number) where {T} =
StepRangeLen{typeof(T(r.ref)*x)}(r.ref*x, r.step*x, length(r), r.offset)
broadcasted(::DefaultArrayStyle{1}, ::typeof(*), r::LinRange, x::Number) = LinRange(r.start * x, r.stop * x, r.len)
Update
Ok, eventually, I found for-loops in copyto! in broadcast.jl. But this is probably to deep into the rabbit hole.