I want to have a curried version of a function. So, I write the code as follows:
f(x::Int64, y::Int64) = x + y
f(x::Int64) = (y::Int64) -> f(x, y)
But I am not sure if Julia considers this an example of a type-unstable definition. On the face of it, one of the methods returns an anonymous function, while another returns an Int64. Yet, when the curried version is applied, the final result is also an Int64.
So, my questions are:
Is this code type-stable?
If not, is there a way to have a curried version of a function without writing type-unstable code?
Thanks in advance.
Yes, it is.
According to the official doc, you can investigate it by using the #code_warntype macro:
julia> #code_warntype f(1, 5)
MethodInstance for f(::Int64, ::Int64)
from f(x::Int64, y::Int64) in Main at REPL[2]:1
Arguments
#self#::Core.Const(f)
x::Int64
y::Int64
Body::Int64
1 ─ %1 = (x + y)::Int64
└── return %1
The arguments of this function have the exact type Int64, and as we can see in the Body::Int64, the inferred return type function is Int64.
Furthermore, we have f(x) which is based on the type-stable function f(x, y):
julia> #code_warntype f(1)
MethodInstance for f(::Int64)
from f(x::Int64) in Main at REPL[15]:1
Arguments
#self#::Core.Const(f)
x::Int64
Locals
#3::var"#3#4"{Int64}
Body::var"#3#4"{Int64}
1 ─ %1 = Main.:(var"#3#4")::Core.Const(var"#3#4")
│ %2 = Core.typeof(x)::Core.Const(Int64)
│ %3 = Core.apply_type(%1, %2)::Core.Const(var"#3#4"{Int64})
│ (#3 = %new(%3, x))
└── return #3
Here as well, there's not any unstable defined parameter type.
Look at the following as an example of an unstable-typed function:
julia> unstF(X) = x*5
unstF (generic function with 1 method)
julia> #code_warntype unstF(1)
MethodInstance for unstF(::Int64)
from unstF(X) in Main at REPL[17]:1
Arguments
#self#::Core.Const(unstF)
X::Int64
Body::Any
1 ─ %1 = (Main.x * 5)::Any
└── return %1
If you try this in the REPL, you'll see the Any appears with a red color. Since we have the Body::Any (Any with the red color), we can conclude that the returned object by this function is a non-concrete type object. Because the compiler doesn't know what is the x (Note that the input is X). So the result can be Anything! So this function is type-unstable for (here, for integer inputs. Note that you should investigate the type-stability of your function by your desired input(s). E.g., #code_warntype f(5) and #code_warntype f(5.) should be observed if I can pass it Float64 or Int64 either).
I have a number of very large CSV files which I would like to parse into custom data structures for subsequent processing. My current approach involves CSV.File and then converting each CSV.Row into the custom data structure. It works well for small test cases but gets really inefficient for the large files (GC very high). The problem is in the second step and I suspect is due to type instability. I'm providing a mock example below.
(I'm new to Julia, so apologies if I misunderstood something)
Define data structure and conversion logic:
using CSV
struct Foo
a::Int32
b::Float32
end
Foo(csv_row::CSV.Row) = Foo(csv_row.a, csv_row.b)
Using the default constructor causes 0 allocations:
julia> #allocated foo1 = Foo(1, 2.5)
0
However, when creating the object from CSV.Row all of a sudden 80 bytes are allocated:
julia> data = CSV.File(Vector{UInt8}("a,b\n1,2.5"); threaded = false)
1-element CSV.File{false}:
CSV.Row: (a = 1, b = 2.5f0)
julia> #allocated foo2 = Foo(data[1])
80
In the first case all types are stable:
julia> #code_warntype Foo(1, 2)
Variables
#self#::Core.Compiler.Const(Foo, false)
a::Int64
b::Int64
Body::Foo
1 ─ %1 = Main.Foo::Core.Compiler.Const(Foo, false)
│ %2 = Core.fieldtype(%1, 1)::Core.Compiler.Const(Int32, false)
│ %3 = Base.convert(%2, a)::Int32
│ %4 = Core.fieldtype(%1, 2)::Core.Compiler.Const(Float32, false)
│ %5 = Base.convert(%4, b)::Float32
│ %6 = %new(%1, %3, %5)::Foo
└── return %6
Whereas in the second case they are not:
julia> #code_warntype Foo(data[1])
Variables
#self#::Core.Compiler.Const(Foo, false)
csv_row::CSV.Row
Body::Foo
1 ─ %1 = Base.getproperty(csv_row, :a)::Any
│ %2 = Base.getproperty(csv_row, :b)::Any
│ %3 = Main.Foo(%1, %2)::Foo
└── return %3
So I guess my question is: How can I make the second case type-stable and avoid the allocations?
Providing the types explicitly in CSV.File does not make a difference by the way.
While this does not focus on type stability, I would expect the highest performance combined with flexibility from the following code:
d = DataFrame!(CSV.File(Vector{UInt8}("a,b\n1,2.5\n3,4.0"); threaded = false))
The above efficiently transforms a CSV.File into a type stable structure, additionally avoiding data copying in this process. This should work for your case of huge CSV files.
And now:
julia> Foo.(d.a, d.b)
2-element Array{Foo,1}:
Foo(1, 2.5f0)
Foo(3, 4.0f0)
I want to generate data from a linear model with noise (Y = X*w + e) where I can specify the distributions of input vector X and scalar noise e. For this, I specify the below struct
using Distributions
struct NoisyLinearDataGenerator
x_dist::ContinuousMultivariateDistribution
noise_dist::ContinuousUnivariateDistribution
weights::Vector{Float64}
end
and a function to generate N points from it:
function generate(nl::NoisyLinearDataGenerator, N)
x = rand(nl.x_dist, N)'
e = rand(nl.noise_dist, N)
return x, x*nl.weights + e
end
This seems to be working, however not type stable, as
nl = NoisyLinearDataGenerator(MvNormal(5, 1.0), Normal(), ones(5))
#code_warntype generate(nl,1)
yields
Variables
#self#::Core.Compiler.Const(generate, false)
nl::NoisyLinearDataGenerator
N::Int64
x::Any
e::Any
Body::Tuple{Any,Any}
1 ─ %1 = Base.getproperty(nl, :x_dist)::Distribution{Multivariate,Continuous}
│ %2 = Main.rand(%1, N)::Any
│ (x = Base.adjoint(%2))
│ %4 = Base.getproperty(nl, :noise_dist)::Distribution{Univariate,Continuous}
│ (e = Main.rand(%4, N))
│ %6 = x::Any
│ %7 = x::Any
│ %8 = Base.getproperty(nl, :weights)::Array{Float64,1}
│ %9 = (%7 * %8)::Any
│ %10 = (%9 + e)::Any
│ %11 = Core.tuple(%6, %10)::Tuple{Any,Any}
└── return %11
I am not sure why this is, since I would expect the type of the sampled data to be specified by using ContinuousMultivariateDistribution and ContinuousUnivariateDistribution.
What is leading to type instability here and what should a type stable implementation look like?
The problem is that ContinuousMultivariateDistribution and ContinuousUnivariateDistribution are abstract types. While your knowledge of statistics tells you that they probably should return Float64, there is no guarantee on a language level that someone won't implement, say, a ContinuousUnivariateDistribution that returns some other object. Therefore the compiler can't know that all ContinuousUnivariateDistribution produces any particular type.
For example, I might write:
struct BadDistribution <: ContinuousUnivariateDistribution end
Base.rand(::BadDistribution, ::Integer) = nothing
Now, you could make a NoisyLinearDataGenerator containing a BadDistribution as x_dist. What would the output type be then?
In other words, the output of generate simply can't be predicted only from its input types.
To solve this, you need to either specify specific distributions for your new type, or else make your new type parametric. In Julia, whenever we have a field of a type that cannot be specified to a concrete type, we usually leave it as a type parameter. Thus, one possible solution is this:
using Distributions
struct NoisyLinearDataGenerator{X,N}
x_dist::X
noise_dist::N
weights::Vector{Float64}
function NoisyLinearDataGenerator{X,N}(x::X, n::N, w::Vector{Float64}) where {
X <: ContinuousMultivariateDistribution,
N <: ContinuousUnivariateDistribution}
return new{X,N}(x,n,w)
end
end
function NoisyLinearDataGenerator(x::X, n::N, w::Vector{Float64}) where {
X <: ContinuousMultivariateDistribution,
N <: ContinuousUnivariateDistribution}
return NoisyLinearDataGenerator{X,N}(x,n,w)
end
function generate(nl::NoisyLinearDataGenerator, N)
x = rand(nl.x_dist, N)'
e = rand(nl.noise_dist, N)
return x, x*nl.weights + e
end
nl = NoisyLinearDataGenerator(MvNormal(5, 1.0), Normal(), ones(5))
Here, the type of nl is NoisyLinearDataGenerator{MvNormal{Float64,PDMats.ScalMat{Float64},FillArrays.Zeros{Float64,1,Tuple{Base.OneTo{Int64}}}},Normal{Float64}} (yes, I know, awful to read), but its type contain all information needed for the compiler to fully predict the output type of generate.
I have read that Julia has Macros but I am unsure if the Macros Julia provide are the ones I am thinking about.
I have the following expression:
Global.data[
Dates.value(Dates.Year(dtCursor)) - 2000,
Dates.value(Dates.Month(dtCursor)),
Dates.value(Dates.Day(dtCursor)),
Dates.value(Dates.Hour(dtCursor)) + 1,
Dates.value(Dates.Minute(dtCursor)) + 1,
1
]
And I repeat this a lot. I am wondering if I could have a macro that with dtCursor as parameter (it might be other variables in other cases) types all that for me. I am therefore looking for the Macro expansion functionality which was traditionally found in Macro assemblers.
I definitively do not want to include this as a function as this code is executed tens of thousands of times and therefore I do not want to add the overhead of a function call.
I have tried:
macro readData(_dtCursor, value)
return :(
Global.data[
Dates.value(Dates.Year(_dtCursor)) - 2000,
Dates.value(Dates.Month(_dtCursor)),
Dates.value(Dates.Day(_dtCursor)),
Dates.value(Dates.Hour(_dtCursor)) + 1,
Dates.value(Dates.Minute(_dtCursor)) + 1,
value
]
)
end
And later be invoked by:
println(#readData(dtCursor, 1))
Where dtCursor is a DateTime variable.
But I am getting:
ERROR: LoadError: UndefVarError: _dtCursor not defined
I have read https://docs.julialang.org/en/v1/manual/metaprogramming/index.html#man-macros-1 but a bit of help understanding what to do in this case is really welcomed.
Use a function
I definitively do not want to include this as a function as this code is executed tens of thousands of times and therefore I do not want to add the overhead of a function call.
You are definitively wrong.
You might be right in some languages, but not in JuliaLang.
(I do think this is a very useful questiom though because can highlight for others not to do this 😀)
That function-call in-lines away, and even if it didn't we have other tools (#inline) we would want to use before using a macro.
Macro's are for syntactic transformations.
If you are not doing a syntactic tranformation think again before using macros.
Here is a link to a good point made during by Steven G. Johnson at his keynote in juliacon:
"Functions are mostly good enough for Jeff Bezanson. Don't try to outsmart Jeff Bezason"
How to write it as a Macro and as a Function
The following answers your original question question
using Dates
using BenchmarkTools
macro readData(_dtCursor, value)
return :(
Global.data[
Dates.value(Dates.Year($(esc(_dtCursor)))) - 2000,
Dates.value(Dates.Month($(esc(_dtCursor)))),
Dates.value(Dates.Day($(esc(_dtCursor)))),
Dates.value(Dates.Hour($(esc(_dtCursor)))) + 1,
Dates.value(Dates.Minute($(esc(_dtCursor)))) + 1,
$value
]
)
end
function readData(_dtCursor, value)
Global.data[
Dates.value(Dates.Year(_dtCursor)) - 2000,
Dates.value(Dates.Month(_dtCursor)),
Dates.value(Dates.Day(_dtCursor)),
Dates.value(Dates.Hour(_dtCursor)) + 1,
Dates.value(Dates.Minute(_dtCursor)) + 1,
value
]
end
Benchmark it.
You say this is going to be run on 10,000s of times.
So I will benchmark on 100_000 uses, just to be safe.
const Global = (; data=[join((y, m, d, h, M, s)," ") for y in 2000:2010, m in 1:3, d in 1:20, h in 1:10, M in 1:30, s in 1:30]);
size(Global.data)
length(Global.data)
const sample_dts = map(1:100_000) do _
y, m, d, h, M, s = rand.(axes(Global.data))
dt = DateTime(y+2000, m, d, h-1, M-1)
end;
func_demo() = [readData(dt, 3) for dt in sample_dts];
macro_demo() = [#readData(dt, 3) for dt in sample_dts];
#btime func_demo()
#btime macro_demo()
They benchmark as identical
julia> #btime macro_demo();
5.409 ms (3 allocations: 781.34 KiB)
julia> #btime func_demo();
5.393 ms (3 allocations: 781.34 KiB)
Infact they specialize into (basically) the same code.
julia> #code_typed macro_demo()
CodeInfo(
1 ─ %1 = Main.sample_dts::Core.Compiler.Const(DateTime[2002-01-18T04:19:00, 2001-01-19T08:22:00, 2006-02-08T04:07:00, 2011-01-08T09:03:00, 2006-02-10T06:18:00, 2002-03-12T00:05:00, 2011-02-20T08:29:00, 2011-02-20T07:12:00, 2005-01-13T03:22:00, 2006-01-01T00:29:00 …
2005-03-10T04:29:00, 2002-03-12T09:11:00, 2002-03-11T00:28:00, 2007-02-12T02:26:00, 2003-02-15T07:29:00, 2009-01-01T02:02:00, 2009-
01-03T02:11:00, 2001-02-16T03:16:00, 2004-01-17T05:12:00, 2010-02-02T05:10:00], false)
│ %2 = %new(Base.Generator{Array{DateTime,1},getfield(Main, Symbol("##50#51"))}, getfield(Main, Symbol("##50#51"))(), %1)::Base.Gen
erator{Array{DateTime,1},getfield(Main, Symbol("##50#51"))}
│ %3 = invoke Base.collect(%2::Base.Generator{Array{DateTime,1},getfield(Main, Symbol("##50#51"))})::Array{String,1}
└── return %3
) => Array{String,1}
julia> #code_typed getfield(Main, Symbol("##50#51")).instance(1) # check the internals
│ %1 = %1 = Main.Global::Core.Compiler.Const((#==GIANT Inlined Const ==#)
│ %2 = Base.getfield(%1, :data)::Array{String,6}
│ %3 = Base.sub_int(dt, 2000)::Int64
│ %4 = Base.add_int(dt, 1)::Int64
│ %5 = Base.add_int(dt, 1)::Int64
│ %6 = Base.arrayref(true, %2, %3, dt, dt, %4, %5, 3)::String
└── return %6
) => String
julia> #code_typed func_demo()
CodeInfo(
1 ─ %1 = Main.sample_dts::Core.Compiler.Const(DateTime[2002-01-18T04:19:00, 2001-01-19T08:22:00, 2006-02-08T04:07:00, 2011-01-08T09:03:00, 2006-02-10T06:18:00, 2002-03-12T00:05:00, 2011-02-20T08:29:00, 2011-02-20T07:12:00, 2005-01-13T03:22:00, 2006-01-01T00:29:00 … 2005-03-10T04:29:00, 2002-03-12T09:11:00, 2002-03-11T00:28:00, 2007-02-12T02:26:00, 2003-02-15T07:29:00, 2009-01-01T02:02:00, 2009-
01-03T02:11:00, 2001-02-16T03:16:00, 2004-01-17T05:12:00, 2010-02-02T05:10:00], false)
│ %2 = %new(Base.Generator{Array{DateTime,1},getfield(Main, Symbol("##43#44"))}, getfield(Main, Symbol("##43#44"))(), %1)::Base.Gen
erator{Array{DateTime,1},getfield(Main, Symbol("##43#44"))}
│ %3 = invoke Base.collect(%2::Base.Generator{Array{DateTime,1},getfield(Main, Symbol("##43#44"))})::Array{String,1}
└── return %3
) => Array{String,1}
julia> #code_typed getfield(Main, Symbol("##43#44")).instance(1)
CodeInfo(
1 ─ %1 = Main.Global::NamedTuple{(:data,),Tuple{Array{String,6}}}
│ %2 = Base.getfield(%1, :data)::Array{String,6}
│ %3 = Base.sub_int(dt, 2000)::Int64
│ %4 = Base.add_int(dt, 1)::Int64
│ %5 = Base.add_int(dt, 1)::Int64
│ %6 = Base.arrayref(true, %2, %3, dt, dt, %4, %5, 3)::String
└── return %6
) => String
There is a very minor difference in the generors function between the two.
Wheree the value became a Compliler.Const or a NamedTuple when inlining,
but after that goes the LLVM that difference goes way too I think
(Check #code_llvm if your really interested. But we are already super deap into the weeds.)
This is probably the wrong code to be optimizing in the first place.
A long with the guidance to benchmark any optimization you do.
One should also profile the code to decide what is worth optimizing.
A function that is only called 10,000s of times and not allocating giant arrays etc, probably not worth worrying too much about.
Especially if you are just worrying about function call overhead,
which is only a handful of CPU cycles.
You have to splice in the variables you pass as macro arguments:
julia> macro readData(dtCursor, value)
return :(
Global.data[
Dates.value(Dates.Year($dtCursor)) - 2000,
Dates.value(Dates.Month($dtCursor)),
Dates.value(Dates.Day($dtCursor)),
Dates.value(Dates.Hour($dtCursor)) + 1,
Dates.value(Dates.Minute($dtCursor)) + 1,
$value
]
)
end
#readData (macro with 1 method)
julia> #macroexpand #readData(dtCursor, 1)
:((Main.Global).data[(Main.Dates).value((Main.Dates).Year(Main.dtCursor)) - 2000, (Main.Dates).value((Main.Dates).Month(Main.dtCursor)), (Main.Dates).value((Main.Dates).Day(Main.dtCursor)), (Main.Dates).value((Main.Dates).Hour(Main.dtCursor)) + 1, (Main.Dates).value((Main.Dates).Minute(Main.dtCursor)) + 1, 1])
Furthermore, Julia macros are hygenic; that means that there will be no confusion about the name _dtCursor in the macro definition, and the name dtCursor at call site. One thing you might need to do is to escape the inputs, though.
Also, this might be an overkill. You should benchmark the macro version against the function version; maybe, there's enough inlining happening that the macro doesn't actually matter.
Consider the following functions whose definitions contain re-definitions of functions.
function foo1()
x = 1
if x != 1
error("Wrong")
end
x = 2
end
function foo()
function p(t) return t + 1 end
if p(1) != 2
error("Wrong")
end
function p(t) return 1 end
end
foo1() runs without error, but foo() gives the error Wrong. I thought it may have something to do with Julia not supporting redefining functions in general, but I'm not sure. Why is this happening?
I would say that this falls into a more general known problem https://github.com/JuliaLang/julia/issues/15602.
In your case consider a simpler function:
function f()
p() = "before"
println(p())
p() = "after"
nothing
end
Calling f() will print "after".
In your case you can inspect what is going on with foo in the following way:
julia> #code_typed foo()
CodeInfo(
4 1 ─ invoke Main.error("Wrong"::String)::Union{} │
│ $(Expr(:unreachable))::Union{} │
└── $(Expr(:unreachable))::Union{} │
) => Union{}
and you see that Julia optimizes out all internal logic and just calls error.
If you inspect it one step earlier you can see:
julia> #code_lowered foo()
CodeInfo(
2 1 ─ p = %new(Main.:(#p#7)) │
3 │ %2 = (p)(1) │
│ %3 = %2 != 2 │
└── goto #3 if not %3 │
4 2 ─ (Main.error)("Wrong") │
6 3 ─ return p │
)
any you see that p in top line is assigned only once. Actually the second definition is used (which is not visible here, but could be seen above).
To solve your problem use anonymous functions like this:
function foo2()
p = t -> t + 1
if p(1) != 2
error("Wrong")
end
p = t -> 1
end
and all will work as expected. The limitation of this approach is that you do not get multiple dispatch on name p (it is bound to a concrete anonymous function, but I guess you do not need multiple dispatch in your example).