In many applications of map(f,X), it helps to create closures that depending on parameters apply different functions f to data X.
I can think of at least the following three ways to do this (note that the second for some reason does not work, bug?)
f0(x,y) = x+y
f1(x,y,p) = x+y^p
function g0(power::Bool,X,y)
if power
f = x -> f1(x,y,2.0)
else
f = x -> f0(x,y)
end
map(f,X)
end
function g1(power::Bool,X,y)
if power
f(x) = f1(x,y,2.0)
else
f(x) = f0(x,y)
end
map(f,X)
end
abstract FunType
abstract PowerFun <: FunType
abstract NoPowerFun <: FunType
function g2{S<:FunType}(T::Type{S},X,y)
f(::Type{PowerFun},x) = f1(x,y,2.0)
f(::Type{NoPowerFun},x) = f0(x,y)
map(x -> f(T,x),X)
end
X = 1.0:1000000.0
burnin0 = g0(true,X,4.0) + g0(false,X,4.0);
burnin1 = g1(true,X,4.0) + g1(false,X,4.0);
burnin2 = g2(PowerFun,X,4.0) + g2(NoPowerFun,X,4.0);
#time r0true = g0(true,X,4.0); #0.019515 seconds (12 allocations: 7.630 MB)
#time r0false = g0(false,X,4.0); #0.002984 seconds (12 allocations: 7.630 MB)
#time r1true = g1(true,X,4.0); # 0.004517 seconds (8 allocations: 7.630 MB, 26.28% gc time)
#time r1false = g1(false,X,4.0); # UndefVarError: f not defined
#time r2true = g2(PowerFun,X,4.0); # 0.085673 seconds (2.00 M allocations: 38.147 MB, 3.90% gc time)
#time r2false = g2(NoPowerFun,X,4.0); # 0.234087 seconds (2.00 M allocations: 38.147 MB, 60.61% gc time)
What is the optimal way to do this in Julia?
There's no need to use map here at all. Using a closure doesn't make things simpler or faster. Just use "dot-broadcasting" to apply the functions directly:
function g3(X,y,power=1)
if power != 1
return f1.(X, y, power) # or simply X .+ y^power
else
return f0.(X, y) # or simply X .+ y
end
end
Related
Im working with some roughly 100000x100000 hermitian complex sparse-matrices, with roughly 5% of entries populated, and want to calculate the eigenvalues/eigenvectors.
Sofar ive been using Arpack.jl eigs(A).
But this is not working well as soon as i crank the size to higher then 5000.
For the benchmarks ive been using the following code to generate some TestMatrices:
using Arpack
using SparseArrays
using ProgressMeter
pop = 0.05
n = 3000 # for example
A = spzeros(Complex{Float64}, n, n)
#showprogress for _ in 1:round(Int,pop * (n^2))
A[rand(1:n), rand(1:n)] = rand(Complex{Float64})
end
# make A hermite
A = A + conj(A)
t = #elapsed eigs(A,maxiter=1500) # ends up being ~ 13 seconds
For n ~ 3000 the eigs() call already takes 13 seconds on my machine, and for bigger n it doesn't finish in any 'reasonable' time or outright quits.
Is there a specialized package/method for this ?
Any help is appreciated
https://github.com/JuliaLinearAlgebra/ArnoldiMethod.jl seems to be what you want:
julia> let pop=0.05, n=3000
A = sprand(Complex{Float64},n,n, 0.05)
A = A + conj(A)
#time eigs(A; maxiter=1500)
#time decomp, history = partialschur(A, nev=10, tol=1e-6, which=LM());
end;
10.521786 seconds (50.73 k allocations: 3.458 MiB, 0.04% gc time)
2.244129 seconds (19 allocations: 1.892 MiB)
sanity check:
julia> a,(b,c) = let pop=0.05, n=300
A = sprand(Complex{Float64},n,n, 0.05)
A = A + conj(A)
eigs(A; maxiter=2500), partialschur(A, nev=6, tol=1e-6, which=LM());
end;
julia> a[1]
6-element Vector{ComplexF64}:
14.5707071003175 + 8.218901803015509e-16im
4.493079744504954 - 0.8390429567118733im
4.493079744504933 + 0.8390429567118641im
-0.3415176925293196 + 4.254184281244591im
-0.3415176925293088 - 4.25418428124452im
0.49406553681541177 - 4.229680489599233im
julia> b
PartialSchur decomposition (ComplexF64) of dimension 6
eigenvalues:
6-element Vector{ComplexF64}:
14.570707100307926 + 7.10698633463049e-12im
4.493079906269516 + 0.8390429076809746im
4.493079701528448 - 0.8390430155670777im
-0.3415174262177961 + 4.254183175902487im
-0.34151626930774975 - 4.25418321627979im
0.49406543866702 + 4.229680079205066im
I'm trying to do a gradation plotting.
using Plots
using LinearAlgebra
L = 60 #size of a matrix
N = 10000 #number of loops
E = zeros(Complex{Float64},N,L) #set of eigenvalues
IPR = zeros(Complex{Float64},N,L) #indicator for marker_z
Preparing E & IPR
function main()
cnt = 0
for i = 1:N
cnt += 1
H = rand(Complex{Float64},L,L)
eigenvalue,eigenvector = eigen(H)
for j = 1:L
E[cnt,j] = eigenvalue[j]
IPR[cnt,j] = abs2(norm(abs2.(eigenvector[:,j])))/(abs2(norm(eigenvector[:,j])))
end
end
end
Plotting
function main1()
plot(real.(E),imag.(E),marker_z = real.(IPR),st = scatter,markercolors=:cool,markerstrokewidth=0,markersize=1,dpi=300)
plot!(legend=false,xlabel="ReE",ylabel="ImE")
savefig("test.png")
end
#time main1()
358.794885 seconds (94.30 M allocations: 129.882 GiB, 2.05% gc time)
Comparing with a uniform plotting, a gradation plotting takes too much time.
function main2()
plot(real.(E),imag.(E),st = scatter,markercolor=:blue,markerstrokewidth=0,markersize=1,dpi=300)
plot!(legend=false,xlabel="ReE",ylabel="ImE")
savefig("test1.png")
end
#time main2()
8.100609 seconds (10.85 M allocations: 508.054 MiB, 0.47% gc time)
Is there a way of gradation plotting as fast as a uniform plotting?
I solved the problem by myself.
After updating from Julia 1.3.1 to Julia1.6.3, I checked the main1 became faster as Bill's comments.
I have an array that contains repeated nonnegative integers, e.g., A=[5,5,5,0,1,1,0,0,0,3,3,0,0]. I would like to find the position of the last maximum in A. That is the largest index i such that A[i]>=A[j] for all j. In my example, i=3.
I tried to find the indices of all maximum of A then find the maximum of these indices:
A = [5,5,5,0,1,1,0,0,0,3,3,0,0];
Amax = maximum(A);
i = maximum(find(x -> x == Amax, A));
Is there any better way?
length(A) - indmax(#view A[end:-1:1]) + 1
should be pretty fast, but I didn't benchmark it.
EDIT: I should note that by definition #crstnbr 's solution (to write the algorithm from scratch) is faster (how much faster is shown in Xiaodai's response). This is an attempt to do it using julia's inbuilt array functions.
What about findlast(A.==maximum(A)) (which of course is conceptually similar to your approach)?
The fastest thing would probably be explicit loop implementation like this:
function lastindmax(x)
k = 1
m = x[1]
#inbounds for i in eachindex(x)
if x[i]>=m
k = i
m = x[i]
end
end
return k
end
I tried #Michael's solution and #crstnbr's solution and I found the latter much faster
a = rand(Int8(1):Int8(5),1_000_000_000)
#time length(a) - indmax(#view a[end:-1:1]) + 1 # 19 seconds
#time length(a) - indmax(#view a[end:-1:1]) + 1 # 18 seconds
function lastindmax(x)
k = 1
m = x[1]
#inbounds for i in eachindex(x)
if x[i]>=m
k = i
m = x[i]
end
end
return k
end
#time lastindmax(a) # 3 seconds
#time lastindmax(a) # 2.8 seconds
Michael's solution doesn't support Strings (ERROR: MethodError: no method matching view(::String, ::StepRange{Int64,Int64})) or sequences so I add another solution:
julia> lastimax(x) = maximum((j,i) for (i,j) in enumerate(x))[2]
julia> A="abžcdž"; lastimax(A) # unicode is OK
6
julia> lastimax(i^2 for i in -10:7)
1
If you more like don't catch exception for empty Sequence:
julia> lastimax(x) = !isempty(x) ? maximum((j,i) for (i,j) in enumerate(x))[2] : 0;
julia> lastimax(i for i in 1:3 if i>4)
0
Simple(!) benchmarks:
This is up to 10 times slower than Michael's solution for Float64:
julia> mlastimax(A) = length(A) - indmax(#view A[end:-1:1]) + 1;
julia> julia> A = rand(Float64, 1_000_000); #time lastimax(A); #time mlastimax(A)
0.166389 seconds (4.00 M allocations: 91.553 MiB, 4.63% gc time)
0.019560 seconds (6 allocations: 240 bytes)
80346
(I am surprised) it is 2 times faster for Int64!
julia> A = rand(Int64, 1_000_000); #time lastimax(A); #time mlastimax(A)
0.015453 seconds (10 allocations: 304 bytes)
0.031197 seconds (6 allocations: 240 bytes)
423400
it is 2-3 times slower for Strings
julia> A = ["A$i" for i in 1:1_000_000]; #time lastimax(A); #time mlastimax(A)
0.175117 seconds (2.00 M allocations: 61.035 MiB, 41.29% gc time)
0.077098 seconds (7 allocations: 272 bytes)
999999
EDIT2:
#crstnbr solution is faster and works with Strings too (doesn't work with generators). There difference between lastindmax and lastimax - first return byte index, second return character index:
julia> S = "1š3456789ž"
julia> length(S)
10
julia> lastindmax(S) # return value is bigger than length
11
julia> lastimax(S) # return character index (which is not byte index to String) of last max character
10
julia> S[chr2ind(S, lastimax(S))]
'ž': Unicode U+017e (category Ll: Letter, lowercase)
julia> S[chr2ind(S, lastimax(S))]==S[lastindmax(S)]
true
Consider the following simple Julia code operating on four complex matrices:
n = 400
z = eye(Complex{Float64},n)
id = eye(Complex{Float64},n)
fc = map(x -> rand(Complex{Float64}), id)
cr = map(x -> rand(Complex{Float64}), id)
s = 0.1 + 0.1im
#time for j = 1:n
for i = 1:n
z[i,j] = id[i,j] - fc[i,j]^s * cr[i,j]
end
end
The timing shows a few million memory allocations, despite all variables being preallocated:
0.072718 seconds (1.12 M allocations: 34.204 MB, 7.22% gc time)
How can I avoid all those allocations (and GC)?
One of the first tips for performant Julia code is to avoid using global variables. This alone can cut the number of allocations by 7 times. If you must use globals, one way to improve their performance is to use const. Using const prevents change of type but change of value is possible with a warning.
consider this modified code without using functions:
const n = 400
z = Array{Complex{Float64}}(n,n)
const id = eye(Complex{Float64},n)
const fc = map(x -> rand(Complex{Float64}), id)
const cr = map(x -> rand(Complex{Float64}), id)
const s = 0.1 + 0.1im
#time for j = 1:n
for i = 1:n
z[i,j] = id[i,j] - fc[i,j]^s * cr[i,j]
end
end
The timing shows this result:
0.028882 seconds (160.00 k allocations: 4.883 MB)
Not only did the number of allocations get 7 times lower, but also the execution speed is 2.2 times faster.
Now let's apply the second tip for high performance Julia code; write every thing in functions. Writing the above code into a function z_mat(n):
function z_mat(n)
z = Array{Complex{Float64}}(n,n)
id = eye(Complex{Float64},n)
fc = map(x -> rand(Complex{Float64}), id)
cr = map(x -> rand(Complex{Float64}), id)
s = 1.0 + 1.0im
#time for j = 1:n
for i = 1:n
z[i,j] = id[i,j] - fc[i,j]^s * cr[i,j]
end
end
end
and running
z_mat(40)
0.000273 seconds
#time z_mat(400)
0.027273 seconds
0.032443 seconds (429 allocations: 9.779 MB)
That is 2610 times fewer allocations than the original code for the whole function because the loop alone does zero allocations.
I would like to initialize a 3d tensor (multi-dimensional array) with the values of the "diagonal Gaussian"
exp(-32*(u^2 + 16*(v^2 + w^2)))
where u = 1/sqrt(3)*(x+y+z) and v,w are any two coordinates orthogonal to u, discretised on a uniform mesh on [-1,1]^3. The following code achieves this:
function gaussian3d(n)
q = qr(ones(3,1), thin=false)[1]
x = linspace(-1.,1., n)
p = Array(Float64,(n,n,n))
square(x) = x*x
Base.#nloops 3 i p begin
#inbounds p[i_1,i_2,i_3] =
exp(
-32*(
square(q[1,1]*x[i_1] + q[2,1]*x[i_2] + q[3,1]*x[i_3])
+ 16*(
square(q[1,2]*x[i_1] + q[2,2]*x[i_2] + q[3,2]*x[i_3]) +
square(q[1,3]*x[i_1] + q[2,3]*x[i_2] + q[3,3]*x[i_3])
)
)
)
end
return p
end
It seems to be quite slow, however. For example, if I replace the defining function with exp(x*y*z), the code runs 50x faster. Also, the #time macro reports ~20% GC time for the above code which I do not understand where they come from. (These numeric values were obtained with n = 128.) My questions therefore are
How can I speed up this piece of code?
Where is the memory allocation hidden which causes the GC overhead?
Knowing nothing of 3D tensors with values of the "diagonal Gaussian", using thesquare comment from the original post, "typing" q (#code_warntype helps here: Big performance jump!), and further specializing the #nloops, this works much faster on the platforms I tried it on.
julia> square(x::Float64) = x * x
square (generic function with 1 method)
julia> function my_gaussian3d(n)
q::Array{Float64,2} = qr(ones(3,1), thin=false)[1]
x = linspace(-1.,1., n)
p = Array(Float64,(n,n,n))
Base.#nloops 3 i p d->x_d=x[i_d] begin
#inbounds p[i_1,i_2,i_3] =
exp(
-32*(
square(q[1,1]*x_1 + q[2,1]*x_2 + q[3,1]*x_3)
+ 16*(
square(q[1,2]*x_1 + q[2,2]*x_2 + q[3,2]*x_3) +
square(q[1,3]*x_1 + q[2,3]*x_2 + q[3,3]*x_3)
)
)
)
end
return p
end
my_gaussian3d (generic function with 1 method)
julia> #time gaussian3d(128);
elapsed time: 3.952389337 seconds (1264 MB allocated, 4.50% gc time in 57 pauses with 0 full sweep)
julia> #time gaussian3d(128);
elapsed time: 3.527316699 seconds (1264 MB allocated, 4.42% gc time in 58 pauses with 0 full sweep)
julia> #time my_gaussian3d(128);
elapsed time: 0.285837566 seconds (16 MB allocated)
julia> #time my_gaussian3d(128);
elapsed time: 0.28476448 seconds (16 MB allocated, 1.22% gc time in 0 pauses with 0 full sweep)
julia> my_gaussian3d(128) == gaussian3d(128)
true