What is the simplest way to iterate over an array of arrays? - julia

Let x::Vector{Vector{T}}. What is the best way to iterate over all the elements of each inner vector (that is, all elements of type T)? The best I can come up with is a double iteration using the single-line notation, ie:
for n in eachindex(x), m in eachindex(x[n])
x[n][m]
end
but I'm wondering if there is a single iterator, perhaps in the Iterators package, designed specifically for this purpose, e.g. for i in some_iterator(x) ; x[i] ; end.
More generally, what about iterating over the inner-most elements of any array of arrays (that is, arrays of any dimension)?

Your way
for n in eachindex(x), m in eachindex(x[n])
x[n][m]
end
is pretty fast. If you want best speed, use
for n in eachindex(x)
y = x[n]
for m in eachindex(y)
y[m]
end
end
which avoids dereferencing twice (the first dereference is hard to optimize out because arrays are mutable, and so getindex isn't pure). Alternatively, if you don't need m and n, you could just use
for y in x, for z in y
z
end
which is also fast.
Note that column-major storage is irrelevant, since all arrays here are one-dimensional.
To answer your general question:
If the number of dimensions is a compile-time constant, see Base.Cartesian
If the number of dimensions is not a compile-time constant, use recursion
And finally, as Dan Getz mentioned in a comment:
using Iterators
for z in chain(x...)
z
end
also works. This however has a bit of a performance penalty.

I'm wondering if there is a single iterator, perhaps in the Iterators package, designed specifically for this purpose, e.g. for i in some_iterator(x) ; x[i] ; end
Today (in Julia 1.x versions), Iterators.flatten is exactly this.
help?> Iterators.flatten
flatten(iter)
Given an iterator that yields iterators, return an iterator that
yields the elements of those iterators. Put differently, the
elements of the argument iterator are concatenated.
julia> x = [1:5, [π, ℯ, 42], 'a':'e']
3-element Vector{AbstractVector}:
1:5
[3.141592653589793, 2.718281828459045, 42.0]
'a':1:'e'
julia> for el in Iterators.flatten(x)
print(el, " ")
end
1 2 3 4 5 3.141592653589793 2.718281828459045 42.0 a b c d e
julia>

Related

Inclusivity of Julia ranges

I hate that ranges include the end. Here is an example where I've deliberately removed the end of the range.
N = 100
for x in 0.0 : 2*pi/N : 2*pi*(N-1)/N
println(x)
end
Is there any way to avoid the ugliness of this for loop?
Yes, there is
N = 100
for x in range(0; step=2π/N, length=N)
println(x)
end
Maybe not the most elegant way... take the first n-1 elements
r = 0.0 : 2*pi/N : 2*pi
r = Iterators.take(r,length(r)-1)
Unfortunately, inclusive ranges (and 1-based indexing) is baked into the idioms of Julia at a fundamental level.
However, for this specific case, do note that stepping with floating point values can be problematic, as adding N values might be less than, equal to, or greater than the final value, giving different results for the for loop. Although julia tries really hard, there's no way to quite do the right thing in all circumstances. As a bonus, working in integer values only for the ranges simplifies things. You might want to consider:
for ix in 0:N-1
x = ix * 2 * pi / N
println(x)
end
Alternatively, the range() function has a form with a len parameter:
for x in range(0, 2*pi*(N-1)/N, length=n)
println(x)
end
Or indeed, combining this with the other answer of only taking (N-1) could work.
You could actually define your own operator such as:
▷(a,b) = a:b-1
Now you can write:
julia> 3▷6
3:5
Julia also natively supports custom indices for arrays. There is a package CustomUnitRanges that is maybe an overkill here.

Trouble understanding OCAML power function

I'm trying to make sense of this Caml function to calculate the n-th power of a number.
let rec iterate n f d =
if n = 0 then d
else iterate (n-1) f (f d)
let power i n =
let i_times a = a * i in
iterate n i_times 1
I understand what it does conceptually, but I am having trouble understanding the i_times bit.
Per my understanding, i_times takes a value a and returns a*i, whre i is passed onto power upon calling it. In the expression where i_times is defined, it's followed by 1, so wouldn't this mean that i_times 1, all together, is evaluated to 1*i?
In this case, I fail to see how 3 parameters are being passed to iterate. Why doesn't it just end up being 2, that is n and i_times 1?
I know this is a pretty basic function, but I'm just getting started with functional programming and I want to make sense of it.
Basically you're asking how OCaml parses a series of juxtaposed values.
This expression:
f a b c
is iterpreted as a call to a function f that passes 3 separate parameters. It is not parsed like this:
f a (b c)
which would be passing 2 parameters to f.
So indeed, three parameters are being passed to iterate. There are no subexpressions in the list of parameters, just three separate parameters.
Why didn't you figure that the subexpression n i_times would be evaluated before calling iterate? The reason (I suspect) is that you know that n isn't a function. But the parsing of the language doesn't depend on the types of things. If you wrote (n i_times) (with parentheses) it would be parsed as a call to n as a function. (This would, of course, be an error.)

Cumulative Integration Options With Julia

I have two 1-D arrays in which I would like to calculate the approximate cumulative integral of 1 array with respect to the scalar spacing specified by the 2nd array. MATLAB has a function called cumtrapz that handles this scenario. Is there something similar that I can try within Julia to accomplish the same thing?
The expected result is another 1-D array with the integral calculated for each element.
There is a numerical integration package for Julia (see the link) that defines cumul_integrate(X, Y) and uses the trapezoidal rule by default.
If this package didn't exist, though, you could easily write the function yourself and have a very efficient implementation out of the box because the loop does not come with a performance penalty.
Edit: Added an #assert to check matching vector dimensions and fixed a typo.
function cumtrapz(X::T, Y::T) where {T <: AbstractVector}
# Check matching vector length
#assert length(X) == length(Y)
# Initialize Output
out = similar(X)
out[1] = 0
# Iterate over arrays
for i in 2:length(X)
out[i] = out[i-1] + 0.5*(X[i] - X[i-1])*(Y[i] + Y[i-1])
end
# Return output
out
end

How to inherit from a regular matrix

using ShiftedArrays
struct CircularMatrix{T} <: AbstractArray{T,2}
data::Array{T,2}
view::CircShiftedArray
currentIndex::Int
function CircularMatrix{T}(dims...) where T
data = zeros(T, dims...)
CircularMatrix(data, ShiftedArrays.circshift(data, (0, -1)), 1)
end
end
Base.size(M::CircularMatrix) = size(M.data)
Base.eltype(::Type{CircularMatrix{T}}) where {T} = T
function shift_forward!(M::CircularMatrix)
M.shift_forward!(1)
end
function shift_forward!(M::CircularMatrix, n)
# replace the view with a view shifted forwards.
M.currentIndex += n
M.view = ShiftedArrays.circshift(M.data, (n, M.currentIndex))
end
#inline Base.#propagate_inbounds function Base.getindex(M::CircularMatrix, i) = M.view[i]
#inline Base.#propagate_inbounds function Base.setindex!(M::CircularMatrix, data, i) = M.view[i] = data
How can I make CircularMatrix act just like a regular matrix.
So that I can access it like
m = CircularMatrix{Int}(4,4)
m[1, 1] = 5
x = view(m, 1, :)
Your matrix type is defined to be a subtype of AbstractArray{T, 2}. You need to implement a few methods in the informal array interface of Julia for your type to make functions and features that work on AbstractArray{T, 2} to also work on your custom type, that is, to make your CircularMatrix an iterable, indexable, completely functioning matrix.
The methods to implement are
size(M::CircularMatrix)
getindex(M::CircularMatrix, i::Int)
getindex(M::CircularMatrix, I::Vararg{Int, N})
setindex!(M::CircularMatrix, v, i::Int)
setindex!(M::CircularMatrix, v, I::Vararg{Int, N})
You already implement 1, 2 and 4 but have not yet set your indexing style. You might not need 3 and 5 if you choose linear indexing style. You only need to set IndexStyle to be IndexLinear() and maybe a few modifications, then everything should just work for your matrix.
1. size(M::CircularMatrix)
The first one is size. size(A::CircularMatrix) returns a Tuple of dimensions of A. I believe for your matrix probably something like the following
Base.size(M::CircularMatrix) = size(M.data)
2. getindex(M::CircularMatrix, i::Int)
This method is needed if you choose linear indexing style. getindex(M, i::Int) should give you the value at linear index i. You already implement it in your code. If you choose linear indexing, you need to set IndexStyle for your type and then you simply skip 3 and 5. Julia will automatically convert multiple index accesses, e.g. a[3, 5], to a linear index access.
Base.IndexStyle(::Type{<:CircularMatrix}) = IndexLinear()
Base.#propogate_inbounds function Base.getindex(M::CircularMatrix, i::Int)
#boundscheck checkbounds(M, i)
#inbounds M.view[i]
end
It might be better to use #inbounds here on the second line. If the caller doesn't use #inbounds, we check the bounds first and this hopefully makes the subsequent bounds check unnecessary. You might want to omit this during development, though.
3. getindex(M::CircularMatrix, I::Vararg{Int, N})
The third one is for Cartesian indexing style. If you choose this style you need to implement this method. Vararg{Int, N} in the signature stands for "exactly N Int arguments". Here N should be equal to the dimensionality of CircularMatrix. Since this is a matrix, N should be two. If you choose this style, you need to define something like the following
Base.#propogate_inbounds function Base.getindex(A::CircularMatrix, I::Vararg{Int, 2})
#boundscheck checkbounds(A, I...)
#inbounds A.view[# convert I[1]` and `I[2]` to a linear index in `view`]
end
or since your dimensionality is not parametric and a matrix is 2D, simply
Base.#propogate_inbounds function Base.getindex(A::CircularMatrix, i::Int, j::Int)
#boundscheck checkbounds(A, i, j)
#inbounds A.view[# convert i` and `j` to a linear index in `view`]
end
4. setindex!(M::CircularMatrix, v, i::Int)
The fourth one is similar to the second. This method should set the value at linear index i, if you choose linear indexing style.
5. setindex!(M::CircularMatrix, v, I::Vararg{Int, N})
The fifth one should be similar to the third, if you choose Cartesian indexing style.
After the implementations for 1, 2, and 4 and setting IndexStyle, you should have a custom matrix type that just works.
m[1, 1] = 5
x = view(m, 1, :)
for e in
...
end
for i in eachindex(m)
...
end
display(m)
println(m)
length(m)
ndims(m)
map(f, A)
....
These should all work.
A few notes
There is a documentation for Abstract Arrays interface here with a few examples. You can also see Optional Methods to implement.
There is a JuliaArray organization on GitHub that provides lots of useful custom array implementations including StaticArrays, OffsetArrays, etc. and also a JuliaMatrices organization that provides custom matrix types. You might want to take a look at their implementations.
#inline is redundant if you use Base.#propogate_inbounds.
#propagate_inbounds
Tells the compiler to inline a function while retaining the caller's
inbounds context.
You do not need to define eltype for your matrix, since there is already a definition for AbstractArray{T, N} which returns T.

Internal manipulation of complex hermitian matrix / explain the use of "RealHermSymComplexHerm" in symmetric.jl

I think Julia handles matrices with complex elements correctly.
My task is to modify the spectrum of a Hermitian matrix H and return just the matrix with modified spectrum. i.e. I have a function f(real_vec)->real_vec that modifies the spectrum s(H) of a hermitian matrix H=U[s(H)]U'. I need the result f(H) = U[f(s(H))]U'. I think it is possible to optimize by not computing explicitly the eigfact(H).
Therefore I tried to write my own eigmodif based on the Julia realization of eigfact. That was difficult because I was lost on the line 4816 in lapack.jl, where syevr() is wrapped up.
I need to understand where and how, Julia has converted a COMPLEX HERMITIAN matrix to a REAL-SYMMETRIC one. Theoretically it is possible, since we have a 2n by 2n matrix J that squares to minus identity; for any n by n COMPLEX HERMITIAN matrix H we then turn it into real(H).I + imag(H).J, or in a block form
[ real(H) -imag(H) ]
[ imag(H) real(H) ]
But how does Julia do this?
Not an expert on LAPACK, but perhaps the use of macros in the definition of the eigensolvers is unclear. From linalg/lapack.jl (around line 4900):
# Hermitian eigensolvers
for (syev, syevr, sygvd, elty, relty) in
((:zheev_,:zheevr_,:zhegvd_,:Complex128,:Float64),
(:cheev_,:cheevr_,:chegvd_,:Complex64,:Float32))
#eval begin
# SUBROUTINE ZHEEV( JOBZ, UPLO, N, A, LDA, W, WORK, LWORK, RWORK, INFO )
# * .. Scalar Arguments ..
# CHARACTER JOBZ, UPLO
⋮
⋮
So the macro code uses $syevr as a placeholder to refer to :zheevr_ and :cheevr_ in the two passes through the loop, defining the same syevr! for different type signatures. These are LAPACK functions dedicated to Hermitian matrices and accept complex inputs. So the meat of the calculation and complex number handling goes on inside LAPACK.

Resources