Converting some code from v0.6.4 to v1.0.1, I noticed a strange behaviour of basic mathematic operators with Array{Union{Float64, Missing}} in v0.6.4. I am not sure this issue is still there for v0.7+. I would like to find a solution which works for v0.6.4, so that I can compare the output of my code with its current working version.
Suppose I have,
using Missings;
A = Array{Union{Float64, Missing}}([2.0 1.0 missing]);
B = [2.0 1.0 0.5];
In v0.6.4, the sum A+B and the element-wise sum A.+B give an output matrix which is Array{Any,2}. I was expecting an Array{Union{Float64, Missing},2}, since the first argument of the sum has this type. Same applies to other basic mathematical operations.
I found an ugly workaround for A+B, but not for the element-wise sum A.+B. What would you suggest to do in order to get the correct type for both operations (and similar - e.g., A.-B)?
I found a (relatively) simple solution to handle A.+B. I defined an ad-hoc broadcast method for the sum operator that converts the output of Base.Broadcast.broadcast_c into a Array{Union{T, Missing}}.
import Base: +, broadcast, Broadcast.broadcast_c, Broadcast.containertype;
broadcast(+, A::Array{Union{T, Missing}}, B::Array{T}) where {T<:Number} =
broadcast_c(+, containertype(A, B), A, B) |> Array{Union{T, Missing}};
broadcast(+, A::Array{T}, B::Array{Union{T, Missing}}) where {T<:Number} =
broadcast_c(+, containertype(A, B), A, B) |> Array{Union{T, Missing}};
broadcast(+, A::Array{Union{T, Missing}}, B::Array{Union{T, Missing}}) where {T<:Number} =
broadcast_c(+, containertype(A, B), A, B) |> Array{Union{T, Missing}};
As mentioned, this is not a problem for Julia 0.7+, but it is for 0.6.4.
Related
I am trying to wrap my head around the LinearMaps.jl package. In their documentation they have only two examples and I am stuck at the first one.
# Define a linear map from a matrix and one from a function (?)
A = LinearMap(rand(10, 10))
B = LinearMap(cumsum, reverse∘cumsum∘reverse, 10)
# Possible operations
3.0A + 2B
A + I
A*B'
[A B; B A]
kron(A, B)
How does LinearMap() work when its argument(s) are functions? For instance the cumsum() function doesn't work on its own as it requires the dims argument, so I am completely puzzled as to what this is doing.
In many Machine Learning use cases, you need to create an array filled with ones, with specific dimensions. In Python, I would use np.ones((2, 1)). What is the analog version of this in Julia?
Julia has a built in ones function which can be used as follows:
julia> ones(1,2)
1×2 Matrix{Float64}:
1.0 1.0
You can read more about the ones function in the Julia docs.
The answer by Logan is excellent. You can just use the ones function.
BUT, you can also often not use it.
For instance, a common use of a vector of ones is to multiply that vector times another vector so you get a matrix where each row just has the same value as in the corresponding element of the matrix. Then you can add that matrix to something. This allows you add the values of a vector to the corresponding rows of a matrix. You get code like this:
>>> A = np.random.rand(4,3)
>>> x = np.random.rand(4)
array([0.01250529, 0.9620139 , 0.70991563, 0.99795451])
>>> A + np.reshape(np.ones(3), (1,3)) * np.reshape(x, (4,1))
array([[0.09141967, 0.83982525, 0.16960596],
[1.39104681, 1.10755182, 1.60876696],
[1.14249757, 1.68167344, 1.64738165],
[1.10653393, 1.45162139, 1.23878815]])
This is actually a lot of extra work for the computer because Python can't optimize this and a lot of extra work is going on. You could also use so called broadcasting to do this extension more simply, but you still have to get x into the right shape:
>>> A + x
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: operands could not be broadcast together with shapes (4,3) (4,)
>>> A + np.reshape(x, (4,1))
array([[0.09141967, 0.83982525, 0.16960596],
[1.39104681, 1.10755182, 1.60876696],
[1.14249757, 1.68167344, 1.64738165],
[1.10653393, 1.45162139, 1.23878815]])
In Julia, the extension of the vector to the same shape as the matrix you to which you want to add can be done more simply using the broadcast operator. Thus, the code above simplifies to
julia> A = rand(4,3)
4×3 Matrix{Float64}:
0.885593 0.494999 0.534039
0.915725 0.479218 0.229797
0.739122 0.670486 0.247376
0.419879 0.857314 0.652547
julia> x = rand(4)
4-element Vector{Float64}:
0.9574839624590326
0.9736140903654276
0.6051487944513263
0.3581090323172089
julia> A .+ x
4×3 Matrix{Float64}:
1.84308 1.45248 1.49152
1.88934 1.45283 1.20341
1.34427 1.27563 0.852524
0.777988 1.21542 1.01066
One reason that this works better is because there is less noise in the syntax because arrays are primitive to Julia.
Much more importantly, though, compiler sees the use of the broadcast operator and it can generate very efficient code (and can even vectorize it). In fact, x doesn't even have to be an actual vector as long as it has a few of the same methods defined for it.
In fact, if you really do need a vector or matrix of all ones (or some other constant) you can use broadcast with scalars as well
julia> A .+ 1
4×3 Matrix{Float64}:
1.88559 1.495 1.53404
1.91572 1.47922 1.2298
1.73912 1.67049 1.24738
1.41988 1.85731 1.65255
Given two vectors a = [1, 2] and b = [3, 4], how do I obtain the concatenated vector c = [1, 2, 3, 4]? It seems that hcat or vcat could be used as they work on arrays, but when using vectors to store collections of elements it seems unfitting to first think about the orientation of the data; it's just supposed to be a list of values.
You can write
[a; b]
Under the hood this is the same as vcat, but it's terser, looks better, and is easier to remember, as it's also consistent with literal matrix construction syntax.
An alternative for concatenating multiple vectors is
reduce(vcat, (a, b))
Most Array methods treat arrays as general "tensors" of arbitrary ranks ("data cubes"), so you do need to think about the orientation. In the general case, there's cat(a, b; dims), of which hcat and vcat are special cases.
There is another class of methods treating Vectors as list like. From those, append! is the method that, well, appends a vector to another. The problem is that it is mutable. So you can, for example, append!(copy(a), b), or use something like BangBang.NoBang.append (which just selects the right method internally, though).
For the case of more than two vectors to be concatenated, I like the pattern of
reduce(append!, (a, b), init=Int[])
I couldn't find an already made function in Julia to compute Pearson's r so I resorted to trying to make it myself however I run into trouble.
code:
r(x,y) = (sum(x*y) - (sum(x)*sum(y))/length(x))/sqrt((sum(x^2)-(sum(x)^2)/length(x))*(sum(y^2)-(sum(y)^2)/length(x)))
if I attempt to run this on two arrays:
b = [4,8,12,16,20,24,28]
q = [5,10,15,20,25,30,35]
I get the following error:
ERROR: `*` has no method matching *(::Array{Int64,1}, ::Array{Int64,1})
in r at none:1
Pearson's r is available in Julia as cor:
julia> cor(b,q)
1.0
When you're looking for functions in Julia, the apropos function can be very helpful:
julia> apropos("pearson")
Base.cov(v1[, v2][, vardim=1, corrected=true, mean=nothing])
Base.cor(v1[, v2][, vardim=1, mean=nothing])
The issue you're running into with your definition is the difference between elementwise multiplication/exponentiation and matrix multiplication/exponentiation. In order to use elementwise behavior as you intend, you need to .* and .^:
r(x,y) = (sum(x.*y) - (sum(x)*sum(y))/length(x))/sqrt((sum(x.^2)-(sum(x)^2)/length(x))*(sum(y.^2)-(sum(y)^2)/length(x)))
With only those three changes, your r definition seems to match Julia's cor to within a few ULPs:
julia> cor(b,q)
1.0
julia> x,y = randn(10),randn(10)
([-0.2384626335813905,0.0793838075714518,2.395918475924737,-1.6271954454542266,-0.7001484742860653,-0.33511064476423336,-1.5419149314518956,-0.8284664940238087,-0.6136547926069563,-0.1723749334766532],[0.08581770755520171,2.208288163473674,-0.5603452667737798,-3.0599443201343854,0.585509815026569,0.3876891298047877,-0.8368409374755644,1.672421071281691,0.19652240951291933,0.9838306761261647])
julia> r(x,y)
0.23514468093214283
julia> cor(x,y)
0.23514468093214275
Julia's cor is defined iteratively (this is the zero-mean implementation — calling cor first subtracts the mean and then calls corzm) which means fewer allocations and better performance. I can't speak to the numerical accuracy.
Your function is trying to multiply two column vectors. You will need to invert transpose one of them. Consider:
> [1,2]*[3,4]
ERROR: `*` has no method matching *(::Array{Int64,1}, ::Array{Int64,1})
but:
> [1,2]'*[3,4]
1-element Array(Int64,1)
11
and:
> [1,2]*[3,4]'
2x2 Array(Int64,2):
3 4
6 8
I have a five-dimensional rootfinding problem I'd like to solve from within a Sage notebook, but the functions I wish to solve depend on other parameters that shouldn't be varied during the rootfinding. Figuring out how to set up a call to, say, scipy.optimize.newton_krylov has got me stumped. So let's say I have (with a,b,c,d,e the parameters I want to vary, F1,F2,F3,F4,F5 the five expressions I which to solve to be equal to F1Val,F2Val,F3Val,F4Val,F5Val, values I already know, and posVal another known parameter)
def func(a, b, c, d, e, F1Val, F2Val, F3Val, F4Val, F5Val, posVal):
F1.subs(x1=a,x2=b,x3=c,x4=d,x5=e,position=posVal)
F2.subs(x1=a,x2=b,x3=c,x4=d,x5=e,position=posVal)
F3.subs(x1=a,x2=b,x3=c,x4=d,x5=e,position=posVal)
F4.subs(x1=a,x2=b,x3=c,x4=d,x5=e,position=posVal)
F5.subs(x1=a,x2=b,x3=c,x4=d,x5=e,position=posVal)
return (F1-F1Val, F2-F2Val, F3-F3Val, F4-F4Val, F5-F5Val)
and now I want to pass this to a rootfinding function to yield func = (0,0,0,0,0). I want to pass an initial guess (a0, b0, c0, d0, e0) vector and a set of arguments (F1Val, F2Val, F3Val, F4Val, F5Val, posVal) for the evaluation, but I can't figure out how to do this. Is there a standard technique for this sort of thing? The multidimensional rootfinders in scipy seem to be lacking the args=() variable that the 1D rootfinders offer.
Best,
-user2275987
Well, I'm still not sure how to actually employ the Newton-Raphson method here, but using fsolve works, for functions that accept a vector of variables and a vector of constant arguments. I'm reproducing my proof of concept here
def tstfunc(xIn, constIn):
x = xIn[0]
y = xIn[1]
a = constIn[0]
b = constIn[1]
out = [x+2*y+a]
out.append(a*x*y+b)
return out
from scipy.optimize import fsolve
ans = fsolve(tstfunc, x0=[1,1], args=[0.3, 2.1])
print ans