Related
Consider:
x <- 5
replicate(10, x <- x + 1)
This has output c(6, 6, 6, 6, 6, 6, 6, 6, 6, 6). However:
x <- 5
replicate(10, x <<- x + 1)
has output c(6, 7, 8, 9, 10, 11, 12, 13, 14, 15).
What does this imply about the environment that x <- x + 1 is evaluated in? Am I to believe that x is treated as if it is an internal variable for replicate? That appears to be what I'm seeing, but when I consulted the relevant section of the language definition, I saw the following:
It is also worth noting that the effect of foo(x <- y) if the argument is evaluated is to change the value of x in the calling environment and not in the evaluation environment of foo.
But if x really was changed in the calling environment, then why does:
x <- 5
replicate(10, x <- x + 1)
x
Return 5 and not 15? What part have I misunderstood?
The sentence you quoted from the language definition is about standard evaluation, but replicate uses non-standard evaluation. Here's its source:
replicate <- function (n, expr, simplify = "array")
sapply(integer(n), eval.parent(substitute(function(...) expr)),
simplify = simplify)
The substitute(function(...) expr) call takes your expression x <- x + 1 without evaluating it, and creates a new function
function(...) x <- x + 1
That's the function that gets passed to sapply(), which applies it to a vector of length n. So all the assignments take place in the frame of that anonymous function.
When you use x <<- x + 1, the evaluation still takes place in the constructed function, but its environment is the calling environment to replicate() (because of the eval.parent call), and that's where the assignment happens. That's why you get the increasing values in the output.
So I think you understood the manual correctly, but it didn't make clear it was talking there about the case of standard evaluation. The following paragraph hints at what's happening here:
It is possible to access the actual (not default) expressions used as arguments inside the function. The mechanism is implemented via promises. When a function is being evaluated the actual expression used as an argument is stored in the promise together with a pointer to the environment the function was called from. When (if) the argument is evaluated the stored expression is evaluated in the environment that the function was called from. Since only a pointer to the environment is used any changes made to that environment will be in effect during this evaluation. The resulting value is then also stored in a separate spot in the promise. Subsequent evaluations retrieve this stored value (a second evaluation is not carried out). Access to the unevaluated expression is also available using substitute.
but the help page for replicate() doesn't make clear this is what it's doing.
BTW, your title asks about apply family functions: but most of them other than replicate ask explicitly for a function, so this issue doesn't arise there. For example, it's obvious that this doesn't affect the global x:
sapply(integer(10), function(i) x <- x + 1)
I would like to create a pointer to an array with smaller dimension.
For example, I have some array arr(1:2, 1:10, 1:10).
Now I want to create a pointer to arr(1:1, 1:10, 1:10) but I want to delete first I don't know how I should name it by it look like index, and second pointer to (2:2, 1:10, 1:10).
I need it because I would like to send array with 2 dimensions (matrix) to a function.
Here is an indication of what I want to do, with pseudocode.
INTEGER, DIMENSION (1:2, 1:10, 1:10), TARGET :: BOUNDRIES
INTEGER, DIMENSION (:,:), POINTER : LEFT_BOUNDRY
LEFT_BOUNDRY => BOUNDRIES(1,1:10,1:10)
DO i = 1,n
DO j = 1,10
write(*,*) LEFT_BOUNDRY(i,j)
END DO
END DO
Is it possible to do it?
When we have a dummy argument in a function or a subroutine (collectively, procedure) we have a corresponding actual argument when we execute that procedure. Consider the subroutine s:
subroutine s(x)
real x(5,2)
...
end subroutine s
The dummy argument x is in this case an explicit shape array, of rank 2, shape [5,2].
If we want to
call s(y)
where y is some real thing we don't need to have y a whole array which is of rank 2 and shape [5,2]. We simply need to have y have at least ten elements and a thing called storage association maps those ten elements to x when we are in the subroutine.
Imagine, then
real y1(10), y2(1,10), y3(29)
call s(y1)
call s(y2)
call s(y3)
Each of these works (in the final case, it's just the first ten elements that become associated with the dummy argument).
Crucially, it's a so-called element sequence that is important when choosing the elements to associate with x. Consider
real y(5,12,10,10)
call s (y(1,1,1:2,3:7))
This is an array section of y of ten elements. Those ten elements together become x in the subroutine s.
To conclude, if you want to pass arr(2,1:10,1:10) (which is actually a rank 2 array section) to a rank 2 argument which is an explicit shape array of no more than 100 elements, everything is fine.
I have a task in which I will have several data types together; character, several integers, and a double precision value, which represent a solution to a problem.
At the moment, I have a "toy" F90 program, that uses MPI with random numbers and a contrived character string for each processor. I want to have a data type that has the character and the double precision random number together.
I will use MPI_REDUCE to get the minimum value for the double precision values. I will have the data type for each process brought together to the root (rank = 0) via the MPI_GATHERV function.
My goal is to match up the minimum value from the random values to the data type. That would be the final answer. I have tried all sort of ideas up to this point, but to no avail. I end up with "forrtl: severe SIGSEGV, segmentation fault occurred".
Now I have looked at several of the other postings too. For instance, I cannot use the "use mpif.h" statement on this particular system.
But, at last, here is the code:
program fredtype
implicit none
include '/opt/apps/intel15/mvapich2/2.1/include/mpif.h'
integer rank,size,ierror,tag,status(MPI_STATUS_SIZE),i,np,irank
integer blocklen(2),type(2),num,rcount(4)
double precision :: x,aout
character(len=4) :: y
type, BIND(C) :: mytype
double precision :: x,aout,test
character :: y
end type mytype
type(mytype) :: foo,foobag(4)
integer(KIND=MPI_ADDRESS_KIND) :: disp(2),base
call MPI_INIT(ierror)
call MPI_COMM_SIZE(MPI_COMM_WORLD,size,ierror)
call MPI_COMM_RANK(MPI_COMM_WORLD,rank,ierror)
aout = 99999999999.99
call random_seed()
call random_number(x)
if(rank.eq.0)y="dogs"
if(rank.eq.1)y="cats"
if(rank.eq.2)y="tree"
if(rank.eq.3)y="woof"
print *,rank,x,y
call MPI_GET_ADDRESS(foo%x,disp(1),ierror)
call MPI_GET_ADDRESS(foo%y,disp(2),ierror)
base = disp(1)
call MPI_COMM_SIZE(MPI_COMM_WORLD,size,ierror)
call MPI_COMM_RANK(MPI_COMM_WORLD,rank,ierror)
aout = 99999999999.99
call random_seed()
call random_number(x)
if(rank.eq.0)y="dogs"
if(rank.eq.1)y="cats"
if(rank.eq.2)y="tree"
if(rank.eq.3)y="woof"
print *,rank,x,y
call MPI_GET_ADDRESS(foo%x,disp(1),ierror)
call MPI_GET_ADDRESS(foo%y,disp(2),ierror)
base = disp(1)
call MPI_COMM_SIZE(MPI_COMM_WORLD,size,ierror)
call MPI_COMM_RANK(MPI_COMM_WORLD,rank,ierror)
aout = 99999999999.99
call random_seed()
call random_number(x)
if(rank.eq.0)y="dogs"
if(rank.eq.1)y="cats"
if(rank.eq.2)y="tree"
if(rank.eq.3)y="woof"
print *,rank,x,y
call MPI_GET_ADDRESS(foo%x,disp(1),ierror)
call MPI_GET_ADDRESS(foo%y,disp(2),ierror)
base = disp(1)
disp(2) = disp(2) - base
blocklen(1) = 1
blocklen(2) = 1
type(1) = MPI_DOUBLE_PRECISION
type(2) = MPI_CHARACTER
call MPI_TYPE_CREATE_STRUCT(2,blocklen,disp,type,foo,ierror)
call MPI_TYPE_COMMIT(foo,ierror)
call MPI_REDUCE(x,aout,1,MPI_DOUBLE_PRECISION,MPI_MIN,0,MPI_COMM_WORLD,i\
error)
call MPI_GATHER(num,1,MPI_INT,rcount,1,MPI_INT,0,MPI_COMM_WORLD)
call MPI_GATHERV(foo,num,type,foobag,rcount,disp,type,0,MPI_COMM_WORLD)
if(rank.eq.0)then
print *,'fin ',aout
end if
end program fredtype
Thank you for any help.
Sincerely,
Erin
Your code is definitely too confusing for me to try to fully fix it. So let's just assume that you have your type mytype defined as follow:
type, bind(C) :: mytype
double precision :: x, aout, test
character(len=4) :: y
end type mytype
(Rk: I've add len=4 to the definition of y as it seemed to be missing from your original code. I might be wrong it that and if so, just adjust blocklen(2) in the subsequent code accordingly)
Now let's assume that you only want to transfer the x and y fields of your variables of type mytype. For this, you'll need to create an appropriated derived MPI type using first MPI_Type_create_struct() to define the basic types and their location into your structure, and then MPI_Type_create_resized() to define the true extent and lower bound of the type, including holes.
The tricky part is usually to evaluate what the lower bound and extent of your Fortran type is. Here, as you include into the fields that you transfer the first and last of them, and as you added bind(C), you can just use MPI_Type_get_extend() to get these informations. However, if you hadn't included x or y (which are first and last fields of the type) into the MPI data type, MPI_Type_get_extent() wouldn't have return what you would have needed. So I'll propose you an alternative (slightly more cumbersome) approach which will, I believe, always work:
integer :: ierror, typefoo, tmptypefoo
integer :: blocklen(2), types(2)
type(mytype) :: foobag(4)
integer(kind=MPI_ADDRESS_KIND) :: disp(2), lb, extent
call MPI_Get_address( foobag(1), lb, ierror )
call MPI_Get_address( foobag(1)%x, disp(1), ierror )
call MPI_Get_address( foobag(1)%y, disp(2), ierror )
call MPI_Get_address( foobag(2), extent, ierror )
disp(1) = MPI_Aint_diff( disp(1), lb )
disp(2) = MPI_Aint_diff( disp(2), lb )
extent = MPI_Aint_diff( extent, lb )
lb = 0
blocklen(1) = 1
blocklen(2) = 4
types(1) = MPI_DOUBLE_PRECISION
types(2) = MPI_CHARACTER
call MPI_Type_create_struct( 2, blocklen, disp, types, tmptypefoo, ierror )
call MPI_Type_create_resized( tmptypefoo, lb, extent, typefoo, ierror )
call MPI_Type_commit( typefoo, ierror )
So as you can see, lb serves as base address for the displacements into the structure, and the type extent is computed by using the relative addresses of two consecutive elements of an array of type mytype.
Then, we create an intermediary MPI data type tmptypefoo which only contains the information about the actual data we will transfer, and we extent it with information about the actual lower bound and extent of the Fortran type into typefoo. Finally, only this last one needs to be committed as only it will serve for data transfers.
The following returns 1, indicating a local x is created.
x = 1
bar() = (x = 2)
bar() # 2
x # 1
This returns 5, indicating both x refer to the global one.
x = 1
for i = 1:5
x = i
end
x # 5
A reference example: this time for loop fails to update the global.
x = 10
function foo(n)
for i = 1:n
x = i
end
1
end
foo(2), x # 1, 10
Update
The link from #matt-b is very useful. This is in fact the result of Soft Scope vs Hard Scope, see here. To wrap up, function scope used to work like loop scope, until there was a break change with the introduction of soft scope & hard scope. The documentation is not quite up to speed.
If you want to use global x in function scope you must declare it global
x = 10
function foo(n)
global x
for i = 1:n
x = i
end
1
end
foo(2), x # 2
As #colinfang have commented, in Julia function scope and loop scope are treat differently and I think the following sentence from documentation try to address this fact:
Julia uses lexical scoping, meaning that a function’s scope does not
inherit from its caller’s scope, but from the scope in which the
function was defined.
The link from #matt-b is very useful. This is in fact the result of Soft Scope vs Hard Scope, see here. To wrap up, function scope used to work like loop scope, until there was a break change with the introduction of soft scope & hard scope. The documentation is not quite up to speed.
I recently started learning Julia by coding a simple implementation of Self Organizing Maps. I want the size and dimensions of the map to be specified by the user, which means I can't really use for loops to work on the map arrays because I don't know in advance how many layers of loops I will need. So I absolutely need broadcasting and slicing functions that work on arrays of arbitrary dimensions.
Right now, I need to construct an array of indices of the map. Say my map is defined by an array of size mapsize = (5, 10, 15), I need to construct an array indices of size (3, 5, 10, 15) where indices[:, a, b, c] should return [a, b, c].
I come from a Python/NumPy background, in which the solution is already given by a specific "function", mgrid :
indices = numpy.mgrid[:5, :10, :15]
print indices.shape # gives (3, 5, 10, 15)
print indices[:, 1, 2, 3] gives [1, 2, 3]
I didn't expect Julia to have such a function on the get-go, so I turned to broadcasting. In NumPy, broadcasting is based on a set of rules that I find quite clear and logical. You can use arrays of different dimensions in the same expression as long as the sizes in each dimension match or one of it is 1 :
(5, 10, 15) broadcasts to (5, 10, 15)
(10, 1)
(5, 1, 15) also broadcasts to (5, 10, 15)
(1, 10, 1)
To help with this, you can also use numpy.newaxis or None to easily add new dimensions to your array :
array = numpy.zeros((5, 15))
array[:,None,:] has shape (5, 1, 15)
This helps broadcast arrays easily :
A = numpy.arange(5)
B = numpy.arange(10)
C = numpy.arange(15)
bA, bB, bC = numpy.broadcast_arrays(A[:,None,None], B[None,:,None], C[None,None,:])
bA.shape == bB.shape == bC.shape = (5, 10, 15)
Using this, creating the indices array is rather straightforward :
indices = numpy.array(numpy.broadcast_arrays(A[:,None,None], B[None,:,None], C[None,None,:]))
(indices == numpy.mgrid[:5,:10,:15]).all() returns True
The general case is of course a bit more complicated, but can be worked around using list comprehension and slices :
arrays = [ numpy.arange(i)[tuple([None if m!=n else slice(None) for m in range(len(mapsize))])] for n, i in enumerate(mapsize) ]
indices = numpy.array(numpy.broadcast_arrays(*arrays))
So back to Julia. I tried to apply the same kind of rationale and ended up achieving the equivalent of the arrays list of the code above. This ended up being rather simpler than the NumPy counterpart thanks to the compound expression syntax :
arrays = [ (idx = ones(Int, length(mapsize)); idx[n] = i;reshape([1:i], tuple(idx...))) for (n,i)=enumerate(mapsize) ]
Now I'm stuck here, as I don't really know how to apply the broadcasting to my list of generating arrays here... The broadcast[!] functions ask for a function f to apply, and I don't have any. I tried using a for loop to try forcing the broadcasting:
indices = Array(Int, tuple(unshift!([i for i=mapsize], length(mapsize))...))
for i=1:length(mapsize)
A[i] = arrays[i]
end
But this gives me an error : ERROR: convert has no method matching convert(::Type{Int64}, ::Array{Int64,3})
Am I doing this the right way? Did I overlook something important? Any help is appreciated.
If you're running julia 0.4, you can do this:
julia> function mgrid(mapsize)
T = typeof(CartesianIndex(mapsize))
indices = Array(T, mapsize)
for I in eachindex(indices)
indices[I] = I
end
indices
end
It would be even nicer if one could just say
indices = [I for I in CartesianRange(CartesianIndex(mapsize))]
I'll look into that :-).
Broadcasting in Julia has been modelled pretty much on broadcasting in NumPy, so you should hopefully find that it obeys more or less the same simple rules (not sure if the way to pad dimensions when not all inputs have the same number of dimensions is the same though, since Julia arrays are column-major).
A number of useful things like newaxis indexing and broadcast_arrays have not been implemented (yet) however. (I hope they will.) Also note that indexing works a bit differently in Julia compared to NumPy: when you leave off indices for trailing dimensions in NumPy, the remaining indices default to colons. In Julia they could be said to default to ones instead.
I'm not sure if you actually need a meshgrid function, most things that you would want to use it for could be done by using the original entries of your arrays array with broadcasting operations. The major reason that meshgrid is useful in matlab is because it is terrible at broadcasting.
But it is quite straightforward to accomplish what you want to do using the broadcast! function:
# assume mapsize is a vector with the desired shape, e.g. mapsize = [2,3,4]
N = length(mapsize)
# Your line to create arrays below, with an extra initial dimension on each array
arrays = [ (idx = ones(Int, N+1); idx[n+1] = i;reshape([1:i], tuple(idx...))) for (n,i) in enumerate(mapsize) ]
# Create indices and fill it one coordinate at a time
indices = zeros(Int, tuple(N, mapsize...))
for (i,arr) in enumerate(arrays)
dest = sub(indices, i, [Colon() for j=1:N]...)
broadcast!(identity, dest, arr)
end
I had to add an initial singleton dimension on the entries of arrays to line up with the axes of indices (newaxis had been useful here...).
Then I go through each coordinate, create a subarray (a view) on the relevant part of indices, and fill it. (Indexing will default to returning subarrays in Julia 0.4, but for now we have to use sub explicitly).
The call to broadcast! just evaluates the identity function identity(x)=x on the input arr=arrays[i], broadcasts to the shape of the output. There's no efficiency lost in using the identity function for this; broadcast! generates a specialized function based on the given function, number of arguments, and number of dimensions of the result.
I guess this is the same as the MATLAB meshgrid functionality. I've never really thought about the generalization to more than two dimensions, so its a bit harder to get my head around.
First, here is my completely general version, which is kinda crazy but I can't think of a better way to do it without generating code for common dimensions (e.g. 2, 3)
function numpy_mgridN(dims...)
X = Any[zeros(Int,dims...) for d in 1:length(dims)]
for d in 1:length(dims)
base_idx = Any[1:nd for nd in dims]
for i in 1:dims[d]
cur_idx = copy(base_idx)
cur_idx[d] = i
X[d][cur_idx...] = i
end
end
#show X
end
X = numpy_mgridN(3,4,5)
#show X[1][1,2,3] # 1
#show X[2][1,2,3] # 2
#show X[3][1,2,3] # 3
Now, what I mean by code generation is that, for the 2D case, you can simply do
function numpy_mgrid(dim1,dim2)
X = [i for i in 1:dim1, j in 1:dim2]
Y = [j for i in 1:dim1, j in 1:dim2]
return X,Y
end
and for the 3D case:
function numpy_mgrid(dim1,dim2,dim3)
X = [i for i in 1:dim1, j in 1:dim2, k in 1:dim3]
Y = [j for i in 1:dim1, j in 1:dim2, k in 1:dim3]
Z = [k for i in 1:dim1, j in 1:dim2, k in 1:dim3]
return X,Y,Z
end
Test with, e.g.
X,Y,Z=numpy_mgrid(3,4,5)
#show X
#show Y
#show Z
I guess mgrid shoves them all into one tensor, so you could do that like this
all = cat(4,X,Y,Z)
which is still slightly different:
julia> all[1,2,3,:]
1x1x1x3 Array{Int64,4}:
[:, :, 1, 1] =
1
[:, :, 1, 2] =
2
[:, :, 1, 3] =
3
julia> vec(all[1,2,3,:])
3-element Array{Int64,1}:
1
2
3