I am wanting to have a signal / function that outputs the moving average of x periods. I have come up with two approaches, both of which work, but I believe the second is more properly reactive. Does anyone know of a better method?
Method 1
#time (
x = Signal(100);
col = foldp(Int[], x) do acc, val
push!(acc, val)
end;
for i in rand(90:110, 100)
push!(x, i)
end;
)
This code executes for me in 0.102663 seconds and can easily yield an average over the last 100 signals: mean(value(col))
Method 2
#time (
a = Signal(100);
sar = Vector{Signal}(101);
sar[1] = a;
for i in 1:100
sar[i+1] = previous(sar[i])
end;
for i in rand(90:110, 100)
push!(a, i)
end;
)
This code executes for me in 0.034911 seconds and can also easily yeild an average of the last 100 signals:
sarval = map(value, sar)
mean(sarval[2:end])`
Neither method above directly provides an output signal; here is Method 1 applied to create a moving average of specified length as a continuing signal:
Method 1 Applied
x = Signal(initial_value)
col = foldp(Float64[], x) do acc, elem
push!(acc, elem)
end
macro moving_average(per, collec)
quote
map($collec) do arr
length(arr) < $per ? mean(arr) : mean(arr[(end-$per+1):end])
end
end
end
ma_period = #moving_average(period_length, col)
This code uses the first method to generate a signal of arrays of all past signals, the length of which increases linearly with the number of signals. For signals that update thousands of times, this seems unwieldy (at best). This does not seem ideal, and I am looking for some ideas for a better approach. Not sure how to take Method 2 and turn it into a signal output, and not sure if it is the best approach I could take.
Also related question: there is a nice built in function previous(signal) that provides a signal lagging 1 update behind signal. Is there a way to specify a signsl that legs a specified, arbitrary number of updates behind signal?
Related
Say I have some int x. I want to check if all MPI ranks get the same value for x. What's a good way to achieve this using MPI collectives?
The simplest I could think of is, broadcast rank0's x, do the comparison, and allreduce-logical-and the comparison result. This requires two collective operations.
...
x = ...
x_bcast = comm.bcast(x, root=0)
all_equal = comm.allreduce(x==x_bcast, op=MPI.LAND)
if not all_equal:
raise Exception()
...
Is there a better way to do this?
UPDATE:
From the OpenMPI user list, I received the following response. And I think it's quite a nifty trick!
A pattern I have seen in several places is to allreduce the pair p =
{-x,x} with MPI_MIN or MPI_MAX. If in the resulting pair p[0] == -p[1],
then everyone has the same value. If not, at least one rank had a
different value. Example:
bool is_same(int x) {
int p[2];
p[0] = -x;
p[1] = x;
MPI_Allreduce(MPI_IN_PLACE, p, 2, MPI_INT, MPI_MIN, MPI_COMM_WORLD);
return (p[0] == -p[1]);
}
Solutions based on logical operators assume that you can convert between integers and logicals without any data loss. I think that's dangerous. You could do a bitwise AND where you make sure you use all the bytes of your int/real/whatever.
You could do two reductions: one max and one min, and see if they give the same result.
You could also write your own reduction operator: operate on two ints, and do a max on the first, min on the second. Then test if the two are the same.
I have been given some initialisation code and told to write a function knight(n,y,x) to print a solution to the problem.
This initialisation code is
size = 8
board = []
for y in range(size) :
board = board + [[-1]*size]
moves= [[1,2],[1,-2],[-1,2],[-1,-2],
[2,1],[2,-1],[-2,1],[-2,-1]]
And my code so far, which just seems to run forever is
def knight(n,y,x):
if n == 0:
board[y][x] = 0
n = 1
elif n == (size**2) :
return board
for j,i in moves:
y1 = y+j
x1 = x+i
if y1 < size and x1 < size and y1>=0 and x1>=0 and board[y1][x1] == -1:
board[y1][x1] = n
knight(n+1,y1,x1)
board[y1][x1] = -1
return
But I can't figure why it is a problem. I have looked through a few of the existing questions on here but they use multiple functions where we have been told to just use the one. Could anybody please help?
I have found a problem in your code, which fixes it for everything up to 7x7 boards. I assume it works on 8x8, it just takes exponentially longer. You might have to implement some algorithm improvements to get it to be quicker. Anyway the error I found was:
You ARE finding the board, but you only return it one frame back. You need to set it up so that, if a board is found, the code returns it all the way back.
Try replace the following line:
knight(n+1,y1,x1)
with the following:
sol = knight(n+1,y1,x1)
if sol is not None:
return sol
I'll have a go at trying to get an answer from the 8x8, and update this if I can help any more.
Which python code written in the client side of #dask is really added to the task graph?
In this script for example, I am reading an hdf5 dataset of 4 dim, using a loop for the fourth dimension.
I calculate the sum for each dim called here g for generation and subtract the result of this generation and the one before it.
Then i am calling the deriv.visualize() to see how it generates the graph.
alive = []
derivate = []
board = []
deriv = 0
rest_1 = 0
hf5 = h5py.File('Datata.h5', 'r')
hds5 = hf5.get('dataset')
list(hf5.keys())
last_gen = hds5.attrs.get('last_gen')
for g in range(0,generations):
board = hds5[g]
arr = da.asarray(board, chunks=(4,5,4))
res = arr.sum()
if g!=0 :
deriv = res - rest_1
rest_1 = res
deriv.visualize()
Here is the graph i am getting
Here without calling .compute() the subtract operator is added to the dask graph apparently, how do we explain this ?
If i add a .compute() in the "res = arr.sum().compute()" and keep the rest as it is, where the subtraction will be executed ? in the client side or in one of the workers ?
An other question which is more general, if i wanna keep the partial sums in the workers, and perform the subtraction (on partial sum of the current and last generation) in the workers, is there a way to say that i want theses operations to be performed on the same chunks over different generations? (for example the worker 0 will operate always on the 3 first rows of each generation, like in mpi even if it's not the same thing at all)?
Dask does not look at your Python code, and so can not see anything other than what you give to it. In this case it is these two lines:
arr = da.asarray(x, chunks=(4,5,4))
res = arr.sum().compute()
I'm a "write Fortran in all languages" kind of person trying to learn modern programming practices. I have a one dimensional function ft(lx)=HT(x,f(x),lx), where x, and f(x) are one dimensional arrays of size nx, and lx is the size of output array ft. I want to apply HT on a multidimensional array f(x,y,z).
Basically I want to apply HT on all three dimensions to go from f(x,y,z) defined on (nx,ny,nz) dimensional grid, to ft(lx,ly,lz) defined on (lx,ly,lz) dimensional grid:
ft(lx,y,z) = HT(x,f(x,y,z) ,lx)
ft(lx,ly,z) = HT(y,ft(lx,y,z) ,ly)
ft(lx,ly,lz) = HT(z,ft(lx,ly,z),lz)
In f95 style I would tend to write something like:
FTx=zeros((lx,ny,nz))
for k=1:nz
for j=1:ny
FTx[:,j,k]=HT(x,f[:,j,k],lx)
end
end
FTxy=zeros((lx,ly,nz))
for k=1:nz
for i=1:lx
FTxy[i,:,k]=HT(y,FTx[i,:,k],ly)
end
end
FTxyz=zeros((lx,ly,lz))
for j=1:ly
for i=1:lx
FTxyz[i,j,:]=HT(z,FTxy[i,j,:],lz)
end
end
I know idiomatic Julia would require using something like mapslices. I was not able to understand how to go about doing this from the mapslices documentation.
So my question is: what would be the idiomatic Julia code, along with proper type declarations, equivalent to the Fortran style version?
A follow up sub-question would be: Is it possible to write a function
FT = HTnD((Tuple of x,y,z etc.),f(x,y,z), (Tuple of lx,ly,lz etc.))
that works with arbitrary dimensions? I.e. it would automatically adjust computation for 1,2,3 dimensions based on the sizes of input tuples and function?
I have a piece of code here which is fairly close to what you want. The key tool is Base.Cartesian.#nexprs which you can read up on in the linked documentation.
The three essential lines in my code are Lines 30 to 32. Here is a verbal description of what they do.
Line 30: reshape an n1 x n2 x ... nN-sized array C_{k-1} into an n1 x prod(n2,...,nN) matrix tmp_k.
Line 31: Apply the function B[k] to each column of tmp_k. In my code, there are some indirections here since I want to allow for B[k] to be a matrix or a function, but the basic idea is as described above. This is the part where you would want to bring in your HT function.
Line 32: Reshape tmp_k back into an N-dimensional array and circularly permute the dimensions such that the second dimension of tmp_k ends up as the first dimension of C_k. This makes sure that the next iteration of the "loop" implied by #nexprs operates on the second dimension of the original array, and so on.
As you can see, my code avoids forming slices along arbitrary dimensions by permuting such that we only ever need to slice along the first dimension. This makes programming much easier, and it can also have some performance benefits. For example, computing the matrix-vector products B * C[i1,:,i3] for all i1,i3can be done easily and very efficiently by moving the second dimension of C into the first position of tmp and using gemm to compute B * tmp. Doing the same efficiently without the permutation would be much harder.
Following #gTcV's code, your function would look like:
using Base.Cartesian
ht(x,F,d) = mapslices(f -> HT(x, f, d), F, dims = 1)
#generated function HTnD(
xx::NTuple{N,Any},
F::AbstractArray{<:Any,N},
newdims::NTuple{N,Int}
) where {N}
quote
F_0 = F
Base.Cartesian.#nexprs $N k->begin
tmp_k = reshape(F_{k-1},(size(F_{k-1},1),prod(Base.tail(size(F_{k-1})))))
tmp_k = ht(xx[k], tmp_k, newdims[k])
F_k = Array(reshape(permutedims(tmp_k),(Base.tail(size(F_{k-1}))...,size(tmp_k,1))))
# https://github.com/JuliaLang/julia/issues/30988
end
return $(Symbol("F_",N))
end
end
A simpler version, which shows the usage of mapslices would look like this
function simpleHTnD(
xx::NTuple{N,Any},
F::AbstractArray{<:Any,N},
newdims::NTuple{N,Int}
) where {N}
for k = 1:N
F = mapslices(f -> HT(xx[k], f, newdims[k]), F, dims = k)
end
return F
end
you could even use foldl if you are a friend of one-liners ;-)
fold_HTnD(xx, F, newdims) = foldl((F, k) -> mapslices(f -> HT(xx[k], f, newdims[k]), F, dims = k), 1:length(xx), init = F)
What's happening, folks.
So, I've done a fair amount of research on merge sort, and in spite of getting the "gist" of it, I am still baffled by how one is supposed to store the subarrays in order to merge them back together—in other words, save them somewhere so that they would "know" each other, as you would otherwise—in classic recursive fashion—have all these independent function calls returning data that I would assume would go out of scope.
Here's what I first thought: create a new array named "subs" to store the subarrays in upon each division (I also considered using a closure to do this and would like to know whether this is advisable). But, as you proceed to the next division, what are you gonna do—replace each element in subs with its subarrays? Then, you would be facing more costly work, especially once you consider how you're gonna move things around in subs in order to ensure that each subarray has its own index.
Heh—I have a bad feeling that this might be a far cry from what's actually supposed to be done. I understand that this algorithm is a classic example of the divide-and-conquer approach, but it's just strange to me that one couldn't just cut to the chase by splitting the array into all of its elements right off the bat (after all, that's the base case, and what would be wrong with throwing in a greedy approach to solving the problem?).
Thanks!
EDIT:
Alright, so I figured it out.
To sum it up: I used indices to track where to place elements (and obviate the need for built-in list functions that may slow down runtime).
By using nested functions and a (hidden) pointer to the new array, I kept data in scope. An auxiliary array buffers data from the subarrays.
In retrospect, what I originally had in mind vaguely resembled insertion sort was, in fact, bottom-up merge sort. Having previously questioned the efficiency and purpose of top-down merge sort, I now understand that by breaking down the problem, it expedites comparisons and swaps (especially when operating on larger lists, which insertion sort would prove to be less efficient in sorting). I did not attempt to implement my initial idea because I did not have a clear enough picture of recursion and how data is passed.
#!/bin/python
import sys
def merge_sort(arr):
def merge(*indices): # indices = first, last, and pivot indices, respectively
head, tail = indices[0], indices[1]
pivot = indices[2]
i = head
j = pivot+1
k = 0
while (i <= pivot and j <= tail):
if new[i] <= new[j]:
aux[k] = new[i]
i += 1
k += 1
else:
aux[k] = new[j]
j += 1
k += 1
while (i <= pivot):
aux[k] = new[i]
i += 1
k += 1
while (j <= tail):
aux[k] = new[j]
j += 1
k += 1
for x in xrange(head, tail+1):
new[x] = aux[x-head]
# end merge
def split(a, *indices): # indices = first and last indices, respectively
head, tail = indices[0], indices[1]
pivot = (head+tail) / 2
if head < tail:
l_sub = a[head:pivot+1]
r_sub = a[pivot+1:tail+1]
split(l_sub, head, pivot)
split(r_sub, pivot+1, tail)
merge(head, tail, pivot)
# end split
new = arr
aux = list(new)
tail = len(new)-1
split(new, 0, tail)
return new
# end merge_sort
if __name__ == "__main__":
loops = int(raw_input().strip())
for _ in xrange(loops):
arr = map(int, raw_input().strip().split(' '))
result = merge_sort(arr)
print result