Julia - Iterating over combinations of keys in a dictionary - dictionary

Question

Is there a nifty way to iterate over combinations of keys in a dictionary?
my dictionary has values like:
[1] => [1,2], [2,3] => [15], [3] => [6,7,8], [4,9,11] => [3], ...
what I need to do is fetch all combinations of keys that are of length 1:n where n might be fx 3
So as in the example above, I would want to iterate over
[[1], [3], [2,3], [[1],[1,2]], [[3],[2,3]], [4,9,11]]
I know I could just collect the keys, but my dictionary is rather large and I am in the middle of redesigning the entire algorithm because it starts swapping insanely when n > 3, reducing efficiency terribly
tl;dr is there a way to create a combinatoric iterator from a dictionary without collect-ing the dictionary?

Answer 1

The following is a straight forward implementation, which tries to minimize a bit on going through the dictionary. Additionally it uses OrderedDict so holding key indices makes sense (since Dicts don't promise consistent key iteration each time and thus meaningful key indexing).
using Iterators
using DataStructures
od = OrderedDict([1] => [1,2], [2,3] => [15], [3] => [6,7,8], [4,9,11] => [3])
sv = map(length,keys(od)) # store length of keys for quicker calculations
maxmaxlen = sum(sv) # maximum total elements in good key
for maxlen=1:maxmaxlen # replace maxmaxlen with lower value if too slow
#show maxlen
gsets = Vector{Vector{Int}}() # hold good sets of key _indices_
for curlen=1:maxlen
foreach(x->push!(gsets,x),
(x for x in subsets(collect(1:n),curlen) if sum(sv[x])==maxlen))
end
# indmatrix is necessary to run through keys once in next loop
indmatrix = zeros(Bool,length(od),length(gsets))
for i=1:length(gsets) for e in gsets[i]
indmatrix[e,i] = true
end
end
# gkeys is the vector of vecotrs of keys i.e. what we wanted to calculate
gkeys = [Vector{Vector{Int}}() for i=1:length(gsets)]
for (i,k) in enumerate(keys(od))
for j=1:length(gsets)
if indmatrix[i,j]
push!(gkeys[j],k)
end
end
end
# do something with each set of good keys
foreach(x->println(x),gkeys)
end
Is this more efficient that what you currently have? It would also be better to put the code in a function or turn it into a Julia task which produces the next keys set each iteration.
--- UPDATE ---
Using the answer about iterators from tasks in https://stackoverflow.com/a/41074729/3580870
An improved iterator-ified version is:
function keysubsets(n,d)
Task() do
od = OrderedDict(d)
sv = map(length,keys(od)) # store length of keys for quicker calculations
maxmaxlen = sum(sv) # maximum total elements in good key
for maxlen=1:min(n,maxmaxlen) # replace maxmaxlen with lower value if too slow
gsets = Vector{Vector{Int}}() # hold good sets of key _indices_
for curlen=1:maxlen
foreach(x->push!(gsets,x),(x for x in subsets(collect(1:n),curlen) if sum(sv[x])==maxlen))
end
# indmatrix is necessary to run through keys once in next loop
indmatrix = zeros(Bool,length(od),length(gsets))
for i=1:length(gsets) for e in gsets[i]
indmatrix[e,i] = true
end
end
# gkeys is the vector of vecotrs of keys i.e. what we wanted to calculate
gkeys = [Vector{Vector{Int}}() for i=1:length(gsets)]
for (i,k) in enumerate(keys(od))
for j=1:length(gsets)
if indmatrix[i,j]
push!(gkeys[j],k)
end
end
end
# do something with each set of good keys
foreach(x->produce(x),gkeys)
end
end
end
Which now enables iterating over all keysubsets up to combined size 4 in this way (after running the code from the other StackOverflow answer):
julia> nt2 = NewTask(keysubsets(4,od))
julia> collect(nt2)
10-element Array{Array{Array{Int64,1},1},1}:
Array{Int64,1}[[1]]
Array{Int64,1}[[3]]
Array{Int64,1}[[2,3]]
Array{Int64,1}[[1],[3]]
Array{Int64,1}[[4,9,11]]
Array{Int64,1}[[1],[2,3]]
Array{Int64,1}[[2,3],[3]]
Array{Int64,1}[[1],[4,9,11]]
Array{Int64,1}[[3],[4,9,11]]
Array{Int64,1}[[1],[2,3],[3]]
(the definition of NewTask from the linked StackOverflow answer is necessary).