Creating a Map Reduce Function in Python from edge list - dictionary

I need some help creating a Map Reduce function in Python from an edge list.
Given the following list:
A,B
A,C
A,D
B,C
C,A
C,B
D,A
My code should follow the format below. My goal is to display a degree list along with the count of degrees.
map(key, value):
//key: document name; value: text of the document
for each word w in value:
emit(w, 1)
reduce(key, values):
//key: a word; value: an iterator over counts
result = 0
for each count v in values:
result += v
emit(key, result)
Loops have always been a struggle for me. Can someone point me in the right direction?
The output should be as follows:
Degree Count
1 2
2 1
3 1
The code should work for any data set similarly formatted
So, as I understand it, I need to count the number of different values each letter is paired with--this would be the "degree" (A, for example, is degree 3), and then total the pairs according to the degree--this would be the "count".
Could the key be the pair of values?
map(key, value):
//key: document name; value: text of the document
for each word w in value:
emit([x,y], 1)

I'm not sure if MapReduce is the best way to approach this problem, but I think the following makes sense.
First Map each Vertex-Edge pair to the (Vertex, 1). Then Reduce by summing the counts for each Vertex.
map(key, value):
//key: vertex; value: edge
emit(key, 1)
reduce(key, values):
//key: vertex; value: an iterator over counts
result = 0
for each count v in values:
result += v
emit(key, result)
This assumes that the input rows are unique.

Related

Differences in result in two similar functions: finding the key with maximun value

I am currently having an issue. Basically, I have 2 similar functions in terms of concept but the results do not align. These are the codes I learned from Bioinformatics I on Coursera.
The first code is simply creating a dictionary of occurrences of each k-mer pattern from a text (which is a long stretch of nucleotides). In this case, k is 5.
def FrequencyMap(text,k):
freq ={}
for i in range (0, len(text)-k+1):
freq[text[i:i+k]]=0
for j in range (0, len(text)-k+1):
if text[j:j+k] == text[i:i+k]:
freq[text[i:i+k]] +=1
return freq, max(freq)
The text and the result dictionary are kinda long, but the main point is when I call max(freq), it returns the key 'TTTTC', which has a value of 1.
Meanwhile, I wrote another code that is simply based on the previous code to generate the 5-mer patterns that have the max values (number of occurrences in the text).
def FrequentWords(text, k):
a = FrequencyMap(text, k)
m = max(a.values())
words = []
for i in a:
if a[i]==m:
words.append(i)
return words,m
And this code returns 'ACCTA', which has the value of 99, meaning it appears 99 times in the text. This makes total sense.
I used the same text and k (k=5) for both codes. I ran the codes on Jupyter Notebook. Why does the first one not return 'ACCTA'?
Thank you so much,
Here is the text, if anyone wants to try:
"ACCATCCCTAGGGCATACCTAAGTCTACCTAAAAGGCTACCTAATACCATACCTAATTACCTAACTACCTAAAATAAGTCTACCTAATACCTAATACCTAAAGTTACCTAACGTACCTAATACCTAATACCTAACCACTACCTAATCCGATTTACCTAACAACCGATCGAGTACCTAATCGATACCTAAATAACGGACAATATACCTAATTACCTAATACCTAATACCTAAGTGTACCTAAGACGTCTACCTAATTGTACCTAACTACCTAATTACCTAAGATTAATACCTAATACCTAATTTACCTAATACCTAACGTGGACTACCTAATACCTAACTTTTCCCCTACCTAATACCTAACTGTACCTAAATACCTAATACCTAAGCTACCTAAAGAACAACATTGTACGTGCGCCGTACCTAAATACCTAACAACTACCTAACTGATACCTAATAGTGATTACCTAACGCTTCTACCTAACTACCTAAGTACCTAACGCTACCTAACTACCTAATGTCCACAAAATACCTAATACCTAATAGCTACCTAATTGTGTACCTAAGTACCTAACCTACCTAATAATACCTAAAAATACCTAAGTACCTAACGTACCTAAATTTTACCTAATCTACCTAACGTACCTAATACCTAATTATACCTAATTACCTAATGGTTACCTAAGTTACCTAATATGCCACTACCTAACCTTACCTAAGACCTACCTAATAGGTACCTAACTGGGTACCTAAGGCAGTTTACCTAATTCAGGGCTACCTAATGTACCTAATACCTAAGTACCTAATACCTAATCCCATACCTAATATTTACCTAAGGGCACCGGTACCTAATACCTAATACCTAATACCTAAACCTTCGTACCTAAATACCTAATCTACCTAATGTACCTAAGGTACCTAATACCTAAGTCACTACCTAATACCTAATACCTAATGGGAGGAGCTTACCTAAGGTTACCTAATTACCTAAATACCTAATCGTTACCTAA"
Why does the first one not return 'ACCTA'?
Because max(freq) returns the maximum key of the dictionary. In this case the keys are strings (the k-mers), and strings are compared alphabetically. Hence the maximum one is the last string when the are sorted alphabetically.
If you want the first function to return the k-mer that occurs most often, you should change max(freq) to max(freq.items(), key=lambda key_value_pair: key_value_pair[1])[0]. Here, you are sorting the (kmer, count) pairs (that's the key_value_pair parameter of the lambda expression) based on the frequency and then selecting the kmer.

Encode numbers with letters with fixed lentgh?

I have two unique numbers, 100000 - 999999 (fixed 6 chars length [0-9]), second
1000000 - 9999999 (fixed 7 char length [0-9]). How can i encode/decode this numbers (they need to remain separate after decoding), using only uppercase letters [A-Z] and [0-9] digits and have a fixed length of 8 chars in total?
Example:
input -> num_1: 242404, num_2 : 1002000
encode -> AX3B O3XZ
decode -> 2424041002000
Is there any algorithm for this type of problem?
This is just a simple mapping from one set of values to another set of values. The procedure is always the same:
List all possible input and output values.
Find the index of the input.
Return the value of the output list at that index.
Note that it's often not necessary to make an actual list (i.e. loading all values into some data structure). You can typically compute the value for any index on-demand. This case is no different.
Imagine a list of all possible input pairs:
0 100'000, 1'000'000
1 100'000, 1'000'001
2 100'000, 1'000'002
...
K 100'000, 9'999'999
K+1 100'001, 1'000'000
K+2 100'001, 1'000'001
...
N-1 999'999, 9'999'998
N 999'999, 9'999'999
For any given pair (a, b), you can compute its index i in this list like so:
// Make a and b zero-based
a -= 100'000
b -= 1'000'000
i = a*1'000'000 + b
Convert i to base 36 (A-Z and 0-9 gives you 36 symbols), pad on the left with zeros as necessary1, and insert a space after the fourth digit.
encoded = addSpace(zeroPad(base36(i)))
To get back to the input pair:
Convert the 8-character base 36 string to base 10 (this is the index into the list, remember), then derive a and b from the index.
i = base10(removeSpace(encoded))
a = i/1'000'000 + 100'000 // integer divison (i.e. ignore remainder)
b = i%1'000'000 + 1'000'000
Here is an implementation in Go: https://play.golang.org/p/KQu9Hcoz5UH
1 If you don't like the idea of zero padding you can also offset i at this point. The target set of values is plenty big enough, you need only about 32% of all base 36 numbers with eight digits or less.

Generate Unique Combinations of Integers

I am looking for help with pseudo code (unless you are a user of Game Maker 8.0 by Mark Overmars and know the GML equivalent of what I need) for how to generate a list / array of unique combinations of a set of X number of integers which size is variable. It can be 1-5 or 1-1000.
For example:
IntegerList{1,2,3,4}
1,2
1,3
1,4
2,3
2,4
3,4
I feel like the math behind this is simple I just cant seem to wrap my head around it after checking multiple sources on how to do it in languages such as C++ and Java. Thanks everyone.
As there are not many details in the question, I assume:
Your input is a natural number n and the resulting array contains all natural numbers from 1 to n.
The expected output given by the combinations above, resembles a symmetric relation, i. e. in your case [1, 2] is considered the same as [2, 1].
Combinations [x, x] are excluded.
There are only combinations with 2 elements.
There is no List<> datatype or dynamic array, so the array length has to be known before creating the array.
The number of elements in your result is therefore the binomial coefficient m = n over 2 = n! / (2! * (n - 2)!) (which is 4! / (2! * (4 - 2)!) = 24 / 4 = 6 in your example) with ! being the factorial.
First, initializing the array with the first n natural numbers should be quite easy using the array element index. However, the index is a property of the array elements, so you don't need to initialize them in the first place.
You need 2 nested loops processing the array. The outer loop ranges i from 1 to n - 1, the inner loop ranges j from 2 to n. If your indexes start from 0 instead of 1, you have to take this into consideration for the loop limits. Now, you only need to fill your target array with the combinations [i, j]. To find the correct index in your target array, you should use a third counter variable, initialized with the first index and incremented at the end of the inner loop.
I agree, the math behind is not that hard and I think this explanation should suffice to develop the corresponding code yourself.

Generate sets from given overlap matrix

Note: I edited the original question to explain more precisely.
While I was doing a simulation for my new method, I needed to generate a special type of dataset consists of multiple subset. The problem is that there is some "shared" variables across the subsets, and the number of shared variable is called "overlap" here. Since the distribution of overlap proportion is given, I need to generate an appropriate list of variables and their overlap follows the given distribution. But I have failed to implement such algorithm...
I am not sure whether there is a specific algorithm for this kind of question,
but I have failed to find such thing after a long search.
I prefer R solution, but anything others also will be very appreciated. Please help me to solve this problem! Thank you so much in advance!
The below is a standardized explanation for my problem. I tried to explain as general as possible I can, but please give me any suggestion if it is not sufficient.
Purpose: Generate n sets from given overlap matrix of elements. Each set contains k elements.
Input: There is a n*n matrix whose (i,j)th cell value represents a number of overlapped elements from (i)th set to (j)th set.
Output: A list of k element identifiers (whatever can be used such as number) for n sets.
Assumption: The number of elements for each set is k, and it is same across all n sets. Hence, the input matrix is symmetric.
Example (assumes k=3 and n=3)
Input
3 1 0
1 3 1
0 1 3
Output
Set 1: A B C
Set 2: A D E
Set 3: D F G
In the above example input, (1,2)th and (2,1)th cells are 1 because set 1 and 2 share "A" element and vice versa, and diagonal cells are 3(=k) because each set shares all elements with itself.
I would repeat the following process until I had accounted for all the matrix entries:
1) Treat the matrix as the adjacency matrix of a graph, and find the largest clique in it. That is, find the largest possible set S of indexes such that for all i, j in set S M(i,j) > 0
2) Create an item that is in all of the sets which correspond to the indexes in S - in fact, if the minimum value of M(i,j) = v, create v such items.
3) subtract v from M(i,j) for all i, j in set S, accounting for the counts generated by the items you have just created.

Completing a list of possible binary sequences give a binary sequence with gaps

So, I am working on a program in Scilab which solves a binary puzzle. I have come across a problem however. Can anyone explain to me the logic behind solving a binary sequence with gaps (like [1 0 -1 0 -1 1 -1] where -1 means an empty cell. I want all possible solutions of a given sequence. So far I have:
function P = mogelijkeCombos(V)
for i=1:size(V,1)
if(V(i) == -1)
aantalleeg = aantalleeg +1
end
end
for i=1:2^aantalleeg
//creating combos here
end
endfunction
sorry that some words are in dutch
aantalleeg means amountempty by which I mean the amount of empty cells
I hope I gave you guys enough info. I don't need any code written, I'd just like ideas of how I can make every possible rendition as I am completely stuck atm.
BTW this is a school assignment, but the assignment is way bigger than this and it's just a tiny part I need some ideas on
ty in advance
Short answer
You could create the combos by extending your code and create all possible binary words of the length "amountempty" and replacing them bit-for-bit in the empty cells of V.
Step-by-step description
Find all the empty cell positions
Count the number of positions you've found (which equals the number of empty cells)
Create all possible binary numbers with the length of your count
For each binary number you generate, place the bits in the empty cells
print out / store the possible sequence with the filled in bits
Example
Find all the empty cell positions
You could for example check from left-to-right starting at 1 and if a cell is empty add the position to your position list.
V = [1 0 -1 0 -1 1 -1]
^ ^ ^
| | |
1 2 3 4 5 6 7
// result
positions = [3 5 7]
Count the number of positions you've found
//result
amountempty = 3;
Create all possible binary numbers with the length amountempty
You could create all possible numbers or words with the dec2bin function in SciLab. The number of possible words is easy to determine because you know how much separate values can be represented by a word of amountempty bits long.
// Create the binary word of amountEmpty bits long
binaryWord = dec2bin( i, amountEmpty );
The binaryWord generated will be a string, you will have to split it into separate bits and convert it to numbers.
For each binaryWord you generate
Now create a possible solution by starting with the original V and fill in every empty cell at the position from your position list with a bit from binaryWordPerBit
possibleSequence = V;
for j=1:amountEmpty
possibleSequence( positions(j) ) = binaryWordPerBit(j);
end
I wish you "veel succes met je opdracht"

Resources