So I was playing with the newly standardized unordered_map from the STL. The code I have is kinda like this, I just create an unordered_map, fill it up, and print it out:
unordered_map<int,string> m1;
m1[5]="lamb";
m1[2]="had";
m1[3]="a";
m1[1]="mary";
m1[4]="little";
m1[7]="fleece";
m1[6]="whose";
m1[10]="fleecey";
m1[8]="was";
m1[9]="all";
for(unordered_map<int,string>::const_iterator i = m1.begin(); i != m1.end(); ++i)
cout<<i->first<<" "<<i->second<<endl;
However, the output I get is ordered thusly:
1 mary
2 had
3 a
4 little
5 lamb
6 whose
7 fleece
8 was
9 all
10 fleecey
But I don't want to pay the price to have my map ordered! That is why I am using an unordered_map... What is going on here?
additional note: I am using gcc version 4.3.4 20090804 (release) 1 (GCC) and am compiling like this g++ -std=c++0X maptest.cpp
"Unordered" doesn't mean it will store the items randomly or maintain the order you put them in the map. It just means you can't rely on any particular ordering. You don't pay a price for ordering, quite the contrary - the implementation isn't explicitly ordering the items, it's a hashmap and stores its elements in whatever way it pleases, which usually is a pretty performant way. It just so happens that the the hashing algorithm and other internal workings of the map, when using exactly these keys and this number and order of operations on the map, end up storing the items in a order that looks ordered. Strings, for example, may lead to an apparently randomized layout.
On a side note, this is probably caused by the map using a hash that maps (at least some) integers to itself and using the lower bits (as many as the map size mandates) of the hash the to determine the index for the underlying array (for instance, CPython does this - with some very clever additions to handle collisions relatively simply and efficiently; for the same reason the hashes of CPython strings and tuples are very predictable).
For your amusement, here's the output from libc++, which also has an identity function for std::hash<int>.
9 all
8 was
10 fleecey
6 whose
7 fleece
4 little
1 mary
3 a
2 had
5 lamb
There are several ways to implement a hash container, each with its own tradeoffs.
Related
APL is great for array type problems but I'm curious as to how best work with graphs in APL. I'm playing around with leet questions, for example question 662. Maximum Width of Binary Tree, the exercise works with Node objects with a value/left/right pointer style, however the test-case uses a basic array like [1,3,null,5,3]. The notation is compressed; uncompressed would be [[1], [3,null], [5,3,null,null]]. Reading layer-by-layer give [[1], [3], [5,3]] (so 2 is the widest layer).
Another example,
[5,4,7,3,null,2,null,-1,null,9] gives the answer 2
So I'm not sure the idiomatic way to work with trees. Do I use classes? Or are arrays best? In either case how do I convert the input?
I came up with a couple of solutions, but both feel inelegant. (Apologies for lack of comments)
convert←{
prev ← {(-⌈2÷⍨≢⍵)↑⍵}
nxt←{
⍵≡⍬:⍺
m←2/×prev ⍺
cnt←+/m
(⍺,(m\cnt↑⍵))nxt(cnt↓⍵)
}
(1↑⍵)nxt(1↓⍵)
}
Alternatively,
convert ← {
total←(+/×⍵)
nxt←{
double←×1,2↓2/0,⍵
(((+/double)↑⍺)#⊢)double
}
⍵ nxt⍣{(+/×⍺)=total}1
}
Both solutions are limited in they assume that 0 is null.
Once I've decompressed the input it's simply just a matter of stratifying by it's order
⌈/(1+⌈/-⌊/)∘⍸¨×nodes⊆⍨⍸2*¯1+⍳⌈2⍟≢nodes
In Python though I could use other methods to traverse i.e. keep track of the left/right-most node on a per-depth basis.
NOTE: This may be two questions, one to decompress and the other how to traverse graphs in general, but one depends on the other
Any ideas?
The work of Co-dfns compiler has given lots of insights on working tree/graph like data structures with APL.
Thesis: A Data Parallel Compiler Hosted on the GPU
GitHub repo: github.com/Co-dfns/Co-dfns (Many related goodies in project README file)
However the thesis is quite lengthy so for this particular exercise I would give a brief explanation on how to approach it.
the exercise works with Node objects with a value/left/right pointer style, however the test-case uses a basic array like [1,3,null,5,3].
Do we really actually build the tree with Node type objects to get an answer to the question? You can write the solution in something like Python and translate to APL, but that would be, losing the whole point of writing it in APL...
Notice the input is already an array! It is a bfs traverse of the binary tree. (The co-dfns compiler uses dfs traverse order, though)
so, actually what we need to do is just built a matrix like below for the input like [1,3,2,5,3,null,9] (⍬ is a placeholder value for for null):
1 ⍬ ⍬ ⍬ ⍝ level 0
3 2 ⍬ ⍬ ⍝ level 1
5 3 ⍬ 9 ⍝ level 2
For this problem we don't need to know which node's parent is which.
We can even do something like, by abusing the fact that input has no negative value (even the number could be negative, actually we only care about if it is null), and change ⍬ to ¯1 or 0 and make it easier to compute the answer.
So the problem has became: "compute the matrix representation of the tree as variable tree from the input array, then calculate the width of each level by +/0<tree, then the output is just 2*level (notice the first level is level-0)" This is using wrong definition for the width. I'll show how to correct it below
And it is actually very easy to do the conversion from input to matrix, hint: ↑.
1 (3 2) 5
┌─┬───┬─┐
│1│3 2│5│
└─┴───┴─┘
↑1 (3 2) 5
1 0
3 2
5 0
Thanks for pointing out that my original solution has problem on constructing the tree matrix.
This is the corrected method for constructing the tree. To distinguish from 0 for null and the padding, I add one to the input array so 2 is for non-null and 1 is for null.
buildmatrix←{
⎕IO←0
in←1+(⊂⊂'null')(≢⍤1 0)⎕JSON ⍵
⍝ Build the matrix
loop←{
(n acc)←⍺
0=≢⍵:acc
cur←n↑⍵
(2×+/2=cur)(acc,⊂cur)∇ n↓⍵
}
↑1 ⍬ loop in
}
However since the definition for width here is:
The width of one level is defined as the length between the end-nodes (the leftmost and rightmost non-null nodes), where the null nodes between the end-nodes are also counted into the length calculation.
We can just compute the width while attempting to reconstructing the tree (compute each level's width using \ and / with patterns from previous level):
If last level is 1011 and next level is 100010
1 0 1 1
1 0 0 0 0 0 1 0
(2/1 0 1 1)\1 0 0 0 1 0
1 0 0 0 0 0 1 0
So the it isn't needed to construct the complete matrix, and the answer for the exercise is just:
width←{
⎕IO←0
in←(⊂⊂'null')(≢⍤1 0)⎕JSON ⍵
⍝ strip leading trailing zero
strip←(⌽⍳∘1↓⊢)⍣2
⍝ Build the matrix
loop←{
(prev mw)←⍺
0=≢⍵:mw
cur←⍵↑⍨n←2×+/prev
((⊢,⍥⊂mw⌈≢)strip cur\⍨2/prev)∇ n↓⍵
}
(,1)1 loop 1↓in
}
width '[1,null,2,3,null,4,5,6]'
2
And the interesting fact is, you can probably do the same in other non-array based languages like Haskell. So instead of translating existing algorithms between similar looking languages, by thinking in the APL way you find new algorithms for problems!
What is the maximum number of elements that can be stored in a Map in GO? If I need to access data from Map frequently, is it a good idea to keep on adding items to Map and retrieving from it, in a long running program?
There is no theoretical limit to the number of elements in a map except the maximum value of the map-length type which is int. The max value of int depends on the target architecture you compile to, it may be 1 << 31 - 1 = 2147483647 in case of 32 bit, and 1 << 63 - 1 = 9223372036854775807 in case of 64 bit.
Note that as an implementation restriction you may not be able to add exactly max-int elements, but the order of magnitude will be the same.
Since the builtin map type uses a hashmap implementation, access time complexity is usually O(1), so it is perfectly fine to add many elements to a map, you can still access elements very fast. Note that however adding many elements will cause a rehashing and rebuilding the internals, which will require some additional calculations - which may happen occasionally when adding new keys to the map.
If you can "guess" or estimate the size of your map, you can create your map with a big capacity to avoid rehashing. E.g. you can create a map with space for a million elements like this:
m := make(map[string]int, 1e6)
"A maximum number"? Practically no.
"A good idea"? Measure, there cannot be a general answer.
So I'm not sure if this question belongs here or maybe Math overflow. In any case, my question is about information theory.
Let's say I have a 16 bit word. There are 65,536 unique configurations of 1's and 0's in that number. What each one of those configurations represents is unimportant as depending on your notation (2's complement vs signed magnitude etc.) the same configuration can mean different things.
What I'm wondering is are there any techniques to store more information than that in a 16 bit word?
My original ideas were like odd/even parity or something but then I realized that's already determined by the configuration... i.e. there is no extra information encoded in that. I'm beginning to wonder if no such thing exists.
EDIT For example, let's say some magical computer (thinking quantum or something here) could understand 0,1,a. Then obviously we have 3^16 configurations and can now store more than the numbers [0 - 65,536]. Are there any other properties of a 16 bit word that you can mess with in order to encode extra information in your bit stream?
EDIT2 I am really struggling to put this into words. Right now when I look at a 16 bit word in the computer, the property which conveys information to me the relative ordering of individual 1's and 0's. Is there another property or way of looking at a 16 bit word which would allow more than 2^16 unique "configurations"? (Note it would no longer be a configuration, but 2^16 xxxx's where xxxx is a noun describing an instance of that property). The only thing I can really think of is something like if we looked at the number of 1 to 0 transitions or something rather than whether each bit was actually a 1 or 0? Now transitions does not yield more than 2^16 combinations because it is ultimately solely dependent on the configuration of 1's and 0's. I'm looking for properties that would derive from the configuration of 1's and 0's AND something else thus resulting in MORE than 2^16. Does anyone even know what this would be called if it did exist?
EDIT3 Ok I got it. My question boils down to this: How do we prove that the configuration of 1's and 0's in a word completely defines it? I.E. How do we prove that you need no other information besides the bitmap to show equality between two 16 bit words?
FINAL EDIT
I have an example... If instead of looking at the presence of 1's and 0's we look at transition between bits we can store 2^16 alphabet characters. If the bit to left is the same, treat it as a 1, if it transitions, treat it as a 0. Using the 16 bit word as a circularly linked list type structure where each link represent 0/1 we basically for a 16 bit word out of the transition between bits. That is an exact example of what I was looking for but that results in 2^16, nothing better. I am convinced that you cannot do better and am marking the correct answer =(
The amount of information in a particular configuration of 16 0/1s is determined by the probability of this configuration (this is called self-information). This can be bigger than 16 bits if the configuration is less likely than 1/(2^16), but that means that some other configurations are more likely than 1/(2^16) and so will contain less information than 16 bits.
To take into account all the possible configurations, you have to use the expected value of self-information (called entropy) of individual configurations. This value will reach its maximum when the probabilities of all configurations are equal (that is 1/(2^16)) and then it will be exactly 16 bits.
So the answer is no, you cannot store more than 16 bits of information in 16 0/1s.
See
http://en.wikipedia.org/wiki/Information_theory
http://en.wikipedia.org/wiki/Self-information
EDIT It is important to realize that bit does not stand for 0 or 1, but it is a unit of information, that is -log_2 P(w) where P(w) is the probability of a particular configuration.
You cannot store more than 2 states in one digit of a semiconductor device. You answered it yourself. The only way more information can be fitted into 16 digits is if each digit were to have many possible values.
Is it a bad idea to use QSet to keep track of a very large set of fairly large strings? Each string is 54 characters (108 bytes). The set may contain thousands of entries (I'm not sure on the exact number yet). The QSet will only be used for insertion and membership query.
If it is a bad idea, I'm definitely open to suggestions. My 54 character strings are composed of only 6 different characters (e.g. "AAAAAAAAABBBBBBBBBCCCCCCCCCDDDDDDDDDEEEEEEEEEFFFFFFFFF"). This seems like a good candidate for compression, perhaps? Any other suggestions are welcome.
Realize that by using a built-in set, you're going to have some path-level compression based on the nature of your data. Of course, this depends on the container's implementation.
Look at some information on radix trees, digital search trees, red-black trees, etc. You'll see that you don't need to store each and every string, but rather the patterns. For instance, let's simplify your problem: we have only 3 characters that can appear an maximum of 2 times each, and each string is 6 characters long. Three possible strings are:
AABBCC, AABCBC, and AACBCB
With these examples, we could get away with using a maximum of 6 + 3 + 4 = 13 nodes instead of a full 18 nodes. not substantial, but I don't know what you're doing either. As with any type of compression, the more your prefix patterns are reused, the more compression you have.
Edit:
The numbers 13 and 18 come from the path-level compression. For instance, in straight C (for argument/discussion), if I am implementing my string storage class as a wrapper around an array I would probably just have an array of character pointers with each pointer referencing a spot in memory that contains a pattern. In the example I gave above, this would take 18 characters ( 6 * 3 = 18). Adding on the size of the array (let's say that sizeof(char*) is 4, our array would take 3 * 4 bytes of storage = 12 + 18 or 30 bytes total to store our patterns.
If I am instead storing the patterns in a sort of digital search tree, I make a small tradeoff. The nodes in my tree are going to be larger than 1 byte apiece (1 byte for the character in the node, 4 bytes for the "next" pointer in each node, 5 bytes apiece). The first pattern we store is AABBCC. This is 6 nodes in the tree. Next is AABCBC. We reuse the path AAB from the first tree and need only an additional 3 nodes for CBC. The last pattern is AACBCB. We reuse AA, and need 4 new nodes for CBCB. This is a total of 13 nodes * 5 bytes = 65 bytes of storage. However, if you have a lot of long, repeating patterns in the prefix of your data, then you'll see some prefix path-level compression.
If this isn't the case for you, I would look into Huffman or LZW compression. This will require you to build a dictionary of patterns that have integer numbers tied to them. When you compress, you build the dictionary and create integer id's for each pattern in your text. You then replace the patterns in your text with the integer id's. When uncompressing, you do the opposite. I don't have the time to describe these algorithms in more detail, so you'll need to look them up.
It's a tradeoff in simplicity/time. If your data will allow it, take the shorter method and just use the built-in container. If not, you will need something more tailored to your data.
I don't think you'd have any additional problems using QSet over another sort of container, such as std::set, a map, or a vector. If you are wondering about running out of memory, that probably depends on how many thousands of the strings you need to store, and if there was a way to encode them more concisely. (For example, if the characters always occur in the same order but vary in relative lengths, store the length for each character rather than all of the characters.) However, even 50,000 of these strings is only around 5 MB, and 500,000 of them is only 50 MB to store, discounting storage overhead, which is a moderate amount of memory on modern machines.
QSet does sound like a good idea. It's basically just a hash-table and it can optimize its bucket size dynamically. Perfect.
Another suggestion for compressing the key:
Treat it as a base-6 number string (think A=0, B=1, ... F=5) and convert it into binary (int).
QByteArray ba("112"); // instead of "BBC"
int num = ba.toInt(0, 6 /*base*/); // num == 44
6^3 < 2^8, so we can represent every 3 chars in your string with a 1 byte int (or char) and make a bytearray of it. That would cut down the size of the key from 54 bytes to 18 bytes.
From your earlier comment: "In my strings, there will always be 54 characters, and there will always be 9 of each character. The order is the only thing that changes."
Don't store raw strings then. You could just compress them into the 6 characters actually used, and then make a QSet of those. A trivial compression would be {a,b,c,d,e,f}, and if the character set is known beforehand (and only those 6 characters) you could even pack things into a 16-bit integer.
first off I'm going to say I don't know a whole lot about theory and such. But I was wondering if this was an NP or NP-complete problem. It specifically sounds like a special case of the subset sum problem.
Anyway, there's this game I've been playing recently called Alchemy which prompted this thought. Basically you start off with 4 basic elements and combine them to make other elements.
So, for instance, this is a short "recipe" if you will for making elements
fire=basic element
water=basic element
air=basic element
earth=basic element
sand=earth+earth
glass=sand+fire
energy=fire+air
lightbulb=energy+glass
So let's say a computer could create only the 4 basic elements, but it could create multiple sets of the elements. So you write a program to make any element by combining other elements. How would this program process the list the create a lightbulb?
It's clearly fire+air=energy, earth+earth=sand, sand+fire=glass, energy+glass=lightbulb.
But I can't think of any way to write a program to process a list and figure that out without doing a brute force type method and going over every element and checking its recipe.
Is this an NP problem? Or am I just not able to figure this out?
How would this program process the list the create a lightbulb?
Surely you just run the definitions backwards; e.g.
Creating a lightbulb requires 1 energy + 1 glass
Creating an energy requires 1 fire + 1 air
and so on. This is effectively a simple tree walk.
OTOH, if you want the computer to figure out that energy + glass means lightbulb (rather than "blob of molten glass"), you've got no chance of solving the problem. You probably couldn't get 2 gamers to agree that energy + glass = lightbulb!
You can easily model your problem as a graph and look for a solution with any complete search algorithm. If you don't have any experience, it might also help to look into automated planning. I'm linking to that text because it also features an introduction on complexity and search algorithms.