Can someone please explain the memory layout of the data stored in a vector<bool>?
like what layout does the memory have from address &myVec[0] upwards? Does it depend on endianness? Is the memory continguous for all stored values? (i'm aware that vector<bool> doesn't actually store booleans). can I dump the content of a vector<bool> into a file using memcopy to get a bitmap of my values?
please no questions like "what do you need it for" or suggestions like using bitsets or boost.
Thank you for an accurate explanation
The std::vector will simply manage a raw array on the heap. So whan you do &myVec[0] you get the address of the first element of this array. As it's an array it...follows the rules of a raw array..
BUT
std::vector is a special case, a specific implementation, a mistake of the C++ commitee that is NOT a vector of bool but a container managing bits. So avoid using this one.
A vector is essentially a wrapper around an array so yes, the memory is contiguous. This also means you can use memcpy on it (if that's what you want).
The endianness of each element depends on your current architecture.
vector<bool> myvector;
myvector.push_back(1);
myvector.push_back(0);
myvector.push_back(0);
myvector.push_back(1);
myvector.push_back(0);
myvector.push_back(1);
myvector.push_back(1);
myvector.push_back(0);
would appear like this in memory:
1 0 0 1 0 1 1 0
I'm not sure if that is what you are asking.
Related
I want to create a union of 2 pointers which have 1 byte alignment, but I don't want to store an additional tag, to distinguish them. I know that on some platforms, at least the most significant bit of a pointer is unused, but that might change in the future.
Please does someone know a portable way to do this?
I'm hoping for a technique that does not invalidate the original pointer. That is, is there a function for which when I apply it to some number n, frees up at least one bit in the number and when I "unapply" it, gives me back the original number.
I have:
typedef QHash<QString,QString> CueTrack;
QVector<CueTrack> tracks;
Each QHash have to store 12 pairs.
The QVector should contain 0 to 99 elements, but I cannot know how many they will be in advance as it depends on data that will be read runtime from an external user file. On average they are expected to be about 10; in most cases are between 5 and 15; rarely are more than 20 and almost never more than 30 (but, in some very rare cases, they can reach up to 99).
I thought of 2 scenarios:
a) no initial sizing and resize when needed
typedef QHash<QString,QString> CueTrack;
QVector<CueTrack> tracks;
...
tracks.resize(i+1);
tracks[i]["cmd"] = "val";
...
b) initial sizing
typedef QHash<QString,QString> CueTrack;
QVector<CueTrack> tracks(99);
I am in doubt, and I would like to understand:
What is the preferred scenario, especially in terms of performance?
Is there any other possible scenario that is preferable?
Should squeeze() be used here? How and when?
An empty QHash doesn't take up much space, and 99 is (for a computer) a very small data set.
Furthermore, Qt containers have an exponential growth policy. When the allocated container space gets too small, it will be doubled.
So if you resize(+1), it will grow (about) in powers of two.
QHash has an advantage over QMap in that it can be preallocated, too.
Probably your best strategy would be:
tracks.reserve(20);
// When adding a new track:
tracks.resize(i+1);
tracks[i].reserve(12); // number of pairs
But as with all questions concerning performance: Profile, then ignore :-)
Having just discovered alignment issues for the first time I am unsure on which method is the best/safest way to deal with them. I have a record which I am serialising to send over a Stream and vice-versa so it must meet the interface spec and contain no padding.
Given the example record:
type MyRecord is record
a : Unsigned_8;
b : Unsigned_32;
end record;
This by default would require 8 bytes but I am able to remove packing using 2 methods:
for MyRecord'Alignment use 1;
or
pragma Pack (MyRecord);
I have found a few questions relating to C examples but haven't been able to find a clear answer on which method is the most appropriate, how to determine which method to use or if they are equivalent?
UPDATE
When I tried both on my 'real' code rather than a basic example I found that the Alignment attribute achieved what I was looking for. pragma Pack significantly reduced the size, not confirmed but I assume it has packed the many enumerated types I'm using, overriding the 'Size use 8 attribute applied to each type.
For Streams you could leave MyRecord without any representation clauses and use the default MyRecord’Write and MyRecord’Read; ARM 13.13.2(9) says
For elementary types, Read reads (and Write writes) the number of stream elements implied by the Stream_Size for the type T; the representation of those stream elements is implementation defined. For composite types, the Write or Read attribute for each component is called in canonical order, which is last dimension varying fastest for an array (unless the convention of the array is Fortran, in which case it is first dimension varying fastest), and positional aggregate order for a record.
One possible disadvantage of the GNAT implementation (and maybe of others) is that the ’Write and ’Read calls each end in a call to the underlying network software. Not a problem (aside from possible inefficiency) normally, but if you’re using TCP_NODELAY (or worse, UDP) this is not the behaviour you’re looking for.
Overloading ’Write leads back to your original problem (but at least it’s confined to the overloading procedure, so the rest of your program can deal with properly aligned data).
I’ve used an in-memory stream for this (especially the UDP case); ’Write to the in-memory stream, then send the Stream_Element_Array to the socket. One example is ColdFrame.Memory_Streams (.ads, .adb).
I think you want the record representation clauses, if you want full control:
for MyRecord'Size use 40;
for MyRecord use record
a at 0 range 0 .. 7;
b at 1 range 0 .. 31;
end record;
(or some such, I might have messed up some of the indices here).
NB: edited as per comment by Simon
I have an application that is pretty memory hungry. It holds a large amount of data in some big arrays.
I have recently been noticing the occasional OutOfMemoryException. These OutOfMemoryExceptions are occurring long before my application (ASP.Net) has used up the 800mb available to it.
I have track the issue down to the area of code where the array is resized. The array contains a structure that is 74bytes in size. (I know that you shouldn't create struct's that are bigger than 16bytes), but this application is a port from a Vb6 application). I have tried changing the struct to a class and this appears to have fixed the problem for now.
I think the reason that changing to a class solves the problem has to do with the fact that when using a struct and the array is resized, a segment of memory that is large enough to store the new array needs to be reserved (e.g. (currentArraySize + increaseBySize)*74) cannot be found. This leads to the OutOfMemoryException.
This isn't the case with a class as each element of the array only needs 8bytes to store a pointer to the new object.
Is my thinking correct here?
Your assumptions regarding how arrays are stored are correct. Changing from struct to class will add a bit of overhead to each instance and you'll loose the advantages of locality as all data must be collected via a reference, but as you have observed it may solve your memory problem for now.
When you resize an array it will create a new one to hold the new data, then copy over the data, and you will have two copies of the same data in memory at the same time. Just as you expected.
When using structs the array will occupy the struct size * number of elements. When using a class it will only contain the pointer.
The same scenario is also true for List which increase in size over time, thus it's smart to initialize it with the expected number of items to avoid resizing and copying.
On 32bit systems you will hit outofmem around ~800mb, as you are aware of. One solution you can try is to but your structs on disk and read them when needed. Since they are a fixed size you can easily jump into the correct position at the file.
I have a project on Codeplex for handling large amounts of data. It has a type of Array with possibility for autogrowing, which might help your scenario if you run into problems with keeping it all in memory again.
The issue you are experiencing might be caused by fragmentation of the Large Object Heap rather than a normal out of memory condition where all memory really is used up.
See http://msdn.microsoft.com/en-us/magazine/cc534993.aspx
The solution might be as simple as growing the array by large fixed increments rather than smaller random increments so that as arrays are freed up the blocks of LOH memory can be reused for a new large array.
This may also explain the struct->class issue as the struct is likely stored in the array itself while the class will be a small object on the small object heap.
The .NET Framework 4.5.1, has the ability to explicitly compact the large object heap (LOH) during garbage collection.
GCSettings.LargeObjectHeapCompactionMode = GCLargeObjectHeapCompactionMode.CompactOnce;
GC.Collect();
See more info in: GCSettings.LargeObjectHeapCompactionMode
And a question about it: Large Object Heap Compaction, when is it good?
I'm designing a game server and I have never done anything like this before. I was just wondering what a good structure for a packet would be data-wise? I am using TCP if it matters. Here's an example, and what I was considering using as of now:
(each value in brackets is a byte)
[Packet length][Action ID][Number of Parameters]
[Parameter 1 data length as int][Parameter 1 data type][Parameter 1 data (multi byte)]
[Parameter 2 data length as int][Parameter 2 data type][Parameter 2 data (multi byte)]
[Parameter n data length as int][Parameter n data type][Parameter n data (multi byte)]
Like I said, I really have never done anything like this before so what I have above could be complete bull, which is why I'm asking ;). Also, is passing the total packet length even necessary?
Passing the total packet length is a good idea. It might cost two more bytes, but you can peek and wait for the socket to have a full packet ready to sip before receiving. That makes code easier.
Overall, I agree with brazzy, a language supplied serialization mechanism is preferrable over any self-made.
Other than that (I think you are using a C-ish language without serialization), I would put the packet ID as the first data on the packet data structure. IMHO that's some sort of convention because the first data member of a struct is always at position 0 and any struct can be downcast to that, identifying otherwise anonymous data.
Your compiler may or may not produce packed structures, but that way you can allocate a buffer, read the packet in and then either cast the structure depending on the first data member. If you are out of luck and it does not produce packed structures, be sure to have a serialization method for each struct that will construct from the (obviously non-destination) memory.
Endiannes is a factor, particularly on C-like languages. Be sure to make clear that packets are of the same endianness always or that you can identify a different endian based on a signature or something. An odd thing that's very cool: C# and .NET seems to always hold data in little-endian convention when you access them using like discussed in this post here. Found that out when porting such an application to Mono on a SUN. Cool, but if you have that setup you should use the serialization means of C# anyways.
Other than that, your setup looks very okay!
Start by considering a much simpler basic wrapper: Tag, Length, Value (TLV). Your basic packet will look then like this:
[Tag] [Length] [Value]
Tag is a packet identifier (like your action ID).
Length is the packet length. You may need this to tell whether you have the full packet. It will also let you figure out how long the value portion is.
Value contains the actual data. The format of this can be anything.
In your case above, the value data contains a further series of TLV structures (parameter type, length, value). You don't actually need to send the number of parameters, as you can work it from the data length and walking the data.
As others have said, I would put the packet ID (Tag) first. Unless you have cross-platform concerns, I would consider wrapping your application's serialised object in a TLV and sending it across the wire like that. If you make a mistake or want to change later, you can always create a new tag with a different structure.
See Wikipedia for more details on TLV.
To avoid reinventing the wheel, any serialization protocol will work for on the wire data (e.g. XML, JSON), and you might consider looking at BEEP for the basic protocol framework.
BEEP is summed up well in its FAQ document as 'kind of a "best hits" album of the tricks used by experienced application protocol designers since the early 80's.'
There's no reason to make something so complicated like that. I see that you have an action ID, so I suppose there would be a fixed number of actions.
For each action, you would define a data structure, and then you would put each one of those values in the structure. To send it over the wire, you just allocate sum(sizeof(struct.i)) bytes for each element in your structure. So your packet would look like this:
[action ID][item 1 (sizeof(item 1 bytes)][item 1 (sizeof(item 2 bytes)]...[item n (sizeof(item n bytes)]
The idea is, you already know the size and type of each variable on each side of the connection is, so you don't need to send that information.
For strings, you can just throw 'em in in a null terminated form, and then when you 'know' to look for a string based on your packet type, start reading and looking for a null.
--
Another option would be to use '\r\n' to delineate your variables. That would require some overhead, and you would have to use text, rather then binary values for numbers. But that way you could just use readline to read each variable. Your packets would look like this
[action ID]
[item 1 (as text)]
...
[item n (as text)]
--
Finally, simply serializing objects and passing them down the wire is a good way to do this too, with the least amount of code to write. Remember that you don't want to prematurely optimize, and that includes network traffic as well. If it turns out you need to squeeze out a little bit more performance later on you can go back and figure out a more efficient mechanism.
And check out google's protocol buffers, which are supposedly an extreemly fast way to serialize data in a platform-neutral way, kind of like a binary XML, but without nested elements. There's also JSON, which is another platform neutral encoding. Using protocol buffers or JSON would mean you wouldn't have to worry about how to specifically encode the messages.
Do you want the server to support multiple clients written in different languages? If not, it's probably not necessary to specify the structure exactly; instead use whatever facility for serializing data your language offers, simply to reduce the potential for errors.
If you do need the structure to be portable, the above looks OK, though you should specify stuff like endianness and text encoding as well in that case.