I would like to write JSON to a datapoint value in Xively (so I can store a value dictionary in a single datapoint). What is the max size (e.g. string length) I can write to a datapoint?
255 bytes, so basically 255 utf-8 characters.
Related
Since SQLite can store any data in any column, my question - how SQLite engine knows which is which?
Lets say a table has column A and there are 2 values 123 and '#$%'. How these two are stored (and I would like to know byte-by-byte layout to understand)?
What happens with these values if I do select * from table where a > 0 ?
Obviously the engine needs to figure out each value type in column A.. How exactly does it do?
Does it store a type marker with each value so it examines marker on select? or does it evaluates type every time it retrieves a value?
There is a type marker called serial type. It is a variable length integer encoding both the type and the length of the value.
NULL, 0, 1, and '' (empty text) need 1 byte (1 byte for the serial type, no value)
Integers needs 2 to 9 bytes (1 byte serial type, 1 to 8 byte(s) value), so 123 needs 2 bytes.
For text values, the serial type may expand to multiple bytes. Assuming UTF-8, '#$%' needs 4 bytes (1 byte serial type because the encoded length is <= 57)
There is more details in the Database File Format.
I checked the storage size but I'm confused when it comes to storing numbers.
In case of Bytes, what does "Byte length" mean? If I store -128 what's the length? And in case of 12?
In case of Floating-point number and Integer, it doesn't matter if I store 325 or 9.9999999999999 it will always be 8 bytes?
In case of Array? Let' say we have ["ab", "bcd"], what's the size, (2+3=5) or (2+1)+(3+1)=7
If you store an array of bytes, the size will simply be the length of that array. An array with a single byte value of -128 is still just one byte.
Yes, all numbers occupy the same 8-byte size, even if you don't see a fractional part.
The documentation says it's the sum of the array element sizes, so I would expect 7, the sum of the two individual string sizes, each encoded in UTF-8 + 1
I have a data that needs to be stored in a database as encrypted, the maximum length of the data before encryption is 50 chars (English or Arabic), I need to encrypt the data using AES-128 bit, and store the output in the database (base64string).
How to know the length of the data after encryption?
Try it with your specified algorithm, block size, IV size, and see what size output you get :-)
First it depends on the encoding of the input text. Is it UTF8? UTF16?
Lets assume UTF8 so 1 Byte per character means 50 Bytes of input data to your encryption algorithm. (100 Bytes if UTF16)
Then you will pad to the Block Size for the algorithm. AES, regardless of key size is a block of 16 Bytes. So we will be padded out to 64 Bytes (Or 112 for UTF 16)
Then we need to store the IV and header information. So that is (usually, with default settings/IV sizes) another 16Bytes so we are at 80 Bytes (Or 128 for UTF16)
Finally we are encoding to Base64. I assume you want string length, since otherwise it is wasteful to make it into a string. So Base 64 bloats the string using the following formula: Ceil(bytes/3) * 4. So for us that is Ceil(80/3) = 27 * 4 = 108 characters (Or 172 for UTF 16)
Again this is all highly dependent on your choices of how you encrypt, what the text is encoded as, etc.
I would try it with your scenario before relying on these numbers for anything useful.
I am trying to determine what the correct value would be to input a number into an SQli query. I have been able to deduce (I believe) that the database is using little endian by playing with the hex values. For example:
To modify a value to 255, I am having to use 00000000FF000000.
For values 1-255 this has been fine, it has worked every time.
My question is, for the values 265 and 275, which in hex is converted to 109 and 113, how would I position the numbers within this 16 digit series?
0000000010900000 does not work and returns a value of 36880
Likewise, shifting the 109 to the left instead, returns a value of 9
I am using http://www.scadacore.com/field-applications/programming-calculators/online-hex-converter/ to try and determine the correct value, using the little endian boxes.
The value string seem to be a structure or an array consisting of 16-bit values in little-endian format.
You can see this by examining the values this way:
for example:
00000000FF000000
can be broken down into 16-bit values:
0000 0000 FF00 0000
And when byte-order for 0xff00 is swapped we get 0xff, ie. 255.
For this value:
0000000010900000
we get:
0000 0000 1090 0000
When byte-order is swapped to 0x9010 we get 36880, as in the result.
You don't state why there is a 16-char limit that needs to be fulfilled, but in general - this seem to be part of a structure or an 16-bit array rather than a 64-bit value (which is normally represented by 8 bytes, or 16 chars) but since the value then should have been at either end it is likely not a 64-bit value.
This is normally not something one can guess automatically and would require you know the structure or array definition (talking about binary arrays). One can of course use a pattern to find values matching a range using various positions, width and byte-order, but I cannot see you would benefit in this case compared to finding out the structure/array format (above I guess 4x 16-bit values, but it could just as well have been 1x32-bit, 1x16-bit and 2x8-bit etc.).
My question is, for the values 265 and 275, which in hex is converted to 109 and 113, how would I position the numbers within this 16 digit series?
To fill the binary array you would place them at position 4 (0-base) as 16-bit values in little-endian order.
You can use the DataView for this as it has built-in support for endianess.
Example
Converting value 275 and placing it in the array with correct endianess and position:
var arr = new Uint8Array(8);
var view = new DataView(arr.buffer);
// set value in correct byte-order and position
view.setUint16(4, 275, true); // pos, value, little-endian
// show result (convert to hex-string)
for (var str = "", i = 0; i < arr.length; i++) str += pad(arr[i].toString(16));
console.log(str);
function pad(s) {return s.length < 2 ? "0"+s : s}; // helper: pads to format 0x00
This produces the string:
0000000013010000
Which if we use the same method as earlier becomes:
0000 0000 1301 0000
And then byte-order swap 0x1301 it becomes 0x0113, or 275 as expected.
Blobs are just an array of bytes; the database does not modify their contents or their order in any way.
The endianness (or any other way in which the bytes are to be interpreted) is determined only by the application that writes and reads the blob.
I have a question about the data types in sqlite3.
As a value of SQLITE_INTEGER can be stored in 1, 2, 3, 4, 6, or 8 bytes depending on the magnitude of the value, if I only know that a column in SQlite database stores SQLITE_INTEGER, how can I know a value in this column is 4 bytes or 6-8 bytes integer, or which one should be used to get the value, sqlite3_column_int() or sqlite3_column_int64()?
Can I use sqlite3_column_bytes() in this case? but according to the documentation, sqlite3_column_bytes() is primarily used for TEXT or BLOB.
Thanks!
When SQLite steps into a record, all integer values are expanded to 64 bits.
sqlite3_column_int() returns the lower 32 bits of that value without checking for overflows.
When you call sqlite3_column_bytes(), SQLite will convert the value to text, and return the number of characters.
You cannot know how large an integer value is before reading it.
Afterwards, you can check the list in the record format documentation for the smallest possible format for that value, but if you want to be sure that integer values are never truncated to 32 bits, you have to always use sqlite3_column_int64(), or ensure that large values get never written to the DB in the first place (if that is possible).