I have noticed int64 type precision lost, in my project Firebase realtime database:
When I add a new child or edit the child value in browser;
When I add a new child or edit the child value via my (C++) code: as SetValue(int64_t) or even as SetValue(firebase::Variant::kTypeInt64);
The precision lost starts after 53 bits:
// 9007199254740991 <- I set 53 bits value.."11111111111111111111111111111111111111111111111111111";
// 9007199254740991 -> it records correctly;
// 18014398509481983 <- I set 54 bits value."111111111111111111111111111111111111111111111111111111";
// 18014398509481984 -> it records as......"1000000000000000000000000000000000000000000000000000000";
// seems it declared as int64_t but saved as float?
Can someone reproduce it?
Is it bug or feature?
Any time you work with data in the Firebase console, it's going to be subject to the limits of JavaScript. JS always stores numbers of 64 bit floating point numbers (not integers). So, if you're dealing with 64 bit integers, some of that precision gets lost if you're using JS.
Try ignoring what you see in the console, and only perform reads and writes with code that handles 64 bit integers correctly. If that doesn't work, then there is something inside Realtime Database that imposes a similar limitation. The documentation doesn't make any claims about the precision of numbers, as far as I can see. However, Firestore (the successor to Realtime Database) does make a claim that 64bit integers are OK. So if this is important to you, you might want to switch (though it still does have problems with those numbers in the console, due to JS limits).
Related
I am trying to write firmware code for RFID device which will have config data storage as well as the temporary storage that maybe can be read and then if convenient be removed.
I am using Arduino IDE to program this on an ESP32 Wroom32. I have tried to understand how the storage actually works, finding various resources. One being datasheet of the same, that says that there could be 4 MB of program code storage possible, and that sounds fantastic, my question is if for example I take EEPROM library and save about 214 bytes to config which will rarely be touched, where is it exactly being stored? Is it simply in NVS? I can see that the default settings show me about 1310720 Bytes of storage and I know that I can utilise other partitions as well to store more in case I ever try to have more sketch storage than 1310720 Bytes.
My question is if I am trying to store data such as config and real time data, how much would I possibly be able to store? Is there a limit? Would it cause any kind of problems if I try to use the other such partitions to write the code? Will it be only NVS that is storing that data or can I utilise the other app0, app1, spiffs etc to store extra Bytes? A lot of the resources are confusing me, here are the data that I am referring to from online 1 and 2. Any idea would help me proceed very further.
P.S. I am aware that the EEPROM library has been deprecated and I shall use either Preferences or littlefs for better management but if I am aware correctly I can still utilise them, and without much issue that will work since there is still compatibility for that. I am also curious about using inbuilt SRAM of RTC with the RTC attribute RTC_DATA_ATTR, since I hope to also utilise deep sleep mode incorporated.
My question is if I am trying to store data such as config and real time data, how much would I possibly be able to store? Is there a limit?
It depends. First on the module; there is ESP32-WROOM with 4MB flash but you could also order different flash sizes.
Then the question is: how big is your application (code)? Obviously this needs to be saved on the flash as well, reducing the total usable amount for data storage (by the size of the application). Also there is a bootloader which needs some small space as well.
Next, ESP32 is using a partition scheme. One partition is reserved for the bootloader. The rest can be divided between one or more application partitions, NVS partitions, and possibly other utility partitions (i.e. OTAData).
If you are using the OTA functions, there will be at least 3 application partitions of equal size, further reducing the total usable amount for data storage.
So the absolute upper limit of what you can store using NVS functions is the size of your NVS partition. However since it's a key-value storage, you must take into account the size of the key, which can be considerably larger than the data you store (up to 12 times for a 12 character key and a uint8 value).
So there is no way to say exactly how much data you can put into the system without knowing exactly how you're going to use it. For example, you could store one very large "blob" value that could take "up to 97.6%" of the partition size. But you could not store 10 "blob" values of 1/10 (9.76%) the size since you must take into account the keys and some flash metadata used internally.
Would it cause any kind of problems if I try to use the other such partitions to write the code?
That depends on what these partitions are used for. If you override the partition table, or bootloader, or your application code, yes there will be problems. If there is "free space" then it won't be a problem, but then you should redefine this free space as NVS space. It's nice of Espressif to provide this NVS library, dont work around it, work with it.
Using Espressif's esptool you can create custom partition tables where you could minimize the size of the application partition to just barely fit your application, and maximize the NVS partition size. This way you will get the most storage out of your device without manually implementing a filesystem. If you are using OTA, you should leave some empty room in your application partition, in case your application code grows, as it usually does.
Will it be only NVS that is storing that data or can I utilise the other app0, app1, spiffs etc to store extra Bytes?
You absolutely can, but you will destroy whatever data is on that partition. And you will have lots of work to do, because you'll have to implement all of this yourself (basically roll your own flash driver).
If you don't need OTA, you dont need app0/app1 partitions at all.
Note that SPIFFS is also a way to store data, except it's not key-value but file-based. If you dont need it, remove that partition, and fill the space with your NVS partition.
On the other hand, SPIFFS is probably a better alternative if you are really tight on flash space, since you can omit the key and do your own referencing.
I have checked the Docs but I got confused a bit. When storing a long integer such as 265628143862153200. Would it be more efficient to store it as a string of integer.
This is what I need help with is the below calculation corrent?
Integer:
length of 265628143862153200 *8 ?
String:
length of 265628143862153200 +1 ?
The Firebase console is not meant to be part of administrative workflows. It's just for discovery and development. If you have production-grade procedures to follow, you should only write code for that using the provided SDKs. Typically, developers make their own admin sites to deal with data in Firesotre.
Also, you should know that JavaScript integers are not "big" enough to store data to the full size provided by Firestore. If you want to use the full size of a number field, you will have to use a system that supports Firestore's full 64 bit signed integer type.
If you must store numbers bigger than either limit, and be able to modify them by hand, consider instead storing multiple numbers, similar to the way Firestore's timestamp stores seconds and nanoseconds as separate numbers, so that the full value can overflow greater than signed 64 bits.
I've got some really strange behavior with subtracting numbers in Google Firestore with Google Cloud Functions.
Here is my test Google Functions code:
exports.testCounter = functions.https.onCall((data, context) => {
db.collection('counter').doc('test').update({
count: admin.firestore.FieldValue.increment(-0.0005)
});
});
And then I call the function from the cli:
firebase functions:shell
testCounter({test: "demo"})
This is the document I created to test:
Initial Schema
Then after the first execution everything works exactly as expected and the number 5 is now 4.9995.:
Document after first execution
However after the second execution 4.9995 is not 4.9990 as expected, but it changes to 4.9990000000000006.:
Document after second execution
Does anyone have an idea how to fix this? Or is this somehow expected behavior?
Thanks and have a great day!
Rick
This is standard behavior for floating point numbers using the IEEE 754 standard, which is what Firestore uses. Put briefly, computers lose precision when storing floating point data efficiently.
If you aren't able to accept this behavior, you should not store floating point numbers at all, and instead just store integers. The integer should contain all of the required precision that you would normally need for the floating point equivalent, except you mulitiply the float by the required precision.
For example, if you need three decimal points of precision (e.g. 5.005), multiply that float by 1000, drop the fractional part, and store the integer 5005. If you need to add .005, then you should instead add 5. Then you can format that number any way you want on the client.
It's actually not a Firebase problem, just inherent to floating point numbers.
(4.9990000006).toFixed(4); // 4.9990
Replace the parameter 4 in toFixed with whatever precision you need.
As explained by Doug, the problem comes from the floating point numbers behaviour.
One solution is to use, in your Cloud Function, the big.js library.
It works well in Cloud Functions (as in any Node.js code), but it means that you will not be able to use admin.firestore.FieldValue.increment() anymore.
I've run into a strange problem that I've been wrestling with for a few days. I'm hoping someone will have some insights.
I have a Flex app that uses standard remoting to access a SQL db via ColdFusion. (Flex sdk 3.4 (yes, old), CF 10)
I've recently found a problem where if the arguments to a cffunction includes a string that could be a very large number ("2e4361251", for example), the remoting result/fault handler in my AS code will not get called for a VERY long time, effectively locking the app. (App logic requires that I wait for this operation to finish before moving on.)
My most recent test took 45 minutes for the result handler to be called.
What the cffunction does does not matter.
If I am attempting to update my db, the update is successful (whether the string is used in that update or not), my app just doesn't get the result for a very long time;
If the cffunction does NOTHING, like
< cffunction name="myFunc" access="remote">
< !-- do nothing -->
< /cffunction>
it STILL takes an extremely long time to return.
Repeated testing has a noticeable effect on the server; CF starts eating up processor and RAM, though it will eventually recover.
Passing the string IN as a parameter is key; if I'm retrieving the string from the db, there is no problem.
These strings could represent VERY large numbers. If I were trying to do any sort of calculations or other manipulations, I could maybe understand this. But as I said, the function doesn't have to do anything at all, just the presence of the string causes the problem. (And I have verified that it is being passed as a String, not an int, Number, float, decimal, etc)
The string that alerted me to this issue originated with a user. It is a catalogue number of some sort. They had many with different letter/number combos; this one happened to use a single 'E'. I cannot simply 'disallow' strings of this nature.
Anyone have any ideas on why this is happening and, more importantly at the moment, how I can avoid it? A 45 min delay = 'frozen app' to the user.
Sometimes, on various Unix architectures, recompiling a program while it is running causes the program to crash with a "Bus error". Can anyone explain under which conditions this happens? First, how can updating the binary on disk do anything to the code in memory? The only thing I can imagine is that some systems mmap the code into memory and when the compiler rewrites the disk image, this causes the mmap to become invalid. What would the advantages be of such a method? It seems very suboptimal to be able to crash running codes by changing the executable.
On local filesystems, all major Unix-like systems support solving this problem by removing the file. The old vnode stays open and, even after the directory entry is gone and then reused for the new image, the old file is still there, unchanged, and now unnamed, until the last reference to it (in this case the kernel) goes away.
But if you just start rewriting it, then yes, it is mmap(3)'ed. When the block is rewritten one of two things can happen depending on which mmap(3) options the dynamic linker uses:
the kernel will invalidate the corresponding page, or
the disk image will change but existing memory pages will not
Either way, the running program is possibly in trouble. In the first case, it is essentially guaranteed to blow up, and in the second case it will get clobbered unless all of the pages have been referenced, paged in, and are never dropped.
There were two mmap flags intended to fix this. One was MAP_DENYWRITE (prevent writes) and the other was MAP_COPY, which kept a pure version of the original and prevented writers from changing the mapped image.
But DENYWRITE has been disabled for security reasons, and COPY is not implemented in any major Unix-like system.
Well this is a bit complex scenario that might be happening in your case. The reason of this error is normally the Memory Alignment issue. The Bus Error is more common to FreeBSD based system. Consider a scenario that you have a structure something like,
struct MyStruct {
char ch[29]; // 29 bytes
int32 i; // 4 bytes
}
So the total size of this structure would be 33 bytes. Now consider a system where you have 32 byte cache lines. This structure cannot be loaded in a single cache line. Now consider following statements
Struct MyStruct abc;
char *cptr = &abc; // char point points at start of structure
int32 *iptr = (cptr + 1) // iptr points at 2nd byte of structure.
Now total structure size is 33 bytes your int pointer points at 2nd byte so you can 32 byte read data from int pointer (because total size of allocated memory is 33 bytes). But when you try to read it and if the structure is allocated at the border of a cache line then it is not possible for OS to read 32 bytes in a single call. Because current cache line only contains 31 bytes data and remaining 1 bytes is on next cache line. This will result into an invalid address and will give "Buss Error". Most operating systems handle this scenario by generating two memory read calls internally but some Unix systems don't handle this scenario. To avoid this, it is recommended take care of Memory Alignment. Mostly this scenario happen when you try to type cast a structure into another datatype and try reading the memory of that structure.
The scenario is bit complex, so I am not sure if I can explain it in simpler way. I hope you understand the scenario.