It is well known that read/write cost of firebase rtdb is free. With little bit more digging, i could've find out that read/write can actually cost in non-direct ways. Ok, so i've been searching through docs and SO questions to figure out of "what is the exact difference between FIRESTORE READING COST($0.06 per 100,000 documents) and REALTIME DATABASE DOWNLOAD($1/GB) COST", but sadly i couldn't have managed to complete it.
Stored data cost for RTDB($5/GB) is really clear, and i understand that the price billed monthly(this one would be true, right?). But what is exactly a DOWNLOAD cost? Through a few SO questions and official docs, i could've figure out that rtdb download cost is really similar to firestore reading cost, and it is important to specify db.ref path clearly by diving into the final path. But, if the download cost is all about these operations, such as reading json data in a specific field or path, what is difference between concept of firestore reading and concept of these rtdb download operations?
If all these things are already happening in the Earth, the cost of RTDB when it comes to 'conceptual reading' is never free, even if we speak in direct manner. Then why some community members and articles always say "read/write cost for RTDB is free"? I was considering migration of some features from firestore to RTDB since it is well-known that rtdb is free for read and write. The feature is updating a single path(document for firestore) of 500B size hundreds time every month. But this issue makes me really confusing.
Let's say that 100,000 read for firestore is $0.04 and download for RTDB(which seems like reading) is $1/GB. In my calculation, 2,500,000 document reads from firestore is equal to a single GB download from RTDB. It means that if a single operation reads bunch of data larger than 400B(approx.), firestore read-cost is even cheaper than RTDB read-cost. Then there is no reason for me to use RTDB for reading data if single operation needs to retrieve data larger than 400B per operation. It feels like i've got caught by wrong concepts, but it is not easy to get out of this swamp.. ]:
So i hope to make clear of RTDB read/write cost(if it is really free of charge by itself), and the reason why it is better to use RTDB than firestore, when the app have to do lots of read operations(for me, ex. approx. 1,000 operations retrieving 400B-size data per month per a single user). I understand that a few firebase gurus are thankfully contributing SO's firebase tag. I've tried to write the question as clear as possible, but think there would be some unclear parts in the question. So, comments will be really appreciated! Hope this question would reach to you.. Thanks in advance [:
I have created a very handy spreadsheet calculator that calculates the rough size of the payload and scales per user while also factoring in the free tier usage as well. You can enter your values at the top and get a decent result.
But to summarize, Realtime DB is highly expensive to read per KB while Firestore is rated for up to 1mb (potential) per read while writing to Realtime is extremely cheap, I have confirmed that besides overhead, it is free to write to realtime db.
Realtime db is not as economical compared to Firestore and is designed to cover some caveats of Firestore. Realtime Billing for reads (download) is the (data + overhead) rounded up to nearest kb
TLDR:
Firestore is ideal for high reads, low writes, static information.
Realtime is better suited with low reads, high writes, volatile information.
When reading documents from Firestore you pay for:
Document reads - The cost to read the document on the server.
Network egress - The cost to download the data to the client.
In most scenarios we see the cost for developers using Firestore coming more from document reads, as the cost per GB is comparatively low.
When reading data from the Realtime Database, you only pay for:
GB downloaded - The cost to download the data to the client.
Here the cost mostly comes from the size of the data you download. It's quite similar to the Network egress from Firestore, but at a higher cost per byte read (and of course you then don't pay for the read operation on the server itself).
While a calculator (such as the one from DIGI Byte, or the one on the pricing page) is going to be best, the rough guidance is that if you perform many small reads and writes, RTDB is going to a better choice, while if you perform fewer writes and/or more larger reads, then Firestore is often the better choice.
Related
I'm working on a product that displays the results of running races. Races could have thousands of participants. So, in the days after a medium-sized event, there might be 3000 non-authenticated users wanting to browse 3000 results.
Although not every visitor will view all the results, the maximum damage at 3000 * 3000 would be 9,000,000 reads which at $.06 (Google cloud pricing) would cost $540,000 (Update: I'm a dummy, I missed the "per 100,000 documents" part, so this would only be $540).
Obviously, I wouldn't deliver all 3000 results for each visit - there would be paging and limits. Though, there's something inherently scary about the possibility of those costs.
Questions:
Is firebase simply the wrong technology for this type of product?
Is firebase not really intended for non-authenticated apps? Obviously DDOS becomes a concern for public access and there's no real protection in FB for this.
Every post I've read on these topics assumes developers are building apps for authenticated users.
9,000,000 reads which at $.06 (Google cloud pricing) would cost $540,000
The Firestore pricing of $0.06 is for 100,000 document reads, so 9 million document reads cost $540.
Aside from that: you should model your data in a way that ensures you read the data that the user actually sees. For example, if all users will read the entirety of all 3,000 documents, consider using a data bundle to distribute that to them.
Realistically though it is more likely that each user will read just a subset of the documents, and probably not of all 3,000 documents. So consider if you can combine the part that they'll read into a more cost-efficient structure. For if these were news articles: you could store the headline and intro paragraph of the first 100 articles in a single document, and just read that document (let's call it the frontpage) into each client when it starts.
There are many more ways to model the data, depending on the use-cases of your app. To learn more on how to think about such data modeling, I recommend reading NoSQL data modeling and watching the excellent Get to know Cloud Firestore video series.
I have watched videos about Firestore on YouTube. It is said that there is a limitation for a where which the max size is 1 Mb and also maximum 1 write per second.
How about the query to a collection? Is there a limitation for this? Because I will heavily rely on a parent collections to perform different queries for a lot of users. That's why I need to know the worst case scenario. I need to know if there are any limitations.
I mean something like, maximum number of query per second, max concurrent queries? Maximum number to get data from a collection in a second ? Do such limitations exis for querying a collection?
I have tried to read the documentation from here and it seems there is no limitation for query in a collection. I need to make sure, maybe there is documentation that I have not read yet?
There is no documented limit to the number of queries you can execute against Firestore. While there is probably a physical limit, you're extremely unlikely to hit it before running into any of the documented limits (such as the 1 million concurrent users).
In other words: it is quite unlikely you'll need to worry about the read scalability or limitations of Firestore for your application. It is made to scale very well on read operations, which is precisely the reasons why it supports a more limited set of functionality, and why it has a write throughput limit on individual documents.
Firestore scales massively for read operations. When using the Blaze payment plan, there are no fundamental read limits like there are for write limits. You just need to be willing to pay for all those documents reads, and the bandwidth required for all that data. Please read the pricing page about billing.
There is limit for Your reads and writes.
They have provided in their document where in Free service you have limited read writes.
Each read will be counted in normal queries it acts same with writing document
I've built an app that let people sell tickets for events. Whenever a ticket is sold, I update the document that represents the ticket of the event in firestore to update the stats.
On peak times, this document is updated quite a lot (10x a second maybe). Sometimes transactions to this item document fail due to the fact that there is "too much contention", which results in inaccurate stats since the stat update is dropped. I guess this is the result of the high load on the document.
To resolve this problem, I am considering to move the stats of the items from the item document in firestore to the realtime database. Before I do, I want to be sure that this will actually resolve the problem I had with the contention on my item document. Can the realtime database handle such load better than a firestore document? Is it considered good practice to move such data to the realtime database?
The issue you're running into is a documented limit of Firestore. There is a limit to the rate of sustained writes to a single document of 1 per second. You might be able to burst writes faster than that for a while, but eventually the writes will fail, as you're seeing.
Realtime Database has different documented limits. It's measured in the total volume of data written to the entire database. That limit is 64MB per minute. If you want to move to Realtime Database, as long as you are under that limit, you should be OK.
If you are effectively implementing a counter or some other data aggregation in Firestore, you should also look into the distributed counter solution that works around the per-document write limit by sharding data across multiple documents. Your client code would then have to use all of these document shards in order to present data.
As for whether or not any one of these is a "good practice", that's a matter of opinion, which is off topic for Stack Overflow. Do whatever works for your use case. I've heard of people successfully using either one.
On peak times, this document is updated quite a lot (10x a second maybe). Sometimes transactions to this item document fail due to the fact that there is "too much contention"
This is happening because Firestore cannot handle such a rate. According to the official documentation regarding quotas for writes and transactions:
Maximum write rate to a document: 1 per second
Sometimes it might work for two or even three writes per second but at some time will definitely fail. 10 writes per second are way too much.
To resolve this problem, I am considering to move the stats of the items from the item document in Firestore to the realtime database.
That's a solution that I even I use it for such cases.
According to the official documentation regarding usage and limits in Firebase Realtime database, there is no such limitation there. But it's up to you to decide if it fits your needs or not.
There one more thing that you need to into consideration, which is distributed counter. It can solve your problem for sure.
When using Firestore and subscribing to document updates, it states a limit of 1M concurrent mobile/web connections per database.
https://firebase.google.com/docs/firestore/quotas#realtime_updates
Is that a hard limit (enforced/throttled in code)? Or is it a theoretical limit (like you're safe up to 1M, then things get dicey)? Is it possible to get an uplift?
Trying to understand how to support a large user base without needing to shard the database (which is one of the advantages of Firestore). Even at 5M users, it seems you would start having problems because you'd probably hit times when >20% of those users were on your app simultaneously.
As you already noticed, the maximum size of a single document in Firestore is 1 Megabyte. Trying to store large number of objects (maps) that may exceed this limitation, is generally considered a bad design.
You should reconsider the logic of you app and think at the reson why you need to have more than 1Mib in single a document, rather than each object being their own document. So to be able to use Firestore, you should change the way you are holding the data from within a single documents to a collection. In case of collections, there are no limitations. You can add as many documents as you want. According to the official documentation regarding Cloud Firestore Data model:
Cloud Firestore is optimized for storing large collections of small documents.
IMHO, you should take advantage of this feature.
For details, I recommend you see my answer from this post where I have explained some practices regarding storing data in arrays (documents), maps or collections.
Edit:
Without sharding, I'm affraid it is not an option. So in this case, sharding will work for sure. So in my opinion, that's certainly a reasonable option.
Edit: After posting the question I thought I could also make this post a quick reference for those of you needs a quick peek at some of the differences between these two technologies which might help you decide on one of them eventually. I will be editing this question and adding more info as I learn more.
I have decided to use firebase for the backend of my project. For firestore is says "the next generation of the realtime database". Now I am trying to decide which way to go. Realtime database or cloud firestore?
Billing:
At a first glance, it looks like firestore charges per number of results returned, number of reads, number of writes/updates etc. Real-time database charges based on the data transmitted. The number of read-write operations is irrelevant. They both also charge on the data stored on the google servers too (I think in this respect firestore is cheaper one). Why am I mentioning this price point? Because from my point of view, although it might a lower weight, it is also a point to consider while choosing the one over the other.
Scaling:
Cloudstore seems to scale horizontally seamlessly. I think this is not possible with the real-time database.
Edit:
In the real-time database, you need to shard your data yourself using multiple databases. And you can only do this if you are in BLAZE pracing plan.
ref: https://firebase.google.com/docs/database/usage/sharding
Performance & Indexing:
Another thing is the real-time database data structure is different in both. The real-time database stores the data as a JSON object in any way we structure them. Firestore structures the data as collections and documents. And hence the querying also changes between the two.
I think firestore does auto indexing which increases the read performance greatly too (which will decrease read performance). I am not sure if this is also the case with the real-time database.
Edit:
The real-time database does not automatically index your data. You need to do it yourself after a solid inspection of your data and your needs.
ref:https://firebase.google.com/docs/database/security/indexing-data
What other differences can you think of?
What would be (or has been) your choice for different types of projects?
Do you still go with the real-time database or have you migrated from that to the firestore? If so why?
And one last thing. How would you compare the SDKs of these two?
Thanks a lot!
What other differences can you think of?
what i think, ok. I use realtime-database for 6 months experience and difference is, firestore easy for sorting data. As Example, i want to retrieving user name based timestamp.
Query firstQuery = firestore.collection("Names").orderBy("timestamp", Query.Direction.DESCENDING).limit(10); // load 10 names
What would be (or has been) your choice for different types of
projects?
For me, Realtime-Database for Data Streaming when i work with Arduino, i want to store Drone Speed.
And Firestore for SMART OFFICE, like Air Conditioner, or light-room and Enterprise like Inventory Quantities, etc.
Do you still go with the real-time database or have you migrated from
that to the firestore? If so why?
still go with real-time because i need TREE for displaying streaming data strucure instead of query TABLE like firestore.