I have been playing around with Google Datastore and was thinking of using it in production. My concern is that because it autoscales and you pay for the queries (after the free tier), if you get a surge in traffic, you will also get an increased bill. Is there a way to limit the amount Google Datastore scales? I would rather have users experience slow traffic then get a huge bill.
And before anyone suggests setting a budget: I don't want to shut down the site, just have it slower.
Based on Pricing and Quota documentation, Google Cloud Datastore is charging per Stored data, Entity Reads, Entity Writes, Entity Deletes and Small Operations. It is not charging for autoscaling. Which means that either they are served fast or slow, since you are accessing the data, you will be billed anyway.
e.g. Currently the price is $0.036 per 100,000 entities for Entity Reads operations daily. Which means that if during the day your users have already read 50,000 entities (for free) in total, you are going to get billed $0.036 for that day for the next 100,000 entities read operations.
The only way to limit this is to actually avoid any read operations for that day, which will make your application unusable.
Related
i've been searching for what is the concurrent users limit for the cloud firestore spark plan but couldn't find it.
https://firebase.google.com/docs/firestore/quotas
It did said 1.000.000 concurrent users limit, but did not mention whether it is for the spark plan or the blaze plan. I've also tried searching answer elswhere, but did not find it answered specifically (with a source).
Help would be appreciated, thank you.
Per the Cloud Firestore pricing information (which Firebase uses):
When you use Firestore, you are charged for the following:
The number of documents you read, write, and delete.
The amount of storage that your database uses, including overhead for metadata and indexes.
The amount of network bandwidth that you use.
There is also no mention of any connection limits on Firebase's pricing page or the quotas documentation that you linked.
Unlike the Realtime Database, Cloud Firestore does not charge on a per-connection basis.
This video series also covers the ins and outs of Firebase products and is well worth sitting through.
Think of Cloud Firestore like a folder on your computer, which can contain thousands of little text files, similar to how documents in Cloud Firestore are stored. Users can update them with little chance of collision and grabbing a single document file would only require feeding 1s and 0s back to the requestor. This is why you are charged for network bandwidth rather than by individual connection.
In comparison, the RTDB was similar to one large JSON text file, with many people all trying to update it at once. Because parsing this text file on the server side was required to read and write data from it, it required compute resources to be able to do so. For this reason (among others), the number of connections the RTDB manager processes handled on behalf of spark plans were rate-limited to prevent abuse.
Despite not performing too many read/write operations - 170r, 4w to my Firestore database, I appear to have a huge number of API calls - 15,093 (See image below). The cause of this high calls to read/write ratio should be accounted to my application's use of network streams. My question is, should this be considered as a billable metric, or should I not worry if this runs into the millions so long as read/write are within limits (theoretically, I've never seen this happen on my own account).
I'm inclined to believe that I needn't worry about this metric, as I can't seem to find it under either the Firebase or GCP quotas page.
It may also be considered that I use the google maps and directions API from GCP, although this aren't nearly used as much as Firestore.
Thanks.
According to this doc, you are charged for the following:
The number of documents you read, write, and delete.
The amount of storage that your database uses, including overhead
for metadata and indexes.
The amount of network bandwidth that you use.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 2 years ago.
Improve this question
Is there any situation where it makes sense to use both realtime and firestore in conjunction? What situations lend themselves more favorably to firebase realtime vs firestore, or a combination? I keep reading about horror stories of people getting hit with huge costs is there anyway to test before hand.
For context I am looking to work with an auction based market place of over 50,000 products. The idea is to be able to filter those products as needed, create, modify and delete bids for those products, favorite items and retrieve Users bids. From what I was reading the general suggestion (to keep cost low) for market places using firebase seems to suggest storing products in realtime db and the user objects, sales etc. in firestore. Kinds of queries I will need are find products with the lowest/highest bids, most favorited items, as well as fetching users current and purchases.
When would it be optimal to store in realtime vs firestore, from a cost perspective?
My current logic is to store the product objects in realtime since they will be referenced more frequently. Alternatively I am thinking it makes sense to store the user info, their bids, and purchases in one document in a firestore as that would incur just one read cost, and for a highly active user could result in a large amount of data to be transferred. Where I am confused comes with things like viewing the previous sales of a given product vs getting a user's previous sales, should sales be stored in realtime (as their own object or embedded in the product object) or firestore (embedded in the User doc) or both?
Looking at your app that you plan to make, let have a short talk regarding it.
A bidding app, first someone wants to sell their stuff so they post it in your app. Then every single user of your app may see it start bidding on it. Now as I don't know how your app is going to work but here's my assumption you will store the data of bidders and the bids they make in firebase realtime database.
This will involve lots of read, write operations. Now Firestore does offer you 20K operations/day, but if you cross the limit it will barely cost you $0.18/100K writes and $0.06/100K reads. Now the choice entirely depends on scale of your app. If your app has large number of audience, go for Realtime-Database. You can download upto 10GB of data per month for free and a dollar per GB beyond that. But this has a catch, if you stick to the spark plan, you can have only 100 simultaneous connections to the database so I doubt the performance if you have large number of users. It can go upto 200K using Blaze plan and that too per database. So if you create another database you will have more. I will personally suggest create multiple databases as per the region or any parameter to spread the traffic. [Again it's upto how many people use your app]
In my opinion, you should use the Firebase Realtime database your app. [Make sure you utilize the firebase storage as well for storing large photos of the things on sale].
Lastly, use firestore when you have less number of operations but are larger in size. Use firebase realtime database when you have many small tasks like updating the highest bid value or number of users currently bidding for a particular thing, use Realtime DB.
In my opinion, go for realtime database. I too use it for some game stuff like to store user stats and update it as the user progresses. This involves lots of read/write/update/delete operations so I stick with realtime-database.
When to use Firestore along with the real-time database?
As you have mentioned user profile, I will suggest use Firestore to store those credentials. Because user's won't generally update their profile so this won't cost much writes. Also the bidders would be much interest in bidding rather than watching others profiles. So even if if a few users check other's profile. This won't cost you much reads. But even if your app is designed in such a way that bidder must check seller's profile once, then firestore will definitely help you reduce usage of realtime database's [GB Downloaded] quota.
Every time someone queries data from your realtime database, you consume some part of the 10 GB of free download limit.
Also as I have mentioned the simultaneous connections to the database, if you host user profile data in Firestore then firestore will take care of profile visits so that bidders get faster response from your application. Just make sure you utilise all the free quotas from firebase storage, firestore and the realtime database and make sure your app is designed in such a way that it spreads traffic evenly between all services. Use the cloud functions on your back-end, and don't make your application [.apk] too heavy on client side as the app needs a lot to code.
So the conclusion, use firestore to store data which won't be accessed frequently like the user credentials and whatever stuff they have on sell. Use realtime database to store bidding data. Oh and yes, if you also want to store some stats like how many purchases has someone made or some information that changes too frequently put that in firebase realtime database. You can simply create a child node users/${username} and keep the frequent stuff in realtime database. This won't cost you much storage but take of that download limit. Shouldn't be expensive much especially talking of your app is going to address 50000 products XD.
I am looking to work with an auction based market place of over 50,000 products.
If you have comparatively less number of users, realtime database is sufficient but who knows when there may be a huge rise in your app users. So it's better to spread the data in both Firestore and Real-time database as mentioned above.
Just a caution: This is what I faced, then searched over stackoverflow and found this. Firestore counts READS even if you are just scrolling over the data tab in firestore. So make sure you don't just get surfing over there. I made 2 writes and was just looking at how the data is being stored and I already got 27 reads ...
I am a Google blaze plan user and I have an express server containing a simple endpoint that just pulls from firestore. During high traffic hours, I can retrieve 5000+ simultaneous read requests which eventually throws this error below,
Error: 8 RESOURCE_EXHAUSTED: Quota exceeded.
After I wait a few minutes I am able to read the collection again.
Update:
Unsure why the downvotes without any explainations.. but I also have a Mutex system which I think may be leading to hitting these limits. If fails to lock a document using transactions, it goes down an array of snapshots until a lock is given. If the array becomes empty, it does another read to firebase for another set of N documents, which is only 50 in my case.
So my question is, is there a limit on the amount of transactions or reads per second that we're allowed to do from a single connection (my express server)? I don't think it's stated anywhere in the documents.
It looks like you're reaching one of the read/write/transaction limits stated in this page.
Might be this one Maximum writes per second per database: 10,000 (up to 10 MiB per second) but I'm only guessing...
To answer your question, according to this link, the maximum concurrent connections for mobile/web clients per Firebase database is 1,000,000. Thus, your connections seem to not exceed the limits.
For the Blaze Plan project, the limit for Cloud Firestore Document Read is 50K/day, since free usage from Spark Plan is included in Blaze Plan. The limit is such, unless you have set any budget limit in your Billing account. The usage will be reset at midnight of PST. If you upgrade your plan to Flame Plan, the limit is 250K/day.
Here you may read about the official Cloud Firestore Quotas and limits, such as maximum document reads, maximum size for a document, that can be useful. Furthermore you may monitor your database usage and check your plan's limits from the “Usage” tab in the Firebase console. You can check usage over the current billing period, the last 30 days, or the last 24 hours.
Stackdriver Monitoring is also a practical tool for monitoring document reads/writes/deletes, active connections and snapshot listeners.
A good practice, if you want to avoid unexpected charges on your billing account, would be to create an alerting policy based on the Cloud Firestore metrics, as stated here.
Additionally, you can estimate and verify your monthly costs on the “Blaze Plan” by using this Blaze Plan calculator.
For anyone who runs into this issue in the future, please check your App Engine budget settings under "Application Settings". I set the daily spending limit to avoid unnecessary charges during testing and it slipped my mind. I increased the budget and the error is currently gone.
AWS usually sends me an email when my budget has been exceeded.
The application I'm currently working on needs real-time communication that is scalable. We have been looking into and tried out Firebase real-time database and firestore. It seems Firebase real-time database is more mature and tested out, while firestore is still in beta, which is why we are leaning towards the real-time database.
We are however worried about its scaling capabilities in our context. Our queries will mainly be geo spatial based on the user's location. According to Firebase simultaneous realtime connections to my database and https://firebase.google.com/pricing/#faq-simultaneous the maximum number of concurrent users is 100.000, which will be too low for our needs.
According to their documentation, it seems like database sharding is the way to scale beyond 100.000 concurrent users https://firebase.google.com/docs/database/usage/sharding. Since our queries are based on the user's location, we could group the data into regions, e.g. US West, US Central, and US East and have a database instance for each of those three regions.
While this method may work, it seems very cumbersome to set it up. We would probably need to have a service which the user initially connects to in order to be redirected to the correct database instance that fits the region which the user is in. Additionally, it should handle the case where a user moves into another region, and should therefore be redirected to another database instance containing the data for that specific region.
Another complex task would be to distribute the data into the correct database instances.
Is there a more simple approach to scale beyond 100.000 users or is it possible to increase the amount of concurrent connections for a single Firebase real-time database?
To me it seems like almost a waste to use Firebase if it requires you to do so much "load" balancing yourself.
The 100K concurrent connections is a hard cap on the Firebase Realtime Database.
The approach you describe with a two-step connect is quite idiomatic. The first step is usually quite simple. In fact for many apps it is part of their authentication flow, or based on the outcome of that. For example, many apps base the user's shard on a hash of their UID.
In your case, you could inject the users region into their token as a custom claim when they register. Then you'd get that claim when they sign in, and can redirect them to their shard. You could also persist the shard info in the client when they first connect, so that you only have to determine that only once for each client/device.
Is there a more simple approach to scale beyond 100.000 users or is it
possible to increase the amount of concurrent connections for a single
Firebase real-time database?
Yes. use Firestore database.
Scales completely automatically. Currently, scaling limits are:
Around 1 million concurrent connections and 10,000 writes/second. (they plan to increase these limits in the future) (source)
Maximum write rate to a document is 1 per second (source)
Is officially out of beta and in General Availability from 31/1/2019 (source)