I know scalability is not an issue in Firebase and supports up to 100k Simultaneous connections(in general).
Based on pricing documentation:
You can create multiple database instances to go beyond the 100K
concurrent limit. See Pricing FAQ for more information.
Question 1: What if there is more than 200k users using simultaneously on the same database? The other half of the users could not query, connect or the request will be placed in queue?
(As a Firebase plan subscriber, I would like to know how Firebase deals with the problem to ensure the quality of the services provided to our customers are always in top-notch)
Since, App globalisation is common nowadays and many companies' practices are to have servers across multiple regions to provide better and stable performance. Online game for example which required low latency.
As for now, the firebase user is required to set the default location when creating the project which is non-editable afterward. Some issues even rises where the users realised they deployed their app to the wrong regions and do not have clues on how to change the regions.
This represents the country/region of your organisation/company. Your
selection also sets the appropriate currency for your revenue
reporting. The selected country does not determine the location of
your data for Firebase features. Google may process and store Customer
Data anywhere Google or its agents maintain facilities.
Question 2: Will or does Firebase provide a solution / tailor-made to such practice which having our database in multiple regions while having a headquartered region and multiple other regions sharing all the databases, functions and auth across the regions?
(For now to have multiple servers location, we have to create different projects and the user and data syncing will be a problem)
Hope the language does not offend, cheers!
It seems like your question (or at least your assumptions) is based on the Firebase Realtime Database, so I'll answer for that below.
Q1) You can create more than 2 databases in a single project, each of which allows 100K connections. So it can scale beyond 200K connections. All of these are hosted in the same region though, so you can't use each database for a separate region.
Q2) For a database solution that handles multiple regions, I'd recommend looking at Cloud Firestore. Also see: Cloud Firestore - selecting region to store data?
Related
So I have set the location of my Firestore database to asia-south-1, let suppose now I start getting users from the US as well as from India, how will I serve both the groups with lower latency?
You can't change the properties of your project once it's set up, nor even simply pay more for better service. If you want improved service in different regions, you will need to buy computing resources in each of those regions. This requires setting up multiple projects, with each Firestore product configured for the region you want improved service. Note that not all Firebase and GCP products are available in all regions.
If you want all users in all regions to be using the same set of data with improved latency, that's not possible to configure. You will have to build a way to mirror data between the different projects. There is nothing terribly easy about that, nor are there any existing solutions offered by Firebase or GCP to do this for you.
Doug nailed it for Firestore. So 👍
Note that Firebase's Realtime Database (unlike Firestore) can have multiple databases per project, which means you can set up a database instances in each region (it support three right now) and redirect your user's traffic to the region closest to them. You may still have to replicate data between the database instances, similar to Doug's answer, but (unlike in Doug's answer) with Realtime Database this can happen within a single project.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 2 years ago.
Improve this question
Is there any situation where it makes sense to use both realtime and firestore in conjunction? What situations lend themselves more favorably to firebase realtime vs firestore, or a combination? I keep reading about horror stories of people getting hit with huge costs is there anyway to test before hand.
For context I am looking to work with an auction based market place of over 50,000 products. The idea is to be able to filter those products as needed, create, modify and delete bids for those products, favorite items and retrieve Users bids. From what I was reading the general suggestion (to keep cost low) for market places using firebase seems to suggest storing products in realtime db and the user objects, sales etc. in firestore. Kinds of queries I will need are find products with the lowest/highest bids, most favorited items, as well as fetching users current and purchases.
When would it be optimal to store in realtime vs firestore, from a cost perspective?
My current logic is to store the product objects in realtime since they will be referenced more frequently. Alternatively I am thinking it makes sense to store the user info, their bids, and purchases in one document in a firestore as that would incur just one read cost, and for a highly active user could result in a large amount of data to be transferred. Where I am confused comes with things like viewing the previous sales of a given product vs getting a user's previous sales, should sales be stored in realtime (as their own object or embedded in the product object) or firestore (embedded in the User doc) or both?
Looking at your app that you plan to make, let have a short talk regarding it.
A bidding app, first someone wants to sell their stuff so they post it in your app. Then every single user of your app may see it start bidding on it. Now as I don't know how your app is going to work but here's my assumption you will store the data of bidders and the bids they make in firebase realtime database.
This will involve lots of read, write operations. Now Firestore does offer you 20K operations/day, but if you cross the limit it will barely cost you $0.18/100K writes and $0.06/100K reads. Now the choice entirely depends on scale of your app. If your app has large number of audience, go for Realtime-Database. You can download upto 10GB of data per month for free and a dollar per GB beyond that. But this has a catch, if you stick to the spark plan, you can have only 100 simultaneous connections to the database so I doubt the performance if you have large number of users. It can go upto 200K using Blaze plan and that too per database. So if you create another database you will have more. I will personally suggest create multiple databases as per the region or any parameter to spread the traffic. [Again it's upto how many people use your app]
In my opinion, you should use the Firebase Realtime database your app. [Make sure you utilize the firebase storage as well for storing large photos of the things on sale].
Lastly, use firestore when you have less number of operations but are larger in size. Use firebase realtime database when you have many small tasks like updating the highest bid value or number of users currently bidding for a particular thing, use Realtime DB.
In my opinion, go for realtime database. I too use it for some game stuff like to store user stats and update it as the user progresses. This involves lots of read/write/update/delete operations so I stick with realtime-database.
When to use Firestore along with the real-time database?
As you have mentioned user profile, I will suggest use Firestore to store those credentials. Because user's won't generally update their profile so this won't cost much writes. Also the bidders would be much interest in bidding rather than watching others profiles. So even if if a few users check other's profile. This won't cost you much reads. But even if your app is designed in such a way that bidder must check seller's profile once, then firestore will definitely help you reduce usage of realtime database's [GB Downloaded] quota.
Every time someone queries data from your realtime database, you consume some part of the 10 GB of free download limit.
Also as I have mentioned the simultaneous connections to the database, if you host user profile data in Firestore then firestore will take care of profile visits so that bidders get faster response from your application. Just make sure you utilise all the free quotas from firebase storage, firestore and the realtime database and make sure your app is designed in such a way that it spreads traffic evenly between all services. Use the cloud functions on your back-end, and don't make your application [.apk] too heavy on client side as the app needs a lot to code.
So the conclusion, use firestore to store data which won't be accessed frequently like the user credentials and whatever stuff they have on sell. Use realtime database to store bidding data. Oh and yes, if you also want to store some stats like how many purchases has someone made or some information that changes too frequently put that in firebase realtime database. You can simply create a child node users/${username} and keep the frequent stuff in realtime database. This won't cost you much storage but take of that download limit. Shouldn't be expensive much especially talking of your app is going to address 50000 products XD.
I am looking to work with an auction based market place of over 50,000 products.
If you have comparatively less number of users, realtime database is sufficient but who knows when there may be a huge rise in your app users. So it's better to spread the data in both Firestore and Real-time database as mentioned above.
Just a caution: This is what I faced, then searched over stackoverflow and found this. Firestore counts READS even if you are just scrolling over the data tab in firestore. So make sure you don't just get surfing over there. I made 2 writes and was just looking at how the data is being stored and I already got 27 reads ...
I have seen a firebase api in which you can create firebase projects and it occurred to me for example, when a user wants to counter a SaaS, make it easier for me to create a project and connect it to their SaaS but I have the following questions:
If there are supposed to be limits to creating firebase projects in a google account, will there be a time when databases can no longer be created for new users?
If the above is true, how can this be solved?
I have seen that you can ask for more projects, but how many can I have?
I know that with firestore, I can model data and only in a database have all the information, but for example, each user may have special requirements in their system and also provide security and information saving operations that would be impossible if all information is in a single database.
Thanks for the help.
EDIT
"How many projects can you have as a developer?" Yes, that's what I mean, having all the user bases in a single firebase google account. For example, on a web page, the user wants to pay for the application, with firebase admin and google cloud functions, I can automatically create a project and have all the databases in a google account. And what I want to know is if you can have multiple databases. I have seen that you can ask google to give you permission to host several projects but, for example, can you have more than 100 projects or even 1000? (I may never have reached that number, but in that case, I would like to know the limits that can be reached).
Edit 2
This first structure I have all the documents in a "universal" collection and I identify them with an id to know the user who used it.
But the second structure seems to me more organized and clean, in addition to the fact that users at no time need to share information among others. I have read that having nested collections is not good but over the years and the progress that firestore has had, this is no longer or was a problem only that the limitation is that you can only have up to 100 collections anidades but I never think to reach that quantity.
Inside of list, have all products for that user, because inside of document, only can have 1 mb of data and download 1 mb and is not the best option.
in the firestore documentation I see that it does not reveal other problems, it only says that it is difficult to delete the subcollection, but I do not allow users to do that and if I need to delete the subcollections, with the Google Cloud functions i can do it.
So, the second structure is more intuitive for me, but is the best option for that? or actually firestore is not good for this strcucutre?
The application I'm currently working on needs real-time communication that is scalable. We have been looking into and tried out Firebase real-time database and firestore. It seems Firebase real-time database is more mature and tested out, while firestore is still in beta, which is why we are leaning towards the real-time database.
We are however worried about its scaling capabilities in our context. Our queries will mainly be geo spatial based on the user's location. According to Firebase simultaneous realtime connections to my database and https://firebase.google.com/pricing/#faq-simultaneous the maximum number of concurrent users is 100.000, which will be too low for our needs.
According to their documentation, it seems like database sharding is the way to scale beyond 100.000 concurrent users https://firebase.google.com/docs/database/usage/sharding. Since our queries are based on the user's location, we could group the data into regions, e.g. US West, US Central, and US East and have a database instance for each of those three regions.
While this method may work, it seems very cumbersome to set it up. We would probably need to have a service which the user initially connects to in order to be redirected to the correct database instance that fits the region which the user is in. Additionally, it should handle the case where a user moves into another region, and should therefore be redirected to another database instance containing the data for that specific region.
Another complex task would be to distribute the data into the correct database instances.
Is there a more simple approach to scale beyond 100.000 users or is it possible to increase the amount of concurrent connections for a single Firebase real-time database?
To me it seems like almost a waste to use Firebase if it requires you to do so much "load" balancing yourself.
The 100K concurrent connections is a hard cap on the Firebase Realtime Database.
The approach you describe with a two-step connect is quite idiomatic. The first step is usually quite simple. In fact for many apps it is part of their authentication flow, or based on the outcome of that. For example, many apps base the user's shard on a hash of their UID.
In your case, you could inject the users region into their token as a custom claim when they register. Then you'd get that claim when they sign in, and can redirect them to their shard. You could also persist the shard info in the client when they first connect, so that you only have to determine that only once for each client/device.
Is there a more simple approach to scale beyond 100.000 users or is it
possible to increase the amount of concurrent connections for a single
Firebase real-time database?
Yes. use Firestore database.
Scales completely automatically. Currently, scaling limits are:
Around 1 million concurrent connections and 10,000 writes/second. (they plan to increase these limits in the future) (source)
Maximum write rate to a document is 1 per second (source)
Is officially out of beta and in General Availability from 31/1/2019 (source)
If you are going to implement a firebase backend service that goes insanely viral and has a growth rate of over 40% per 10 days and you hit the quota of the blaze plan of nothing over 100,000 simultaneous connections. What kind of working strategies could be implemented with firebase for handling simultaneous connections of over 100k? 1M? 100M? 1B!!?
100k concurrents is quite a lot of concurrent connections, but not out of the question for large apps, as we've had applications with > 1MM.
In general, the strategy for doing this involves sharding data across multiple databases. This is pretty trivial if the data is all independent (e.g. per user todo list), since you can assign a developer to a database and you never have to sync across databases.
Read only data (such as 1:N chat) is generally also pretty straightforward, as you can perform the same "allow someone to connect to any of N copies of the same database", and have a single user/job update all of them with the same information.
For more complicated schemes (particularly 1:1 chat), allowing developers to connect to any database, then syncing data across databases using Cloud Functions or another system is probably recommended.
If you're expecting to realistically hit 100k+ concurrents, feel free to reach out to our support team with more info on the use case and we're happy to work with you.