Here is our use case:
We have way more than 200,000 clients need to connect to firebase realtime db. So we created multiple database with same data and load blance the connections.
Here is the problem:
If we update one database, we will have to initiate connection and udpate the rest of the database as well. I would like to check if there is a way to auto sync up data between multiple databases.
Docs I have went through:
https://firebase.google.com/docs/database/usage/limits
https://firebase.google.com/docs/database/usage/sharding
Also I checked rules, and it seems that rules is not meant to be used to sync data.
Thanks
firebaser here
There is nothing built into Firebase to automatically synchronize data between multiple database instances. A common way to implement this when writing through a server-side process, is to simply write to each database in turn there.
If the data you want to write comes from a client-side SDK, I'd have the client write it to a staging area (just a temporary node in the database), and then use Cloud Functions to write the data the permanent location in all database instances.
Related
I am new to Flutter and mobile development in general and currently trying to wrap my head around certain database principles. I am building an note-taking app which uses Firestore for storing note data to the cloud. Users first need to register an account and log in, after which they are able to view and store notes to the cloud. The way I initially designed this was by using StreamProvider connected to the Firestore instance to update the user's list of notes after they add a note to their list.
After some reflection and worries that my app would be 'read' intensive, I realised that what I am trying to build does not require the notes to be constantly fetched from the server, as the data is private to them and should only be fetched by the user that creates it. My solution is instead of creating notes and listening to changes on the Firestore server to update the list, when a user creates a new note, a function should run and update the Firestore server as well as a local nosql database, removing the need for a read operation on the server after writing to it. This would allow me to have a local duplicate copy of the Firestore server and to periodically update it with write operations when needed. The only read operation would be on the app's startup to fetch a copy of the Firestore server.
My questions are:
Is storing a local version of the Firestore server a viable solution to minimise the number of read operations in my app, or am I overcomplicating things?
The StreamProvider allowed me to easily access data relating to the user, notes, etc throughout the app using Provider.of<Model>(Context). How can I easily access this data regardless of where I am in the app?
Given that my Firestore database structure for accessing notes is users/{user.id}/notes/{note.id}/note , how can I get a snapshot of the current logged in user ID and all the child fields in a single request (i.e. all the user's notes in users/{user.id}/notes and user data under users/{user.id}/)
I had already saved all data like users or rooms contents on Firebase Database. Now I want to move all posts to Cloud Firestore for better queries.
But,
Can I take conditions from Realtime Database?
I mean all users data are in realtime database. So I want to get some informations from there, then decide whether it's accessible or not.
The security rule systems are different between Realtime Database and Firestore. There is currently no easy to way to simply copy from one to the other, or to make one recognize the other. If you're porting your data to Firestore, you'll have to write new rules by hand that functionally match the ones you were using in Realtime Database.
No, this isn't functionality that exists. You'll need to have migrated the data that the rules depend upon to Cloud Firestore first.
It seems odd to me that Firestore would charge me for read queries to locally cached data, but I can't find any clarification to the contrary in the Firestore Pricing document. If I force Firebase into offline mode and then perform reads on my locally cached data, am I still charged for each individual entity that I retrieve?
Second, offline users in my app write many small updates to a single entity. I want the changes to persist locally each time (in case they quit the app), but I only need eventually consistent saves to the cloud. When a user reconnects to the internet and Firestore flushes the local changes, will I be charged a single write request for the entity or one per update call that I made while offline?
Firestore could potentially fit my use case very well, but if offline reads and writes are charged at the same rate as online ones it would not be an affordable option.
As the offical documentation says,
Cloud Firestore supports offline data persistence. This feature caches a copy of the Cloud Firestore data that your app is actively using, so your app can access the data when the device is offline. You can write, read, listen to, and query the cached data. When the device comes back online, Cloud Firestore synchronizes any local changes made by your app to the data stored remotely in Cloud Firestore.
So, every client that is using a Firestore database and sets PersistenceEnabled to true, maintains it's own internal (local) version of the database. When data is inserted/updated, it is first written to this local version of the database. As a result, all writes to the database are added to a queue. This means that all the operations that where stored there will be commited on Firebase servers once you are back online. This also means that those operations will be seen as independent operations and not as a whole.
But remeber, don't use Firestore as an offline-only database. It is really designed as an online database that came work for short to intermediate periods of being disconnected. While offline it will keep queue of write operations. As this queue grows, local operations and app startup will slow down. Nothing major, but over time these may add up.
If Google Cloud Firestore priceing model does not fit your use case very well then use Firebase Realtime Database. As mentioned also in this post from the Firebase offical blog, one the reasons you still might want to use the Realtime Database is:
As we noted above, Cloud Firestore's pricing model means that applications that perform very large numbers of small reads and writes per second per client could be significantly more expensive than a similarly performing app in the Realtime Database.
So it's up to you which option you choose.
According to this If you want to work completely offline with Cloud Firestore you can disable network by :
FirebaseFirestore.getInstance().disableNetwork()
but firestore will cause client offline error for first user get request, that you must consider this error as empty response.
Firebase's documentation has the following paragraph:
Using Cloud Firestore and Realtime Database: You can use both databases within the same Firebase app or project. Both NoSQL databases can store the same types of data and the client libraries work in a similar manner. Keep in mind the differences outlined above if you decide to use both databases in your app.
I can't find any documentation on how to add a Cloud Firestore to an existing project with a Realtime Database, though. I will ultimately upgrade to Cloud Firestore, but would like some time to experiment and learn before I convert the production database.
Does anybody know how to use both databases in the same Firebase project?
When you go to your project in the console and choose the Database product, you should see something like this the first time:
This is a selector that lets you choose to see either Realtime Database or Firestore in your project. You can switch between the two with this selector.
The first time you select Cloud Firestore, it will ask you to configure things. Start in "test mode" to set things up for full read and write without authentication to get started quickly, but of course your should always have rules set up in production.
After you set up Firestore, you should be able to use both client SDKs to access both databases independently.
I am doing some experiments on Firebase realtime database. Though I came across firebase-queue. This is what docs says https://github.com/firebase/firebase-queue.
A fault-tolerant, multi-worker, multi-stage job pipeline built on the Firebase Realtime Database.
What does it mean ?
firebase-queue lets you use Realtime Database to submit jobs to a backend server you control. Clients write into the database, and that data is received by firebase-queue, where you can do whatever you want with it.
If you have interest in running backend code that reacts to changes in your Firebase project, you are much better off looking into Cloud Functions for Firebase, because you won't have to manage a backend server at all. You just write and deploy code to it.