We are currently running on MongoDB but are now interested in moving over to Cloud Firestore. We have not found any tools for such a task so our planned approach is to create a "schema aware" script that makes a special sequence of queries from the MongoDB, and for each document create a new entry in Firestore while creating a map between the old mongo _id and the newly assigned id in Firestore. These maps will be used in later queries in the sequence that need to make reference to other documents in other collections.
Has anyone else tried something similar or found a better, more general purpose tools for the task?
Related
This question already has answers here:
Couple Firebase Firestore and Firebase Storage calls together into a batch?
(2 answers)
Closed last month.
I'm developing an app in Flutter, I have a method called **CreateUser **which takes as parameters the user information, his profile picture, and a list of strings, I need to save the information in the Realtime Database, the picture in the Cloud Storage and the list in the Firestore.
I would like all these operations to be successful, if one of these should fail then I would like the others to undo the data they wrote. How can I implement the rollback of the other operations? Can I use transactions?
I've tried using transactions but I'm not sure if I can use them on different Databases.
I need to save the information in the Realtime Database, the picture in the Cloud Storage, and the list in the Firestore.
That's indeed possible, by performing one operation, right after another, only when the operation succeeds. For example, as soon as the operation for writing data to the Realtime Database completes, then inside the callback, perform the addition of the image to Storage. As soon as the addition of the image to Storage succeeds, perform the last operation of writing the data to Firestore.
I would like all these operations to be successful, if one of these should fail then I would like the others to undo the data they wrote.
There is no built-in mechanism for that. If you thought you can add to a batch operation, a Realtime Database write operation, a Firebase Storage file upload
and Firestore write operation and be sure that all three are complete, so you can have consistent data, please note that this is not possible. These operations are a part of different Firebase services and unfortunately, at the moment I'm writing this answer there is no way you can make them atomic, meaning all succeed or all fail with an exception.
How can I implement the rollback of the other operations?
You have to write code for that because none of the Firebase products support cross-product transactional operations. To solve this, you'll have to nest the calls during your write/upload operations and handle the error if the next operation fails. This means that you either have to delete the data from the Realtime Database and the file from Storage if the write operation in Firestore fails. Or only delete the data from the Realtime Database if the file upload to Storage fails.
But note, at some point in time, there will be a failure that the client can't roll back one of the delete operations. The most common approach for these inevitable failures which might happen is to make your code robust by handling exceptions and performing occasional cleanups in both places, Firebase Storage and Firestore, considering that the first operation is the one that writes data to the Realtime Database.
As discussed with the Firebase engineers, the reason is quite clear. Even if the Realtime Database and Cloud Firestore are both Firebase products, they are still different products. Besides that, Firebase Storage is a service within Google Cloud. So now, 2023-01-12 there is no way you we can do that. Hopefully, it will be available in the near future.
Can I use transactions?
No, and that's for the exact same reason as above.
One way I might address this, is to use the Firestore document write operation to trigger a Workflow [1] that can handle the three operations and rollback depending on failure state. That way you can also have a constant transaction record follow the process.
If you wanted to provide app feedback say to the user, you could have your app. wait for a DB record of completion (or error) get written and based on that report back to the user.
[1] https://cloud.google.com/firestore/docs/solutions/workflows
I am working on the following topic right now:
about 8000 stores/objects(uniqueID, name, adress, latitude, longitude, geohash, amount of productX, amount of productY.)
in the future there will be new stores and also some of the stores will be deleted
I am planning to add a version attribute to each data: for example I start with version1. If I receive an update or a new store then it will have version2
it is not needed to receive realtime changes. This data will be changed only a couple of times each month
Flutter app with a feature: find n stores at this location
because the data is not changed frequently I was planning to put an updated local sqlite database to each build
I will use Realtime Database or Firestore to receive updated data within my app to avoid to publish a new version only because database was changed
Firestore: I will add only a new attribute to my document and I will request all data that is newer than my local database version and I will also have a collection that has all deleted items
Realtime database: here I would have something like this
-updates
version1
newData: List of unique IDs
deletedData: List of unique IDs
I read a lot about geo queries on firestore and I also found a working flutter plugin to do this filtering on serverside, but because my data is not changed that frequently I am very confused about the best approach.
Questions:
Is my approach a good idea or should I forget about the local sqlite DB and use only Firebase/Firestore?
If I will use the local database and will only observe updates from the backend, which of the services should I use in terms of pricing etc.? (Advantage Firestore: If I use Firestore I can easily switch to it by using the flutter plugin and forget about my local database)
If you have only a couple of thousand objects, and the data hardly every changes, I'd typically consider querying the data locally on the device. You can likely very quickly iterate over the 8,000 items to check their lat/lon against the user location.
You could then use something like Firebase Storage or even Firebase Hosting to distribute updated data sets to the users. Of course you could use Firestore or Realtime Database for that too, but if you're not using their querying or realtime capabilities, there are cheaper options to distribute a static file.
I have built an app using firestore as we are interested in the realtime updates portion of things. However,we are not building a website that has CRM component where a lot of reports will be generated. The contents of that CRM are all new. There is only one report that would need firebase data as well as the new data (you can say 1 report out of 20).
I was thinking of building the CRM backend off mysql DB? Do you recommend to go with this approach or shall I do the CRM in the same firebase/firestore db?
Thanks
If you are looking for a real-time backend database for your CRM, then the Firebase RTDB / Cloud Firestore would be ideal for this. I'm not sure why you'd want to add a mySQL component, unless you are going to create some reports that require complex joins. However, if your data is modelled correctly, this also shouldn't be an issue.
Take a look at this video to get a better understanding: What is a NoSQL Database? How is Cloud Firestore structured? - Get to Know Cloud Firestore Ep.1
I had already saved all data like users or rooms contents on Firebase Database. Now I want to move all posts to Cloud Firestore for better queries.
But,
Can I take conditions from Realtime Database?
I mean all users data are in realtime database. So I want to get some informations from there, then decide whether it's accessible or not.
The security rule systems are different between Realtime Database and Firestore. There is currently no easy to way to simply copy from one to the other, or to make one recognize the other. If you're porting your data to Firestore, you'll have to write new rules by hand that functionally match the ones you were using in Realtime Database.
No, this isn't functionality that exists. You'll need to have migrated the data that the rules depend upon to Cloud Firestore first.
Firebase's documentation has the following paragraph:
Using Cloud Firestore and Realtime Database: You can use both databases within the same Firebase app or project. Both NoSQL databases can store the same types of data and the client libraries work in a similar manner. Keep in mind the differences outlined above if you decide to use both databases in your app.
I can't find any documentation on how to add a Cloud Firestore to an existing project with a Realtime Database, though. I will ultimately upgrade to Cloud Firestore, but would like some time to experiment and learn before I convert the production database.
Does anybody know how to use both databases in the same Firebase project?
When you go to your project in the console and choose the Database product, you should see something like this the first time:
This is a selector that lets you choose to see either Realtime Database or Firestore in your project. You can switch between the two with this selector.
The first time you select Cloud Firestore, it will ask you to configure things. Start in "test mode" to set things up for full read and write without authentication to get started quickly, but of course your should always have rules set up in production.
After you set up Firestore, you should be able to use both client SDKs to access both databases independently.