I have a Firebase project which basically have two environments: Staging and Production. The way I organized them is by creating different Firebase projects. Each of my projects uses Firebase Cloud Functions and Firestore. Except for that, I have each of the projects associated with a specific GIT branch. Both of the branches are integrated into CI/CD pipeline in Google Cloud Build.
So, in order to make it absolutely clear, I will share a simple diagram:
As you can see, I have the source code for the cloud functions under source control and there's nothing to worry about there. The issue comes in when I have the following situation:
A Firestore schema change is present on Staging
Cloud function (on Staging) is adjusted to the new schema.
Merge staging branch into production.
Due to the old Firestore schema on production, the new functions there won't work as expected.
In order to work around it, I need to manually go to the production Firestore instance and adjust the schema there (there's a risk to mess up production data).
In the perfect case, I would have that operation automated and existing project data would be adjusted to a new schema which comes in dynamically after merge.
Is that possible somehow? Something like migrations in .NET Core.
Cloud Firestore is schema-less - documents have no enforced schema. Code is able to write whatever fields it wants at any time that it wants. (For web and mobile clients, this is gated by security rules, but for backend code, there are no restrictions.) As such, there is no such thing as a formal migration in Cloud Firestore.
The "schema" of your documents is effectively defined by your code that reads and writes those documents. This means that migrating a data to a new format means that you're going to have to write code to perform the required changes. There is really no easy way around this. All you can really do is design your updates so that they are not disruptive to existing code when it comes time to move them to another environment. This means your code should be resilient to breaking changes, or simply do not perform breaking changes until after all code has been updated to deal with those changes.
You have to use Google Cloud to download an archive of the Firestore data. Run a migration script yourself on the archive, and then upload the archive to restore your Firestore database.
https://cloud.google.com/firestore/docs/manage-data/export-import
Google Cloud gives you a lot of command line access for managing your Firestore service.
// manage indexes
gcloud firestore indexes
// export all data to a bucket
gcloud firestore export gs://[BUCKET_NAME]
// import data from a bucket
gcloud firestore import gs://[BUCKET_NAME]/[filename]
// manage admin "functions" currently running (i.e. kill long processes)
gcloud firestore operations
To download/upload your JSON archive from Google Cloud buckets
// list files in a bucket
gsutil ls gs://[BUCKET_NAME]
// download
gsutil cp gs://[BUCKET_NAME]/[filename] .
// upload
gsutil cp [filename] gs://[BUCKET_NAME]/[filename]
Once you setup Google Cloud to be accessible from your build scripts. It's possible to automate data migration scripts to download, transform and then upload data.
It's recommended to maintain a "migrations" document in your Firestore so you can track which reversion of the migration needs to be done.
To avoid heavy migration tasks try adding a "version" property to documents, and then use the converter callbacks on the query builder to mutate data to the latest schema on "client side". While this won't help you handle changes with Firestore rules or functions. It's often easier to make tiny changes that are mostly cosmetic.
Related
still new to flutter and firebase. I understand how to store and retrieve images and display it on the app.
But how do I go about synchronizing files from the app's local asset folder to an asset folder stored in firebase storage? My intention is to check the cloud folder if a new image like an icon is recently uploaded, and download it to the app's local folder. If a file is removed in the cloud storage, it should also remove it from the local assets folder, mirroring it.
I need a way to compare local AssetManifest.json to the one on firebase storage. I Just need a little direction/algorithm to start with. Thanks for the help!
There is nothing specific built into Cloud Storage's Firebase SDK for this, so you'll have to build it in your own application code.
Using only the Cloud Storage for Firebase API
If you're just using Cloud Storage, you'll have to:
List all files that you're interested in from Storage.
Loop over the files.
Get the metadata for each file and check then the files was last updated
Compare that to when you wrote the local file.
Download the file if it is new or modified.
This approach will work, but it requires quite some calls to the Storage API, because there's no specific API to give you files that were modified since a specific date/time.
Storing metadata in a cloud database
You could also consider storing the metadata in a cloud database, like Firebase's Realtime Database or Cloud Firestore, and then use the query capabilities of that database to retrieve only files that were modified since your device last synchronized.
The recipe then becomes:
Determine when we last synchronized, which is a value you'll want to store in local storage/shared preferences.
Execute a query to the database to determine which files were added/modified since then.
Loop over the query results and...
Download each file that was modified.
In here, only step 2 and 4 make calls to the external APIs, so it is likely to be faster and cheaper to execute (but more work for you to write initially).
I have to Firebase projects, one is for dev and another is for production. I create a bunch of collections and after they successfully pass the test, I'll need to move them into production database. How can I do this without using Cloud Shell, or are there any alternative suggestions of database?
Thank you!
You can export the collection using cloud shell
Manage export ad import
In case you want to move collections between your dev and your production databases, without using Cloud Shell, there is an alternative that you can follow.
For you to achieve that, you will need to follow the below steps.
Create a Cloud Storage bucket to hold the data from your source project.
Export the data from your source project to the bucket.
Give your destination project permission to read from the bucket.
Import the data from the bucket into your destination project.
With these steps, you should be able to migrate data between your projects and have this way, the structure you want of a Development database and a Production database, where you can easily transfer the data. I would recommend you to check the official documentation Move Data Between Projects, to get the whole tutorial on how to achieve the above steps, in case you have doubts on how to achieve them.
Let me know if the information helped you!
I'm hoping to run integration and cypress browser tests against my React app that uses Firestore. There's a local emulator, but it only is targeted at testing security rules right now (see Local offline development with Firestore).
I can point the tests at a 'development' instance of the database, but that doesn't work if I want to run this in CI across all open pull requests.
In other worlds I’d just create a fresh database and drop it afterwards, but you can only have one firestore database per google cloud project.
Do you prefix the table names everywhere? Automate creating new google projects and dropping them again? Something else obvious I'm missing?
Context, I have a project with datastore which already has information loaded, currently we wanted to use cloud firestore (native mode), but we realized that migration is not possible, what alternatives do I have to use cloud firestore (native mode)?
Update June 16, 2021:
You can now do gcloud datastore export in your first project, followed by gcloud firestore import in your new project. The longer more involved migration below is no longer needed.
Just keep in mind that the Datastore export goes to a Cloud Storage bucket. Make sure that the account running the Firestore import has access to that bucket.
Original answer from 2019
I just migrated from Datastore to Firestore (native mode) for one of my web apps. Here is what I needed to do:
Create a new GCP project, as Firestore (native mode) and Datastore
can't co-exist in the same project.
Migrate the data from Datastore in my old project to Firestore (native mode) in my new project. As of this writing, there are no tools to do that in an automatic way. I wrote Python scripts that read all records from Datastore and wrote them to Firestore in the new project. These scripts ran locally on my machine, using service account keys downloaded from the Cloud Console.
(Side note: You might be tempted to use gcloud datastore export followed by gcloud firestore import. It seems to work and no error messages pop up when you do. But doc IDs and JSON properties don't translate well. This was a big time-sink for me. Don't go down this road.)
Rewrite the data access layer in your app. Firestore (native mode) has a different API than Datastore.
This was a fair amount of work, but it was worth it in my case:
I was able to retire a lot of server-side code because the clients can access the database directly.
I was able to retire a lot of client-side code for supporting offline mode because the Firestore client library implements it already.
Best of luck!
Unfortunately, you'll need to create your Cloud Firestore database in a new project that allow your existing service accounts to access that new database.
I have a firebase function that makes use of a SQLite database (read-only) which is currently uploaded along with the function.
The problem is that the db file is quite large and gets uploaded every time the function is changed. Is there a way to fetch this file from cloud storage during the installation process (during firebase deploy) - without hard-coding the URL in the source files?
What you're trying to do is problematic because your code running in Cloud Functions may actually be running on any number of server instances, determined by the load on your project. As such, downloading a file once at the time of deployment isn't going to naturally affect all the instances that maybe created or destroyed at any given moment.
It's far better to keep doing what you're doing, and include your extra data during deployment. When a new instance is spun up to handle events for your function, the file will be immediately ready to help service requests.