Context, I have a project with datastore which already has information loaded, currently we wanted to use cloud firestore (native mode), but we realized that migration is not possible, what alternatives do I have to use cloud firestore (native mode)?
Update June 16, 2021:
You can now do gcloud datastore export in your first project, followed by gcloud firestore import in your new project. The longer more involved migration below is no longer needed.
Just keep in mind that the Datastore export goes to a Cloud Storage bucket. Make sure that the account running the Firestore import has access to that bucket.
Original answer from 2019
I just migrated from Datastore to Firestore (native mode) for one of my web apps. Here is what I needed to do:
Create a new GCP project, as Firestore (native mode) and Datastore
can't co-exist in the same project.
Migrate the data from Datastore in my old project to Firestore (native mode) in my new project. As of this writing, there are no tools to do that in an automatic way. I wrote Python scripts that read all records from Datastore and wrote them to Firestore in the new project. These scripts ran locally on my machine, using service account keys downloaded from the Cloud Console.
(Side note: You might be tempted to use gcloud datastore export followed by gcloud firestore import. It seems to work and no error messages pop up when you do. But doc IDs and JSON properties don't translate well. This was a big time-sink for me. Don't go down this road.)
Rewrite the data access layer in your app. Firestore (native mode) has a different API than Datastore.
This was a fair amount of work, but it was worth it in my case:
I was able to retire a lot of server-side code because the clients can access the database directly.
I was able to retire a lot of client-side code for supporting offline mode because the Firestore client library implements it already.
Best of luck!
Unfortunately, you'll need to create your Cloud Firestore database in a new project that allow your existing service accounts to access that new database.
Related
I'm using the Firebase Emulator to run all Firebase services. I have managed to run the emulator with a backup of my Firestore data by running:
firebase emulators:start --import ./my-directory
... but I can't find a way to do the same with my Storage data.
Firestore has an option to do import and export, while Firebase storage don't have this feature yet (storage uses upload and download). Currently, there's no native way to import Google Cloud Storage data in the Storage Emulator.
Additionally, you can study how the emulator register the object by uploading sample files within the emulator and then running the export, you will see that the emulator will require 1 object and 1 JSON file that contains it's metadata. For now, it'll be up to you to download the objects from your production bucket along with a separate JSON file containing its metadata and then structure it for import.
There's also an opened issue for this in github that you can monitor as well.
Although #RJC's answer is correct, I want to share what I did.
You can add the option --export-on-exit when starting the emulator, and it will export the state of the local Firebase instance to a folder of your choice. So I downloaded all my data from the real Firebase project, uploaded everything with the Emulator, then killed the emulator, everything (including storage) got exported and now I start it with the --import option using the same folder where I exported everything.
This was not very awful since I didn't that much stuff in my storage.
I have a Firebase project which basically have two environments: Staging and Production. The way I organized them is by creating different Firebase projects. Each of my projects uses Firebase Cloud Functions and Firestore. Except for that, I have each of the projects associated with a specific GIT branch. Both of the branches are integrated into CI/CD pipeline in Google Cloud Build.
So, in order to make it absolutely clear, I will share a simple diagram:
As you can see, I have the source code for the cloud functions under source control and there's nothing to worry about there. The issue comes in when I have the following situation:
A Firestore schema change is present on Staging
Cloud function (on Staging) is adjusted to the new schema.
Merge staging branch into production.
Due to the old Firestore schema on production, the new functions there won't work as expected.
In order to work around it, I need to manually go to the production Firestore instance and adjust the schema there (there's a risk to mess up production data).
In the perfect case, I would have that operation automated and existing project data would be adjusted to a new schema which comes in dynamically after merge.
Is that possible somehow? Something like migrations in .NET Core.
Cloud Firestore is schema-less - documents have no enforced schema. Code is able to write whatever fields it wants at any time that it wants. (For web and mobile clients, this is gated by security rules, but for backend code, there are no restrictions.) As such, there is no such thing as a formal migration in Cloud Firestore.
The "schema" of your documents is effectively defined by your code that reads and writes those documents. This means that migrating a data to a new format means that you're going to have to write code to perform the required changes. There is really no easy way around this. All you can really do is design your updates so that they are not disruptive to existing code when it comes time to move them to another environment. This means your code should be resilient to breaking changes, or simply do not perform breaking changes until after all code has been updated to deal with those changes.
You have to use Google Cloud to download an archive of the Firestore data. Run a migration script yourself on the archive, and then upload the archive to restore your Firestore database.
https://cloud.google.com/firestore/docs/manage-data/export-import
Google Cloud gives you a lot of command line access for managing your Firestore service.
// manage indexes
gcloud firestore indexes
// export all data to a bucket
gcloud firestore export gs://[BUCKET_NAME]
// import data from a bucket
gcloud firestore import gs://[BUCKET_NAME]/[filename]
// manage admin "functions" currently running (i.e. kill long processes)
gcloud firestore operations
To download/upload your JSON archive from Google Cloud buckets
// list files in a bucket
gsutil ls gs://[BUCKET_NAME]
// download
gsutil cp gs://[BUCKET_NAME]/[filename] .
// upload
gsutil cp [filename] gs://[BUCKET_NAME]/[filename]
Once you setup Google Cloud to be accessible from your build scripts. It's possible to automate data migration scripts to download, transform and then upload data.
It's recommended to maintain a "migrations" document in your Firestore so you can track which reversion of the migration needs to be done.
To avoid heavy migration tasks try adding a "version" property to documents, and then use the converter callbacks on the query builder to mutate data to the latest schema on "client side". While this won't help you handle changes with Firestore rules or functions. It's often easier to make tiny changes that are mostly cosmetic.
I'm writing a script for setting up a Google Cloud project and I'd like to use Firestore. A new project, however, is by default in Datastore mode.
Switching to Firestore can easily be done using the web interface in the cloud console but I'd like to automate it. Is there any API or command line tool (gcloud? firebase?) for enabling it?
Unfortunately, as of today (30 Apr 2019), there is no API or command line to set the mode for Cloud Firestore.
Things seem to have changed by now (03 Jan 2022). I managed to set up a new Firestore with gcloud using the following commands.
Native mode is chosen by default:
gcloud app create --region=us-central
gcloud firestore databases create --region=us-central
also see the documentation for
app create and firestore database create.
Is there a command or parameter to disable FireBase's storage option? I never need it and it adds extra time to the deployment process that I don't need. Or should I just create a new project and start over?
Thanks.
Simply do not implement the storage library into your gradle file, then your app will not connect at all with Firebase Storage and your build will be a slightly faster since it won't be loading all Firebase Storage library
I have my android app running firebase cloud message using uat#gmail.com. and now, i need to change the administration to production account bbb#mycompany.com
what i have done, using bbb#mycompany.com to create new project and generate a new google-services.json and replace existing google-services.json i created with aaa#gmail.com
using android studio connect to firebase and show
Firebase: A config file "D:\xxx\xxx\google-services.json" was found, but it indicates a project ID "xxx-1vv" that you dont appear to have access to. Request access or delete the file in order to proceed with the connection process.
anyone could help??
You can transfer ownership of a Cloud (Firebase) product without having to change the way your users user the project. Please read this for details. Note that all Firebase projects are also Cloud projects, so anything you do in the Cloud console applies to Firebase.