I trying to set up a development environment from an existent project on Firebase, we are currently using Firestore as a database. I've successfully exported the Firestore data from one project to another but I can't figure out a way of importing the data, it keeps saying
ERROR: (gcloud.firestore.import) PERMISSION_DENIED: The caller does not have permission
I've tried adding the [SOURCE_PROJECT_ID]#appspot.gserviceaccount.com as a Storage admin on the bucked but got no success, also tried adding all possible permissions and accessing through the cli the services accounts.
Which permissions do I need in order to import this data?
So you have a source_project and a new dev_project, and you're trying to run gcloud firestore import for your dev_project. Some things to check:
Make sure gcloud is connected to your dev_project:
gcloud config set project DEV_PROJECT_ID
From the error, it sounds like the account you're using with gcloud doesn't have permission to run gcloud firestore import for your dev_project. Make sure your user account is a project owner or has the datastore.owner role for your dev_project.
For the bucket permissions, it's the [DEV_PROJECT_ID]#appspot.gserviceaccount.com service account that needs access to the bucket with the data files.
Also see docs for moving Firestore data between projects.
Related
I'm following the official docs on how to export and import firebase firestore data between 2 projects.
I'm able to export firestore data to a bucket. https://console.cloud.google.com/firestore/export
But I don't see that bucket when I try to import in a different firebase project. https://console.cloud.google.com/firestore/import
I gave Storage Admin permission to the destination service account i.e. dest-proj#appspot.gserviceaccount.com and both projects and this bucket is stored in the same multi-region (us-centeral)
I'm aware of the other way to import-export using gcloud shell but why this method is not working?
The console only browse buckets that exist inside your current project. If data is coming from an outside bucket, you can simply type its entire file path in the Filename field as shown on the image below.
It will succeed if the service account running the import have the right IAM permission on the separate source bucket.
I managed to move data between projects using the instructions as per this guide: Moving data between projects.
During the import, I got an error:
ERROR: (gcloud.firestore.import) PERMISSION_DENIED: The Cloud Firestore API has not been used in project XXX before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/firestore.googleapis.com/overview?project=project-name and then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.
I enabled Cloud Firestore API and completed the import process.
Now, on Firebase console, when I go the Cloud Firestore page of the destination project, I see this message:
This project is set up to use Cloud Firestore in Datastore mode. This mode can only be accessed from Google Cloud Platform
The import command creates the Cloud Firestore database in Datastore mode. And it's not possible to switch it back Native mode as the first write on the database executed.
How can I import the data in Cloud Firestore Native mode?
I understand the import operation coerced the type of the new project database to Firestore in Datastore mode instead of Firestore Native, which was the mode of the database you exported the data from.
I have opened the following issue tracker so that this behavior could be investigated in further detail. I would recommend to star the issue and add your mail in the CC list so that you're aware of the updates made therein.
Separately, once the first write has been commited to the database it's no longer possible to change the database mode, therefore you'll have to create another project to use Firestore in Native mode.
To that effect I have tested the following workaround so that you could succesfully make the import in a new project:
Create a new project. Then go to the Firebase Console and create a Firestore database.
Using the console, create a mock collection and a mock document within it.
Proceed with the import as outlined in the documentation.
After completing these steps you could delete the mock collection and keep with your development as usual.
I am running scripts to interact with Firestore (e.g. creating a doc). I've provided a service account key file to do so (tried with key files from Firebase and GCloud) - scripts run the same.
I am aware that Firebase is now built / closely linked to GCloud so I am expecting the key files to be similar.
However, the docs recommend initializing the sdk with a key from Firebase console when adding Firebase admin to your server and from GCloud Console for unit testing.
So here are my questions:
Why recommend 2 different ways to generate them? Is one more appropriate than the other?
Do service account key files generated in Firebase (1) and GCloud (2) consoles differ in terms of scope? Do they allow the same operations? Are they granted equivalent permissions?
Firebase Console > Project Settings > Service Accounts > Firebase Admin SDK > Create new private key
https://console.firebase.google.com/project/[my-project-id]/settings/serviceaccounts/adminsdk
Google Cloud Console > IAM & Admin > Service Accounts > App Engine default service account > Create key
https://console.cloud.google.com/iam-admin/serviceaccounts?project=[my-project-id]
The service account and credentials provided by the Firebase console are no different than one that you'd create in the Cloud console. Firebase is just making it easier to get started, so you don't have to learn the Cloud console. If you want to use the one provided by Firebase, fine. If you want to use one you create and configure in the Cloud console, fine. The Firebase service account should be visible in the Cloud console just like any other. When it comes to assigning permissions to individual products, you can still choose either one to work with. It's up to you.
I have created/forked a lil Google Apps Script Library to manage Firebase Firestore and Firebase Remote Config called FirebaseGoogleAppsScript. The goal is to simply manage the contents of your collections in an apps script as well as update your remote config.
My issue is I can't get the a service account to do both.
Firebase creates two service accounts upon creating a project:
The first is listed in the Firebase Console -> Project Settings -> Service Accounts. This one I use within my cloud functions to retrieve the Remote Config just fine. However in the Apps Script Project it is unable to retrieve any data from firestore. I tried adding all kinds of roles including Owner and Editor yet no firestore data, but I still can get the RemoteConfig.
The second is only visible in the GCP service accounts and has the title: Firebase Admin SDK Service Agent with the roles Firebase Admin SDK Administrator Service Agent and Service Account Token Creator. This one is able to retrieve all the data from firestore within an Apps Script Project. However in the apps script project I can't get it to retrieve the RemoteConfig even if I add the role Firebase Remote Config Admin.
I have also made my own service account which was able to get the Remote config and just about everything else from Firebase except the Firestore data. Seems only the one service account created by Firebase is able to get any data.
To recreate the issue simply deploy my lil FirebaseGoogleAppsScript project and associate it to the same GCP project Firebase is connected to. There is a test file in it which can recreate the issue assuming you have some data in RemoteConfig and a collection called posts with some docs.
What the heck is going on here? Why can't I make a service account who can access Firestore and RemoteConfig? Any ideas on what to do to create a proper role to do both? Do I really have to use two separate service accounts?
I am trying out the new export feature of the FireStore as shown on this link, and I keep getting error message:
gcloud alpha firestore export gs://backup --async
ERROR: (gcloud.alpha.firestore.export) PERMISSION_DENIED: Service account does not have access to Google Cloud Storage file: /backup. See https://cloud.google.com/datastore/docs/export-import-entities#permissions for a list of permissions needed. Error details: projectABC#appspot.gserviceaccount.com does not have storage.buckets.get access to backup.
I've run the command to add my service account as a storage.admin, and that command succeed:
gcloud projects add-iam-policy-binding projectABC --member='serviceAccount: projectABC#appspot.gserviceaccount.com' --role='roles/storage.admin'
Could someone help?
You can't select an arbitrary bucket name for your backup. You're using "gs://backup". Instead, you need to use the url for an actual bucket that you've created an you fully control.
Give storage admin access to IAM role