We are attempting to deploy Firebase Functions, Rules, and Indexes to multiple projects for tenant isolation of data. We are attempting to use Google Cloud Source Repository, but Cloud Build in each project does not have the ability to connect to the Central Project Source Repository - and we have added the required Source Repo IAM rules on our Cloud Build service account.
What is a good solution for deploying our Firebase Functions, Rules, and Indexes from a central repository?
You can't access to event from a source repository in another project mode. Thereby, you can't set up a trigger on the source repository that don't belong to your project
So, you can imagine this workaround to achieve what you want
Source Project
Create a PubSub topic (push-event for example)
Configure the trigger that you want which run a Cloud Build
In this Cloud Build, format a JSON message with all the push data that you want (commit SHA, type of event, repo name,...) and publish this message to push-event topic
Tenant Projects
Create a cloud function that trigger Cloud Build (focus on that bellow)
Create a push subscription on the pubsub push-event topic located in the source projet (be sure that the current account that run the terraform has the roles topicViewer and topicSubscriber on the push-event topic (or on the source project))
Note: the first thing that you have to do in the Cloud Build execution is to clone the source repository because you won't have the data automatically downloaded (get the correct source according with the branch, tag or pull event.)
Cloud Functions
I don't know your dev language, but the principle is to perform an API call to the Cloud Build API to launch the build. This API call require the content of the cloudbuild.json. So, in the cloud function,
You can also clone the source repo (grant the reader permission) in the /tmp directory and then read the cloudbuild.json file to run in your Cloud Build. But it could be difficult in case of branch, tag, or pull context.
You can publish, in addition of other data in the PubSub message published in the source project, the content of the cloudbuild.json file to run by the Cloud Functions in the tenant project.
Related
Following the Firebase examples to create and deploy a function, I keep failing at the deploy phase.
The error, with --debug enabled, shows:
<<< [apiv2][body] POST
https://cloudfunctions.googleapis.com/v1/projects/actus-poc2/locations/us-central1/functions
{"error":{"code":403,"message":"Cloud Functions uses Artifact Registry
to store function docker images. Artifact Registry API is not enabled
in your project. To enable the API, visit
https://console.developers.google.com/apis/api/artifactregistry.googleapis.com/overview?project=...
Now I was expecting to be able to stay within the confines of the firebase console but this message seems to imply I need to open the Google Cloud Console to enable additional permissions.
Should the code samples better document this?
Or is this a recent change in firebase functions that breaks many of the existing examples?
I need to open the Google Cloud Console to enable additional permissions.
The reason why you need to use the Google Cloud Console is because Cloud Functions for Firebase relies on some Google Cloud services. See.
Function deployments with Firebase CLI 11.2.0 and higher rely on Cloud Build and Artifact Registry.
is this a recent change in firebase functions that breaks many of the existing examples?
Deployments to older versions also do rely on some Google Cloud services. Deployments to older versions use Cloud Build in the same way, but rely on Container Registry and Cloud Storage for storage instead of Artifact Registry.
Should the code samples better document this?
If you do think an update to said documentation could be helpful, here is more about opening Feature requests.
I want to use Firebase cloud functions but when I try to init exists project I get error.
This is the error "Error: HTTP Error: 403, The caller does not have permission"
I created project before and then I moved project folder. When I had tried deploy, it wasn't work. I don't remember which error I got. And then I tried init same project but I got this error.
I tried logout and login again but it wasn't work.
I deleted "firebase-tools" but it wasn't work too.
And after I tried on different computer and I got same error.
And also I have Firebase Admin permission.
The firebase init command initializes/enables the project to use Cloud Functions for Firebase. Owners or Editors should use this command. Owners/Editors have higher permissions which allows APIs to be enabled (e.g. allowing the project to create VMs, Cloud Functions, or create a NoSQL Database).
Therefore I think you should perform this operation with Owner/Editor role.
You need to have Owner or Editor IAM permission on the GCP project to use firebase init command.
The firebase init command initializes/enables the project to use Cloud Functions/Hosting for Firebase. Owners/Editors have higher permissions that allow APIs to be enabled etc.
it is a first-time task when you starting the development, and then init command can be avoided if you have set up your cloud functions code in VCS for multiple developers or you.
i.e, Usually after writing code, you will replicate the project to other developers, who will then also write code and use firebase use yourprojectid further.
How do we know that what cloud functions are running on Firestore with respect to the GIT branch? Is there versioning we can reference? I mean can we know that what is the branch of the cloud function that is deployed on Fire-store?
Cloud Functions doesn't know anything about your git. If you want to add some information to your deployment, perhaps added to your source code, or in another file that you deploy, you are free to arrange for that yourself. None of the provided tooling will do this for you. Once you've added the information you need to your deployment, you can view the source code for the current deployment in the Google Cloud console.
I have a Firebase project which basically have two environments: Staging and Production. The way I organized them is by creating different Firebase projects. Each of my projects uses Firebase Cloud Functions and Firestore. Except for that, I have each of the projects associated with a specific GIT branch. Both of the branches are integrated into CI/CD pipeline in Google Cloud Build.
So, in order to make it absolutely clear, I will share a simple diagram:
As you can see, I have the source code for the cloud functions under source control and there's nothing to worry about there. The issue comes in when I have the following situation:
A Firestore schema change is present on Staging
Cloud function (on Staging) is adjusted to the new schema.
Merge staging branch into production.
Due to the old Firestore schema on production, the new functions there won't work as expected.
In order to work around it, I need to manually go to the production Firestore instance and adjust the schema there (there's a risk to mess up production data).
In the perfect case, I would have that operation automated and existing project data would be adjusted to a new schema which comes in dynamically after merge.
Is that possible somehow? Something like migrations in .NET Core.
Cloud Firestore is schema-less - documents have no enforced schema. Code is able to write whatever fields it wants at any time that it wants. (For web and mobile clients, this is gated by security rules, but for backend code, there are no restrictions.) As such, there is no such thing as a formal migration in Cloud Firestore.
The "schema" of your documents is effectively defined by your code that reads and writes those documents. This means that migrating a data to a new format means that you're going to have to write code to perform the required changes. There is really no easy way around this. All you can really do is design your updates so that they are not disruptive to existing code when it comes time to move them to another environment. This means your code should be resilient to breaking changes, or simply do not perform breaking changes until after all code has been updated to deal with those changes.
You have to use Google Cloud to download an archive of the Firestore data. Run a migration script yourself on the archive, and then upload the archive to restore your Firestore database.
https://cloud.google.com/firestore/docs/manage-data/export-import
Google Cloud gives you a lot of command line access for managing your Firestore service.
// manage indexes
gcloud firestore indexes
// export all data to a bucket
gcloud firestore export gs://[BUCKET_NAME]
// import data from a bucket
gcloud firestore import gs://[BUCKET_NAME]/[filename]
// manage admin "functions" currently running (i.e. kill long processes)
gcloud firestore operations
To download/upload your JSON archive from Google Cloud buckets
// list files in a bucket
gsutil ls gs://[BUCKET_NAME]
// download
gsutil cp gs://[BUCKET_NAME]/[filename] .
// upload
gsutil cp [filename] gs://[BUCKET_NAME]/[filename]
Once you setup Google Cloud to be accessible from your build scripts. It's possible to automate data migration scripts to download, transform and then upload data.
It's recommended to maintain a "migrations" document in your Firestore so you can track which reversion of the migration needs to be done.
To avoid heavy migration tasks try adding a "version" property to documents, and then use the converter callbacks on the query builder to mutate data to the latest schema on "client side". While this won't help you handle changes with Firestore rules or functions. It's often easier to make tiny changes that are mostly cosmetic.
Can someone please advise how to setup a backup for Files in Firebase storage. I am able to make a backup of Database but not sure how to setup a regular backup for files (I have images) in firebase storage.
How to make local backup of Firebase Storage
There is no built-in method via Firebase. However, since Firebase uses Google Cloud Storage behind the scenes for Firebase Storage it's possible to use the gutils Tool.
Prerequisites
Make sure Python (2.7.9+) is installed on your machine python -V
Go to the Google Cloud SDK page and follow the directions to download and install Google Cloud SKD on your OS.
Steps
At the end of the Google SDK installation you should have run gcloud init. This will ask you to select your project and authenticate you. Since Firebase uses Google Cloud Platform behind the scenes your Firebase project should be available as a choice.
In order for Google Cloud Utils to download the files that were uploaded with Firebase permissions you need to give your account Firebase Privileges. Go to the IAM page and select your email address you signed into cloud init with. In the list of available permissions you need to select Firebase Rules System from the Other category.
Get your Google Storage URL from the Firebase Storage Page in the dashboard (Towards the top) Should look something like this: gs://<bucket_name>
In command line on your local machine navigate to the folder you want to do a local backup to. Make sure you are in the folder you want as the following command will download all files right there in current folder
Run the gutil command gsutil -m cp -R gs://<bucket_name> .
-m enables multithreading for faster downloads if you have many files.
cp is the copy command
-R is recursive. If enabled it will download all files and folders in the specified tree.
You're done! This will run for some time depending on the size of your storage.
This can be used to also make a copy(backup) to another Google Cloud Storage Bucket or AWS etc.
Use Google Cloud Transfer Service.
Select your current project
Create Transfer Job
Select source (storage bucket url)
Select destination (click browse and create new bucket)
Use created bucket URL as destination
Configure transfer settings (This is where you can schedule how often the backup runs.)
Click "Create"
If you follow the wizard in the link it will guide you through pretty easily.
There is no built-in backup feature in Cloud Storage for Firebase.
But since it is built on top of Google Cloud Storage, any backup solution for GCS can work for Firebase too. Typically this will involve creating a separate bucket that is the target of the regular bucket where you store/read files.