I'm hoping to run integration and cypress browser tests against my React app that uses Firestore. There's a local emulator, but it only is targeted at testing security rules right now (see Local offline development with Firestore).
I can point the tests at a 'development' instance of the database, but that doesn't work if I want to run this in CI across all open pull requests.
In other worlds I’d just create a fresh database and drop it afterwards, but you can only have one firestore database per google cloud project.
Do you prefix the table names everywhere? Automate creating new google projects and dropping them again? Something else obvious I'm missing?
Related
The Main part
I am trying to create a comfortable workflow for myself. I just want to know is there any way to automate deploys from GitHub Repo to Firebase hosting when I commit changes in the repo's main branch at my desktop and push origin to GitHub?
...possibly using "GitHub actions"...
Context and what I've tried
As on 15-Jan-2022 there are two ways I found of deploying website files to firebase hosting.
The de facto firebase cli way, where you use firebase deploy command to push files to firebase hosting.
The automated way using github actions, where the site is deployed automatically every time a pull request is created.
The first one is tedious, the second one is not my cup of cake. Especially because the workflow I want to setup is not too much git oriented, I don't use "pull requests" as far as I understand. I just commit changes directly in the main branch.
(That is so because, I am not a programmer, I just trying to create a blogging site using the Firebase hosting + GitHub repo)
I also referred https://medium.com/firebase-developers/the-comprehensive-guide-to-github-actions-and-firebase-hosting-818502d86c31
Use Deploy to live & preview channels via GitHub pull requests. It will work if you use pull requests to review changes or if you push commits directly to the primary branch.
I would recommend that you use pull requests though. It's a great flow to double check your changes and with the Firebase action it will deploy a temporary preview of the the changes so you can validate everything looks correct.
I am using Firebase for a long time (since 2018) and loving it. In that time there was not Location southamerica-east1 (São Paulo). Now I would like to store the project (web app, cloud function, and database) in southamerica to reduce cost and make it near to my end-users (also based in Brazil).
I have source control, all environment parameters values stored in Custom Environment Variables. The application works fine when no data is found. No concerns with backup data. No problem about downtime. This is not a critical app.
Anyway, I can't delete the application because I already have some users logged in there and IoT devices sending data through PubSub.
How can I rebuild my Firebase/Firestore/Web application/Function from the ground up, and make sure the new location is southamerica? If possible, I need to keep user and passwords, and web
Looking forward, (I don't think moving the bucked location would be the best solution here) but based on this page Select locations for your project I can't update the location, but since it is based on bucked location, if it doesn't break the project, I will use Google Cloud Transfer Page to Moving and renaming buckets
May is it a better solution than rebuild the app (Firebase/Firestore/Web application/Function)?
May I break my Firestore database or cloud function or web app?
May I lost my project domain or any other related URL parameter like authDomain, databaseURL, storageBucket?
May I need to update some web app parameter after the change?
They cannot be moved at present and migrating data is a manual process. Difficulty varies by product.
General guidance
Do not delete the old project before fully migrated.
Hosting
This migration is nearly trivial, with the understanding that there is likely a minor service interruption while moving custom domains.
Deploy to the new site
CNAME your custom domain to the new site (myproject.firebaseapp.com)
Delete custom domain from old site
Add custom domain to new site
Cloud Functions
This migration is trivial.
Create a local directory for your new project
Run firebase init and set up project normally (enable Functions)
Copy your Functions code into the new project's functions/ directory
Deploy to the new project
Database
This migration is tricky, difficult, and highly specific to your use case and tolerance for downtime. What follows is a general template to adapt.
Reference docs for import/export: Firestore import/export, Realtime Database backups
In the old project:
Lock the database using security rules to prevent changes
Export existing database
In the new project:
Import the database backup
You probably need to migrate existing users (see account export/import) as well so user ids stored in your DB will still reference the correct accounts
Point existing apps to the new project
If downtime is not an option, or if you'll be deploying a new mobile app version and need time for changes to propagate, then you'll need to set up a dual write model:
Dual sync: Create a Cloud Function on both the new and old database that duplicate all create/update/delete operations on the respective partner endpoint.
Sync pre-existing data: Perform the export/import process as above on all data created before the dual sync was implemented, excluding the step to lock the old database
Shut down your old mobile app version (once enough accounts have migrated)
Shut down the dual sync Functions and turn down the old site
Based in your information, the main issue is Firestore because other products are globally balanced like Cloud IoT Core and Hosting (these can't be configured on a specific region)
Other products like Functions can be redeployed with the same code and name into another region.
I think that you can create another project only to move the database to the new region and configure all Cloud resources to reach the new located database.
As a caveat, you need to add another domain/subdomain and create new credentials to work with the new project; this step can´t be skipped because it is required for authentication.
On the application side you can add the access to the new database
In case you need assistance during your migration you can start a case with GCP/Firestore support.
This is a hard pill to swallow, but maybe the costs and the time to migrate to another region will be higher than keeping your application as is working today.
I have a Firebase project which basically have two environments: Staging and Production. The way I organized them is by creating different Firebase projects. Each of my projects uses Firebase Cloud Functions and Firestore. Except for that, I have each of the projects associated with a specific GIT branch. Both of the branches are integrated into CI/CD pipeline in Google Cloud Build.
So, in order to make it absolutely clear, I will share a simple diagram:
As you can see, I have the source code for the cloud functions under source control and there's nothing to worry about there. The issue comes in when I have the following situation:
A Firestore schema change is present on Staging
Cloud function (on Staging) is adjusted to the new schema.
Merge staging branch into production.
Due to the old Firestore schema on production, the new functions there won't work as expected.
In order to work around it, I need to manually go to the production Firestore instance and adjust the schema there (there's a risk to mess up production data).
In the perfect case, I would have that operation automated and existing project data would be adjusted to a new schema which comes in dynamically after merge.
Is that possible somehow? Something like migrations in .NET Core.
Cloud Firestore is schema-less - documents have no enforced schema. Code is able to write whatever fields it wants at any time that it wants. (For web and mobile clients, this is gated by security rules, but for backend code, there are no restrictions.) As such, there is no such thing as a formal migration in Cloud Firestore.
The "schema" of your documents is effectively defined by your code that reads and writes those documents. This means that migrating a data to a new format means that you're going to have to write code to perform the required changes. There is really no easy way around this. All you can really do is design your updates so that they are not disruptive to existing code when it comes time to move them to another environment. This means your code should be resilient to breaking changes, or simply do not perform breaking changes until after all code has been updated to deal with those changes.
You have to use Google Cloud to download an archive of the Firestore data. Run a migration script yourself on the archive, and then upload the archive to restore your Firestore database.
https://cloud.google.com/firestore/docs/manage-data/export-import
Google Cloud gives you a lot of command line access for managing your Firestore service.
// manage indexes
gcloud firestore indexes
// export all data to a bucket
gcloud firestore export gs://[BUCKET_NAME]
// import data from a bucket
gcloud firestore import gs://[BUCKET_NAME]/[filename]
// manage admin "functions" currently running (i.e. kill long processes)
gcloud firestore operations
To download/upload your JSON archive from Google Cloud buckets
// list files in a bucket
gsutil ls gs://[BUCKET_NAME]
// download
gsutil cp gs://[BUCKET_NAME]/[filename] .
// upload
gsutil cp [filename] gs://[BUCKET_NAME]/[filename]
Once you setup Google Cloud to be accessible from your build scripts. It's possible to automate data migration scripts to download, transform and then upload data.
It's recommended to maintain a "migrations" document in your Firestore so you can track which reversion of the migration needs to be done.
To avoid heavy migration tasks try adding a "version" property to documents, and then use the converter callbacks on the query builder to mutate data to the latest schema on "client side". While this won't help you handle changes with Firestore rules or functions. It's often easier to make tiny changes that are mostly cosmetic.
What's the best way to work with multiple environments/projects on firebase?
I can switch between firebase projects using the the CLI.
I see here how to add environment variables to a firebase project and access them through firebase-functions's .config() method.
Is there a way to do something similar on the client-side when using firebase hosting.
For example: I'm using Algolia to run searches. I have firebase-functions to keep the indexes up to date, and run the searches from the client. Both functions and the hosted content need to point to the right Algolia project depending on the environment. I'd like to tie both configs to the same switch; firebase use staging vs firebase use production, for example. What's the best way to go about that?
Is there a way to delete an app via Firebase CLI?
Our CI creates a temporary Firebase app for a feature branch (based on the CI build number), and then runs tests on it. At the end it needs to delete the app. I'm not entirely sure how this is done. firebase disable:hosting just disables hosting and does not delete the app. This is similar to how we would have used Heroku review apps or Heroku forked apps. A similar CI workflow can also be achieved on Google App engine via versions.
Any pointers would be much appreciated.
The correct command is firebase hosting:disable, which make your site offline. You can not remove your project, instead you can overwrite it by creating a new one with firebase init.
More info here https://firebase.google.com/docs/cli/.
There is currently no public API to delete a Firebase backend.
The recommended practice is to use the same database for testing and (if needed) put each run in its own node under the root. So instead of creating/deleting a new database, you're just create/deleting a node in a single database.
Using the command firebase use <alias> --unaliasand then delete the project from the console https://console.firebase.google.com/
Check before the list of commands with the command -h