I've tried out the export data from Firebase to 'backup' my database but much to my surprise it didn't export the sub-collections, despite them being uniquely named. Is there a way to export the entire database?
I know there are NPM packages that can do this but it doesn't export to the binary format that the GCP exports to which is more space efficient.
EDIT:
I've tried to export the data via the scheduled cloud function and the GCP console. I followed the instructions exactly and didn't change anything.
They both uploaded a folder with the contents in the images below.
The root directory:
all_namespaces/all_kinds
I imported the 2020-11-05T14:40:04_75653.overall_export_metadata via the GCP console and sure enough all the top level collections and documents came back, but all the sub-collections were not there. I expected that all the collections and subcollections would be restored from the import.
So in summary I tried to export the data via two methods and upload via one.
Here's the scheduled cloud function:
const functions = require('firebase-functions');
const firestore = require('#google-cloud/firestore');
const client = new firestore.v1.FirestoreAdminClient();
// Replace BUCKET_NAME
const bucket = 'gs://BUCKET_NAME';
exports.scheduledFirestoreExport = functions.pubsub
.schedule('every 24 hours')
.onRun((context) => {
const projectId = process.env.GCP_PROJECT || process.env.GCLOUD_PROJECT;
const databaseName =
client.databasePath(projectId, '(default)');
return client.exportDocuments({
name: databaseName,
outputUriPrefix: bucket,
// Leave collectionIds empty to export all collections
// or set to a list of collection IDs to export,
// collectionIds: ['users', 'posts']
collectionIds: []
})
.then(responses => {
const response = responses[0];
console.log(`Operation Name: ${response['name']}`);
})
.catch(err => {
console.error(err);
throw new Error('Export operation failed');
});
});
So with the above code, I'd expect it to export everything in firestore. For example if I had the sub-collection:
orders/{order}/items
I'd expect that when I import the data via the GCP console, I get back the sub-collection listed above. However, I only get back the top-level. i.e.
orders/{order}
Related
I have very simple cloud function truggers firestore document onCreate, when i deploy this function I'm getting Error
Functions deploy had errors with the following functions:
makeUppercase
In cloud logs
ERROR: error fetching storage source: generic::unknown: retry budget exhausted (3 attempts): fetching gcs source: unpacking source from gcs: source fetch container exited with non-zero status: 1
My function .ts
import * as functions from "firebase-functions";
exports.makeUppercase = functions.firestore.document("/clients/{documentId}")
.onCreate((snap, context) => {
// Grab the current value of what was written to Firestore.
const original = snap.data().original;
functions.logger.log("Uppercasing", context.params.documentId, original);
const uppercase = original.toUpperCase();
return snap.ref.set({uppercase}, {merge: true});
});
Also please check my firestore
In package JSON node version is 14. I don't know what went wrong, I have been trying this couple of hours and always same Error.
I have the following https callable cloud function that imports all documents found in a backup.
const path = `${timestamp}`;
const projectId = await auth.getProjectId();
// we change the action for importDocuments
const url = `https://firestore.googleapis.com/v1/projects/${projectId}/databases/(default):importDocuments`;
const backup_route = `gs://${BUCKET_NAME}/${path}`;
return client.request({
url,
method: 'POST',
data: {
inputUriPrefix: backup_route,
}
}).then(async (res: any) => {
console.log(`Began backup restore from folder ${backup_route}`);
return Promise.resolve(res.data.name);
}).catch(async (e) => {
return Promise.reject(new functions.https.HttpsError('internal', e.message));
})
I use to this function to restore the database to the exact state it was when it was exported.
The problem is that the import operation, does not affect documents that are not found in the export. So new documents added after the export will remain in the database.
The following quote from the documentation explains this behaviour:
If a document in your database is not affected by an import, it will remain in your database after the import.
Is deleting the whole database before starting the import operation my only option? I can not find an operation that achieves the desired behaviour.
The provided export mechansim isn't meant for what what most people would call a "backup" that would be "restored" in the event of data loss. It's simply an export. It useful for making a copy of a database to be imported elsewhere, making it easy to duplicate a database across environments without having to write a lot of code.
If you want a fresh copy of the database from an import with no other documents, you will have to wipe out what's there before importing.
I am currently exporting my Firestore data to a timestamped subdirectory in a bucket using the code below. It runs in a scheduled (firebase) cloud function.
import { client } from "../firebase-admin-client";
import { getExportCollectionList } from "./helpers";
export async function createFirestoreExport() {
const bucket = `gs://${process.env.GCLOUD_PROJECT}_backups/firestore`;
const timestamp = new Date().toISOString();
try {
const databaseName = client.databasePath(
process.env.GCLOUD_PROJECT,
"(default)"
);
const collectionsToBackup = await getExportCollectionList();
const responses = await client.exportDocuments({
name: databaseName,
outputUriPrefix: `${bucket}/${timestamp}`,
collectionIds: collectionsToBackup
});
const response = responses[0];
console.log(`Successfully scheduled export operation: ${response.name}`);
return response;
} catch (err) {
console.error(err);
throw new Error(`Export operation failed: ${err.message}`);
}
}
This has the advantage that you will get a list of exports each under its own timestamp. For example my script which imports the data to BigQuery will figure out the latest timestamp and load the data from there.
This approach has some drawbacks I think. There is no way to list directories in cloud storage. In order to figure out the latest timestamp I have to fetch a list of all objects (meaning all files of all exports) and then split and reduce these paths to extract a list of unique timestamps.
When keeping exports of months of data I can imagine this operation becomes quite inefficient.
So I'm thinking of storing the backups as versioned files, but I have no experience with this. If I understand correctly, with versioned files I can export the data each time to the same location, and (configurable) x versions of the files will persist in the bucket. You then have the ability to read any of available the versions if needed.
The bigquery import script could then load the data from the same location each time, automatically getting the latest export data.
So in my mind this would simplify things. Are there any drawbacks to using versioned files for backup data? Is this a recommended approach?
When using Firestore in Cloud Functions, how to use the Create a missing index through an error message feature?
Basically, after I use a new Firestore query in my Cloud Function, where should I expect to find the "error message [that] includes a direct link to create the missing index"?
Update: add sample code based on a comment suggestion
Assume the query below wasn't used before. After I deploy the function to Cloud Functions and use it, where can I find the automatically-created link for creating the missing index?
import * as admin from 'firebase-admin';
import * as functions from 'firebase-functions';
admin.initializeApp(functions.config().firebase);
const db = admin.firestore();
export const posts = functions.https.onCall(async (
data: { uid: string },
context
) => {
// ...
const querySnapshot = await postsRef
.where('uid', '==', data.uid)
.orderBy('timestamp', 'desc')
.get();
// ...
});
When you perform the query, the resulting promise will become rejected. Your code will need to catch that error from the rejected promise, then log the error object. The url will be in the log message. (I'm assuming you're using node here, but the same error handling will apply to any language using its conventions for capturing and logging errors.)
According to the following google I/O (2019) post of the firebase team the new emulator allows us to combine firebase/database plus cloud function to fully simulate our firebase server codes. That should also mean we should be able to write tests for it.
we’re releasing a brand new Cloud Functions emulator that can also
communicate with the Cloud Firestore emulator. So if you want to build
a function that triggers upon a Firestore document update and writes
data back to the database you can code and test that entire flow
locally on your laptop (Source: Firebase Blog Entry)
I could find multiple resources looking/describing each individual simulation, but no all together
Unit Testing Cloud Function
Emulate Database writes
Emulate Firestore writes
To setup a test environment for cloud functions that allows you to simulate read/write and setup test data you have to do the following. Keep in mind, this really simulated/triggers cloud functions. So after you write into firestore, you need to wait a bit until the cloud function is done writing/processing, before you can read the assert the data.
An example repo with the code below can be found here: https://github.com/BrandiATMuhkuh/jaipuna-42-firebase-emulator .
Preconditions
I assume at this point you have a firebase project set up, with a functions folder and index.js in it. The tests will later be inside the functions/test folder. If you don't have project setup use firebase init to setup a project.
Install Dependencies
First add/install the following dependencies: mocha, #firebase/rules-unit-testing, firebase-functions-test, firebase-functions, firebase-admin, firebase-tools into the functions/package.json NOT the root folder.
cd "YOUR-LOCAL-EMULATOR"/functions (for example cd C:\Users\User\Documents\FirebaseLocal\functions)
npm install --save-dev mocha
npm install --save-dev firebase-functions-test
npm install --save-dev #firebase/rules-unit-testing
npm install firebase-admin
npm install firebase-tools
Replace all jaipuna-42-firebase-emulator names
It's very important that you use your own project-id. It must be the project-id of your own project and must exists. Fake ids won't work. So search for all jaipuna-42-firebase-emulator in the code below and replace it with your project-id.
index.js for an example cloud function
// functions/index.js
const functions = require("firebase-functions");
const admin = require("firebase-admin");
// init the database
admin.initializeApp(functions.config().firebase);
let fsDB = admin.firestore();
const heartOfGoldRef = admin
.firestore()
.collection("spaceShip")
.doc("Heart-of-Gold");
exports.addCrewMemeber = functions.firestore.document("characters/{characterId}").onCreate(async (snap, context) => {
console.log("characters", snap.id);
// before doing anything we need to make sure no other cloud function worked on the assignment already
// don't forget, cloud functions promise an "at least once" approache. So it could be multiple
// cloud functions work on it. (FYI: this is called "idempotent")
return fsDB.runTransaction(async t => {
// Let's load the current character and the ship
const [characterSnap, shipSnap] = await t.getAll(snap.ref, heartOfGoldRef);
// Let's get the data
const character = characterSnap.data();
const ship = shipSnap.data();
// set the crew members and count
ship.crew = [...ship.crew, context.params.characterId];
ship.crewCount = ship.crewCount + 1;
// update character space status
character.inSpace = true;
// let's save to the DB
await Promise.all([t.set(snap.ref, character), t.set(heartOfGoldRef, ship)]);
});
});
mocha test file index.test.js
// functions/test/index.test.js
// START with: yarn firebase emulators:exec "yarn test --exit"
// important, project ID must be the same as we currently test
// At the top of test/index.test.js
require("firebase-functions-test")();
const assert = require("assert");
const firebase = require("#firebase/testing");
// must be the same as the project ID of the current firebase project.
// I belive this is mostly because the AUTH system still has to connect to firebase (googles servers)
const projectId = "jaipuna-42-firebase-emulator";
const admin = firebase.initializeAdminApp({ projectId });
beforeEach(async function() {
this.timeout(0);
await firebase.clearFirestoreData({ projectId });
});
async function snooz(time = 3000) {
return new Promise(resolve => {
setTimeout(e => {
resolve();
}, time);
});
}
it("Add Crew Members", async function() {
this.timeout(0);
const heartOfGold = admin
.firestore()
.collection("spaceShip")
.doc("Heart-of-Gold");
const trillianRef = admin
.firestore()
.collection("characters")
.doc("Trillian");
// init crew members of the Heart of Gold
await heartOfGold.set({
crew: [],
crewCount: 0,
});
// save the character Trillian to the DB
const trillianData = { name: "Trillian", inSpace: false };
await trillianRef.set(trillianData);
// wait until the CF is done.
await snooz();
// check if the crew size has change
const heart = await heartOfGold.get();
const trillian = await trillianRef.get();
console.log("heart", heart.data());
console.log("trillian", trillian.data());
// at this point the Heart of Gold has one crew member and trillian is in space
assert.deepStrictEqual(heart.data().crewCount, 1, "Crew Members");
assert.deepStrictEqual(trillian.data().inSpace, true, "In Space");
});
run the test
To run the tests and emulator in one go, we navigate into the functions folder and write yarn firebase emulators:exec "yarn test --exit". This command can also be used in your CI pipeline. Or you can use npm test instead.
If it all worked, you should see the following output
√ Add Crew Members (5413ms)
1 passing (8S)
For anyone struggling with testing firestore triggers, I've made an example repository that will hopefully help other people.
https://github.com/benwinding/example-jest-firestore-triggers
It uses jest and the local firebase emulator.