Backup of Firestore like in Firebase Realtime Database? [duplicate] - firebase

Does the google firestore database service provides a backup?
If so, how do I backup the database and how do I restore in case of an error?

Update: It is now possible to backup and restore Firebase Firestore using Cloud Firestore managed export and import service
You do it by:
Create a Cloud Storage bucket for your project - Make sure it's a regional in us-central1 or 2 / multi regional type of bucket
Set up gcloud for your project using gcloud config set project [PROJECT_ID]
EXPORT
Export all by calling
gcloud firestore export gs://[BUCKET_NAME]
Or Export a specific collection using
gcloud firestore export gs://[BUCKET_NAME] --collection-ids='[COLLECTION_ID_1]','[COLLECTION_ID_2]'
IMPORT
Import all by calling
gcloud firestore import gs://[BUCKET_NAME]/[EXPORT_PREFIX]/
where [BUCKET_NAME] and [EXPORT_PREFIX] point to the location of your export files. For example - gcloud firestore import gs://exports-bucket/2017-05-25T23:54:39_76544/
Import a specific collection by calling:
gcloud firestore import --collection-ids='[COLLECTION_ID_1]','[COLLECTION_ID_2]' gs://[BUCKET_NAME]/[EXPORT_PREFIX]/
Full instructions are available here:
https://firebase.google.com/docs/firestore/manage-data/export-import

Update July 2018: Cloud Firestore now supports managed import and export of data. See the documentation for more details:
https://firebase.google.com/docs/firestore/manage-data/export-import
[Googler here] No, right now we do not offer a managed backup or import/export service. This is something we will definitely offer in the future, we just did not get it ready for the initial Beta release.
The best way to back up right now is to write your own script using our Java/Python/Node.js/Go server SDKs, it should be fairly straightforward to download all documents from each collection and write them back if you need to.

https://www.npmjs.com/package/firestore-backup
Is a tool that has been created to do just this.
(I did not create it, just adding it here as people will find this question)

Local backups
firestore-import-export
This is the one I use for "one-off", local backups, and what I generally recommend. (most straight-forward if you want a single JSON file)
firestore-backup-restore
Drawbacks:
Hasn't been updated in a long time.
Additional options: (not recommended)
python-firebase-admin-firestore-backup
Drawbacks:
Backup only; cannot restore from the backups it creates.
Hasn't been updated in a long time.
firestore-backup
Drawbacks:
Backup only; cannot restore from the backups it creates.
Cloud backups
The official gcloud backup commands.
Drawbacks:
The backup files are difficult/infeasible to parse. (update: how to convert to a json file)
You have to set up the gcloud cli. (update: or use the cloud shell to run the commands)
It doesn't backup locally; instead, it backs up to the cloud, which you can then download. (could also be considered an advantage, depending on what you want)
Note that for the gcloud backup commands, you have multiple options on how to schedule them to run automatically. A few options are shown here.

I am using the following work-around in order to have daily firestore backups:
I installed this globally: https://www.npmjs.com/package/firestore-backup-restore
I have a cron job that looks like this:
0 12 * * * cd ~/my/backup/script/folder && ./backup-script.sh
And my backup-script.sh looks like this:
#!/bin/sh
. ~/.bash_profile
export PATH=/usr/local/bin/
dt=$(/bin/date '+%d-%m-%Y %H:%M:%S');
echo "starting backup for $dt"
firestore-backup-restore -a ~/path/to/account/credentials/file.json -B ./backups/"$dt"

I've written a tool that traverses the collections/documents of the database and exports everything into a single json file. Plus, it will import the same structure as well (helpful for cloning/moving Firestore databases). It's published as an NPM package. Feel free to try it and give some feedback.
https://www.npmjs.com/package/node-firestore-import-export

I had the same issue and created a small npm package which allows you to create a scheduled backup with Cloud Functions. It uses the new import/export feature of Firestore.
const firestoreBackup = require('simple-firestore-backup')
exports.firestore_backup = functions.pubsub.schedule('every 24 hours').onRun(firestoreBackup.createBackupHandler())
Checkout the full readme on how to set it up, it's super simple!

A solution using Python 2.
Fork it on https://github.com/RobinManoli/python-firebase-admin-firestore-backup
First install and setup Firebase Admin Python SDK: https://firebase.google.com/docs/admin/setup
Then install it in your python environment:
pip install firebase-admin
Install the Firestore module:
pip install google-cloud-core
pip install google-cloud-firestore
(from ImportError: Failed to import the Cloud Firestore library for Python)
Python Code
# -*- coding: UTF-8 -*-
import firebase_admin
from firebase_admin import credentials, firestore
import json
cred = credentials.Certificate('xxxxx-adminsdk-xxxxx-xxxxxxx.json') # from firebase project settings
default_app = firebase_admin.initialize_app(cred, {
'databaseURL' : 'https://xxxxx.firebaseio.com'
})
db = firebase_admin.firestore.client()
# add your collections manually
collection_names = ['myFirstCollection', 'mySecondCollection']
collections = dict()
dict4json = dict()
n_documents = 0
for collection in collection_names:
collections[collection] = db.collection(collection).get()
dict4json[collection] = {}
for document in collections[collection]:
docdict = document.to_dict()
dict4json[collection][document.id] = docdict
n_documents += 1
jsonfromdict = json.dumps(dict4json)
path_filename = "/mypath/databases/firestore.json"
print "Downloaded %d collections, %d documents and now writing %d json characters to %s" % ( len(collection_names), n_documents, len(jsonfromdict), path_filename )
with open(path_filename, 'w') as the_file:
the_file.write(jsonfromdict)

Here is my Android Java code for get backup easily for any fire store data collection
First use this method to read the collection data and store in it to serialized file in the mobile device storage
private void readCollection(){
ServerSide.db.collection("Collection_name")
.get()
.addOnCompleteListener(new OnCompleteListener<QuerySnapshot>() {
#Override
public void onComplete(#NonNull Task<QuerySnapshot> task) {
if (task.isSuccessful()) {
HashMap alldata = new HashMap();
for (QueryDocumentSnapshot document : task.getResult()) {
alldata.put(document.getId(),document.getData());
// ServerSide.db.collection("A_Sentences_test").document(document.getId())
// .set(document.getData());
}
try {
FileOutputStream fos = openFileOutput("filename.txt", Context.MODE_PRIVATE);
ObjectOutputStream os = new ObjectOutputStream(fos);
os.writeObject(alldata);
os.close();
fos.close();
Toast.makeText(MainActivity.this, "Stored", Toast.LENGTH_SHORT).show();
FileInputStream fis = openFileInput("filename.txt");
ObjectInputStream is = new ObjectInputStream(fis);
HashMap ad = (HashMap) is.readObject();
is.close();
fis.close();
Log.w("All data",ad+" ");
}catch (Exception e){
Log.w("errrrrrrrr",e+"");
}
} else {
Log.d("Colllllllllll", "Error getting documents: ", task.getException());
}
}
});
}
After that you can check the logcat whether the data is serialized correctly. and here is the restore code
private void writeData(){
try {
FileInputStream fis = openFileInput("filename.txt");
ObjectInputStream is = new ObjectInputStream(fis);
HashMap ad = (HashMap) is.readObject();
is.close();
fis.close();
for (Object s : ad.keySet()){
ServerSide.db.collection("Collection_name").document(s.toString())
.set(ad.get(s));
}
Log.w("ddddddddd",ad+" ");
}catch (Exception e){
e.printStackTrace();
}
}
Hope this would help

Question is old, projects are nice but I have some concerns about the backup.
1-For blaze plan users (free) official solution is off-limit.
2-Since Free users have 50k read quota per day that limit could be a problem in live and large databases.
3-As far as I examined most of the projects does not have a time limit or so, downloading same data every time it is run.
4-Wouldn't it be better to save collections as folders and every document as seperate file and and fetch only updated documents and replace file directly.
I will probably implement my own solution but just wondering your thoughts :)

Related

Accesssing Firebase Firestore from a script

I have a flutter Andriod app that can be used to store data in the firestore. Now I want to access that stored data from a Dart script to perform some operations on the data.
How can I access the firestore with a Dart script?
Updated the question: Looking only for dart package/library to access firebase firestore.
The Firebase docs provides good help with how to set up environment and a lot of examples.
It is possible with firedart.
import 'package:firedart/firedart.dart';
String pi = 'example-app';
void main(List<String> arguments) async {
Firestore.initialize(pi);
var map = await Firestore.instance.collection("users").get();
print(map);
}

Can I backup and restore a Firestore database? [duplicate]

Does the google firestore database service provides a backup?
If so, how do I backup the database and how do I restore in case of an error?
Update: It is now possible to backup and restore Firebase Firestore using Cloud Firestore managed export and import service
You do it by:
Create a Cloud Storage bucket for your project - Make sure it's a regional in us-central1 or 2 / multi regional type of bucket
Set up gcloud for your project using gcloud config set project [PROJECT_ID]
EXPORT
Export all by calling
gcloud firestore export gs://[BUCKET_NAME]
Or Export a specific collection using
gcloud firestore export gs://[BUCKET_NAME] --collection-ids='[COLLECTION_ID_1]','[COLLECTION_ID_2]'
IMPORT
Import all by calling
gcloud firestore import gs://[BUCKET_NAME]/[EXPORT_PREFIX]/
where [BUCKET_NAME] and [EXPORT_PREFIX] point to the location of your export files. For example - gcloud firestore import gs://exports-bucket/2017-05-25T23:54:39_76544/
Import a specific collection by calling:
gcloud firestore import --collection-ids='[COLLECTION_ID_1]','[COLLECTION_ID_2]' gs://[BUCKET_NAME]/[EXPORT_PREFIX]/
Full instructions are available here:
https://firebase.google.com/docs/firestore/manage-data/export-import
Update July 2018: Cloud Firestore now supports managed import and export of data. See the documentation for more details:
https://firebase.google.com/docs/firestore/manage-data/export-import
[Googler here] No, right now we do not offer a managed backup or import/export service. This is something we will definitely offer in the future, we just did not get it ready for the initial Beta release.
The best way to back up right now is to write your own script using our Java/Python/Node.js/Go server SDKs, it should be fairly straightforward to download all documents from each collection and write them back if you need to.
https://www.npmjs.com/package/firestore-backup
Is a tool that has been created to do just this.
(I did not create it, just adding it here as people will find this question)
Local backups
firestore-import-export
This is the one I use for "one-off", local backups, and what I generally recommend. (most straight-forward if you want a single JSON file)
firestore-backup-restore
Drawbacks:
Hasn't been updated in a long time.
Additional options: (not recommended)
python-firebase-admin-firestore-backup
Drawbacks:
Backup only; cannot restore from the backups it creates.
Hasn't been updated in a long time.
firestore-backup
Drawbacks:
Backup only; cannot restore from the backups it creates.
Cloud backups
The official gcloud backup commands.
Drawbacks:
The backup files are difficult/infeasible to parse. (update: how to convert to a json file)
You have to set up the gcloud cli. (update: or use the cloud shell to run the commands)
It doesn't backup locally; instead, it backs up to the cloud, which you can then download. (could also be considered an advantage, depending on what you want)
Note that for the gcloud backup commands, you have multiple options on how to schedule them to run automatically. A few options are shown here.
I am using the following work-around in order to have daily firestore backups:
I installed this globally: https://www.npmjs.com/package/firestore-backup-restore
I have a cron job that looks like this:
0 12 * * * cd ~/my/backup/script/folder && ./backup-script.sh
And my backup-script.sh looks like this:
#!/bin/sh
. ~/.bash_profile
export PATH=/usr/local/bin/
dt=$(/bin/date '+%d-%m-%Y %H:%M:%S');
echo "starting backup for $dt"
firestore-backup-restore -a ~/path/to/account/credentials/file.json -B ./backups/"$dt"
I've written a tool that traverses the collections/documents of the database and exports everything into a single json file. Plus, it will import the same structure as well (helpful for cloning/moving Firestore databases). It's published as an NPM package. Feel free to try it and give some feedback.
https://www.npmjs.com/package/node-firestore-import-export
I had the same issue and created a small npm package which allows you to create a scheduled backup with Cloud Functions. It uses the new import/export feature of Firestore.
const firestoreBackup = require('simple-firestore-backup')
exports.firestore_backup = functions.pubsub.schedule('every 24 hours').onRun(firestoreBackup.createBackupHandler())
Checkout the full readme on how to set it up, it's super simple!
A solution using Python 2.
Fork it on https://github.com/RobinManoli/python-firebase-admin-firestore-backup
First install and setup Firebase Admin Python SDK: https://firebase.google.com/docs/admin/setup
Then install it in your python environment:
pip install firebase-admin
Install the Firestore module:
pip install google-cloud-core
pip install google-cloud-firestore
(from ImportError: Failed to import the Cloud Firestore library for Python)
Python Code
# -*- coding: UTF-8 -*-
import firebase_admin
from firebase_admin import credentials, firestore
import json
cred = credentials.Certificate('xxxxx-adminsdk-xxxxx-xxxxxxx.json') # from firebase project settings
default_app = firebase_admin.initialize_app(cred, {
'databaseURL' : 'https://xxxxx.firebaseio.com'
})
db = firebase_admin.firestore.client()
# add your collections manually
collection_names = ['myFirstCollection', 'mySecondCollection']
collections = dict()
dict4json = dict()
n_documents = 0
for collection in collection_names:
collections[collection] = db.collection(collection).get()
dict4json[collection] = {}
for document in collections[collection]:
docdict = document.to_dict()
dict4json[collection][document.id] = docdict
n_documents += 1
jsonfromdict = json.dumps(dict4json)
path_filename = "/mypath/databases/firestore.json"
print "Downloaded %d collections, %d documents and now writing %d json characters to %s" % ( len(collection_names), n_documents, len(jsonfromdict), path_filename )
with open(path_filename, 'w') as the_file:
the_file.write(jsonfromdict)
Here is my Android Java code for get backup easily for any fire store data collection
First use this method to read the collection data and store in it to serialized file in the mobile device storage
private void readCollection(){
ServerSide.db.collection("Collection_name")
.get()
.addOnCompleteListener(new OnCompleteListener<QuerySnapshot>() {
#Override
public void onComplete(#NonNull Task<QuerySnapshot> task) {
if (task.isSuccessful()) {
HashMap alldata = new HashMap();
for (QueryDocumentSnapshot document : task.getResult()) {
alldata.put(document.getId(),document.getData());
// ServerSide.db.collection("A_Sentences_test").document(document.getId())
// .set(document.getData());
}
try {
FileOutputStream fos = openFileOutput("filename.txt", Context.MODE_PRIVATE);
ObjectOutputStream os = new ObjectOutputStream(fos);
os.writeObject(alldata);
os.close();
fos.close();
Toast.makeText(MainActivity.this, "Stored", Toast.LENGTH_SHORT).show();
FileInputStream fis = openFileInput("filename.txt");
ObjectInputStream is = new ObjectInputStream(fis);
HashMap ad = (HashMap) is.readObject();
is.close();
fis.close();
Log.w("All data",ad+" ");
}catch (Exception e){
Log.w("errrrrrrrr",e+"");
}
} else {
Log.d("Colllllllllll", "Error getting documents: ", task.getException());
}
}
});
}
After that you can check the logcat whether the data is serialized correctly. and here is the restore code
private void writeData(){
try {
FileInputStream fis = openFileInput("filename.txt");
ObjectInputStream is = new ObjectInputStream(fis);
HashMap ad = (HashMap) is.readObject();
is.close();
fis.close();
for (Object s : ad.keySet()){
ServerSide.db.collection("Collection_name").document(s.toString())
.set(ad.get(s));
}
Log.w("ddddddddd",ad+" ");
}catch (Exception e){
e.printStackTrace();
}
}
Hope this would help
Question is old, projects are nice but I have some concerns about the backup.
1-For blaze plan users (free) official solution is off-limit.
2-Since Free users have 50k read quota per day that limit could be a problem in live and large databases.
3-As far as I examined most of the projects does not have a time limit or so, downloading same data every time it is run.
4-Wouldn't it be better to save collections as folders and every document as seperate file and and fetch only updated documents and replace file directly.
I will probably implement my own solution but just wondering your thoughts :)

Cloud Functions Firebase - create directory in default bucket - typescript

I am trying to write a function that updates a node in database and then create a directory in the default storage bucket.
admin.database().ref('messages').push({ original: original })
.then(() => {
//looking for something like this
//functions.storage.object().mkdir('myFolder');
})
Function samples from firebase docs use const gcs = require('#google-cloud/storage')(); but i am having hard time importing this package using typescript.
importing it this way does not work. instead of having access to gcs.bucket(...) i have access to gcs.Bucket
import * as gcs from '#google-cloud/storage';
I am looking for ways to get this import working or other ways i can use in typescript.
thanks.
Google Cloud Storage does not have a concept of "folders" -- as a pure object store, you can store a file with any arbitrary key. Various UIs (including the Firebase console) look for slashes in the object names to provide a virtual structure, but no actual structure exists.
On the import issue -- as of version 5.2.0 of the Admin SDK, you can just do:
admin.storage().bucket()
to get a reference to the Cloud Storage bucket.

How to sync Firebase Database with Google Sheets?

I am working in an Ionic 3 project with ts to integrate Firebase into my app.
The below code I used to integrate firebase with Ionic project
constructor(angFire: AngularFireDatabase){
}
books: FirebaseListObservable<any>;
To send the data from my app to firebase, I used push method and to update entries I used update($key). Now I have all the data's in Firebase backend.
Now, how can I sync the firebase Database with Google Sheets so that each and every entry added to firebase backend has to get updated into sheets. I used a third party ZAPIER for this integration, but it would be nice if I get to learn on how to do this sync on my own.
Upon surfing, there are many tutorials to get the data's from the google sheets into Firebase. But I didn't come across any tutorials for vice versa.
I followed the below tutorial but it doesn't point to spreadsheets.
https://sites.google.com/site/scriptsexamples/new-connectors-to-google-services/firebase
Any help would be greatly appreciated!
I looked into importing Firebase right into Google Scripts either through the JavaScript SDK or or the REST API. Both have requirements/steps that Google Scripts cannot satisfy or that are extremely difficult to satisfy.
There is no foreseeable method of downloading the JavaScript SDK inside a Google Script because almost every method requires a DOM, which you don't have with a Google Sheet.
The REST API requires GoogleCredentials which, at a short glance, appear very difficult to get inside Google Scripts as well
So, the other option is to interact with Firebase in a true server side environment. This would be a lot of code, but here are the steps that I would take:
1) Setup a Pyrebase project so you can interact with your Firebase project via Python.
import pyrebase
config = {
"apiKey": "apiKey",
"authDomain": "projectId.firebaseapp.com",
"databaseURL": "https://databaseName.firebaseio.com",
"storageBucket": "projectId.appspot.com",
"serviceAccount": "path/to/serviceAccountCredentials.json"
}
firebase = pyrebase.initialize_app(config)
...
db = firebase.database()
all_users = db.child("users").get()
2) Setup a Google Scripts/Sheets project as a class that can interact with your Google Sheet
from __future__ import print_function
import httplib2
import os
from apiclient import discovery
from oauth2client import client
from oauth2client import tools
from oauth2client.file import Storage
try:
import argparse
flags = argparse.ArgumentParser(parents=[tools.argparser]).parse_args()
except ImportError:
flags = None
# If modifying these scopes, delete your previously saved credentials
# at ~/.credentials/sheets.googleapis.com-python-quickstart.json
SCOPES = 'https://www.googleapis.com/auth/spreadsheets.readonly'
CLIENT_SECRET_FILE = 'client_secret.json'
APPLICATION_NAME = 'Google Sheets API Python Quickstart'
class GoogleSheets:
...
# The rest of the functions from that link can go here
...
def write(self, sheet, sheet_name, row, col):
"""
Write data to specified google sheet
"""
if sheet == None or sheet == "":
print("Sheet not specified.")
return
day = time.strftime("%m/%d/%Y")
clock = time.strftime("%H:%M:%S")
datetime = day + " - " + clock
values = [[datetime]]
spreadsheetId = sheet
rangeName = sheet_name + "!" + str(row) + ":" + str(col)
body = {
'values': values
}
credentials = self.get_credentials()
http = credentials.authorize(httplib2.Http())
discoveryUrl = ('https://sheets.googleapis.com/$discovery/rest?'
'version=v4')
service = discovery.build('sheets', 'v4', http=http,
discoveryServiceUrl=discoveryUrl)
result = service.spreadsheets().values().update(
spreadsheetId=spreadsheetId, range=rangeName,
valueInputOption="RAW", body=body).execute()
3) Call the Google Sheets somewhere inside your Pyrebase project
from GoogleSheets import GoogleSheets
...
g = GoogleSheets()
g.write(<project-id>, <sheet-name>, <row>, <col>)
...
4) Set up a cron job to run the python script every so often
# every 2 minutes
*/2 * * * * /root/my_projects/file_example.py
You will need some basic server (Heroku, Digital Ocean) to run this.
This is not extensive because there is a lot of code to be written, but you could get the basics done. Makes we want to make a package now.
You can go for Zapier which is a 3rd party service through which you can easily integrate your Firebase and Google spreadsheets and vice versa. It has also got some support for google docs and other features.
https://zapier.com/zapbook/firebase/google-sheets/
Firebase can't be used as a trigger in Zapier, only as an action, so you can't send data from it to Google Sheets.

How to upload files from JAVA server to Firebase Storage by any means necessary [duplicate]

How can I integrate Firebase with a Java desktop application?
Is there a .jar file I could download?
I've seen plain Java docs on the old Firebase version but I'm unable to find any documentation on the latest version of Firebase.
My IDE is Netbeans.
Thanks.
According to the documentation website Firebase will not work just like that, it's an application designed to run only on 3 platforms, namely:
Android
iOS
Web
You can try and use the maven repository for the integration purpose, with any build script. I'm not exactly sure what you expect to do.
For Firebase Storage on the server, I recommend using gcloud-java:
// Authenticate using a service account
Storage storage = StorageOptions.builder()
.authCredentials(AuthCredentials.createForJson(new FileInputStream("/path/to/my/key.json"))
.build()
.service();
// Create blob
BlobId blobId = BlobId.of("bucket", "blob_name");
// Add metadata to the blob
BlobInfo blobInfo = BlobInfo.builder(blobId).contentType("text/plain").build();
// Upload blob to GCS (same as Firebase Storage)
Blob blob = storage.create(blobInfo, "Hello, Cloud Storage!".getBytes(UTF_8));
You can use firebase-server-sdk-3.0.1.jar (current version)
In Netbeans I would recommend to create Maven project and use artifact: GroupId - com.google.firebase, ArtifactId: firebase-server-sdk.
I works perfectly for me.
You can find some documentation here.
To initialize SDK just follow documentation: add service account (I use Owner role, I haven't tried weaker roles), download private key, and use this snippet:
FirebaseOptions options = new FirebaseOptions.Builder()
.setServiceAccount(new FileInputStream("path/to/downloaded private key.json"))
.setDatabaseUrl("https://your database name.firebaseio.com/")
.build();
FirebaseApp.initializeApp(options);

Resources