I would like to get all the data from my Realtime Database (200GB) to migrate it to a different database (PostgreSQL).
I considered:
creating a script with the Admin SDK and request all the entries, but this would be really slow and would make the database reach the peek (that means slow down all the database).
using the "Export JSON" function of the console but it is not possible because the size (200GB) is too large.
I just made a manual backup on the console and hopefully I'm going to receive a huge ZIP file that then I will need to manage with a script.
Any other smarter solution?
I have installed a firebase extension "collections to bigquery" to export the data in my firebase collection to the BigQuery table. I have installed the extension an hour ago and I can still not see any data in the BigQuery. I am new to FireBase so can anyone tell me how long does it usually take until data finally starts showing up in BigQuery?
Thanks!
The Firestore BigQuery extension will automatically sync future writes to a collection to BigQuery - if your database isn't actively being written to, it won't do anything just by installing it.
To bring existing documents into the BigQuery table, see this guide that runs through how to run the import script.
I want to use Export to BigQuery extension, and i'd like to know if this export is charged as "streaming inserts"
Both collection mirroring and initial import operations (using https://www.npmjs.com/package/#firebaseextensions/fs-bq-import-collection) are implemented with a call to the bigquery streaming inserts API (https://cloud.google.com/bigquery/streaming-data-into-bigquery) and billed as such.
I have upgraded my account to blaze which is one of the prerequisite. I have tried to followed the FAQ Link BigQuery to Firebase. still not able to see any of the data from firestore or firebase-realtimeDatabase into bigQuery.
I see option in bigQuery to create Dataset. however after creating dataset it allows me to upload data from [file], [cloud storage], [bigQuery],[google Drive] . but not firestore database.
Plesae help.
Firestore now allows to import / export data. The data is exported to a Cloud Storage bucket and from there it can be imported into Big Query. Here are some links that might be helpful:
https://firebase.google.com/docs/firestore/manage-data/export-import
https://cloud.google.com/bigquery/docs/loading-data-cloud-datastore (Firestore uses the same format as Datastore for imports / exports)
** Edit:
Docs for BigQuery imports from Firestore are now available https://cloud.google.com/bigquery/docs/loading-data-cloud-firestore
In case anyone need it. I ended up automating scripts because current export data option only allows stream data and preserve for 30 days.
export data
gcloud beta firestore export --collection-ids=users gs://mybucket/users
load backup into bq
bq load --source_format=DATASTORE_BACKUP mydataset.users gs://gs://mybucket/users/all_namespaces/kind_users/all_namespaces_kind_users.export_metadata
You can now export data from Cloud Firestore to BigQuery with a Firebase Extension. See: https://firebase.google.com/products/extensions/firestore-bigquery-export
Also see David's answer on how to import/export data.
Outdated answer below:
There is no built-in support to import data from the Firebase Realtime Database or Cloud Firestore into BigQuery.
For now, if you want to import data, you'll have to write code to do so.
You should use the BigQuery export extension built-in in Firebase
See details: https://firebase.google.com/products/extensions/firestore-bigquery-export
Now, this extension will be importing to your BigQuery table new created/updated/deleted documents inside a collection, your current old data before the new one is added will not be placed inside this table.
To import all the previous data you will need first to install the extension because all the writes while doing the export first without installing the extension will be lost.
After you install the extension, just use gcloud to export all the current dataset
https://github.com/firebase/extensions/blob/master/firestore-bigquery-export/guides/IMPORT_EXISTING_DOCUMENTS.md
I made an NPM package that lets you create a BigQuery dataset and tables with autogenerated schemas based on your Firestore data, and then copy and convert chosen Firestore collections.
https://www.npmjs.com/package/firestore-to-bigquery-export
There is now also an extension that does it https://github.com/firebase/extensions/tree/master/firestore-bigquery-export
We manage to produce lots of large json files ~600MB+ (with one json record per line) and we want to import this into Firebase database. Is there any way to do this directly using a gs:// path?
(The fire base console UI has an import function but this is using json from local files not gs:// and https://github.com/firebase/firebase-import is not clear whether is can work from Google Storage.)
There is no direct connection to import data from a file in Google Cloud Storage into the Firebase Database. You will have to use some intermediary tool or code for that. While recommending a specific tool is off-topic on Stack Overflow, I want to point out Firebase's own streaming import tool as one option
But you should really consider what purpose this import serves. The Firebase Database is a great tool for synchronizing data between connected clients. It is not a great tool for advanced querying or reporting on large data sets. Something like BigQuery is much more suited to such tasks.