I was looking at a few ways to export data out from Firestore without using export (expensive operation in the long term as it doesn't support incremental backups) to use in BigQuery and Data Studio.
1) Using Google Pub/Sub.
This will probably require function to both write to pub/sub and then another to trigger to BQ.
2) Using Cloud Functions to trigger from an onCreate event to write directly to a BigQuery dataset and table.
(This is using table.insert)
What would the advantage be to use Pub/Sub - other than it would appear that it will cost more in the long term?
Or is there another way I am unaware of to do this?
I'm new at this. Some advise and pro and cons of the above scenarios are much appreciated.
The official solution is here.
In case of using Cloud Functions to trigger from an onCreate event, what will you create? Create File on Cloud Storage or create Firestore Document?
I think that in case of using Cloud Functions you should use PubSub trigger.
I recommend asynchronous architecture like Pub/Sub. Because rerun is easy and the scope of influence is limited.
I developed sample is here. I'm using Cloud Scheduler not cron.yaml. The cost of Cloud Scheduler is here.
(If you want) Export Firebase Authentication users to Cloud Firestore Collection. Use Firestore, Cloud Functions(PubSub) and Cloud Scheduler.
Export All Cloud Firestore Collections and Specified collections to Cloud Storage. Use AppEngine and Cloud Scheduler.
Export Specified Cloud Firestore Collections to BigQuery( as Partitioned tables). Use AppEngine and Cloud Scheduler.
Related
Being on the Blaze plan, what are the cost implications when it comes to firestore triggers?
Do I assume correctly that a trigger by itself doesn't generate any extra reads from database?
I still have a feeling that triggers are not free from perspective of CPM/memory consumption. And from that perspective do I understand correctly that firestore triggers can be treated just like any other firebase functions?
The Firestore document that triggers your Cloud Function is included in the context. There is no document read or bandwidth charge for accessing this document.
You will be charged for the invocation and CPU/memory usage of the Cloud Function itself, as well as for any additional Firestore access you perform inside your Function's code.
According to the Firebase docs:
Firestore Triggers
With Cloud Functions, you can handle events in
Cloud Firestore with no need to update client code.
So Firestore triggers can only be done via Cloud Functions. For cloud functions there are costs for running a function (https://firebase.google.com/pricing) and the costs of listening to changes (snapshots) or reading/writing to Firestore will be added to that. Depending on what you are planning to do you also have to account for the internet traffic generated between the Firestore and the Cloud Functions, but this depends on some variables.
So, yes you are correct.
I have an app with tens of thousands of users where every second we are capturing data (location data). For analysis we are trying to allow this data to be queryable to generate reports for the user.
The idea is to pass the data in the Realtime database to Spanner so that I can query the data in SQL format and generate reports based on that.
I want to be able to do this from my trigger everytime that I update the realtime database.
So data follows this flow:
iOS / Android location data -> writes Firebase Realtime DB -> Function trigger based on write event -> Add to Spanner
Is this possible and if so how do you suggest it being done?
I am already using firebase functions to trigger notifications and keep database consistency, but not sure how to connect to spanner. The documentation that I have found is all about Cloud functions and when I do
const Spanner = require('#google-cloud/spanner');
I get the "Error: Error parsing triggers: Cannot find module '#google-cloud/spanner'"
My first limitation is: Are firebase functions limited to firebase itself or can I integrate with other cloud tools such as spanner?
Thanks,
Ricardo
I don't have experience with Realtime Database, PubSub and Functions, but I'm also interested :)
I imagine 2 steps: [1] Firestore triggers can publish message to PubSub, and [2] Functions can subscribe and handle PubSub events which interact with Spanner.
[1] How to publish message in Google Pub/Sub from Firebase Cloud Function?
[2] Google Cloud Pub/Sub Triggers | Cloud Functions
[2] Using Cloud Spanner with Cloud Functions | Cloud Functions
The error means you haven't (correctly) installed the Cloud Spanner client library. To fix this, in your functions folder run:
npm install #google-cloud/spanner
Unless you actually need the data in RTDB, you can avoid that step entirely and call a Cloud Function directly: See HTTP Functions.
There is a handy 'How-to Guide' on writing a Cloud Function for interacting with Cloud Spanner in the documentation, just modify it for put instead of get: See Using Cloud Spanner with Cloud Functions
I want to read and write data in database real time across cloud functions but I don't know if this is possible, also I don't know if this is good practice or if is better make this through with SDK database realtime
I learning firebase and react native, I am new learning these topics.
From the docs:
Cloud Functions for Firebase lets you automatically run backend code in response to events triggered by Firebase features and HTTPS requests. Your code is stored in Google's cloud and runs in a managed environment. There's no need to manage and scale your own servers.
Cloud Functions runs Node v.6.11.5, so we recommend that you develop locally with this version.
You are able to use realtime database trigggers like onCreate(), onWrite(). Also you are able to use set() to send data to the realtime database.
https://firebase.google.com/docs/functions/
https://firebase.google.com/docs/functions/database-events
I'm building a Firebase app, and plan to use the real-time database when I need real-time updates. However, most of the application data is more traditional.
Now that Functions is a thing, how do I also leverage either DataStore or CloudSQL? Can anyone point me to specific documentation or examples how to read/write with either of those services from a function?
Neither Cloud Datastore nor Cloud SQL support Cloud Functions yet, which means you aren't yet able to trigger Cloud Functions based on their events the way you can with the Firebase Realtime Database.
Fortunately, once a Cloud Function has been triggered (for example via HTTP), you can still read and write from Datastore and SQL as you would from any other Node.js code. Here is documentation for Cloud Datastore, and here it is for Cloud SQL.
Finally, if you're adventurous and might like to provide early feedback on upcoming integrations like Datastore, fill out this form!
As I know currently Cloud Functions doesn't support triggering functions from Firebase.
For now I'm planing to use an basic Engine instance to trigger the functions based on the queue.
Is this the right way to go? or should I trigger the cloud function directly from the clients device after the data is inserted in the db?
thank you
Cloud Functions for Firebase was just launched today! You can use the SDK to trigger cloud functions from your Firebase database, storage bucket, authentication, and analytics events. https://firebase.google.com/docs/functions/