I am researching of a way to regularly sync Firebase data to BigQuery, then display that data to Data Studio. I saw this instruction in the documentation:
https://support.google.com/firebase/answer/6318765?hl=en
According to the above instruction, it says once Firebase is linked to BigQuery, the data from Firebase is being streamed to BigQuery real-time.
Let's say I have initial export of Firebase data to BigQuery (before linking) and I made a Data Studio visualization out of that initial data, we call it Dataset A. Then I started linking Firebase to BigQuery. I want Dataset A to be in sync with Firebase every 3 hours.
Based on the documentation, does this mean I don't have to use some external program to synchronize Firebase data every 3 hours to BigQuery, since it is streaming real-time already? After linking, does the streamed data from Firebase automatically goes to Dataset A?
I am asking because I don't want to break the visualization if the streaming behaves differently than the expected (expected means that Firebase streams to BigQuery's Dataset A consistent with the original schema). Because if it does (break the original dataset or it doesn't stream to the original dataset), I might as well write a program that does the syncing.
Once you link your Firebase project to BigQuery, Firebase will continuously export the data to BigQuery, until you unlink the project. As the documentation says, the data is exported to daily tables, and a single fixed intraday table. There is no way for you to control the schedule of the data export beyond enabling/disabling it.
If you're talking about Analytics data, schema changes to the exported data are very rare. So far there's been a schema change once, and there are currently no plans to make any more schema changes. If a schema change ever were to happen again though, all collaborators on the project will be emailed well in advance of the change.
Related
Is it possible to set the default table expiration that is used when enabling the Crashlytics Big Query integration in Firebase?
We are trying to reduce our monthly Firebase costs (Blaze plan) that are due to the amount of data exported automatically and now exists in our BigQuery tables. These are the costs that appear in our Firebase billing reports as "non Firebase services".
To reduce the costs we would like to allow the data to expire automatically and adjust the "time to expire" shown below for all ongoing data exported from Firebase to BigQuery.
Is this possible from within the Firebase console itself? Or can this only be done in BigQuery using the CLI? This page doesn’t seem to give any indication that this is possible from the Firebase Console itself: https://firebase.google.com/docs/crashlytics/bigquery-export
But we can see from the BigQuery docs that Table Expiration appears to be what we need to set, our question is essentially how to do this to apply for all existing and future tables streamed from Firebase Crashlytics (but also Events and Performance) data.
Thanks for any advice!
You can limit data in BigQuery by setting the retention time right in the BigQuery console to whatever length of time you prefer:
Set default table expiration times here
Update a particular table's expiration time here
The size of exported data highly depends on the product usage. Moreover, especially for Crashlytics, stacktrace in the data is completely unpredictable.
In order for you to have an idea of the cost, You can check following links:
Schema of the exported table
Columns presenting regardless of the stack trace
BigQuery Free operations
Additionally, please follow following documentation, which has clearer insight on the export data to BigQuery.
Is it possible to filter the data to be exported to BigQuery? For example I just want to have fatal crashes (is_fatal=TRUE) to be exported but not non fatal exceptions which allocates much more space in my case.
I checked out data transfer options but could not find anything related to filtering or schema customization.
The only configuration options for the exporting Crashlytics data to BigQuery are to:
Turn it on or off
Enable streaming of intra-day events (if your project is on the Blaze plan)
It's not possible to control what crash data is exported beyond that.
If you want less data to be stored in BigQuery, you'll have to copy the data you want to keep over to new tables, and delete the ones generated by the integration.
The scenario here is that I have my Firebase( just Analytics-Data) project linked to BigQuery. but when i check on the bigQuery to see the dataset it's not appear there and i don't know which name/id has.
I highly appreciate your support. Thanks
According to the documentation, when you link your Firebase Project to BigQuery,a corresponding dataset will be created. This dataset can be found, in BigQuery, under your project id and it will be named as your app. In case, that you have both IOS and Android versions of your app, two datasets will be created as follows:
The above image was taken from the documentation, here.
Furthermore, in addition to your app_events table, under your app's name, you will have apps_events_intraday, which will receive data near real time from Firebase. In other words, as soon as Firebase receives the data from the app it will transfer it to the intraday table in BigQuery. Whereas, the app_events table will be uploaded once per day, link.
Lastly, keep in mind that the data generated by your app can take up to 1 hour to be sent to Firebase which then will be nearly instantly sent to BigQuery. You can read more about the latency here.
Need some help with accessing historical data for Firebase Crashlytics and Events data in BigQuery.
We have linked BigQuery to firebase and we are able to get only last 2 months of data in BigQuery at this moment.
Can you please suggest a way to get the data since the inception of the app?
Firebase doesn't keep the events data indefinitely which makes this feature not feasible at the moment.
Currently, your data will start being exported since the moment you enable the BigQuery connection, i.e. you can't access your historical data.
If you think this feature would be useful for you and for other people, I encourage you to request it in this link.
I hope it helps
I have linked Bigquery to my app through Firebase and it has worked well for more than 3 months. However, couple days ago, a random dataset called "analytics_156951304" was created and then the app data was streamed into this dataset instead of the original dataset that Bigquery created for me when I first linked my app to Bigquery, "my_app_name_IOS". And the table schema was changed too. I checked Stackdriver logging and it said an account called "firebase-measurement#system.gserviceaccount.com" created this job at midnight of my local time. However, I have no clue what happened and how to get my streaming data back to my original dataset, "my_app_name_IOS". Thank you for all the answers!!!
Firebase recently scheduled a huge schema migration - your previous queries won't work anymore.
(AFAIK this was communicated via email)
There's a script that helps migrating your previous datasets to the new schema:
https://support.google.com/analytics/answer/7029846?visit_id=1-636661955728880760-637850888&rd=1#migrationscript
(tho the script won't help with modifying the existing queries)
From the email I got:
Currently, the data from each app in a Firebase project are exported to a distinct BigQuery dataset in the corresponding BigQuery project. Going forward, the data from each app in a Firebase project will be exported to a single dataset in that BigQuery project. Additionally, the data in this new dataset will follow a new schema.