Is it possible to set the default table expiration that is used when enabling the Crashlytics Big Query integration in Firebase?
We are trying to reduce our monthly Firebase costs (Blaze plan) that are due to the amount of data exported automatically and now exists in our BigQuery tables. These are the costs that appear in our Firebase billing reports as "non Firebase services".
To reduce the costs we would like to allow the data to expire automatically and adjust the "time to expire" shown below for all ongoing data exported from Firebase to BigQuery.
Is this possible from within the Firebase console itself? Or can this only be done in BigQuery using the CLI? This page doesn’t seem to give any indication that this is possible from the Firebase Console itself: https://firebase.google.com/docs/crashlytics/bigquery-export
But we can see from the BigQuery docs that Table Expiration appears to be what we need to set, our question is essentially how to do this to apply for all existing and future tables streamed from Firebase Crashlytics (but also Events and Performance) data.
Thanks for any advice!
You can limit data in BigQuery by setting the retention time right in the BigQuery console to whatever length of time you prefer:
Set default table expiration times here
Update a particular table's expiration time here
The size of exported data highly depends on the product usage. Moreover, especially for Crashlytics, stacktrace in the data is completely unpredictable.
In order for you to have an idea of the cost, You can check following links:
Schema of the exported table
Columns presenting regardless of the stack trace
BigQuery Free operations
Additionally, please follow following documentation, which has clearer insight on the export data to BigQuery.
Related
Is it possible to filter the data to be exported to BigQuery? For example I just want to have fatal crashes (is_fatal=TRUE) to be exported but not non fatal exceptions which allocates much more space in my case.
I checked out data transfer options but could not find anything related to filtering or schema customization.
The only configuration options for the exporting Crashlytics data to BigQuery are to:
Turn it on or off
Enable streaming of intra-day events (if your project is on the Blaze plan)
It's not possible to control what crash data is exported beyond that.
If you want less data to be stored in BigQuery, you'll have to copy the data you want to keep over to new tables, and delete the ones generated by the integration.
The scenario here is that I have my Firebase( just Analytics-Data) project linked to BigQuery. but when i check on the bigQuery to see the dataset it's not appear there and i don't know which name/id has.
I highly appreciate your support. Thanks
According to the documentation, when you link your Firebase Project to BigQuery,a corresponding dataset will be created. This dataset can be found, in BigQuery, under your project id and it will be named as your app. In case, that you have both IOS and Android versions of your app, two datasets will be created as follows:
The above image was taken from the documentation, here.
Furthermore, in addition to your app_events table, under your app's name, you will have apps_events_intraday, which will receive data near real time from Firebase. In other words, as soon as Firebase receives the data from the app it will transfer it to the intraday table in BigQuery. Whereas, the app_events table will be uploaded once per day, link.
Lastly, keep in mind that the data generated by your app can take up to 1 hour to be sent to Firebase which then will be nearly instantly sent to BigQuery. You can read more about the latency here.
Need some help with accessing historical data for Firebase Crashlytics and Events data in BigQuery.
We have linked BigQuery to firebase and we are able to get only last 2 months of data in BigQuery at this moment.
Can you please suggest a way to get the data since the inception of the app?
Firebase doesn't keep the events data indefinitely which makes this feature not feasible at the moment.
Currently, your data will start being exported since the moment you enable the BigQuery connection, i.e. you can't access your historical data.
If you think this feature would be useful for you and for other people, I encourage you to request it in this link.
I hope it helps
I am researching of a way to regularly sync Firebase data to BigQuery, then display that data to Data Studio. I saw this instruction in the documentation:
https://support.google.com/firebase/answer/6318765?hl=en
According to the above instruction, it says once Firebase is linked to BigQuery, the data from Firebase is being streamed to BigQuery real-time.
Let's say I have initial export of Firebase data to BigQuery (before linking) and I made a Data Studio visualization out of that initial data, we call it Dataset A. Then I started linking Firebase to BigQuery. I want Dataset A to be in sync with Firebase every 3 hours.
Based on the documentation, does this mean I don't have to use some external program to synchronize Firebase data every 3 hours to BigQuery, since it is streaming real-time already? After linking, does the streamed data from Firebase automatically goes to Dataset A?
I am asking because I don't want to break the visualization if the streaming behaves differently than the expected (expected means that Firebase streams to BigQuery's Dataset A consistent with the original schema). Because if it does (break the original dataset or it doesn't stream to the original dataset), I might as well write a program that does the syncing.
Once you link your Firebase project to BigQuery, Firebase will continuously export the data to BigQuery, until you unlink the project. As the documentation says, the data is exported to daily tables, and a single fixed intraday table. There is no way for you to control the schedule of the data export beyond enabling/disabling it.
If you're talking about Analytics data, schema changes to the exported data are very rare. So far there's been a schema change once, and there are currently no plans to make any more schema changes. If a schema change ever were to happen again though, all collaborators on the project will be emailed well in advance of the change.
I have a problem to understand my data in the Firebase Realtime Database,
it is not always clear and easy to navigate and query the data.
The database entries are not easily understood and interpreted. Adding more data to make the entries more clear will add to the data storage costs.
Adding extra data for a certain duration, cannot be cleared out easily. It is also not visible when database read operations are performed. Dates are stored as an integers and cannot be interpreted easily. I have to use a bash command "date -r ", etc.
Clearly the database alone is not enough to debug flow of events,
and as the database grows, analysis of data will be harder.
I have no record of the sequence of the events that describes the database entries.
Possible solutions:
(1) Use Realtime Database
I create another Realtime Firebase "node" and log all the events in a human friendly way to this node". This way I have control over this data and can clear this at any time I wish, minimising the data "costs". The only problem I see with this is that I will have to periodically have to remember to clear this data. (Perhaps Firebase has some periodic scheduler to call some process). Or use some mobile clients to trigger the events...
(2) Fabric
My other option is to use Fabric's Answers, but after looking at the reporting of this data, it does not really suite my needs, no filtering and the details of the messages are not really as verbose as I have expected.
(3) Firebase Analytics
I am not sure about Firebase Analytics, I see no mechanism to clear events,
will this add to my costs? Will it be easy enough to filter / query logs
to analyse a certain sequence of events.
Typically I would like to see data something like this:
data_time_user_friendly,user_id,user_friendly_id,event_action,payload
What is the best practice to have a remote syslog for analysing my data and flow of events.
Edited ....
After some searching, I found there are numerous products that seem more suitable and are specifically developed for logging.
I did a quick comparison of some "free" products for system logging:
Papertrail 100 MB/month 7 days retention search 48 hours
Loggly 200 MB/day 7 days retention
Bugfender 100K lines/day 1 day retention
Logz.io 1GB/day 3 days retention
This is just a quick comparison and have not evaluated any of the options.
Does Firebase have a solution for this or is it best to use one of the above mentioned products?