I am trying to extract near real-time analytics data from Firebase Analytics. The data is exported to BigQuery in real-time to intraday table. In order to sync real-time data to my database. I need to know when the record has logged to the table to avoid reading same records multiple times.
However, According to the Firebase document below, event_timestamp field is the timestamp when it is logged in a client device. event_server_timestamp_offset field is the offset between the collection time and upload time.
https://support.google.com/firebase/answer/7029846?hl=en
So, I assumed the server logged time can be found by event_timetamp + event_server_timestamp_offset.
But I've found that event_server_timestamp_offset have both negative and positive number.
Does anyone know what the event_server_timestamp_offset is for? Unloading data before Collecting data is impossible.
Related
Is it possible to set the default table expiration that is used when enabling the Crashlytics Big Query integration in Firebase?
We are trying to reduce our monthly Firebase costs (Blaze plan) that are due to the amount of data exported automatically and now exists in our BigQuery tables. These are the costs that appear in our Firebase billing reports as "non Firebase services".
To reduce the costs we would like to allow the data to expire automatically and adjust the "time to expire" shown below for all ongoing data exported from Firebase to BigQuery.
Is this possible from within the Firebase console itself? Or can this only be done in BigQuery using the CLI? This page doesn’t seem to give any indication that this is possible from the Firebase Console itself: https://firebase.google.com/docs/crashlytics/bigquery-export
But we can see from the BigQuery docs that Table Expiration appears to be what we need to set, our question is essentially how to do this to apply for all existing and future tables streamed from Firebase Crashlytics (but also Events and Performance) data.
Thanks for any advice!
You can limit data in BigQuery by setting the retention time right in the BigQuery console to whatever length of time you prefer:
Set default table expiration times here
Update a particular table's expiration time here
The size of exported data highly depends on the product usage. Moreover, especially for Crashlytics, stacktrace in the data is completely unpredictable.
In order for you to have an idea of the cost, You can check following links:
Schema of the exported table
Columns presenting regardless of the stack trace
BigQuery Free operations
Additionally, please follow following documentation, which has clearer insight on the export data to BigQuery.
The scenario here is that I have my Firebase( just Analytics-Data) project linked to BigQuery. but when i check on the bigQuery to see the dataset it's not appear there and i don't know which name/id has.
I highly appreciate your support. Thanks
According to the documentation, when you link your Firebase Project to BigQuery,a corresponding dataset will be created. This dataset can be found, in BigQuery, under your project id and it will be named as your app. In case, that you have both IOS and Android versions of your app, two datasets will be created as follows:
The above image was taken from the documentation, here.
Furthermore, in addition to your app_events table, under your app's name, you will have apps_events_intraday, which will receive data near real time from Firebase. In other words, as soon as Firebase receives the data from the app it will transfer it to the intraday table in BigQuery. Whereas, the app_events table will be uploaded once per day, link.
Lastly, keep in mind that the data generated by your app can take up to 1 hour to be sent to Firebase which then will be nearly instantly sent to BigQuery. You can read more about the latency here.
Need some help with accessing historical data for Firebase Crashlytics and Events data in BigQuery.
We have linked BigQuery to firebase and we are able to get only last 2 months of data in BigQuery at this moment.
Can you please suggest a way to get the data since the inception of the app?
Firebase doesn't keep the events data indefinitely which makes this feature not feasible at the moment.
Currently, your data will start being exported since the moment you enable the BigQuery connection, i.e. you can't access your historical data.
If you think this feature would be useful for you and for other people, I encourage you to request it in this link.
I hope it helps
I am researching of a way to regularly sync Firebase data to BigQuery, then display that data to Data Studio. I saw this instruction in the documentation:
https://support.google.com/firebase/answer/6318765?hl=en
According to the above instruction, it says once Firebase is linked to BigQuery, the data from Firebase is being streamed to BigQuery real-time.
Let's say I have initial export of Firebase data to BigQuery (before linking) and I made a Data Studio visualization out of that initial data, we call it Dataset A. Then I started linking Firebase to BigQuery. I want Dataset A to be in sync with Firebase every 3 hours.
Based on the documentation, does this mean I don't have to use some external program to synchronize Firebase data every 3 hours to BigQuery, since it is streaming real-time already? After linking, does the streamed data from Firebase automatically goes to Dataset A?
I am asking because I don't want to break the visualization if the streaming behaves differently than the expected (expected means that Firebase streams to BigQuery's Dataset A consistent with the original schema). Because if it does (break the original dataset or it doesn't stream to the original dataset), I might as well write a program that does the syncing.
Once you link your Firebase project to BigQuery, Firebase will continuously export the data to BigQuery, until you unlink the project. As the documentation says, the data is exported to daily tables, and a single fixed intraday table. There is no way for you to control the schedule of the data export beyond enabling/disabling it.
If you're talking about Analytics data, schema changes to the exported data are very rare. So far there's been a schema change once, and there are currently no plans to make any more schema changes. If a schema change ever were to happen again though, all collaborators on the project will be emailed well in advance of the change.
I'm trying to analyze an app user data now have access to the Firebase and Google BigQuery data. To do some analysis, I need to link these two database together by identify the users. In the database of Firebase, I have a field named user and I hope I could find a field in the BigQuery database which contains the same information to link these two database. But I only find a field named app instance id in BigQuery, which is not the same as user field in Firebase and I don't know how to join these two data source by a common field. Is anyone can help? Thanks!
Firebase does not automatically record User identification in its Analytics data. If you want that, you should set the relevant value yourself in by calling setUserId().