Firebase real time database by time period - firebase

I would like to ask about firebase Google Analytics (GA)
Can real-time firebase see the data by last 3 hours or more ( Currently, it is limited to last 30 minutes).
How to see the reports filtered by hours? ( I would like to find which period has more active users, let says 6:00PM-7:00PM)

Related

missing data firebase sync to bigquery sandbox

I use Bigquery Sandbox and export raw event data from Google Analytics to Bigquery. But The number of events 2 places are different.Bigquery much less Google Analytics.
My number of events per day usually ranges from 1 to 200 thousand events meanwhile GA usually ranges 7-10 million evets day. And The storage space I have left is 6.97/10 GB.

Discrepancy between Engagement Time Msec Firebase Dashboard Vs BigQuery Data

I use BigQuery to get data from Firebase, and I'm having some issues with data accuracy. Figures I get from the data in BigQuery are not matching the ones on Firebase dashboard.
I recently understood that Firebase does not look at all users, instead it considers only active users. So now I have filtered for active users by putting a filter for engagement_time_msec>0. My active users number is matching up with Firebase dashboard now (just 1-2 digits difference occassionally).
But my main problem is with the average engagement time!
Firebase (and GA for Firebase) shows average engagement time metric under engagement overview. When you hover over it, it gives this definition.
"Average Engagement Time per active user for the time period selected"
However, when I get data through BigQuery and calculate this manually, my numbers are off.
I am calculating Active Users as Distinctcount of user_psuedo_ID where engagement time>0, and engagement time is being summed up where event name = user_engagement. (I have converted engagement time msec to mins)
Average engagement time = SUM(Engagement time mins)/Active Users
This should give me an average engagement time per active user, but this figure doesn't match the one in Firebase console. I have tried so many methods, and I fail to understand what Firebase is doing at the back end to come up with these values.
P.s: I have also tried summing up engagement time without a condition on event name and that gives me an even greater average, making the difference between it and Firebase even bigger.
Please help!!

When should I run daily ETL jobs for Firebase Analytics data exported to BigQuery?

We use Firebase Analytics to collect events from our apps. We have enabled events export to BigQuery. Every day we run some ETL jobs to create more friendly analytics tables in BigQuery (e.g. sessions, purchases).
The question is when should we run these ETL jobs?
We know that Firebase Analytics creates in BigQuery 'events_intraday_' table which is changed to 'events_' after some hours after midnight. We also understand that some events might be reported later if client is not connected with the internet, but this is not the problem.
Our theory is that 'events_intraday_' table is some kind of temporary table and we should run ETL jobs when it changes to 'events_'. Unfortunately we could not find any documentation about it. Is this good solution?
From Announcing Realtime Exporting of your Analytics Data into BigQuery:
At the end of the day [1], this data will be moved into its permanent appevents_ home, and the old intraday table will be automatically cleaned up for you.
With:
[1] This is determined by looking at the developer's time zone.
So it looks like the daily table is created at midnight for your timezone.
Thanks to Frank van Puffelen I've found article on Firebase Blog
How Long Does it Take for My Firebase Analytics Data to Show Up?, which says that analytics data exported to BigQuery can be delayed up to little more than 1 hour. So based on this information ETL jobs should be runned about, lets say 2 AM UTC+0 and query should just UNION ALL events with events_intraday table.
So if today is 2019-04-02 and I want to query data from last month, the query should look like:
SELECT * FROM
(
SELECT *
FROM `<PROJECT_ID>.analytics_<ANALYTICS_ID>.events_*`
WHERE _TABLE_SUFFIX BETWEEN '20190301' AND '20190401'
)
UNION ALL
(
SELECT *
FROM `<PROJECT_ID>.analytics_<ANALYTICS_ID>.events_intraday_*`
WHERE _TABLE_SUFFIX = '20190401'
)

Discrepancy in Daily user engagement between Firebase Analytics dashboard and BigQuery

I'm trying to develop a query against Firebase Analytics data linked to BigQuery to reproduce the "Daily user engagement" graph from the Firebase Analytics dashboard (to include in a Google Data Studio report).
According to Firebase Help documentation, Daily user engagement is defined as "Average daily engagement per user for the date range, including the fluctuation by percentage from the previous date range." So, my attempt is to sum the engagement_time_msec (the additional engagement time (ms) since the last user_engagement event according to https://support.google.com/firebase/answer/7061705?hl=en) for user_engagement events, divided by the count of users (identified by user_dim.app_info.app_instance_id) per day. The query looks like this:
SELECT ((total_engagement_time_msec / 1000) / users) as average_engagement_time_sec, date FROM
(SELECT
SUM(params.value.int_value) as total_engagement_time_msec,
COUNT(DISTINCT(user_dim.app_info.app_instance_id)) as users,
e.date
FROM `com_artermobilize_alertable_IOS.app_events_*`, UNNEST(event_dim) as e, UNNEST(e.params) as params
WHERE e.name = 'user_engagement'
AND params.key = 'engagement_time_msec'
GROUP BY e.date)
ORDER BY date desc
The results are close to what's displayed in the Firebase console graph of Daily user engagement, but the values from my query are consistently a few seconds higher (BigQuery results shown here on the left, Firebase Console graph values on the right).
To note, we're not setting user_dim.user_id and not using IDFA, so my understanding is the correct/only way to count "users" is the user_dim.app_info.app_instance_id, and I imagine the same would be true for the Firebase console.
Can anyone suggest what might be different between how I'm determining the average engagement time from BigQuery, and how that's being determined in the Firebase console graph?
To note, I've seen a similar question posed here, but I don't believe the suggested answer applies for my query since 1) the discrepancies are present over multiple days, 2) I'm already querying for user_engagement events and 3) the event date being used in the query is stated to be based on the registered timezone of your app (according to this).

Google Analytics show real time goal hits but not on the conversions report

I'm trying to report conversions to Google Analytics from the server side of an app after a payment is successfully processed. I'm using the Measurement Protocol from the devguides. https://developers.google.com/analytics/devguides/collection/protocol/v1/
The problem is that it successfully shows the goal hits on the real time conversions report, but this are not showed in the normal conversions report as goal completions.
Is there any difference between 'goal hit' and 'goal completion' I'm missing? Or is there any delay on the data that makes into the regular conversions report?
There is a delay. Per documentation it's 24-48 hours (4 hours on a 360 account), although usually the data shows up somewhat faster.
Documentation:
Processing latency is 24-48 hours. Standard accounts that send more
than 200,000 sessions per day to Analytics will result in the reports
being refreshed only once a day. This can delay updates to reports and
metrics for up to two days. To restore intra-day processing, reduce
the number of sessions your account sends to < 200,000 per day. For
Analytics 360 accounts, this limit is extended to 2 billion hits per
month.
I used to think there was long delays in data showing up in GA reports as well, until I discovered a small bug in the GA system in regards to time zones. The system automatically selects the date for you on the reports, but if you live in a time zone like Australia or The Philippines, these can be out of sync, and therefore, the most recent data doesn't show up.
I now always set the date to "Today" or to the last few days, and I find all data comes through within minutes, not hours.

Resources