I've integrated my Firebase project with BigQuery. Now I'm facing a data discrepancy issue while trying to get 1 day active users, for the selected date i.e. 20190210, with following query from BigQuery;
SELECT COUNT(DISTINCT user_pseudo_id) AS 1_day_active_users_count
FROM `MY_TABLE.events_*`
WHERE event_name = 'user_engagement' AND _TABLE_SUFFIX = '20190210'
But the figures returned from BigQuery doesn't match with the ones reported on Firebase Analytics Dashboard for the same date. Any clue what's possibly going wrong here?
The following sample query mentioned my Firebase Team, here https://support.google.com/firebase/answer/9037342?hl=en&ref_topic=7029512, is not so helpful as its taking into consideration the current time and getting users accordingly.
N-day active users
/**
* Builds an audience of N-Day Active Users.
*
* N-day active users = users who have logged at least one user_engagement
* event in the last N days.
*/
SELECT
COUNT(DISTINCT user_id) AS n_day_active_users_count
FROM
-- PLEASE REPLACE WITH YOUR TABLE NAME.
`YOUR_TABLE.events_*`
WHERE
event_name = 'user_engagement'
-- Pick events in the last N = 20 days.
AND event_timestamp >
UNIX_MICROS(TIMESTAMP_SUB(CURRENT_TIMESTAMP, INTERVAL 20 DAY))
-- PLEASE REPLACE WITH YOUR DESIRED DATE RANGE.
AND _TABLE_SUFFIX BETWEEN '20180521' AND '20240131';
So given the small discrepancy here, I believe the issue is one of timezones.
When you're looking at a "day" in the Firebase Console, you're looking at the time interval from midnight to midnight in whatever time zone you've specified when you first set up your project. When you're looking at a "day" in BigQuery, you're looking at the time interval from midnight to midnight in UTC.
If you want to make sure you're looking at the events that match up with what's in your console, you should query the event_timestamp value in your BigQuery table (and remember that it might span multiple tables) to match up with what's in your timezone.
Related
In Version 8.12.1 of the Firebase Apple SDK an issue with session_start events being logged during app prewarming on iOS 15+ which inserts additional 'session_start' events. I've noticed that as a result of additional session start rows which inserts additional 'ga_session_id' values into the BigQuery table.
ga_session_id is a unique session identifier associated with each event that occurs within a session and is thus created when this additional session_start fires when the app_warming occurs - using the session_number field and calculating session length it's possible to remove sessions with just one session_start and a small session length but this does not seem to reduce the overall count of sessions by much.
This has impacted the reported number of sessions when querying the BigQuery table when counting distinct user_psuedo_id||ga_session_id.
Is there a way to isolate these sessions in a separate table or constrict them from the query using an additional clause in said query to remove the sessions which are not truly sessions.
https://github.com/firebase/firebase-ios-sdk/issues/6161
https://firebase.google.com/support/release-notes/ios
A simplified version of said query I'm using:
with windowTemp as
(
select
PARSE_DATE("%Y%m%d",event_date) as date_formatted,
event_name,
user_pseudo_id,
(select value.int_value from unnest(event_params) where key = 'ga_session_id') as session_id
from
`firebase-XXXX.analytics_XXX.events_*`
where
_table_suffix between '20210201' and format_date('%Y%m%d',date_sub(current_date(), interval 1 day))
group by
1,2,3,4
)
SELECT
date_formatted,
Count(DISTINCT user_pseudo_id) AS users,
Count(DISTINCT Concat(user_pseudo_id,session_id)) AS sessions,
FROM
windowTemp
GROUP by 1
ORDER BY 1
I am trying to calculate the total time spent by users on my app. We have integrated firebase analytics data in BigQuery. Can I use the sum of the values of engagement_time_msec/1000 in the select statement of my query? This is what I am trying :
SELECT SUM(x.value.int_value) FROM "[dataset]", UNNEST(event_params) AS x WHERE x.key = "engagement_time_msec"
I am getting very big values after executing this query(it giving huge hours per day). I am not sure if is it ok to use SUM("engagement_time_msec") for calculating the total time spent by users on the app.
I am not expecting that users are spending this much time on the app. Is it the right way to calculate engagement_time, or which is the best event to calculate the engagement_time?
Any help would be highly appreciated.
As per google analytics docs in regards to engagement_time_sec, this field is defined as "The additional engagement time (ms) since the last user_engagement event". Therefore, if you only look at this, you are losing all the previous time spent by users before the mentioned user_engagement event is triggered.
What I'd do, since now ga_session_id is defined, would be to grab the maximum and minimum for each ga_session_id timestamp, use the TIMESTAMP_DIFF() function for each case, and sum the results of all the sessions for a given day:
WITH ga_sessions AS (
SELECT
event_timestamp,
event_date,
params.value.int_value AS ga_session_id
FROM
`analytics_123456789.events_*`, UNNEST(event_params) AS params
WHERE
params.key = "ga_session_id"
),
session_length AS (
SELECT
event_date,
TIMESTAMP_DIFF(MAX(TIMESTAMP_MICROS(event_timestamp)), MIN(TIMESTAMP_MICROS(event_timestamp)), SECOND) AS session_duration_seconds
FROM
ga_sessions
WHERE
ga_session_id IS NOT NULL
GROUP BY
1
),
final AS (
SELECT
event_date,
SUM(session_duration_seconds) as total_seconds_in_app
FROM
session_length
GROUP BY
1
ORDER BY
1 DESC
)
SELECT * FROM final
OUTPUT (data extracted from the app I work at):
event_date | total_seconds_in_app
-----------+--------------------
20210920 | 45600
20210919 | 43576
20210918 | 44539
So I have created a daily schedule in BigQuery using "Append to table" preference, so every day it adds yesterday's data to my specified table. I have scheduled to run this query every day at 9AM, but the issue is that sometimes Firebase creates previous day data table in BigQuery later then 9AM.
The example of daily scheduled SELECT I would be using is:
SELECT * FROM `analytics.events_*` WHERE _TABLE_SUFFIX = FORMAT_DATE('%Y%m%d',DATE_SUB(CURRENT_DATE(), INTERVAL 1 DAY))
What would be the best practice to schedule a daily update for the previous day in BigQuery from Firebase, so there are no times where I am missing days?
Bigquery Schedules are set to run at fixed times. If your incoming data is varying in delivery time then BigQuery Schedules are not what you're looking for.
But if you insist in using BigQuery Schedules, you could just relax the WHERE condition and catch "missing" days the next time the schedule runs. Then you flipped your problem and instead need to handle the case of not appending already appended rows (also increasing query cost):
SELECT *
FROM `analytics.events_*`
LEFT JOIN [target dataset].[target table] AS T
USING (event_name, event_timestamp, user_pseudo_id)
WHERE T.event_name IS NULL
AND T.event_timestamp IS NULL
AND T.user_pseudo_id IS NULL
AND _TABLE_SUFFIX >= FORMAT_DATE('%Y%m%d',DATE_SUB(CURRENT_DATE(), INTERVAL 2 DAY))
Or you could alternatively modify the query into an INSERT statement where you insert records and handle duplications similarly:
INSERT `[target dataset].[target table]`
SELECT *
FROM `analytics.events_*`
LEFT JOIN `[target dataset].[target table]` AS T
USING (event_name, event_timestamp, user_pseudo_id)
WHERE _TABLE_SUFFIX >= FORMAT_DATE('%Y%m%d',DATE_SUB(CURRENT_DATE(), INTERVAL 2 DAY))
AND T.event_name IS NULL
AND T.event_timestamp IS NULL
AND T.user_pseudo_id IS NULL
Then you wouldn't need to configure a destination table for the schedule.
Futhermore, if your target table is timestamp partitioned, you can reduce amount of data scanned by limiting the range in which you scan in the target table by adding an additional WHERE condition that strictly limits to a single date instead of the entire table:
...
AND DATE(T.event_timestamp) = DATE_SUB(CURRENT_DATE(), INTERVAL 2)
...
When Firebase Crashlytics is linked to Bigquery, daily Firebase Crashlytics data is pushed to Biquery,
I want to know how to only select this daily crash data from big-query?
Firebase Crashlytics data is exported to BigQuery using the following schema that contains a field named "event_timestamp", which indicates when an event occurred. The Firebase documentation includes Crashlytics queries for querying data in BigQuery, one of these sample queries describes how to query the number of crashes by day for the past month:
SELECT
COUNT(DISTINCT event_id) AS number_of_crashes,
FORMAT_TIMESTAMP("%F", event_timestamp) AS date_of_crashes
FROM
`projectId.firebase_crashlytics.package_name_ANDROID`
GROUP BY
date_of_crashes
ORDER BY
date_of_crashes DESC
LIMIT 30;
Taking the above query as base, you would be able to query the number of crashes in a specific day, e.g., the following query will query the crashes on May 5th, 2019:
SELECT
COUNT(DISTINCT event_id) AS number_of_crashes,
FORMAT_TIMESTAMP("%F", event_timestamp) AS date_of_crashes
FROM
`projectId.firebase_crashlytics.package_name_ANDROID`
WHERE
event_timestamp = TIMESTAMP('2019-05-05')
Im trying to get a the event timestamp from big query (google analytics 360) but I cant seem to find the correct export field to get it.
I have tried hits.eventInfo.timestamp and hits.eventInfo.datetime but none of them seem to work.
My query is
SELECT
hits.eventInfo.timestamp as purchaseDate,
fullVisitorId as visitorId
FROM (tables)
WHERE LOWER(hits.eventInfo.eventAction) == 'purchase'
GROUP BY 2
ORDER BY 1 DESC
LegacySQL
DATE_ADD(TIMESTAMP(FORMAT_UTC_USEC(visitStartTime*1000000)), (hits.time/1000), "SECOND")
This will return YYYY-MM-DD HH:MM:SS in UTC.
Hope it helps.
There is no field for hit timestamp however you can calculate this from hits.time and visitStartTime. There is a row from Schema description:
hits.time INTEGER The number of milliseconds after the visitStartTime when this hit was registered. The first hit has a hits.time of 0