I noticed there are thousands of events duplicated in the events tables of BigQuery (in an integration with Firebase).
My definition of duplicated is: 2 or more events that share the same data in all these fields:
event_timestamp, event_name, user_pseudo_id, app_info.id, device.advertising_id
It happens for automatically collected events, and also custom events. I found some of the parameters that could differ from one result to the other are (what make those events different):
event_server_timestamp_offset, geo.continent, geo.country
I guess there is no reason for a duplicated event at the same moment, same user, same app, same device, but one event is geo.continent=America and the other geo.continent=Asia.
Any thoughts why this is happening? Thanks in advance.
Google's explanation is that Firebase data duplication in BigQuery is mostly related to network issues on the client's side that cause events to be buffered and sent twice.
However there is a way to deduplicate these events by using event_server_timestamp_offset. This field is difference between the time the event was sent to Google's server and when it was received.
This means that given the same event_timestamp, event_name and user_pseudo_id you could take only the event with lower event_server_timestamp_offset to have a correct result.
You can also safely delete duplicates records from your event table.
Sorry I can't share sources for this because the answer came from Google Analytics support, as I was encountering the same issue.
We use the QUALIFY clause for deduplication Firebase events in BigQuery:
SELECT
*
FROM
`project.dataset.events_*`
QUALIFY
ROW_NUMBER() OVER (
PARTITION BY
user_pseudo_id,
event_name,
event_timestamp,
TO_JSON_STRING(event_params)
) = 1
Qualifying columns:
- name: user_pseudo_id
description: Autogenerated pseudonymous ID for the user -
Unique identifier for a specific installation of application on a client device,
e.g. "938642951.1666427135".
All events generated by that device will be tagged with this pseudonymous ID,
so that you can relate events from the same user together.
- name: event_name
description: Event name, e.g. "app_launch", "session_start", "login", "logout" etc.
- name: event_timestamp
description: The time (in microseconds, UTC) at which the event was logged on the client,
e.g. "1666529002225262".
- name: event_params
description: A repeated record (ARRAY) of the parameters associated with this event.
Related
I have a firebase connection with BigQuery and i want to extract the daily user engagement information that is present on the analytics dashboard.
Firebase Analytics Dashboard Daily User Engagement
I´ve already tried to find the numbers using EVENT_NAME filter as 'user_engagement', 'screen_view' and EVENT_PARAMS_KEY as 'engagement_time_msec', 'engaged_session_event'.
With the filter mentioned previously i couldn´t even get near from the firebase values.
Somebody knows how to reach them
According to the documentation, user engagement data collection is triggered periodically by Firebase SDK, while the app is running in the foreground. Saying this, for user_engagement we might be looking over the events, filtering for the entry where event_params.key = "engagement_time_msec" and fetching the event_params.value.int_value from there.
SELECT event_timestamp, user_pseudo_id,
(SELECT value.int_value FROM UNNEST(event_params) WHERE key =
"engagement_time_msec") AS engagement_time
FROM `firebase*.events_*`
WHERE event_name = "user_engagement"
I have a booking system where a booking can be made:
1. immediately by a customer, or
2. on their behalf by a customer service agent.
For 1, I've had success POSTing an event hit to the Measurement Protocol with a cid and our uid for a "purchase". Initially without the cid, it wasn't working despite the docs saying only one of uid or cid are required.
For 2, which isn't customer initiated, when I don't pass a cid nothing comes through to GA (neither the real-time dashboard or the ecomm report).
Here's the hit for option 1, broken in to lines for easier viewing:
t=event
&v=1
&tid=UA-XXXXXXX-2
&ec=Ecommerce
&ea=Purchase
&ti=176410
&ta=Some+Affiliation
&pa=purchase
&tr=123.45
&pr1id=Repair367
&pr1nm=Something+Replacement
&pr1pr=123.45
&pr1qt=1
&pr1ca=Repair%2FInspection
&pr1ps=1
&cd3=None
&ni=1
&uid=469585
OK, so it turns out this is the case:
If using cid, your hit will appear in your property's standard view's real-time Event screen (immediately) and ecomm Sales Performance screen (~10mins later for me)
If using uid and no cid, your hit will appear ONLY in those screens under your property's UserID view.
I'd been looking in the standard view and nothing was appearing as soon as I swapped over to a uid-only hit when it was customer service initiated. So although the docs say only one of either uid or cid are required, it should mention it has a significant affect on where the data is viewable in GA.
Sigh.
Firebase Analytics connected to BigQuery and the BQ table schema is described here:
https://support.google.com/firebase/answer/7029846
I would like to find out how each event record can be uniquely identified.
Originally I thought that a combination of a
user_pseudo_id and event_timestamp
is to be unique. But I found out that it is not unique...
I added: event_date, event_name, event_previous_timestamp, stream_id, etc. into the 'group by' clause, but nothing helps.
Can anybody advise me, what makes the event record unique, please?
We are using Google Advertising ID as unique device ID. A user may have logged on to your app with multiple devices but using the same account, so in this case user_id is not unique, user_pseudo_id for the same device will change if he/she re-installs the app. Only assumption out here is that the user has not intentionally reset his/her GAID.The GAID field can be found under event_params with Key as gaid in BigQuery. Hope this helps!
I'm under the impression fullVisitorId being just a hash of clientId, there should be one-to-one mapping between the two. But here, I've a situation where few of the fullVisitorId are mapped to two different client Id (we're collecting GA Client ID into User scoped custom dimensions)
Is that possible ? under what circumstances?
Thanks for any clarification on this
Cheers!
[edit: ] attaching screenshot
You may be interested in reading about the Google Analytics schema for BigQuery. Some of the relevant parts are:
fullVisitorId: The unique visitor ID (also known as client ID).
visitId: An identifier for this session. This is part of the value usually stored as the _utmb cookie. This is only unique to the user. For a completely unique ID, you should use a combination of fullVisitorId and visitId.
So client ID and full visitor ID are synonymous, and if you want a unique ID for a particular visit, you should use a combination of fullVisitorId and visitId.
I am experiencing a strange behaviour, when GA is exporting data to BigQuery I see lots of hits with type EVENT but their corresponding Event Category, Label and Action are null. I've reached support and all they told me is it is as its send (advising that tagging solution sends null or undefined event type hits) and that in GA portal such events are filtered out.
I tried sending such hits (type=event, event category==null) using GTM or manually from page code and these events are never to be found in BigQuery, meaning that those null events are something different.
Any ideas guys?
Query:
SELECT fullVisitorId, visitId, hits.type, hits.eventInfo.eventCategory, hits.eventInfo.eventAction, hits.eventInfo.eventLabel, hits.eventInfo.eventValue
FROM [XXXXXXX.ga_sessions_20160316]
where hits.type='EVENT' and hits.eventInfo.eventCategory is null
As no answers so far, I will share my findings on this, may be this will help somebody.
So it is perfectly normal to see NULL events in your BigQuery. An easiest way to reproduce would be to construct a hit using measurement protocol omiting the category, action and label parameters but specifying custom dimensions and metrics.
Also, hits from older versions of GA might record in BQ as nullable events.
Hope this helps
As per the Big Query export schema, the timing hits on the website are captured as event hits. Screenshot below:
https://support.google.com/analytics/answer/3437719?hl=en
If you add the variable hits.latencyTracking.pageLoadTime to the above query, you will see the page load time data with category, action and label as null.
Below is the final query:
SELECT fullvisitorid,
visitid,
hits.type,
hits.eventinfo.eventcategory,
hits.eventinfo.eventaction,
hits.eventinfo.eventlabel,
hits.eventinfo.eventvalue,
hits.latencytracking.pageloadtime
FROM [xxxxxxx.ga_sessions_20160316]
WHERE hits.type = 'EVENT'
AND hits.eventinfo.eventcategory IS NULL
This is why null event data is present in the Big Query. Happy coding!