I'm starting to use BigQuery for work so I'm very new to it, and I'm struggling with a request.
So I request data from a mobile game from Firebase. I would like to get the number of types of ads watched per level. For instance:
Type of ads (Inter/Rewarded) - Number of ads watched - level number
I started with this:
SELECT
param.value.string_value AS Type_of_ads,
COUNT(param.value.string_value) AS Nber
FROM
`*Name of the project**`,
UNNEST(event_params) AS param
WHERE
event_name = 'Fullscreen_displayed'
AND param.key = 'Ad_type'
AND user_first_touch_timestamp > 1560587334000000 #15/06/2019
GROUP BY
param.value.string_value
ORDER BY
param.value.string_value
With this, I only have the number of ads and ad_type in total. I would like to have per level. So I did this:
SELECT
param.value.string_value AS Type_of_ads,
COUNT(param.value.string_value) AS Nber,
app_info.version AS Version,
COUNT (event_name) AS Runs
FROM
`Name of the project*`,
UNNEST(event_params) AS param,
UNNEST(event_params) AS param2
WHERE
event_name = 'Fullscreen_displayed'
AND param.key = 'Ad_type'
AND event_name = 'Level_end'
AND param2.key = 'Level'
GROUP BY
Type_of_ads, Version
ORDER BY
Type_of_ads, Version, Runs
But I have the "This query didn't return any result". I can't figure out how to fix it. Could you help me on this matter please? Thank you very much for your help!
Related
Good night,
I am trying to create a query on BigQuery which include the following dimensions: Date, ProductListName, ProductSKU, ProductListPosition and the following metrics:Product List Views, Product List Clicks, Quantity and Number units added to cart.
Nevertheless, Quantity and Units added to cart are not working as expected. Both always show the same result (0). I have already check with Google Analytics the correct results so I know the figure I would have got if the query was correct.
Below these lines, the query I did
Could anyone please help me with that?
Thanks in advance
SELECT
PARSE_DATE("%Y%m%d",date) AS Fecha,
product.productListName AS Lista_Producto,
product.productSKU AS SKU,
product.productListPosition AS Posicion_En_Lista,
SUM(IF(product.isImpression = true,1,0)) AS Vistas_Producto,
SUM(IF(product.isClick = true,1,0)) AS Clics_Producto,
SUM(IF(hits.eCommerceAction.action_type = "3",1,0)) AS AddToCart,
SUM(IF(hits.eCommerceAction.action_type = "6",1,0)) AS Cantidad_Comprada
FROM `bigquery-public-data.google_analytics_sample.ga_sessions_*`
,UNNEST(hits) hits
,UNNEST(hits.product) product
WHERE _TABLE_SUFFIX BETWEEN "20170730" AND "20170731"
--AND product.productSKU = "GGOEYFKQ020699" AND product.productListName = "Category" AND product.productListPosition = 1
AND product.productListName != "(not set)"
GROUP BY Fecha, SKU, Lista_Producto, Posicion_En_Lista
ORDER BY Fecha DESC;
Try use below as add_to_cart:
CASE
WHEN LEAD(productListName) OVER (PARTITION BY sessionID, productSKU ORDER BY hitNumber) = "(not set)" THEN LEAD(product_add_to_cart) OVER (PARTITION BY sessionID, productSKU ORDER BY hitNumber)+product_add_to_cart
ELSE
product_add_to_cart
END
I'm starting using BigQuery these days for work. Until now I managed to request what I wanted but I'm stuck.
I retrieve data from Firebase on my big query console. These data are events from a mobile game we are testing.
I would like to know how many players are there in each level by ABVersion. I can't figure out how to do it.
I did this:
SELECT
param.value.string_value AS Version,
COUNT (DISTINCT user_pseudo_id) AS Players,
param2.value.string_value AS Level
FROM
`*Name of the dataset*`,
UNNEST(event_params) AS param,
UNNEST(event_params) AS param2
WHERE
event_name = 'Level_end'
AND param.key = 'ABVersion'
AND param2.key = 'Level'
GROUP BY Version,Level
And I got this:
I would like to have the number of players per level, with the ABVersion provided.
Thank you for your help!
Level is an integer parameter instead of string. So you should use value.int_value for level.
For the thing you're trying to do, it looks like a better query to me:
SELECT
highest_level,
abversion,
count(*) as players
FROM (
SELECT
user_pseudo_id,
ANY_VALUE((SELECT value.string_value FROM UNNEST(params) WHERE key = 'ABVersion')) as abversion,
MAX((SELECT value.int64_value FROM UNNEST(params) WHERE key = 'Level')) as highest_level
FROM `*Name of the dataset*`,
WHERE
event_name = 'Level_end'
AND EXISTS (SELECT 1 FROM UNNEST(params) WHERE key IN ('Level', 'ABVersion'))
GROUP BY user_pseudo_id
)
GROUP BY 1,2
ORDER BY 1,2
I would like to calculate avg time as seen on the screenshot below using bigquery, but I'm not sure how to add the screen class in my query to be able to yield the same result, can you please help me?
My current query only sum up all values in engagement time msec
SELECT SUM(params.value.int_value) as total_engagement_time_msec,
event_date
FROM `datasetid.events_*`, UNNEST(event_params) as params
WHERE event_name = 'user_engagement'
AND params.key = 'engagement_time_msec'
GROUP BY event_date
I'm taking care of this case to provide you an update.
Your table seems to be grouped by screen class, indeed, that's why the avg aggregation is possible.
I'm not familiar with Firebase, but I found out the BigQuery Export schema and Event Parameter Details that your table in the image is probably using, especially the firebase_screen_class (Screen Class) and engagement_time_msec.
So, after checking your question to include Screen Class as a column, you might want to use two tables to group by firebase_screen_class, for example:
#standardSQL
WITH (
SELECT params.key as screen_class, event_name
FROM `datasetid.events_*`, UNNEST(event_params) as params
WHERE params.key = 'firebase_screen_class'
) as sc
SELECT event_date as eventDate, sc.screen_class as screenClass, AVG(engagement_time_msec) as totalEngagementTime
FROM `datasetid.events_*`, UNNEST(event_params) as params
WHERE event_name = 'user_engagement'
AND params.key = 'engagement_time_msec'
INNER JOIN sc
ON sc.event_name==event_name
GROUP BY event_date, screenClass
Note: The query might need some adjustments
I stumbled unto this standard SQL BigQuery documentation this week, which got me started with a Firebase Analytics Closed Funnel. I however got the wrong results (view image below). There should be no users that had a "Tutorial_LessonCompleted" before they did not start a "Tutorial_LessonStarted >> Lesson = 1 " first. This could be because of various reasons.
Questions:
Is it wise to use the User Property = "first_open_time", or is it better to use the Event = "first_open". How would the latter implementation look like ?
I suspect I am perhaps not correctly drilling down to: Event (String = "Tutorial_LessonStarted") >> parameter (String = "LessonNumber") >> value (String = "lesson1")?
How would a filter on _TABLE_SUFFIX = '20170701' possibly work, I read this will be cheaper. Any optimised code suggestions are received with open arms and an up-vote!
#standardSQL
SELECT
step1, step2, step3, step4, step5, step6,
COUNT(*) AS funnel_count,
COUNT(DISTINCT user_id) AS users
FROM (
SELECT
user_dim.app_info.app_instance_id AS user_id,
event.timestamp_micros AS event_timestamp,
event.name AS step1,
LEAD(event.name, 1) OVER (
PARTITION BY user_dim.app_info.app_instance_id
ORDER BY event.timestamp_micros ASC) as step2,
LEAD(event.name, 2) OVER (
PARTITION BY user_dim.app_info.app_instance_id
ORDER BY event.timestamp_micros ASC) as step3,
LEAD(event.name, 3) OVER (
PARTITION BY user_dim.app_info.app_instance_id
ORDER BY event.timestamp_micros ASC) as step4,
LEAD(event.name, 4) OVER (
PARTITION BY user_dim.app_info.app_instance_id
ORDER BY event.timestamp_micros ASC) as step5,
LEAD(event.name, 5) OVER (
PARTITION BY user_dim.app_info.app_instance_id
ORDER BY event.timestamp_micros ASC) as step6
FROM
`......`,
UNNEST(event_dim) AS event,
UNNEST(user_dim.user_properties) AS user_prop
WHERE user_prop.key = "first_open_time"
ORDER BY 1, 2, 3, 4, 5 ASC
)
WHERE step6 = "Tutorial_LessonStarted" AND EXISTS (
SELECT *
FROM `......`,
UNNEST(event_dim) AS event,
UNNEST(event.params)
WHERE key = 'LessonNumber' AND value.string_value = "lesson1") GROUP BY step1, step2, step3, step4, step5, step6
ORDER BY funnel_count DESC
LIMIT 100;
Note:
Enter your query table FROM, i.e:project_id.com_game_example_IOS.app_events_20170212,
I left out the funnel_count and user_count.
Output:
----------------------------------------------------------
Update since original question above:
#Elliot: I don’t understand why you said: -- ensure that an event with lesson1 precedes Tutorial_LessonStarted.
Tutorial_LessonStarted has a parameter "LessonNumber" with values lesson1,lesson2,lesson3,lesson4.
I want to count all funnels that took place with a last step in the funnel equal to LessonNumber=lesson1.
So, applied to event log-data for a brand new user's first session (aka: an user that fired first_open_time), the answer would be the table below:
View.OnboardingWelcomePage
View.OnboardingFinalPage
View.JamLoading
View.JamLoading
Jam.UserViewsJam
Jam.ProjectOpened
View.JamMixer
Tutorial.LessonStarted (This parameter “LessonNumber"'s value would be equal to “lesson1”)
Jam.ProjectPlayStarted
View.JamLoopSelector
View.JamMixer
View.JamLoopSelector
View.JamMixer
View.JamLoopSelector
View.JamMixer
Tutorial.LessonCompleted
Tutorial.LessonStarted (This parameter “LessonNumber"'s value would be equal to “lesson2”)
So it is important to firstly get all the users that had a first_open_time on a specific day, as well structure the events into a funnel so that the last event in the funnel is one which matches an event and a specific parameter value, and then form the funnel "backwards" from there.
Let me go through some explanation, then see if I can suggest a query to get you started.
It looks like you want to analyze the sequence of events in your analytics data, but the sequence is already there for you--you have an array of the events. Looking at the Firebase schema for BigQuery, event_dim is the relevant column, and unless I'm misunderstanding something, these events are ordered by time. If you want to check what the sixth event's name was, you can use:
event_dim[SAFE_ORDINAL(6)].name
This will evaluate to NULL if there were fewer than six events, or else it will give you the string with the event name.
Another observation is that you are attempting to analyze both event_dim and user_dim, but you are taking the cross product of the two, which will explode the number of rows and make it hard to reason about the results of the query. To look for a specific user property, use an expression of this form:
(SELECT value.value.string_value
FROM UNNEST(user_dim.user_properties)
WHERE key = 'first_open_time') = '<expected property value>'
Combining these two filters, your FROM and WHERE clause would look something like this:
FROM `project_id.com_game_example_IOS.app_events_*`
WHERE _TABLE_SUFFIX = '20170701' AND
event_dim[SAFE_ORDINAL(6)].name = 'Tutorial_LessonStarted' AND
(SELECT value.value.string_value
FROM UNNEST(user_dim.user_properties)
WHERE key = 'first_open_time') = '<expected property value>'
Using the bracket operator to access the steps from event_dim, we can do something like this:
WITH FilteredInput AS (
SELECT *
FROM `project_id.com_game_example_IOS.app_events_*`
WHERE _TABLE_SUFFIX = '20170701' AND
event_dim[SAFE_ORDINAL(6)].name = 'Tutorial_LessonStarted' AND
(SELECT value.value.string_value
FROM UNNEST(user_dim.user_properties)
WHERE key = 'first_open_time') = '<expected property value>' AND
-- ensure that an event with lesson1 precedes Tutorial_LessonStarted
EXISTS (
SELECT 1
FROM UNNEST(event_dim) WITH OFFSET event_offset
CROSS JOIN UNNEST(params)
WHERE key = 'LessonNumber' AND
value.string_value = 'lesson1' AND
event_offset < 5
)
)
SELECT
event_dim[ORDINAL(1)].name AS step1,
event_dim[ORDINAL(2)].name AS step2,
event_dim[ORDINAL(3)].name AS step3,
event_dim[ORDINAL(4)].name AS step4,
event_dim[ORDINAL(5)].name AS step5,
event_dim[ORDINAL(6)].name AS step6,
COUNT(*) AS funnel_count,
COUNT(DISTINCT user_dim.user_id) AS users
FROM FilteredInput
GROUP BY step1, step2, step3, step4, step5, step6;
This will return all unique "paths" along with a count and number of distinct users for each. Note that I'm just writing this off the top of my head--I don't have representative data that I can try it on--so there may be syntax or other errors.
I have managed to calculate total events by ISOweek but not unique events for a given Google Analytics Event using BigQuery. When checking GA, total_events matches the GA interface on the dot but unique_events are off. Do you know how I can solve this?
The query:
SELECT INTEGER(STRFTIME_UTC_USEC(PARSE_UTC_USEC(date),"%V")) iso8601_week_number,
hits.eventInfo.eventCategory,
hits.eventInfo.eventAction,
COUNT(hits.eventInfo.eventCategory) AS total_events,
EXACT_COUNT_DISTINCT(fullVisitorId) AS unique_events
FROM
TABLE_DATE_RANGE([XXXXXX.ga_sessions_], TIMESTAMP('2017-05-01'), TIMESTAMP('2017-05-07'))
WHERE
hits.type = 'EVENT' AND hits.eventInfo.eventCategory = 'BIG_Transaction'
GROUP BY
iso8601_week_number, hits.eventInfo.eventCategory, hits.eventInfo.eventAction
Depending on the scope you need to count(distinct ) different things, but you always need to fulfill these conditions:
unique events refer to the combination of category, action and label
make sure eventAction is not NULL
make sure eventLabel is not NULL
eventCategory is allowed be NULL
I'm using COALESCE() to avoid NULLs
Example Session Scope
SELECT
SUM( (SELECT COUNT(h.eventInfo.eventCategory) FROM t.hits h) ) events,
SUM( (SELECT COUNT(DISTINCT
CONCAT( h.eventInfo.eventCategory,
COALESCE(h.eventinfo.eventaction,''),
COALESCE(h.eventinfo.eventlabel, ''))
)
FROM
t.hits h ) ) uniqueEvents
FROM
`google.com:analytics-bigquery.LondonCycleHelmet.ga_sessions_20130910` t
Example Hit Scope
SELECT
h.eventInfo.eventCategory,
COUNT(1) events,
-- we need to take sessions into account, so we add fullvisitorid and visitstarttime
COUNT(DISTINCT CONCAT(fullvisitorid, CAST(visitstarttime AS string),
COALESCE(h.eventinfo.eventaction,''),
COALESCE(h.eventinfo.eventlabel, ''))) uniqueEvents
FROM
`google.com:analytics-bigquery.LondonCycleHelmet.ga_sessions_20130910` t,
t.hits h
WHERE
h.type='EVENT'
GROUP BY
1
ORDER BY
2 DESC
hth!
The definition of unique events in Google Analytics is:
A count of the number of times an event with the category/action/label
value was seen at least once within a session.
In other words, the number of sessions in which a specific event (defined by category, action AND label) was sent. In your query, you count the number of unique visitors that had the event, while you need to count the number of sessions and keep in mind that events with different labels should be counted as different unique events (although we are only interested in category and action).
A possible way to fix your code is:
SELECT
INTEGER(STRFTIME_UTC_USEC(PARSE_UTC_USEC(date),"%V")) iso8601_week_number,
hits.eventInfo.eventCategory,
hits.eventInfo.eventAction,
COUNT(hits.eventInfo.eventCategory) AS total_events,
EXACT_COUNT_DISTINCT(CONCAT(fullVisitorId,'-',string(visitId),'-',date,'-',ifnull(hits.eventInfo.eventLabel,'null'))) AS unique_events
FROM
TABLE_DATE_RANGE([XXXXXX.ga_sessions_], TIMESTAMP('2017-05-01'), TIMESTAMP('2017-05-07'))
WHERE
hits.type = 'EVENT' AND hits.eventInfo.eventCategory = 'BIG_Transaction'
GROUP BY
iso8601_week_number, hits.eventInfo.eventCategory, hits.eventInfo.eventAction
The results of this query should match with the data in the GA interface.
I believe the issue is that you are only counting the number of unique visitors have completed the specified action, while GA defines unique events as "The number of times during a date range that a session contained the specific dimension".
Therefore, I would just change your code to the below:
SELECT INTEGER(STRFTIME_UTC_USEC(PARSE_UTC_USEC(date),"%V")) iso8601_week_number,
hits.eventInfo.eventCategory,
hits.eventInfo.eventAction,
COUNT(hits.eventInfo.eventCategory) AS total_events,
EXACT_COUNT_DISTINCT(CONCAT(fullVisitorId, STRING(visitId))) AS unique_events
FROM
TABLE_DATE_RANGE([XXXXXX.ga_sessions_], TIMESTAMP('2017-05-01'), TIMESTAMP('2017-05-07'))
WHERE
hits.type = 'EVENT' AND hits.eventInfo.eventCategory = 'BIG_Transaction'
GROUP BY
iso8601_week_number, hits.eventInfo.eventCategory, hits.eventInfo.eventAction
This should give you the distinct count of sessions that had the given events.
We did something similar to what #Martin was suggesting with some cool CTEs and we were able to get an 100% match on what was coming out of Google Analytics from BigQuery.
Checkout the code snippet below that returns a per day sum of sessions + unique Add to Cart events:
#standardSQL
WITH AN_ATC AS
(
SELECT
-- full date w/ hyphens (ie 2021-01-07)
CAST(format_date('%Y-%m-%d', parse_date("%Y%m%d", date)) AS DATE) as DATE,
-- COUNT OF SESSIONS
COUNT(DISTINCT CONCAT(fullVisitorId, CAST(visitStartTime AS STRING))) AS Sessions,
-- COUNT OF UNIQUE EVENTS PER SESSION
COUNT(DISTINCT CONCAT(fullvisitorid, CAST(visitstarttime AS string),
COALESCE(hits.eventinfo.eventaction,''),
COALESCE(hits.eventinfo.eventlabel, ''))) AS EVENTS
FROM `an-big-query.PROJECT_ID.ga_sessions_*` ,
UNNEST(hits) as hits
WHERE
-- start date
_table_suffix BETWEEN '20190101'
-- yesterday
AND FORMAT_DATE('%Y%m%d',DATE_SUB(CURRENT_DATE(),INTERVAL 1 DAY))
AND hits.eventInfo.eventAction = 'add to cart'
GROUP BY
date
)
SELECT
DATE,
SESSIONS,
EVENTS
FROM AN_ATC
ORDER BY date DESC
Where,
SESSIONS = Google Analytics ga:Sessions
and
EVENTS = Google Analytics ga:uniqueEvents
BOTH with eventAction=#add to cart
Hope that helps everyone that was searching/googling!