We use Firebase A/B test product for our mobile apps. We need to reach the parameters of our events and make a deeper analyze. We have worked with BigQuery before for this, but it requires a lot of effort.
Let me tell you briefly about our problem:
Let's say we have an event called add_to_cart. We want to look at the number of times the add_to_cart is triggered from a specific screen in the A/B test results. For example, those whose firebase screen class is category_page. This data can be accessed by writing a query over BigQuery, but create extra effort for different needs.
Is there a short way or tool about doing analysis by event parameters?
As we find Firebase's reporting and analysis insufficient, we will decide to use a different tool. If anyone encounters such a problem, it is possible to make a deep analysis through BigQuery.
Another way you can use Audience as a hacky way.
1. Go to Custom Definitions section and create a custom definition.
Your scope should be "User". Select firebase_exp_<N> as the User property. Because Firebase defines a property for each user it adds to the experience. You can find the <N> number from the link on your A/B test page.
E.g. your A/B test link is like: https://console.firebase.google.com/u/0/project/your-project/config/experiment/results/20. The <N> number is 20 and user property is firebase_exp_20.
2. Create Audience for each control group
Create a new audience according to this created dimension value. A value of 0 corresponds to Baseline. Each control group after that continues with consecutive numbers. (1,2,3..)
3. Go to Analytics
Go to Analytics and do your analysis for each control group with these Audiences.
I hope it helps.
Related
As some other questions pointed out if you're setting up a Remote Config based AB test there's no activation event based on user first opened.
We want to AB test our new onboarding flow against the previous onboarding experience however without a startup trigger we're not sure how to properly create this experiment.
One SO answer talks about sending a custom activation event with a timestamp and then filtering the test participants by that timestamp e.g. custom_first_open > 1234567... however the onboarding flow is the first thing the user is to see.
From my understanding as soon as the user initializes their remote config they will be subscribed to any active experiments. We would have to send the custom event before initialization and it would have to be immediately available to the AB test. AB test data and Firebase events both seem to be very slow to register (hours to days) so I doubt it would properly configure the user for the onboarding test using this trick.
Is there another way to use AB testing to test onboarding efficacy only against new users?
There are a couple of ways to go about this.
First, when you create the experiment you can limit the experiment targeting to only include users on a new version or build of the app -- or country, etc.
example targeting
You can also only target users in an Audience you define, which give you pretty flexible abilities to define whatever group you'd like to roll the tests out to.
creating an audience
Note - we tend to recommend you use first_touch_timestamp for correctly identifying new users. Better than first_open.
Also, outcomes are easier to measure when you're looking at ARPU/LTV outcomes
I'm running an A/B Test targeting 100% of iOS users with specific versions using a regex to match versions 2.1.27 and up, here's the regex in case it's relevant:
2\.1\.([3-9].|2[7-9])
I did not set any activation event for the experiment; meaning I left that field blank.
Now my question is: Who becomes part of the experiment? Anyone who opens the app with a matching version? Anyone who starts a session with a matching version? Anyone who engages with a matching session?
So far, it says the total number of users that have been exposed to the experiment is 9,6K. I'm trying to set up a funnel that only includes these users but I can't figure out one that shows me anything close to that number in the date range of the experiment.
Firebase Support says:
Firebase will determine if a user will be part of the experiment when your app do the activatedFetch method.
That's the answer I was looking for.
What are the drawbacks of using a single custom dimension and a single custom metric in GA to track everything?
Ok, I am limited to 20 custom dimensions and 20 custom metrics for a property in GA. But the number of values for a single dimension is unlimited.
So, I will be able to, for example, create a dimension called Dimensio and a metric called Metri. Then, if I want to track a custom event called Ev1 I will tag on Dimensio with value Ev1 and on Metri with value 1.
I am just suspicious, why having such a solution GA ever worried about having 20 dimensions and 20 metrics (it would be enough to have 1 dimension and 1 metric).
Thank you.
There are numerous reasons/use cases where you'd want more than 1 dimension. I'd suggest some reading on scope.
One use case that will break the 1 dimension usage:
For a music streaming service, you want to track the user type (premium vs free) and as well as the version of the app (v1.0.1 vs v1.0.2) for debugging purposes.
The user type should be on a user scope, as in it will persist as long as the user doesn't clear their cookies. The version might be on a hit scope, as you might push out an emergency fix part way through the user session.
Using the above use case, you cannot report on the user type as well as version in the same report if you were only using one dimension.
I know this has been discussed a lot on SO, but even based on the previous questions and answers I still can't quite understand the details of how event logging is really supposed to work in Firebase.
Let's say somewhere in my code I have this:
FirebaseAnalytics mFirebaseAnalytics = FirebaseAnalytics.getInstance(this);
Bundle bundle = new Bundle();
bundle.putInt("card_id", 912);
mFirebaseAnalytics.logEvent("card_open", bundle);
Do I understand correctly that the "card open" even should now appear in Firebase Console with the "card_id" parameter?
Then I click More > Edit parameter reporting, enter the name of the parameter I added in my code, and it should appear on the list.
I have read the docs several times, but I still can't get it. Can I not even use any of the events and parameters provided by FirebaseAnalytics.Param and FirebaseAnalytics.Event and just send custom events/parameters (within the known limits, of course).
Another question is whether the limit on the number of global parameters force you to use Big Query?
Thank you!
You're correct. You should see the card_open event with card_id parameter in the analytics console.
Note that there's a delay in the analytics reporting, about 4 to 6 hours after it was logged and uploaded by the device.
Just to ensure that your implementation is correct. I mean, to verify that the events, parameters and user properties are logged in the right time and the right place correctly, use the DebugView console.
I also believe that you should be able see the card_id parameter in the custom reporting for card_open event, considering the delay in the reporting.
For your last question, the parameter limit is on a project level perspective now, compared before that it was per app. Parameters with the same name across different apps will be counted as 1. Not sure, maybe Firebase would like us (developers) to use the parameters accordingly.
After how many clicks will the Core Reporting API start to sample the click data, when using samplingLevel=LARGE?
I'm trying to retrieve data from a large account (i.e. more than 30,000 clicks/day on average) and the number of clicks doesn't always match what I can see on Google Analytics. This, however, seems to happen only on this large account, and not every day. Strangely, on those days where the click count doesn't match, transactions and revenue match what I can see in Google Analytics.
In my query, I'm only trying to retrieve the data for a given account, without applying any filter.
EDIT: If I don't retrieve data aggregated at the account level ― thus not including ga:adwordsCampaignID, ga:adwordsAdGroupID and ga:adwordsCriteriaID in the dimensions ― I can retrieve all the clicks.
EDIT2: If add the ga:deviceCategory dimension, along with ga:adwordsCampaignID, ga:adwordsAdGroupID and ga:adwordsCriteriaID I can retrieve all clicks. I'm not sure if this can help narrow down the issue.
Google Analytics has a cardinality of 50K after that you ll receive (others)
Based on the "EDIT" i can safely assume the reason is that when a click doesn't have an 'ga:adwordsCampaignID' associated with it, it ll not retrieve that click. This happened to me with custom dimensions.
EDIT: Try using the 'include-empty-rows' parameter on your query. https://developers.google.com/analytics/devguides/reporting/core/v3/reference#includeEmptyRows