Is it possible on firebase A/B tests to monitor the event counts instead of the conversion rates?
For example, I would like to know if users in Variant A trigger a certain event more times than the Baseline (not only if they trigger the event or not, but how many times they trigger it).
Thanks!
Oriol
That's a really good question. For each event that is triggered, you should be able to derive event count as well as an event DAU metric. Firebase has limited capability to run such analysis. One way to do this is to download the data and run a manual analysis. Or use a tool like Statsig that allows you compute these metrics automatically. Here's a screenshot of what you get for every A/B experiment, and you can see how the tool breaks down Event Count and Event DAU for each metric.
Disclaimer: I work at Statsig.
Related
I want to create one AB Test using Firebase, I can not found the metrics for tracking user engagement durations, is there have one suggestion or solution to solve this problem?
I try to use screen_view event to measure the duration, but this event be track as event count only, I expect using the event property value for measure the metrics.
Thanks!
We use Firebase A/B test product for our mobile apps. We need to reach the parameters of our events and make a deeper analyze. We have worked with BigQuery before for this, but it requires a lot of effort.
Let me tell you briefly about our problem:
Let's say we have an event called add_to_cart. We want to look at the number of times the add_to_cart is triggered from a specific screen in the A/B test results. For example, those whose firebase screen class is category_page. This data can be accessed by writing a query over BigQuery, but create extra effort for different needs.
Is there a short way or tool about doing analysis by event parameters?
As we find Firebase's reporting and analysis insufficient, we will decide to use a different tool. If anyone encounters such a problem, it is possible to make a deep analysis through BigQuery.
Another way you can use Audience as a hacky way.
1. Go to Custom Definitions section and create a custom definition.
Your scope should be "User". Select firebase_exp_<N> as the User property. Because Firebase defines a property for each user it adds to the experience. You can find the <N> number from the link on your A/B test page.
E.g. your A/B test link is like: https://console.firebase.google.com/u/0/project/your-project/config/experiment/results/20. The <N> number is 20 and user property is firebase_exp_20.
2. Create Audience for each control group
Create a new audience according to this created dimension value. A value of 0 corresponds to Baseline. Each control group after that continues with consecutive numbers. (1,2,3..)
3. Go to Analytics
Go to Analytics and do your analysis for each control group with these Audiences.
I hope it helps.
My question: how do I update "this and all future" instances in a recurring event which is limited by count so that the total number of events stays consistent?
What is the problem:
Trying to modify recurring event and I follow the below guide:
https://developers.google.com/calendar/recurringevents
Basically to update all future recurring events using a target event, the doc says one need to do two calls:
update existing event to make so it ends before the target event date
create a new recurring event with the same fields except of those need changes.
That works fine until there is an event that is limited by the number of occurrences.
Let's say there is a recurring event limited by 10 occurrences and target event is 5th event.
Now I need to split the original so that the first 4 events goes to the original one (so I update COUNT from 10 to 4) and then I create a new recurring event that holds the rest 6 events (so COUNT is 6 in this case)
My first observation is that this is not how the split events are displayed in google calendar - if I test that manually, the both events still show 10 occurrences but the second one doesn't produce any extra events (I'd expect 14 events from developer perspective, yet there are 10 as any user would expect). That implies there is a different approach here? Is it?
Also if I end up counting manually the number of events, there are still issues with cases like deleting one of the events first (let's say, the 4th event) - now how do I know that I need to show 6 instances in the new one and not 7?
Those thoughts make me think there is a better approach, but I can't find any other alternatives. Any advice on that?
UPDATE
It seems like google does it differently: for example after changing a title for "this and future" events in calendar view, it doesn't seem to produce two different recurring events since if you try to delete "all" events, that will remove all of them completely (rather than deleting only one chunk, either before or after the target event)
It seems like they are creating a bunch of exceptions or maybe "recurring exception" or something to do that. Can't find any examples on how to do that as of now thought.
Can't find any good solution for this after a few days of research and while I need to move forward I ended up with a sort of "compromise" between "good enough UX for my case" and "breaking best practice".
So I ended up updating each event individually which goes against google's warning as shown below but I limited the max count by 50. This is not necessary what others want to do, but this is good enough for the real world use case in my app.
Warning: Do not modify instances individually when you want to modify
the entire recurring event, or "this and following" instances. This
creates lots of exceptions that clutter the calendar, slowing down
access and sending a high number of change notifications to users.
And if user needs to schedule more than that, the user is asked to use "end date" instead.
Again, not ideal by any means so if anyone knows how to handle that correctly or knows how google handles that, you are very welcome to share it! (meh... and I need that for outlook too now...)
UPDATE: just got an idea: as an improvement, one can edit either "all future events" or alternatively the master event + "all previous events" depending on the index of the target event. In this case one can limit the number of requests by 2 (so in case of 50 events I'll need to do 25 requests maximum)
So if user wants to change the title from "Hello" to "Goodby" and if the user picked event number 5 in the series of 50 events to change all future events, we can change the master event to "Goodby" which will change the title of all events, and then update the first 4 events to the original "Hello".
Obligatory summary of comments and chat:
Updating events:
To update specific events in a recurring event you need to update the individual instance by specifying the event instance ID.
This is just the event ID concatenated with a datetime stamp (you can see this when making an Events: instances request for your eventID; if your event ID is xxxxxxxxxxxx then an instance ID would be something like xxxxxxxxxxxx__20200603T170000Z).
Unfortunately there's no direct update-instances endpoint so to update multiple instances in one request you'd need to use batching
The API doesn't have a dedicated method for updating recurring events regardless of the recurrence type, and I presume this is the reason the documentation says to edit the previous recurring event by cutting it down and inserting a new one, as per Google's warning:
Do not modify instances individually when you want to modify the entire recurring event, or "this and following" instances. This creates lots of exceptions that clutter the calendar, slowing down access and sending a high number of change notifications to users.
Batching:
Making a batch update on event instances does keep count consistency. If you edit instances in a batch and then use the 'this and all future events' option when deleting one of the instances of the recurring event they do all get deleted as they're still a part of the recurrance. There is no new event being created in either scenario, the event instances are being changed.
If you play around with Events: instances and use Events: update to change only some instances of an event, then you can see that they all stay part of the same recurrence chain and there is no count change.
For arbitrary large counts, even if you have a recurring event with 9999999 instances, each event still has an ID which you can retrieve from Events: instances. It's stored as a single event for event use, but the IDs of the instances are the identifiers which are different.
Honestly, it's not great that you have to edit each one manually; for large counts like 9999999 it's basically infeasible because you'll have to make a batch request for each set of 100 instances you want to change, but it's the only option available via the API at the moment.
Feature Request:
You can however let Google know that this is a feature that is important for the Calendar API and that you would like to request they implement it. Google's Issue Tracker is a place for developers to report issues and make feature requests for their development services, I'd urge you to make a feature request there, the Calendar API feature request form can be found here.
I'm tracking events on my websites and in those events I'm sending along an event value. The event value is how many times a person has done that particular event, so it could be a number from 1 to infinity.
I'd like to figure out where on a bell curve (or something like that) the event values are. I'd like to somehow find the distribution of the event values, but Google Analytics only gives me the average values.
How would you go about finding the distribution of event values in Google Analytics?
You could use metric-Filters in a segment to get the number of Events for each value.
If you use the API to extract the data you can write a loop to get every possible Value.
Still that is not the best solution. If you want to do this more than ones it would be better to add the value as Event-Label or Event-Action. GA saves this as a Dimension and you can easaly export and analyze this.
We recently released two typefaces on our website for free (albeit suggesting an optional donation). I decided we should track downloads through Google Analytics using the event feature, so we ended up adding the corresponding JS snippet to the download form (on submit), something akin to this:
_gaq.push(['_trackEvent', 'Typeface', 'Download', 'Typeface #1', parseInt($('input[name=amount]').val(), 10) || 0]);
I also decided we might as well use GA to keep track of donations, so as you might have noticed the optional donation amount is being sent as the event value argument. There's already a browser-side numeric-only verification, and it will set it to 0 in case it's empty (NaN), so we're completely sure it's always an integer (required type for the argument).
I configured two different goals (one for each typeface) in our GA profile, using the two different events as their respective conditions, as recommended by every howto I've been reading about this subject.
However, some of the reported data appears to be somewhat inflated. According to GA there's been, as of now, 455 unique events out of 550 total events, which seems to be okay, but apparently it's worth a value of over a million dollars. And, believe me on this, we have not received such a huge amount, at least just yet.
According to GA: Event Value is the total value of an event or set of events. It is calculated by multiplying the per-event value by the number of times the event occurred.
I assumed I could set individual values to different instances of the same event, even GA documentation leads me to believe so with their examples, so I don't really understand why it's being reported as such an inflated total value.
Is there something wrong with my assumption? Is this the correct approach to what I'm trying to accomplish? should I just forget about keeping track of donations using this method and resort to using the e-commerce feature instead as I've also been reading about?
I'm not checking for any verification of a donation successfully completing, so I'm left with an estimate and I'm okay with that. Maybe someone jokingly wrote off some exaggerated amount then never completed the donation process?
Your assumption is right : you could set individual values to each event and "the report adds the total values based on each event count" (as explain in doc).
The main problem with your approach is the one you mentioned : you count the donation at form validation, before its confirmation and even before you told your visitor that the donation must be made via PayPal. So yes : some people probably wrote off some exaggerated amount or simply not complete the donation process.
I recommend you to use e-commerce tracking after the PayPal payment to avoid unconfirmed donation tracking and the lack of deduplication using goals values to monitor amounts.