As some other questions pointed out if you're setting up a Remote Config based AB test there's no activation event based on user first opened.
We want to AB test our new onboarding flow against the previous onboarding experience however without a startup trigger we're not sure how to properly create this experiment.
One SO answer talks about sending a custom activation event with a timestamp and then filtering the test participants by that timestamp e.g. custom_first_open > 1234567... however the onboarding flow is the first thing the user is to see.
From my understanding as soon as the user initializes their remote config they will be subscribed to any active experiments. We would have to send the custom event before initialization and it would have to be immediately available to the AB test. AB test data and Firebase events both seem to be very slow to register (hours to days) so I doubt it would properly configure the user for the onboarding test using this trick.
Is there another way to use AB testing to test onboarding efficacy only against new users?
There are a couple of ways to go about this.
First, when you create the experiment you can limit the experiment targeting to only include users on a new version or build of the app -- or country, etc.
example targeting
You can also only target users in an Audience you define, which give you pretty flexible abilities to define whatever group you'd like to roll the tests out to.
creating an audience
Note - we tend to recommend you use first_touch_timestamp for correctly identifying new users. Better than first_open.
Also, outcomes are easier to measure when you're looking at ARPU/LTV outcomes
Related
We use Firebase A/B test product for our mobile apps. We need to reach the parameters of our events and make a deeper analyze. We have worked with BigQuery before for this, but it requires a lot of effort.
Let me tell you briefly about our problem:
Let's say we have an event called add_to_cart. We want to look at the number of times the add_to_cart is triggered from a specific screen in the A/B test results. For example, those whose firebase screen class is category_page. This data can be accessed by writing a query over BigQuery, but create extra effort for different needs.
Is there a short way or tool about doing analysis by event parameters?
As we find Firebase's reporting and analysis insufficient, we will decide to use a different tool. If anyone encounters such a problem, it is possible to make a deep analysis through BigQuery.
Another way you can use Audience as a hacky way.
1. Go to Custom Definitions section and create a custom definition.
Your scope should be "User". Select firebase_exp_<N> as the User property. Because Firebase defines a property for each user it adds to the experience. You can find the <N> number from the link on your A/B test page.
E.g. your A/B test link is like: https://console.firebase.google.com/u/0/project/your-project/config/experiment/results/20. The <N> number is 20 and user property is firebase_exp_20.
2. Create Audience for each control group
Create a new audience according to this created dimension value. A value of 0 corresponds to Baseline. Each control group after that continues with consecutive numbers. (1,2,3..)
3. Go to Analytics
Go to Analytics and do your analysis for each control group with these Audiences.
I hope it helps.
I am using Google Analytics 4 (GA4) on the client to track a whole bunch of different events. However, there are 2 scenarios that I can't cover client side:
A user completing check out on a payment page hosted by a third-party (Stripe in this case).
A refund that is made by the support team.
These events are handled by the server using webhooks. To me it seems like the most straightforward solution, would be to let the server send the event to GA4 (as opposed to the client sending it). I believe the Measurement Protocol should be used for this.
For each event submitted through the Measurement Protocol a client_id is required. When the client is submitting an event, this is an automatically generated ID which is used to track a particular device.
My question thus is, what should the client_id be when submitting an event server-side?
Should the same client_id perhaps be used for all events, as to recognize the server as one device? I have read some people proposing to use a randomly generated client_id for each event, but this would result in a new user to be recognized for every server-side event...
EDIT:
One of the answers proposes to use the client_id, which is part of the request as a cookie. However, for both examples given above, this cookie is not present as the request is made by a third-party webhook and not by the user.
I could of course store the client_id in the DB, but the refund in the second example is given by the support team. And thus conceptually it feels odd to associate that event with the user's client_id as the client_id is just a way to recognize the user's device? I.e. it is not the user's device which triggered the refund event here.
Another refund event example would be when user A makes a purchase with user B and user B refunds this purchase a week later. In this situation, should the client_id be the one of user A or of user B? Again, it feels odd to use a stored client_id here. Because, what if user A is logged in on two devices? Which client_id should be used here then?
Great question. Yes, your aim to use Measurement Protocol is a proper solution here.
Do not hardcode the client id. It's gonna be a hellish mess in reports. The nature of user-based reporting (which GA is) demands client ids to uniquely identify users. To your best ability.
GA stores the client id in a cookie. You should have convenient and immediate access to it on every client hit to BE. The cookie name is _ga. GA4 appends the measurement id to the cookie name. Here, google's docs on it: https://developers.google.com/analytics/devguides/collection/analyticsjs/cookie-usage But you can easily find it if you inspect "collect" hits and look at their payloads. There's another cookie named _gid that contains a different value. That would be a unique client id. Set it too if you can, but don't use it for the normal client id. It has a different purpose. Here how the cookie looks here, on stack:
And here it is in Network. You will need it for proper debugging. Mostly to make sure your FE client ids are the same as BE client ids:
Keep an eye on the cases when the cookie is not set. When a cookie is not set, that most frequently means the user is using an ad-blocker. Your analysts will still want to know that the transaction happened even if there's a lack of context about the user. You still can track them properly.
3.1 The laziest solution would be giving them an "AnonymousUser" client id and then append a random number to that so that it would
both indicate that a user is anonymous and still make it possible
for GA to separate them.
3.2 A better solution would be for you to make a fingerprint client id for such users, say, hashing a concatenated string of their
useragent+ip+locale+screen resolution, this is up to your analysts
to actually work on the definition of a unique user if the google
analytics library is unable to do it.
3.3 Finally, one of the best solutions for you would be generating a client id on your own, keeping GA's format and maybe adding an indicator there that it has been generated on your end just for easier debugging in the Future and setting it as a cookie, using it instead of _ga. Just use a different cookie name so that ad-blockers wouldn't know to block it.
If you want to indicate that a hit was sent through the server, that's a good idea. Use custom dimension for that. Just sync it with your analysts first. Maybe they wouldn't want that, or maybe they would want it in a different dimension.
Now, this is very trivial. There are ways to go much deeper and to improve the quality of data from here. Like gluing the order id, the transaction id, the user id to that, using them to generate client id, do some custom client tracking for the future. But I must say that it's better than what more than 90% of, say, shopify clients have.
Also, GA4 is not good enough for deeper production usage. Many things there are still very rudimentary and lacking. I would suggest concentrating on Universal Analytics and having GA4 as a backup for when Google makes GA4 actually good enough to replace UA. That is, unless you're downloading your data elsewhere and not using GA's interface for analysis.
It seems that this page (Relevant portion in the screenshot below), advices to either send the data along with the client_id or user_id. However fails to address the fact client_id is a mandatory field as stated here.
I believe it is probably safe to assume that randomly generating this field should work. At least it seems to on my end however be warned that I am unsure if this has any impact on attribution.
* In the above image, Device ID refers to client_id
Question about goal tracking.
I have an application that I can track from Step 1 to Step 10, and the user submits data on step 11.
Pretty standard form.
We then intake the application, evaluate, then send the user and email to log back in and accept our terms and purchase the product, which are steps 12-15.
If I set up a goal funnel, will analytics know the users has come back and completed steps 12-15?
Or do I basically lose all tracking after step 11?
There are some limitations in GA that could be a bit troublesome:
30-min session window: if user comes back later, new session is initiated.
multiple browsers/devices: if finishing the last 3 steps after receiving your email is done on different laptop/mobile device, the user will have different ClientID and so you won't be able to stitch the data together (in GA reporting interface).
There are many more aspects to it - but if the two limitations listed above doesn't seem like relevant, then the implementation shouldn't be tat difficult. For more technical details, I'd consider using MeasurementProtocol (fancy name for server-side tracking).
This is my current understanding of the events on pressing the Activate button:
If the user has replication permission, the content is sent to the dispatcher.
If the user does not have replicate, the Request for Activate workflow is run. Out of the Box this performs two steps which are both sent to the Administrators group.
If I change the OOB Request for Activation workflow to assign to another group then this group gets the message in their CQ Inbox instead.
My requirement is that I have multiple websites belonging to different divisions within the company, meaning that each has separate groups for users who can only author content and users who can Approve/Activate that content. In one case there is an additional requirement for the content in the careers section to be approved by the HR team only.
How can I alter the workflow to achieve these requirements?
I can think of two ways of doing this:
1) Change the OOB box workflow to use OR steps to switch to differnt groups based on the content path. This could get very complex when dealing with pages for both pages and assets, and the special conditions like the careers example.
2) Create a custom step that runs Java code that uses the AccessControlManager class to look for replicate permissions on the node, and if there aren't any then to traverse up the tree until it finds a node with permissions.
I've gone with option 1) for a similar problem.
If your main concern is having too many branches in your OR Split, maybe there's an opportunity to break down the workflow's decisions into multiple steps, instead of having one decision point where it branches out to many different paths.
For example, you might first split by what site the payload is on, and then split again based on user type, or section of the site. So, something like:
site 1
section 1
role 1
role 2
section 2
site 2
... and so on, where each level of indentation represents a separate OR Split.
If you use the Container Step to trigger a sub-workflow at each of these decision points, that may help keep your workflow more organised.
Because I didn't love the idea of changing the OOB Request for Activation workflow, I minimised that by making the first step an OR Split that does a generic check - basically:
Pseudo-code:
if (we're in one of the sites that's subject to my custom workflows) {
Container step that points to my main custom workflow;
} else {
Continue with the default Request for Activation workflow steps;
}
That way you make minimal changes to the OOB workflow, and leave yourself open to running the default workflow if you set up a new site on the same instance, and don't want it subject to your custom workflow.
We've created a custom property in each page, "page owner", which in fact is a pointer to a group (I wish we have made it inheritable through the tree since beginning). Then workflow was customized so that page owner group receives this in their inbox for approval.
We're investigating Alfresco for doing wideband delphi ("planning poker") based on submitted statements of work (collected user stories). I've been reading through the Alfresco documentation, and there are two questions that I haven't been able to get clear answers to:
Can we set it up so users can write, but not read, to a folder or node? (To support "anonymous" planning, without users knowing what the other users submitted estimates were)
Can workflow tasks be implemented to ask users to comment or submit items to a node or director with the above model, rather than just simple approve or deny?
Workflow:
User submits a statement of work
All users (or selected users at random, or ... ) in group get notice to review
Reviews include estimates on the overall SOW or specific phases
Reviews are anonymous/secret to all but the manager
Have you implemented something similar in Alfresco with fine grained access control? Sharing your experience would be very helpful... i'm not looking for someone to do the work for me, just to confirm it can be done.
I would use some kind of parallel workflow for this.
First the managers starts the workflow and the task type of this first node will have additional info about the user story and such, then the manager selects a people or a group to which it will send this user story.
Here comes the parallel thing into play. Because it's parallel noone sees the results of the other members of the workflow. The members fill in the requested fields (another custom task type with data like: score (estimate) and maybe explanation.
Before the workflow goes back to manager the automatic calculations are made in a non-user task/node where you calculate overall score for the story. You can include each individual user and their score in the result/report if necessary.
Now the results are sent to the manager.