I'm trying to configure Firebase A/B experiment only for new users of my app.
What I did:
1) I set custom_first_open_time user param on the app first launch equal to timestamp in millis from epoch
2) I created a new experiment, set custom_first_open_time > 1581944923450 in the targeting section
3) I used 1 of 10 remote config params for the experiment. Once I started, all users stopped obtaining any remote config param and inside the app was used params set as default in code (inside the app). None of 10 params obtained in the app, though I change only 1 in A/B
Why using value of this param brakes the whole remote config?
Why it doesn't work as I expect?
Any help appreciated!
The problem was because of using millis from epoch and looks like firebase couldn't compare so large numbers. Using just seconds solved the problem
Related
In the firebase docs for remote config loading strategies it recommends that if you decide to fetch and activate remote config behind a loading screen you should add a timeout to the loading screen.
Firstly I assume I can use the fetchTimeout property below to do this?
init() {
#if DEBUG
let settings = RemoteConfigSettings()
settings.minimumFetchInterval = 0
remoteConfig.configSettings = settings
#else
let settings = RemoteConfigSettings()
settings.fetchTimeout = 10
remoteConfig.configSettings = settings
#endif
}
Secondly, is there any guidance on what this timeout should be? the default is 1 minute but that is clearly too long. I have set it to 10 seconds and also see there is a note that With A/B Testing, additional time is required as the user is placed into an experiment and the experimental values are applied.
As we don't control this service, it is very hard to gauge what a good timeout number is. Even 10 seconds seems fairly long for a user to wait.
Thanks
I'm using Firebase Remote Config with Firebase A/B Testing on iOS.
I see a value received from Remote Config sometimes reset to default value.
I think, activated config values does not delete even if server cache on the app has expired.
Is there any situation that activated values are back to the default value?
In-App: Default Value A set.
fetch -> activate
Got B from the server.
Leave apps.
Several hours later, I come back to the app, it seems the value is reset to A.
fetch and activated a strategy
The app activates remote config values once at launch time.
https://firebase.googleblog.com/2017/01/firebase-remote-config-loading.html
Strategy #3: Load values for next time
When your user starts up your app, you immediately call activateFetched(). This will apply any old values you've previously fetched from the cloud. And then your user can immediately start interacting with your app. In the meantime, you kick off an asynchronous fetch() call to fetch new values from the cloud. And in the completion handler for this call… you do nothing. Heck, you don't even need to add a completion handler in the first place. Those values you fetched from the cloud will remain stored locally on the device until your user calls activateFetched the next time they start your app.
ABTesting Settings
target user: 100% of the app
variant-key -> SOME_KEY
Control Group: 50% -> value = A
Variant B: 50% -> value = B
Pods versions from Podfile.lock:
- FirebaseABTesting (2.0.0):
- FirebaseRemoteConfig (3.1.0):
How to set defaults, get values
Default Value is:
RemoteConfig.remoteConfig().setDefaults(["SOME_KEY": "A"])
To get Value from Remote Config:
let value = RemoteConfig.remoteConfig().configValue(forKey: "SOME_KEY")
App Launch Sequences
At first app launch
activateFetched()
got value A(=default).
There is no fetched value yet, activateFetched has no effect.
then fetch.
next time app launch
activateFetched()
got value B (if the user is selected ABTestring's value 'B')
then fetch.
next time app launch
activateFetched()
got value A (this is the issue)
then fetch.
In the above situation, if the app fetched and activated value B,
the value is persisted in activated values cache and
the cached value doesn't reset to A I thought.
Is the assumption right?
Addtional Info
When the app got B from configValue(forKey: "SOME_KEY")
then remoteConfig.allKeys(from: .remote, namespace: NamespaceGoogleMobilePlatform) returns ["SOME_KEY"].
But when the app got A back from 'B' (the last situation in above launch sequence),
then remoteConfig.allKeys(from: .remote, namespace: NamespaceGoogleMobilePlatform) returns empty.
I've configured firebase ab-testing. Everything works fine except there is no impact user on console.
Actually, I can see UI and log show ab-testing is applied.
Moreover, by checking the other StackoverFlow topic, activateFetched also invoked after fetch successfully.
Moreover, I've referenced
Firebase Remote Config A/B testing shows no results after 24 hours
Firebase Remote Config results on initial request
Remote Config A/B Test does not provide results on iOS
But those are no work on my case.
Is there anything miss or any other need to check so that client can response AB testing result to firebase console.
Thanks for your help first.
Code snippet:
[FIRApp configure];
FIRRemoteConfigSettings* configSettings = [[remoteConfig configSettings] initWithDeveloperModeEnabled:YES];
[[FIRRemoteConfig remoteConfig] setConfigSettings:configSettings];
[[FIRRemoteConfig remoteConfig] fetchWithExpirationDuration:duration completionHandler:^(FIRRemoteConfigFetchStatus status, NSError *error) {
if (status == FIRRemoteConfigFetchStatusSuccess) {
BOOL configFound = [[FIRRemoteConfig remoteConfig] activateFetched];
A couple things to check or take note of:
Make sure you're using and have deployed the latest Remote Config SDK. Earlier versions don't work with A/B test experiments.
Be sure to verify your experiment on a test device by following the documentation here
It can take a couple days for data to come in for your experiment.
Please call the functions in the following order:
fetch()
Call activatefetched() in the completion handler of fetch().
Fire activation event. If you need to call activation event immediately after activatefetched(), add a time delay of a few seconds. This is because activatefetched() process asynchronously and hence the function may not execute completely, before the activation event is fired.
Once done, test a running experiment on test device. In the debug logs search with string "exp_X" where 'X' is the experiment Id. You will find the experiment Id in the URL of the experiment. If you find the experiment ID in the debug logs while executing the code on test device, it means the device was covered in experiment.
Also if the experiment setup is correct, the running experiment will show 1 active experiment user in the console.
I configured Firebase Remote Config A/B testing for Android, and we did rollout on at least 10K devices.
For some reason, I see "0 users" in my A/B test after more than 24 hours.
Firebase GMS version is: 11.8.0
Should it show A/B participants in real-time or it's ok to see 0 users after 24 hours?
P.S: We are able to get AB test variants on test devices through Firebase Instance Id, it works well.
The simplest experiment which is running has only app package as a target, with no additional filters. And it shows 0 users as well.
Finally, we found an answer!
Maybe somebody will find it helpful:
For now, it happens (no data in Firebase remote config A/B test experiment) if you have an activation event configured for A/B test experiment.
If you have 2 different experiments, both will fail to get results even if you have "activation event" configured only in 1 of them.
Additionally, remote config will not work as well, you'll be able to get only default values.
We already reported to Google about, so they'll fix it at some point I hope.
Another useful info which is really hard to get:
How long is it ok to see "0 Total Users" in experiment I've just
started?
It takes many hours before you can see any data in your experiment. We were able to see results only after 21 hours after experiment start, so if you configured everything well, don't worry and wait for at least 24 hours. It will show 0 "Total Users" for many hours after the start.
Should I use app versionName or versionCode in "Version" field of
experiment setup?
You should use versionName.
Some useful info from support:
Firebase SDK
Make sure your users have the version of your app with the latest SDK.
Since your experiment is with Remote Config
When activateFetched() is called, all events from that point on will be tagged with the experiment. If you have a goal or activation event that happens before activateFetched(), such as automatic events like first_open, session_start, etc., the experiment setup might be wrong.
Are you using an Activation Event?
Make sure to call fetch() and activateFetched() before the activation event occurs.
Experiment ID of the experiments (if support asks you about)
It's the number at the end of the URL while viewing experiment results.
This debugging log could be useful to get what is going on
Also:
The good way to check if your experiment is working now is to set it to a specific version you didn't publish yet and check logs from remote config with the fresh app install(or erase all app data & restart).
It should show different variant every time you reinstall the app, since your Firebase Instance ID changes after app reinstall/app data erase.
If you see variants change - then A/B test is running well.
In your "build.graddle": don't forget to set the same versionName which you set in experiment setup.
In my case, I was receiving results of A/B testing but suddenly, it stopped to appear. It had continued for 7 days and then results appeared. Firebase Support manager said:
what I suspected here is just a delay in showing the result in the
experiments
Additionally, she said that
With that, I would suggest always using the latest SDK version and
enabling Google Analytics data sharing.
In my case, I used I wasn't using the latest SDK version, but Google Analytics was enabled for "Benchmarking", "Technical Support", "Account Specialists" except for "Google products & services". I believe these settings were enabled by default (the screenshot from Google Analytics):
How are the connections are being calculated?
Let's assume that I have a web app which one load sends a message to all connected clients, and let's say I have 5 connected clients. Does it means that as long as the browser tab with the web app is open it will count as 1 connections, which means that I will have 6 concurrent connections and that's count towards what you define as "Connection" in the pricing page?
If not, please explain how you calculate the "Connection". Thanks
This question was bugging me ever since I ran through the thinkster.io angular+firebase tutorial and I saw my firebase analytics tab showing a peak concurrent of 6 even though I only remember having the one page open. I looked back at the code and thought it could be to do with how the tutorial has you create a new Firebase(url) for each location in your firebase.
I wanted to test the difference between creating a new Firebase(url) vs taking the root reference and then accessing the .child() location. My theory was that new Firebase(url) would create a new connection each time, while .child() would re-use the existing connection.
Setup
Created two new firebases each with identical data
Setup an angularjs project using yeoman
Included angularfire
Code
For simplicity, I just put everything in the main controller of the generated code.
To test out the connections created with new Firebase() I did the following:
$scope.fb_root = $firebase(new Firebase(FBURL_NEW));
$scope.fb_root_apps = $firebase(new Firebase(FBURL_NEW + '/apps'));
$scope.fb_root_someApp = $firebase(new Firebase(FBURL_NEW + '/apps/someApp'));
$scope.fb_root_users = $firebase(new Firebase(FBURL_NEW + '/users'));
$scope.fb_root_mike = $firebase(new Firebase(FBURL_NEW + '/users/mike'));
To test out the connections created with ref.$child() I did the following:
$scope.fb_child = $firebase(new Firebase(FBURL_CHILD));
$scope.fb_child_apps = $scope.fb_child.$child("apps");
$scope.fb_child_someApp = $scope.fb_child_apps.$child("someApp");
$scope.fb_child_users = $scope.fb_child.$child("users");
$scope.fb_child_mike = $scope.fb_child_users.$child("mike");
I then bound these objects in my view so I can see them, and I played around with updating data via my firebase forge and watching the data update live on my app.
Results
I opened up my local app into 17 browser tabs, hoping that a large number of tabs would exaggerate any differences between the connection methods.
What I found is that each tab only opened up one single web socket connection back to firebase for each firebase db. So at the end of the test, both methods resulted in the same peak count of 17 connections.
Conclusion
From this simple test I think it's safe to say that the Firebase JS library does a good job of managing its connection.
Regardless of your code calling new Firebase() a bunch of times, or by referencing child locations via .child(), the library will only create a single connection as far as your metering is concerned. That connection will stay online for as long as your app is open.
So in your example - yes I believe you will see 6 concurrent connections, 1 for the app where someone is sending the message, and 5 for the apps receiving the message.
Update
One other thing worth mentioning is that Firebase measures connections for paid plans based on the 95th percentile of usage during the month. This is listed in the FAQ section of their Pricing page # https://www.firebase.com/pricing.html
Update 11-Mar-16: Firebase no longer appears to measure connections based on 95th %. Instead, the 101st concurrent connection is denied.
https://www.firebase.com/pricing.html :
All our plans have a hard limit on the number of database connections.
Our Free and Spark plans are limited to 100. The limit cannot be
raised. All other plans have a courtesy limit of 10,000 database
connections. This can be removed to permanently allow Unlimited
connections if you email us at firebase-support#google.com.. The
reason we impose this courtesy limit is to prevent abuse and to ensure
that we are prepared to handle our largest customers. Please contact
us at least 24 hours in advance so we can lift this limit and ensure
we have enough capacity available for your needs.