How are the connections are being calculated?
Let's assume that I have a web app which one load sends a message to all connected clients, and let's say I have 5 connected clients. Does it means that as long as the browser tab with the web app is open it will count as 1 connections, which means that I will have 6 concurrent connections and that's count towards what you define as "Connection" in the pricing page?
If not, please explain how you calculate the "Connection". Thanks
This question was bugging me ever since I ran through the thinkster.io angular+firebase tutorial and I saw my firebase analytics tab showing a peak concurrent of 6 even though I only remember having the one page open. I looked back at the code and thought it could be to do with how the tutorial has you create a new Firebase(url) for each location in your firebase.
I wanted to test the difference between creating a new Firebase(url) vs taking the root reference and then accessing the .child() location. My theory was that new Firebase(url) would create a new connection each time, while .child() would re-use the existing connection.
Setup
Created two new firebases each with identical data
Setup an angularjs project using yeoman
Included angularfire
Code
For simplicity, I just put everything in the main controller of the generated code.
To test out the connections created with new Firebase() I did the following:
$scope.fb_root = $firebase(new Firebase(FBURL_NEW));
$scope.fb_root_apps = $firebase(new Firebase(FBURL_NEW + '/apps'));
$scope.fb_root_someApp = $firebase(new Firebase(FBURL_NEW + '/apps/someApp'));
$scope.fb_root_users = $firebase(new Firebase(FBURL_NEW + '/users'));
$scope.fb_root_mike = $firebase(new Firebase(FBURL_NEW + '/users/mike'));
To test out the connections created with ref.$child() I did the following:
$scope.fb_child = $firebase(new Firebase(FBURL_CHILD));
$scope.fb_child_apps = $scope.fb_child.$child("apps");
$scope.fb_child_someApp = $scope.fb_child_apps.$child("someApp");
$scope.fb_child_users = $scope.fb_child.$child("users");
$scope.fb_child_mike = $scope.fb_child_users.$child("mike");
I then bound these objects in my view so I can see them, and I played around with updating data via my firebase forge and watching the data update live on my app.
Results
I opened up my local app into 17 browser tabs, hoping that a large number of tabs would exaggerate any differences between the connection methods.
What I found is that each tab only opened up one single web socket connection back to firebase for each firebase db. So at the end of the test, both methods resulted in the same peak count of 17 connections.
Conclusion
From this simple test I think it's safe to say that the Firebase JS library does a good job of managing its connection.
Regardless of your code calling new Firebase() a bunch of times, or by referencing child locations via .child(), the library will only create a single connection as far as your metering is concerned. That connection will stay online for as long as your app is open.
So in your example - yes I believe you will see 6 concurrent connections, 1 for the app where someone is sending the message, and 5 for the apps receiving the message.
Update
One other thing worth mentioning is that Firebase measures connections for paid plans based on the 95th percentile of usage during the month. This is listed in the FAQ section of their Pricing page # https://www.firebase.com/pricing.html
Update 11-Mar-16: Firebase no longer appears to measure connections based on 95th %. Instead, the 101st concurrent connection is denied.
https://www.firebase.com/pricing.html :
All our plans have a hard limit on the number of database connections.
Our Free and Spark plans are limited to 100. The limit cannot be
raised. All other plans have a courtesy limit of 10,000 database
connections. This can be removed to permanently allow Unlimited
connections if you email us at firebase-support#google.com.. The
reason we impose this courtesy limit is to prevent abuse and to ensure
that we are prepared to handle our largest customers. Please contact
us at least 24 hours in advance so we can lift this limit and ensure
we have enough capacity available for your needs.
Related
I have just added a new feature to an app I'm building. It uses the same working Cosmos/Table storage code that other features use to query and pump results segments from the Cosmos DB Emulator via the Tables API.
The emulator is running with:
/EnableTableEndpoint /PartitionCount=50
This is because I read that the emulator defaults to 5 unlimited containers and/or 25 limited and since this is a Tables API app, the table containers are created as unlimited.
The table being queried is the 6th to be created and contains just 1 document.
It either takes around 30 seconds to run a simple query and "trips" my Too Many Requests error handling/retry in the process, or hangs seemingly forever and no results are returned, the emulator has to be shut down.
My understanding is that with 50 partitions I can make 10 unlimited tables, collections since each is "worth" 5. See documentation.
I have tried with rate limiting on and off, and jacked the RU/s to 10,000 on the table. It always fails to query this one table. The data, including the files on disk, has been cleared many times.
It seems like a bug in the emulator. Note that the "Sorry..." error that I would expect to see upon creation of the 6th unlimited table, as per the docs, is never encountered.
After switching to a real Cosmos DB instance on Azure, this is looking like a problem with my dodgy code.
Confirmed: my dodgy code.
Stand down everyone. As you were.
I am working on a flutter app that fetches 341 documents from the firestore, after 2 days of analysis I found out that my read requests are increasing too much. So I made a chart on the stackdriver metrics explorer from which I get to know that my app is just reading 341 docs a single time, it's the firebase console which is increasing my reads.
Now, comes to what are the questions that are bothering me,
1)How reads are considered when we see data on the console and how can I reduce my read requests? Basically there are 341 docs but it is showing more than 600 reads whenever I refresh my console.
2)As you can see in the picture there are two types of document reads 'LOOKUP' and 'QUERY', what's the exact difference between them?
3)I am getting data from the firestore with a single instance and when I open my app the chart shows 1 active client which is cool but in the next 5 minutes, the number of active clients starts to increase.
Can anybody please explain to me why this is happening?
For the last question, I tried to disable all the service accounts and then again opened my app but got the same thing again.
Firestore.instance.collection("Lectures").snapshots(includeMetadataChanges: true).listen((d){
print(d.metadata.isFromCache);//prints false everytime
print(d.documents.length);// 341
print(d.documentChanges.length);//341
});
This is the snippet I am using. When the app starts it runs only once.
I will try to answer your questions:
How reads are considered when we see data on the console and how can I
reduce my read requests? Basically there are 341 docs but it is
showing more than 600 reads whenever I refresh my console.
Reads are considered depending on your how you query your Firestore database in addition to your access to this database from the console so using of the Firebase console will incur reads and even if you leave the console open to do other stuff, when new changes to database occured these changes will incur reads also, automatically.and any document read from the server is going to be billed. It doesn't matter where the read came from. The console should be included in that.
Check this official documentation under the "Manage data" title you can see there is a note : "Note: Read, write, and delete operations performed in the console count towards your Cloud Firestore usage."
Saying that if you think there is an issue with this, you can contact Firebase support directly to have more detailed answers.
However, If you check the free plan of Firebase you can see that you have 50K free reads per day.
A workaround that I found for this (thanks to Dependar Sethi)
Bookmarking the Usage tab of the Firestore page. (So you basically
'Skip' the Data Tab)
Adding a dummy collection in a certain way that ensures it is the
first collection(alphabetically) which gets loaded by default on
the Firestore page.
you can find his full solution here.
Also, you can optimise your queries however you want to retreive only the data that you want using where() method and pagination with Firebase
As you can see in the picture there are two types of document reads
'LOOKUP' and 'QUERY', what's the exact difference between them?
I guess there are no important difference between them but "QUERY" is getting the actual data(when you call data() method) while "LOOKUP" is getting a reference of these data(without calling data() method).
I am getting data from the firestore with a single instance and when I
open my app the chart shows 1 active client which is cool but in the
next 5 minutes, the number of active clients starts to increase.
For this question, considering the metrics that you are choosing in Stackdriver I can see 3 connected clients. and as per the decription of "connected client" metric:
The number of active connections. Each mobile client will have one connection. Each listener in admin SDK will be one connection. Sampled every 60 seconds. After sampling, data is not visible for up to 240 seconds.
So please check: how many mobiles are connected to this instance and how many listeners do you have in your app. The sum of all of them is the actual number of connected clients that you are seeing in Stackdriver.
I am using Celery with Flask after working for a good long while, my celery is showing a celery.backends.rpc.BacklogLimitExceeded error.
My config values are below:
CELERY_BROKER_URL = 'amqp://'
CELERY_TRACK_STARTED = True
CELERY_RESULT_BACKEND = 'rpc'
CELERY_RESULT_PERSISTENT = False
Can anyone explain why the error is appearing and how to resolve it?
I have checked the docs here which doesnt provide any resolution for the issue.
Possibly because your process consuming the results is not keeping up with the process that is producing the results? This can result in a large number of unprocessed results building up - this is the "backlog". When the size of the backlog exceeds an arbitrary limit, BacklogLimitExceeded is raised by celery.
You could try adding more consumers to process the results? Or set a shorter value for the result_expires setting?
The discussion on this closed celery issue may help:
Seems like the database backends would be a much better fit for this purpose.
The amqp/RPC result backends needs to send one message per state update, while for the database based backends (redis, sqla, django, mongodb, cache, etc) every new state update will overwrite the old one.
The "amqp" result backend is not recommended at all since it creates one queue per task, which is required to mimic the database based backends where multiple processes can retrieve the result.
The RPC result backend is preferred for RPC-style calls where only the process that initiated the task can retrieve the result.
But if you want persistent multi-consumer result you should store them in a database.
Using rabbitmq as a broker and redis for results is a great combination, but using an SQL database for results works well too.
I have a single web client and a few Lambda functions which use the Admin SDK. I've noticed recently that I've bumped into the 100 simultaneous connection limit but I really shouldn't be anywhere near that limit. Also it would appear that the connections established by my Lamba functions are not dropping off even after the function has completed.
Any idea on:
how I can prevent this run-up on connections from happening?
how I can release connections established by past Lambda scripts?
how can I monitor which processes/threads/stacks are holding connections?
Note: this is a testing environment I'm working out of so I'd prefer to keep this in the free tier and my requirements should definitely not be running into the 100 active limit. I am on a paid plan in prod.
I attempt to avoid calling initializeApp more than once by using the following connection code. In the example I'm talking about I only have a single database as a backend and so the default "name" of DEFAULT is used each time.
const runningApps = new Set(firebase.apps.map(i => i.name));
this.app = runningApps.has(name)
? firebase.app()
: firebase.initializeApp({
credential: firebase.credential.cert(serviceAccount),
databaseURL: config.databaseUrl
});
I'm now trying to explicitly close connections with goOffline but that leads to another issue where on the second connection -- aka, where the DEFAULT application is already setup and it just reuses the connection already established I get the following logging:
# Generated as result of `goOnline`
Connecting to Firebase: [https://xyz.firebaseio.com]
appears to be already connected
# Listening on ".info/connected" comes back as true, resulting in:
AbstractedAdmin: connected to [DEFAULT]
# but then I get this error
NotAllowed: You must first connect before using the database() API at Object._getFirebaseType
The fact that you have unexpected incoming connections to the database, makes it seem like the stale instances keep an open connection.
Best I can think off is to call goOffline() in your function before it completes to explicitly disconnect. That would probably also mean you have to call goOnline at the start of the function, since it might be running on an instance that previously went offline. Both goOnline and goOffline are synchronous calls afaik, but there's definitely going to be some time between going online and the data becoming available in your app.
If Lambda has a way for you to detect life-cycle events of its instances, that would be the preferred place to call goOffline and goOnline.
admin.initializeApp should only get called once in your script/node app.
The Firebase SDK's talks HTTP2 to the Firebase cloud system, so I'm not sure why you would encounter max connection issues as unique sockets are not stood up per call.
One thing to look out for is that calls to 3rd part API's (such as sendgrid) are not supported on the free tier.
I currently have over 1k realtime users setup on a MySQL server(only 10-20 users will register simulatneously) for Asterisk. The problem is the sip is not registering evertime. Sometimes I get 'registration timeout'. Is there a setup guide or a setting which I need to configure in order to have >99% successful registrations?
Never faced the issue as I have fewer users.
But according to the aterisk documentation:
If you have problems with your network connection going up and down (e.g. an unreliable cable connection) and you keep losing your sip registry, you may want to add registerattempts and registertimeout settings to the general section above the register definitions.
Setting registerattempts=0 will force Asterisk to attempt to reregister until it can (the default is 10 tries).
registertimeout sets the length of time in seconds between registration attempts (the default is 20 seconds).
About achieving 99% success:
I think you have to study your system and apply setting to the above variables accordingly (dynamically). I suggest using Markovian models like mm1 simulation if your system is not complicated.