I have a single web client and a few Lambda functions which use the Admin SDK. I've noticed recently that I've bumped into the 100 simultaneous connection limit but I really shouldn't be anywhere near that limit. Also it would appear that the connections established by my Lamba functions are not dropping off even after the function has completed.
Any idea on:
how I can prevent this run-up on connections from happening?
how I can release connections established by past Lambda scripts?
how can I monitor which processes/threads/stacks are holding connections?
Note: this is a testing environment I'm working out of so I'd prefer to keep this in the free tier and my requirements should definitely not be running into the 100 active limit. I am on a paid plan in prod.
I attempt to avoid calling initializeApp more than once by using the following connection code. In the example I'm talking about I only have a single database as a backend and so the default "name" of DEFAULT is used each time.
const runningApps = new Set(firebase.apps.map(i => i.name));
this.app = runningApps.has(name)
? firebase.app()
: firebase.initializeApp({
credential: firebase.credential.cert(serviceAccount),
databaseURL: config.databaseUrl
});
I'm now trying to explicitly close connections with goOffline but that leads to another issue where on the second connection -- aka, where the DEFAULT application is already setup and it just reuses the connection already established I get the following logging:
# Generated as result of `goOnline`
Connecting to Firebase: [https://xyz.firebaseio.com]
appears to be already connected
# Listening on ".info/connected" comes back as true, resulting in:
AbstractedAdmin: connected to [DEFAULT]
# but then I get this error
NotAllowed: You must first connect before using the database() API at Object._getFirebaseType
The fact that you have unexpected incoming connections to the database, makes it seem like the stale instances keep an open connection.
Best I can think off is to call goOffline() in your function before it completes to explicitly disconnect. That would probably also mean you have to call goOnline at the start of the function, since it might be running on an instance that previously went offline. Both goOnline and goOffline are synchronous calls afaik, but there's definitely going to be some time between going online and the data becoming available in your app.
If Lambda has a way for you to detect life-cycle events of its instances, that would be the preferred place to call goOffline and goOnline.
admin.initializeApp should only get called once in your script/node app.
The Firebase SDK's talks HTTP2 to the Firebase cloud system, so I'm not sure why you would encounter max connection issues as unique sockets are not stood up per call.
One thing to look out for is that calls to 3rd part API's (such as sendgrid) are not supported on the free tier.
Related
I am using Celery with Flask after working for a good long while, my celery is showing a celery.backends.rpc.BacklogLimitExceeded error.
My config values are below:
CELERY_BROKER_URL = 'amqp://'
CELERY_TRACK_STARTED = True
CELERY_RESULT_BACKEND = 'rpc'
CELERY_RESULT_PERSISTENT = False
Can anyone explain why the error is appearing and how to resolve it?
I have checked the docs here which doesnt provide any resolution for the issue.
Possibly because your process consuming the results is not keeping up with the process that is producing the results? This can result in a large number of unprocessed results building up - this is the "backlog". When the size of the backlog exceeds an arbitrary limit, BacklogLimitExceeded is raised by celery.
You could try adding more consumers to process the results? Or set a shorter value for the result_expires setting?
The discussion on this closed celery issue may help:
Seems like the database backends would be a much better fit for this purpose.
The amqp/RPC result backends needs to send one message per state update, while for the database based backends (redis, sqla, django, mongodb, cache, etc) every new state update will overwrite the old one.
The "amqp" result backend is not recommended at all since it creates one queue per task, which is required to mimic the database based backends where multiple processes can retrieve the result.
The RPC result backend is preferred for RPC-style calls where only the process that initiated the task can retrieve the result.
But if you want persistent multi-consumer result you should store them in a database.
Using rabbitmq as a broker and redis for results is a great combination, but using an SQL database for results works well too.
While implement Fire-base cloud messaging in browser with help of Service Worker getting an error "DOMException: Registration failed - storage error" please help why i am getting this error
This means that your Service Worker has stored so much data that the storage available to it is full, and upon SW installation it tries to store even more which in turn leads to an error.
You have to either manually remove stuff from your storage or use some library that automatically handles these situations for you in a way or another (eg. removes stuff FIFO style).
In my case the problem was caused by create-react-app's own service worker unregistration logic at the very bottom of index.js:
serviceWorker.unregister();
It was conflicting with the fact of using a service worker in the middle of our own application's logic.
Solution was simple – just removing that line from index.js
I created a serverless function that performs that Firebase Token Validation.
Everything works as intended. Except, I have I get errors on subsequent calls to initialize my app that the default app already exists (same container). This raises some questions.
If my serverless infrastructure was to spin up multiple concurrent containers, each working to initialize the app. Would this also cause this error? That the app is initiailized elsewhere? Or is this error isolated to local instances?
If its the latter, If I provide a named app based on the container it is spun up in, is there a firebase limit to the maximum number of apps that can be initialized at once?
This is how I am initializing the app now:
cred = credentials.Certificate(SERVICE)
firebase_admin.initialize_app(cred)
I could do this but am not sure about firebase app limits or concurrent initializations (cant find any specifics in docs):
cred = credentials.Certificate(SERVICE)
firebase_admin.initialize_app(cred, 'APP-NAME-[CONTAINERID]')
Or, should I just re-write this using my own JWT Decoder and grabbing the public keys from google?
And here is the full error:
Error occurred setting firebase credentials: The default Firebase app already exists. This means you called initialize_app() more than once without providing an app name as the second argument. In most cases you only need to call initialize_app() once. But if you do want to initialize multiple apps, pass a second argument to initialize_app() to give each app a unique name.
UPDATE: AWS Lambda, Python.
I am going to test out with the following, to prevent re-initializing the app within the same container on warm function executions and move forward with the assumption that there are no API limits on performing auth.validate_id_token() and that this won't conflict with concurrent container executions. Ill report back if it tests out differently.
try:
firebase_admin.get_app()
logger.info('firebase already intialized.')
except ValueError as e:
logger.info('firebase not initialized. initialize.')
cred = credentials.Certificate(SERVICE)
firebase_admin.initialize_app(cred)
I will probably still migrate to another JWT validation to reduce function size (since I already have a jwt library for my own app use) and migrate away from relying on Firebase API to decode it.
If you get an error when initializing the admin SDK that says the default app already exists, that just means you're trying to init the admin SDK twice in the same process. Obviously, don't do that. If you init once and only once per process, you will never see this error.
You will have to take some care to only call the init method once per server instance. It's not clear exactly what you're doing from the code you've shown. I don't know about python, but with node, you can init once in a global context without problems. If you need to init during a function execution, you should have some flag to check that ensures the default Firebase app hasn't already been initialized, and init only conditionally based on that flag.
Made the mistake of performing a destructive migration on a synchronized realm, which I just now learned I shouldn’t have done according to the docs' statement “However, if the migration makes a destructive change, the Realm will stop syncing with ROS, producing a Bad changeset received error”. The server won't restart our Realm Object Server and the logs say realm-object-server dead but pid file exists. We cannot even access ROS on web at this point.
Is there a way around this without re-installing our realm instance? Also, if the magnitude of this migration is so severe, is there not a way to give a warning to the developer?
Code Sample:
let config = Realm.Configuration(
syncConfiguration: SyncConfiguration(user: curUser, realmURL: RealmURL.userObjects), migrationBlock: { (migration, schema) in
// todo
})
When you perform a schema change, this results in an operation that is appended to the operation log maintained by a Realm. This first occurs on the client copy of a synchronized Realm and is then synced to Realm Object Server. If the operation is a destructive change, the server should simply reject the operation and return an error. The result is that the server's operation log is not affected but the client is now in a state where it can't continue to sync with the server. In this situation, the solution is to reset the client, which is easiest in development by deleting and reinstalling the app.
Your situation, however, sounds like a different problem. The fact that the server is unresponsive implies something else went wrong. You could try removing and reinstalling the server since this does not delete the data or configuration.
How are the connections are being calculated?
Let's assume that I have a web app which one load sends a message to all connected clients, and let's say I have 5 connected clients. Does it means that as long as the browser tab with the web app is open it will count as 1 connections, which means that I will have 6 concurrent connections and that's count towards what you define as "Connection" in the pricing page?
If not, please explain how you calculate the "Connection". Thanks
This question was bugging me ever since I ran through the thinkster.io angular+firebase tutorial and I saw my firebase analytics tab showing a peak concurrent of 6 even though I only remember having the one page open. I looked back at the code and thought it could be to do with how the tutorial has you create a new Firebase(url) for each location in your firebase.
I wanted to test the difference between creating a new Firebase(url) vs taking the root reference and then accessing the .child() location. My theory was that new Firebase(url) would create a new connection each time, while .child() would re-use the existing connection.
Setup
Created two new firebases each with identical data
Setup an angularjs project using yeoman
Included angularfire
Code
For simplicity, I just put everything in the main controller of the generated code.
To test out the connections created with new Firebase() I did the following:
$scope.fb_root = $firebase(new Firebase(FBURL_NEW));
$scope.fb_root_apps = $firebase(new Firebase(FBURL_NEW + '/apps'));
$scope.fb_root_someApp = $firebase(new Firebase(FBURL_NEW + '/apps/someApp'));
$scope.fb_root_users = $firebase(new Firebase(FBURL_NEW + '/users'));
$scope.fb_root_mike = $firebase(new Firebase(FBURL_NEW + '/users/mike'));
To test out the connections created with ref.$child() I did the following:
$scope.fb_child = $firebase(new Firebase(FBURL_CHILD));
$scope.fb_child_apps = $scope.fb_child.$child("apps");
$scope.fb_child_someApp = $scope.fb_child_apps.$child("someApp");
$scope.fb_child_users = $scope.fb_child.$child("users");
$scope.fb_child_mike = $scope.fb_child_users.$child("mike");
I then bound these objects in my view so I can see them, and I played around with updating data via my firebase forge and watching the data update live on my app.
Results
I opened up my local app into 17 browser tabs, hoping that a large number of tabs would exaggerate any differences between the connection methods.
What I found is that each tab only opened up one single web socket connection back to firebase for each firebase db. So at the end of the test, both methods resulted in the same peak count of 17 connections.
Conclusion
From this simple test I think it's safe to say that the Firebase JS library does a good job of managing its connection.
Regardless of your code calling new Firebase() a bunch of times, or by referencing child locations via .child(), the library will only create a single connection as far as your metering is concerned. That connection will stay online for as long as your app is open.
So in your example - yes I believe you will see 6 concurrent connections, 1 for the app where someone is sending the message, and 5 for the apps receiving the message.
Update
One other thing worth mentioning is that Firebase measures connections for paid plans based on the 95th percentile of usage during the month. This is listed in the FAQ section of their Pricing page # https://www.firebase.com/pricing.html
Update 11-Mar-16: Firebase no longer appears to measure connections based on 95th %. Instead, the 101st concurrent connection is denied.
https://www.firebase.com/pricing.html :
All our plans have a hard limit on the number of database connections.
Our Free and Spark plans are limited to 100. The limit cannot be
raised. All other plans have a courtesy limit of 10,000 database
connections. This can be removed to permanently allow Unlimited
connections if you email us at firebase-support#google.com.. The
reason we impose this courtesy limit is to prevent abuse and to ensure
that we are prepared to handle our largest customers. Please contact
us at least 24 hours in advance so we can lift this limit and ensure
we have enough capacity available for your needs.