We have a mobile app service on Firebase.
Our service's concurrent connection is of almost 5,000 ~ 10,000.
I know, it does NOT matter with performance limitations.
But, we have an issue about Realtime Database's pending (1~3 minutes).
It occurs every single day at night time, even with few connections.
We started logging the main realtime database with Elasticsearch because of these pending issues.
This pending issue can be checked in detail.
If occurs db's pending, our app going to disable in 1~3 minutes suddenly (db load 3% -> 100%)
and we can checked 'concurrent-connect/concurrent-disconnect' of almost concurrent users in same time.
I have attached a relevant screen capture. This issue can also be checked in GCP's Stackdriver.
We guess this issue was generated logic of Firebase assigned performance because of usage difference daytime with nighttime.
I've contacted support 3 days ago but haven't received a reply yet.
So, I wondering if anyone has the same problem or knows of this issue.
stackdriver captured
elk captured
It sounds like you're performing a bulk read or write operation every day at the same time. This type of operation typically comes from a batch process and is unrelated to the number of active users of your app. In fact, the bulk read/write will lock out the regular users of your app.
An example of such a batch process could be a bulk read that every night synchronizes the data in the Firebase Realtime Database with Elasticsearch, but there are many more options.
If there's indeed a batch process that is causing this load, you'll want to look into:
Either splitting the process into multiple smaller steps, so that they can intersperse with the traffic from your regular clients.
Or running the same type of process on a backup of the database. Since you'd be running against a local file, it won't interfere with the regular clients, and the backup is made by an out-of-band process (so not causing interference either).
Related
I am developing an Android app which basically does this: On the landing(home) page it shows a couple of words. These words need to be updated on daily basis. Secondly, there is an 'experiences' tab in which a list of user experiences (around 500) shows up with their profile pic, description,etc.
This basic app is expected to get around 1 million users daily who will open the app daily at least once to see those couple of words. Many may occasionally open up the experiences section.
Thirdly, the app needs to have a push notification feature.
I am planning to purchase a managed wordpress hosting, set up a website, and add a post each day with those couple of words, use the JSON-API to extract those words and display them on app's home page. Similarly for the experiences, I will add each as a wordpress post and extract them from the Wordpress database. The reason I am choosing wordpress is that it has ready made interfaces for data entry which will save my time and effort.
But I am stuck on this: will the wordpress DB be able to handle such large amount of queries ? With such a large userbase and spiky traffic, I suspect I might cross the max. concurrent connections limit.
What's the best strategy in my case ? Should I use WP, or use firebase or any other service ? I need to make sure the scheme is cost effective also.
My app is basically very similar to this one:
https://play.google.com/store/apps/details?id=com.ekaum.ekaum
For push notifications, I am planning to use third party services.
Kindly suggest the best strategy I should go with for designing the back end of this app.
Thanks to everyone out there in advance who are willing to help me in this.
I have never used Wordpress, so I don't know if or how it could handle that load.
You can still use WP for data entry, and write a scheduled function that would use WP's JSON API to copy that data into Firebase.
RTDB-vs-Firestore scalability states that RTDB can handle 200 thousand concurrent connections and Firestore 1 million concurrent connections.
However, if I get it right, your app doesn't need connections to be active (i.e. receive real-time updates). You can get your data once, then close the connection.
For RTDB, Enabling Offline Capabilities on Android states that
On Android, Firebase automatically manages connection state to reduce bandwidth and battery usage. When a client has no active listeners, no pending write or onDisconnect operations, and is not explicitly disconnected by the goOffline method, Firebase closes the connection after 60 seconds of inactivity.
So the connection should close by itself after 1 minute, if you remove your listeners, or you can force close it earlier using goOffline.
For Firestore, I don't know if it happens automatically, but you can do it manually.
In Firebase Pricing you can see that 100K Firestore document reads is $0.06. 1M reads (for the two words) should cost $0.6 plus some network traffic. In RTDB, the cost has to do with data bulk, so it requires some calculations, but it shouldn't be much. I am not familiar with the pricing small details, so you should do some more research.
In the app you mentioned, the experiences don't seem to change very often. You might want to try to build your own caching manually, and add the required versioning info in the daily data.
Edit:
It would possibly be more efficient and less costly if you used Firebase Hosting, instead of RTDB/Firestore directly. See Serve dynamic content and host microservices with Cloud Functions and Manage cache behavior.
In short, you create a HTTP function that reads your database and returns the data you need. You configure hosting to call that function, and configure the cache such that subsequent requests are served the cached result via hosting (without extra function invocations).
I just hit a situation which pushed me to ask this question:
I have about 150 active monthly users and I just hit 1k concurrent connections on a single day.
I did research and found many questions on "firebase concurrent connections" topic and those who refers to user-to-concurrent ratio say that on average it's close to 1 concurrent = ~1400 monthly users (like here and here).
I'm now trying to understand if I really did something wrong and if yes, how to fix that?
The questions are:
Is it look ok to get 1k concurrent connections with about 150 active users? Or am I reading it wrong?
Is it possible to profile concurrent connections somehow?
What are the typical "connection leaks" when it comes to chrome extensions and how to avoid them?
So far the architecture of the extension is that all the communication with firebase database is made from the background persistent script which is global to a browser instance.
And as a note, 150 active users is an estimation. For upper boundary I can say that I have 472 user records in total and half of them installed the extension and uninstalled it shortly after that - so they are not using it. And about 20% of the installed instances are also disabled in chrome.
Here is what I get after discussing with the support team:
here are other common use cases that can add up to the number of
connections in your app:
Opening your web app in multiple tabs on your browser (1 connection per tab)
Accessing the Realtime Database dashboard from the Firebase Console (1 connection per tab)
Having Realtime Database triggers
So Realtime Database triggers appeared to be my case.
Further discussion revealed the following:
In the case of uploading 200 data points which each trigger a
function, any number of concurrent connections between 1 and 200 is
possible. It all depends on how Cloud Functions scales. In an extreme
case it could just be one instance that processes all 200 events one
at a time. In another extreme case the Cloud Functions system could
decide to spin up 200 new server instances to handle the incoming
events. There's no guarantee as to what will happen here, Cloud
Functions will try to do the right thing. Each case would cost the
user the same amount on their Cloud Functions bill. What's most likely
in a real application (where it's not a complete cold start) is
something in the middle.
There's no need to worry about the number of concurrent connections
from Cloud Functions to RTDB. Cloud Functions would never spin up
anywhere near 100k server instances. So there's no way Cloud Functions
would eat up the whole concurrency limit. That would only happen if
there are many users on the client app accessing your database
directly from their devices.
So the described behavior in my question seems to be expected and it will not come any close to the limit of 100k connections from server side.
https://firebase.google.com/docs/firestore/manage-data/enable-offline
How does Firestore work with offline data?
How are writes merged by many clients editing the same data offline, that then come online at the same time?
How long is offline data persisted? If my user uses my app for 5 years offline, then comes back online, will this be an issue? Do offline changes persist after device restarts?
Does query performance of offline data degrade as the data set gets larger?
Im specifically interested in the web Firestore client.
Do all the language clients implement the above in the same manner?
Thanks.
How are writes merged by many clients editing the same data offline, that then come online at the same time?
The write operations that will take place on Firebase servers, will be in the order in which that series of operations happened. The last operation (the most recent one) will be the one that will be available in the database by the time the synchronization occurs.
How long is offline data persisted? If my user uses my app for 5 years offline, then comes back online, will this be an issue?
The problem is not about how long is about how many operations do you make while the device is offline. While offline, Firestore will keep in queue all the write operations. As this queue grows, local operations and app startup will slow down. Nothing major, but over time these may add up. The major problem in this case is that the end result of this will be that the data on the server stays unmodified. Then what is the purpose of a realtime database? Firestore is really designed as an online database that came work for short to intermediate periods of being disconnected, not to stay offline for 5 years. Beside that, in 5 years it might be a problem of compatibility and not of the number of writes.
Do offline changes persist after device restarts?
The offline persistence is also called disk persistence. This type of persistence is enabled by default in Cloud Firestore and it means that recently listened data (as well as any pending writes from the app to the database) are persisted to disk. The data in this cache survives app restarts and device reboots.
Does query performance of offline data degrade as the data set gets larger?
Yes it does, like explained above.
Do all the language clients implement the above in the same manner?
No. For iOS and Android, the offline feature works fine while for web, this feature is still experimental.
Each of failed requests was stalled for 10 seconds, so for 40 seconds my app couldn't reach the Firebase database.
I tried to open https://console.firebase.google.com/ and I couldn't react it either.
While https://status.firebase.google.com/ said that database is in normal operations. And any other web pages I opened worked normally.
Is it normal and how often does it happen?
If you think Firebase has downtime, always check the Firebase status dashboard as you did.
You'll see there that we regularly post for interruptions on the individual database servers. But the time of the last status update doesn't match with your post, so it is unlikely you were seeing an outage.
Since you also couldn't reach the console, which hasn't had any reported outage in over a week, it is much more likely that there was an interruption between your machine and the Firebase servers.
I've build a library-style SQLite iOS app using the code in the Recipes sample app, and it works - updates on one device are (eventually) reliably propagated to all other devices running the same app. I've been testing it with multiple events per hour all day long, and all the log transactions do get to every device. However, the time for the updates to propagate is highly variable. If I bring the app up and let it sit, it could be a relatively long time before the cloud sends update transactions to the app, and so what's on-screen remains old data for that same long time. Worse, there's no indication the data is out of date.
If I cause the app to post a change to the cloud, though, updates from the cloud propagate down relatively soon. That suggests that I could put in a hack that periodically posts pointless changes to the database, but even then I won't know if I've received all the changes.
First question: Do methods that will force transactions to propagate exist? a This thread suggests not.
Second question: Is there a way to detect if the local database is out of date? I don't want to tickle the cloud copy incessantly, but doing so now and then until the database is current might not be such a bad idea.
1) if you find such methods, please let me know.
2) same stuff here. I didn't even find any reliable way to detect whether the currently running instance of the iCloud app is not the first instance i.e. other instances have already sent updates to the iCloud until the first push received (which can take some unknown time)