I am implementing a real time chat with text moderation. I'd like for moderators to be able to add words to the filter and have it be applied to the Google Cloud function instantly.
Two ideas I had:
I could make a HTTP request on start up that retrieves an updated word list from my site, but that's not exactly real time.
Checking a Firebase database in onWrite and if the text is there, delete. I am under the impression this scenario would be quite expensive, as each chat sent would cost multiple reads.
Not sure what to do, thoughts?
Related
I have a Firebase realtime DB i am using to track user analytics. Currently there is about 11 000 users and each of them has quite a bit of entries ( from ten to few hundreds based on how long they interacted with the app ). Json file is 76MBs when i export whole DB.
I am using this data only for analytics, so i will have a look once per day or so on all of the data. Ie i need to download whole DB to get all the data.
When i do that, it takes about 3-5 minutes to actually load the data. I can imagine that if there were ten times more users, it would not be usable then anymore, because of load time.
So i am wondering if these load times are normal and if this is realy bad practice to do such thing? The reason i always download whole DB, is that i want to get overall data, ie how many users is registered and then for example how many ads were watched. To do that, i need to go into each user and see how many ads he watched and count them up. I cant do that without having access to data of all users.
This is first time i am doing something like this on a bit larger scale and those 76MBs are a bit surprising to me as well as the load times to get the data. It seems like its not feasable long term to use this setup.
If you only need this data yourself, consider using the automated backups to get access to the JSON. These backups are made out-of-band, meaning that they (unlike your current process) don't interfere with the handling of other client requests.
Additionally, if you're only using the database for gathering user analytics, consider offloading the data to a database that's more suitable for this purpose. So: use Realtime Database for the user's to send the data to you, but remove it from there to a cheaper/better place after that.
For example, it is quite common to transfer the data to BigQuery, which has much better ad-hoc querying capabilities than Realtime Database.
I'm beginner in flutter-fire app [: And here i've got a simple issue but not easy to figure out.
When we use Streambuilder in the app, we can usually see awesome synchronization between UI and DB on time. I'm really feeling joyful when using Streambuilder. It seemed like only beautiful things would happen with Streambuilder.
Meanwhile, a suspicious question popped out. When we enter print('hello world!'); in the Streambuilder, we can see that RUN console is printing out the phrase every milliseconds, otherwise would be just printed once. It means that RAM usage is extremely increased. When it comes to DB synchronization, we can easily guess that the use of Streambuilder leads to huge usage of client-server communication fee, which in my case is Firebase fee.
So, here is my question.
Can i feel free to use Streambuilder when the stream is connected to DB(in my case, firebase)?
I'm worrying about communication fee between UI and firebase, because streambuilder seems like literally using huge amount of energy every milliseconds(IMO additionally, server fee) unlike normal builders. Especially when the length of collection is so long that reading the collection once may cost a lot of energy, the fee would increase a lot more on and on, because the streambuilder have to check thousands of firebase documents only to figure out a single line of condition.
I guess many backend-connected flutter methods use the Streambuilder, so someone could clearly figure it out how much we're gonna pay for Google when we use Streambuilder. I know it's quite ambiguous question, but hope you understand. [:
Content coming from the Firebase database, whether FutureBuilder or StreamBuilder, pays only for the query value that has been processed once, and after that, in case the same response value is the same for the same query, it does not pay for that cost and displays the stored list stored in the cache on the client screen again. .
And check that it's not being called on something like setState. if so, of course StreamBuilder is called again.
I am working on a video based app that keeps track of how many views that video has received. I originally planned on having a field for view_count in my document that I would write to after someone watches a video.
However, knowing how many writes that could end up leading to, I started to wonder if it's possible to see a breakdown of how many reads have been made for each document in a collection and use that number instead. Since the videos are short, I figured this would be an accurate number for the view count.
Is this possible to access this kind of data?
Firestore does not expose any per-document access metrics. The available monitoring options are shown on this page on monitoring usage.
If you want something beyond that you'll have to build it yourself, as you originally intended.
I am creating an application that uses cloud firestore to store data about "events" in our lab on several assets. We collected data for a few months and we are averaging about 2000 events per asset per month. Each event captures a few pieces of meta data that the user can query.
I imported all the data into firestore with a very simple layout at first.
Events (Collection of event data)
-> EventData (documents which contains a few fields for metadata)
From my understanding, even if the collection of events becomes quite large, for billing and speed of queries this won't be a problem (assuming I do some sort of pagination on the query results). The composite indexes are also very manageable with this structure.
The problem I see, is if someone goes and looks at the firestore console and brings that collection up, our read requests go through the roof. It seems that does a full read on the entire collection...which of course will kill us on billing as time goes on. I don't see this as a problem forever as eventually we should get to the point where everything is stable and won't need to go into the console very often, but what if someone does when we have a million or more records.
My next thought was to structure the database like this:
Events -> Assets -> {Asset_Name } -> {year_month} -> {Collection of
Document with field meta-data}
This certainly solves the issue of the ever growing collection of documents. The number of assets that we have is fixed, and the number of events is (effectively) capped to a maximum amount per month as well. The problem with this setup, however, is managing composite indexes. There are about 5 indexes needed for my original setup. I think this alternative setup means I would need to setup the same 5 indexes for each each collection of documents for every asset every month.
I thought maybe there could be a way to have a cloud function manage it for me (it doesn't appear there is an API for this). I think the number of indexes per project is also capped.
So, in the end, I am looking for recommendations on how to structure this database to limit reads if using the console, as well as keeping the indexes manageable. I am pretty new to NoSQL and perhaps I am just completely off.
I recommend you keep your structure as is if that's what's working for you. You should not need to optimize for reducing console reads. Console reads do count towards your usage but the console does not load the entire collection when you open the console.
The console loads just enough documents to let you scroll a bit and then it loads more documents if you scroll down. It will only load the entire collection if you scroll through the entire collection.
In the firebase example (https://gist.github.com/anantn/4323981), to add an user to the game, we attach the transaction method to playerListRef. Now, every time firebase attempts to update data, it will call the callback passed to the transaction method with the list of userid of all players. If my game supports thousands of users to join at a time, every instance this method executes, the entire user list will be downloaded and passed which will be bad.
If this is true, what is the recommended way to assign users then?
This is specifically what Firebase was designed to handle. If your application needs to actually assign player numbers, this example is the way to go. Otherwise, if the players just need to be in the same "game" or "room" without any notion of ordering you could remove the transaction code to speed things up a bit. The snippet as well as the backend have handled the number of concurrent connections you've mentioned—if you're seeing any specific problems with your code or behavior with Firebase that appears to be a bug, please contact us at support#firebase.com and we can dig into it.