does anyone know how to make the firestore on Google Cloud be up-to-date with Firestore on Firebase?
I have tried first time using "Time-to-live" TTL on Google Cloud. when I create the policy to auto delete documents on firestore, I see there are still some options of collections that I deleted few days ago on my Firebase project, and the new collections did not appear.
So I can not set the TTL for the documents of new collections.
These are collections on my Firebase:
And here is what I have on Google Cloud:
As you can see from my Firebase, The collection of "Ads" should no longer exist, but the collection of "messages" should be there on the Firestore of Google Cloud instead.
Do you have any solutions for me? I appreciate it.
To summarize the documentation on managing data retention with TTL policies
Deletion through TTL is not instant and expired documents will continue to appear until the TTL process actually deletes them.
Data will typically be deleted after 72 hours after its expiration date. TTL also does not necessarily delete documents at the same time as their expiration.
If you require data deletion to be done transactionally, perform the deletions using a client library.
TTL deletions are treated with a lower priority to minimize impact on other database activities.
Basically, using TTL will not instantly remove the collection on Google Cloud Console but it is applied on Firestore Database once TTL policy is applied.
Related
how to check firebase cloud firestore size in a project?
according to firebase the free plan gives you 1GB for firebase cloud firestore, i already create some collections with documents inside, but where can i check the total size i am using?
i already check some statistics in firebase console but i can only see the numbers of reads and writes.
You can check your Cloud Firestore size or Cloud Firestore stored data in Google Cloud App Engine Quotas page. You can go direct to Google Cloud using the link because when you create a Firebase project, you're also creating a project in the Google Cloud.
Inside the App Engine Quotas page you can see also the other Cloud Firestore usage information including reads, writes, index writes, deletes, and network egress.
Example images:
UPDATE:
You can check your usage up to last 30 days in Firestore Database > Usage
Go to the Firestore Usage tab
Click "View in Usage and Billing" (bottom right)
This will show a summary of usage including total bytes stored, bandwidth, reads, writes, and deletes.
The answer is you can’t check the usage of bytes stored per month in Firestore. Only per day.
Usage per day can be checked in the app engine
It’s confusing because you don’t know how much data you have stored in total in Firestore.
I have an app that has 50k - 60k document reads a day and I can't afford another plan now, so I'm looking for a way to optimize reads, getting less reads possible I can.
The database has 4 collections with some subcollections, I think I might have around 1000 documents in the whole server.
It is there anyway: I can read documents from cloud Firestore (first time the user opens the app) and store on the Firestore cache, after that, make the app only load data from Firestore cache, and if there is a new document(s), it reads these documents only, store to the cache, and keep reading from the cache?
The app could only load data from cache (using the parameter "source") and never directly from Firestore, if possible.
Firestore.instance.collection("images").getDocuments(source: Source.cache);
Is is there anyway: I can read documents from cloud Firestore (first time the user opens the app) and store on the Firestore cache
That's the default behavior. According to the official documentation regarding Firestore offline persistence:
For Android and iOS, offline persistence is enabled by default. To disable persistence, set the PersistenceEnabled option to false.
For the web, offline persistence is disabled by default. To enable persistence, call the enablePersistence method. Cloud Firestore's cache isn't automatically cleared between sessions. Consequently, if your web app handles sensitive information, make sure to ask the user if they're on a trusted device before enabling persistence.
So there is nothing special that you need to do. Once you open a stream on one of your collections, the data is added to the cache.
after that, make the app only load data from Firestore cache, and if there is a new document(s), it reads these documents only, stores to the cache, and keeps reading from the cache?
That's again the default behavior, but this work as long as the documents in your database are not changed. If a document in the database is changed, you'll be charged with a document read for each document. The mechanism is the same.
Besides that, you can tell Firestore to read data only from the cache if needed, as explained in the answer from the following post:
How to cache Firebase data in Flutter?
According to firestore documentation:
a transaction is a set of read and write operations on one or more documents.
Also:
Transactions will fail when the client is offline.
Now the limitation in firestore is that:
In Cloud Firestore, you can only update a single document about once per second, which might be too low for some high-traffic applications.
So using cloud functions and running transactions to increment/decrement counters when the traffic is high will fail.
So they have discussed to use the approach of distributed counters.
According to the algorithm of distrbuted counter:
create shards
choose a shard randomly
run a transaction to increment/decrement the counter
get all the shards and aggregate the
result to show the value of a counter
Scenerio:
consider you have a counter which is to be updated when a document is added and that counter is being displayed in the UI. Now for good UX, I cannot block the UI when network is offline. So I must allow creation/updation of documents even when client is offline and sync these changes once client is online so that everyone else listening to these changes receive the correct value of the counter.
Now transactions fail when the client is offline.
So my question for best user experience (even when offline) is:
Do you really require a transaction to increment a counter? I know
transactions ensure that writes are atomic and are either
successful/unsuccessful and prevent partial writes. But what's the
point when they fail offline? I was thinking maybe write them to local cache and sync it once the network is back online.
Should this be done via client sdks of via cloud functions?
Do you really require a transaction to increment a counter?
Definitely yes! Because we are creating apps that can be used in a multi user environment, transactions are mandatory, so we can provide consistent data.
But what's the point when they fail offline?
When there is a loss of network connectivity (there is no network connection on user device), transactions are not supported for offline use. This is because a transaction absolutely requires round trip communications with server in order to ensure that the code inside the transaction completes successfully. So, transactions can only execute when you are online.
Should this be done via client sdks of via cloud functions?
Please note, that the Firestore SDK for Android has a local cache that's enabled by default. According to the official documentation regarding Firestore offline persistence:
For Android and iOS, offline persistence is enabled by default. To disable persistence, set the PersistenceEnabled option to false.
So all read operations will come from cache if there are no updates on the server. So Firestore provides this feature for handle offline data.
You can also write a function in Cloud Function that will increment the counter while a new document is added or to decrement the conter while a document is deleted.
I also recommend you to take a look:
How to count the number of documents under a collection in Firestore?
So you may also consider using Firebase realtime database for that. Cloud Firestore and Firebase realtime database work very well together.
Edit:
It allows one to upvote the answer even when the device is offline. After the network is online, it syncs to the server and the counter is updated. Is there a way i can do this in firestore when the device is offline.
This is also happening by default. So if the user tries to add/delete documents while offline, every operation is added to a queue. Once the user regains the connection, every change that is made while offline, will be updated on Firebase servers. With other words, all queries will be commited on the server.
Cloud fnctions are triggered only when the change is received and that can only happen when the device is online.
Yes, that correct. Once the device regains the network connection, the document is added/deleted from the database, moment in which the function fires and increases/decreases the counter.
Edit2:
Suppose I have made around 100 operations offline, will that not put a load on the cloud functions when the device comes online? What's your thought on this?
When offline, pending writes that have not yet been synced to the server are held in a queue. If you do too many write operations without going online to sync them, that queue will grow fast and it will not slow down only the write operations it will also slow down your read operations. So I suggest use this database for its online capabilities.
Regarding Cloud Functions for those 100 offline operations, there will be no issues. Firebase servers work very well with concurent operations.
It seems odd to me that Firestore would charge me for read queries to locally cached data, but I can't find any clarification to the contrary in the Firestore Pricing document. If I force Firebase into offline mode and then perform reads on my locally cached data, am I still charged for each individual entity that I retrieve?
Second, offline users in my app write many small updates to a single entity. I want the changes to persist locally each time (in case they quit the app), but I only need eventually consistent saves to the cloud. When a user reconnects to the internet and Firestore flushes the local changes, will I be charged a single write request for the entity or one per update call that I made while offline?
Firestore could potentially fit my use case very well, but if offline reads and writes are charged at the same rate as online ones it would not be an affordable option.
As the offical documentation says,
Cloud Firestore supports offline data persistence. This feature caches a copy of the Cloud Firestore data that your app is actively using, so your app can access the data when the device is offline. You can write, read, listen to, and query the cached data. When the device comes back online, Cloud Firestore synchronizes any local changes made by your app to the data stored remotely in Cloud Firestore.
So, every client that is using a Firestore database and sets PersistenceEnabled to true, maintains it's own internal (local) version of the database. When data is inserted/updated, it is first written to this local version of the database. As a result, all writes to the database are added to a queue. This means that all the operations that where stored there will be commited on Firebase servers once you are back online. This also means that those operations will be seen as independent operations and not as a whole.
But remeber, don't use Firestore as an offline-only database. It is really designed as an online database that came work for short to intermediate periods of being disconnected. While offline it will keep queue of write operations. As this queue grows, local operations and app startup will slow down. Nothing major, but over time these may add up.
If Google Cloud Firestore priceing model does not fit your use case very well then use Firebase Realtime Database. As mentioned also in this post from the Firebase offical blog, one the reasons you still might want to use the Realtime Database is:
As we noted above, Cloud Firestore's pricing model means that applications that perform very large numbers of small reads and writes per second per client could be significantly more expensive than a similarly performing app in the Realtime Database.
So it's up to you which option you choose.
According to this If you want to work completely offline with Cloud Firestore you can disable network by :
FirebaseFirestore.getInstance().disableNetwork()
but firestore will cause client offline error for first user get request, that you must consider this error as empty response.
I would like some explanation about the pricing for the realtime listener.
On the Firestore documentation (https://firebase.google.com/docs/firestore/pricing) it's stated:
if the listener is disconnected for more than 30 minutes (for example,
if the user goes offline), you will be charged for reads as if you had
issued a brand-new query.
What about if there are no changes (neither locally, nor remotely). Do you still get charged for a read when you reconnect after 30 minutes?
In such case it seems a pricing burden comparing to the Firebase Realtime Database, where if you enable the offline caching, the listeners don't download any new data and hence you are not charged.
According to this, you don't get charged for reads that are fulfilled by the local cache.
However you would still get charged for 1 read for the query itself (regardless of the results).