I’m wondering if accessing the data from the “change” snapshot of the onWrite cloud function count as an extra read on my billings.
Thank you!
The data from the database that is included with the payload that triggers your Cloud Function does not count as a charged read towards your database (either Firestore or Realtime Database), since they are served from within an existing data stream.
As per the documentation,
Cloud Firestore allows you to listen to the results of a query and get realtime updates when the query results change.
When you listen to the results of a query, you are charged for a read each time a document in the result set is added or updated. You are also charged for a read when a document is removed from the result set because the document has changed. (In contrast, when a document is deleted, you are not charged for a read.)
Also, if the listener is disconnected for more than 30 minutes (for example, if the user goes offline), you will be charged for reads as if you had issued a brand-new query.
This means that you'll be charged on the 1st read of the listener and will only charge you if there are changes or writes on the current snapshot.
Related
Currently I am using Firestore for my database and I have a users collection. Whenever a user document is created or updated to the users collection, a cloud function takes the user document and saves it in Elastisearch.
I am starting to be concerned about the scalability to this architecture. For example, suppose that several thousand cloud functions started writing documents to Elasticsearch at once, is Elasticsearch going to handle this load. Is there a better solution to this in Google cloud?
For example, can those cloud functions write the user documents in a queue and have cloud functions at the other end of the queue take a 100 documents and bulk write them to Elasticsearch.
I am new to Google cloud and would appreciate if you give me ideas, videos, and things to read.
Thanks
ElasticSearch has no limits on number of documents it can have per index but there are some limits such as maximum doc size and bulk writes mentioned in their documentation.
Maximum Document Size: 100KB [configurable in 7.7+]
Maximum Indexing Payload Size: 10MB
Bulk Indexing Maximum: 100 documents per batch
As far as I know, Google Cloud has no full text search API.
Talking of bulk writes, if realtime availability (data to be available immediately after adding) if not a concern, then you can store the new documents in Firestore along with a timestamp they were added and a boolean value if a document has been indexes in Elasticsearch.
Then instead of running a cloud function with onCreate trigger, you can run a scheduled cloud function every N minutes which will:
Query documents which have not been added in Elasticsearch
Make batches of 100 (for the 1000/batch limit)
Upload them to Elasticsearch
This way you are are more documents per cloud function run so that'll be a bit efficient but if you need your new data to be available immediately then this won't work.
https://firebase.google.com/docs/firestore says:
Like Realtime Database, Cloud Firestore uses data synchronization to update data on any connected device.
I'm trying to figure out what this "data synchronization" consists of.
https://firebase.google.com/docs/firestore/manage-data/enable-offline says:
When persistence is enabled, Cloud Firestore caches every document received from the backend for offline access.
Does Cloud Firestore update that cache whenever changes happen in the back end?
For example, if I have a certain document in my cache, does the document
update (a) automatically when that document is changed in the back end or
(b) whenever I do a query that uses that document, and I happen to be online
at the time?
UPDATE
I'm going to restate my last paragraph, because it's been misunderstood by
at least one person.
Imagine the following scenario:
I have a smartphone. To make things easy, assume the smartphone
is online all the time.
Installed on this smartphone is a mobile app that contains the Firestore SDK.
The app has enabled the Firestore cache.
The cache contains, say, 1000 documents.
Now my question is: if one of these documents changes in the cloud (not on
the smartphone), under what circumstances will the Firestore SDK update the
document in the cache? Will it (a) update the document automatically, soon
after the change on the cloud, or (b) update the document the next time the
mobile app does a query that uses the document?
Both of the highlighted paragraphs are in my opinion straightforward explanations, and to answer your questions:
Does Firestore update that cache whenever changes happen in the back end?
Yes, if you are offline and some documents are changed, once you are back online, you get all changed documents and vice versa, if you change some documents locally (in your cache), when you're back online, all changed documents are sent to the Firebase servers.
If you have a query that returns 10 documents, it means that you are in sync with 10 documents, which basically means that if of those 10 documents is changed, your listener is triggered only for that change.
does the document update (a) automatically when that document is changed in the back end
Yes, first of all in your cache and once you are back online, also on Firebase servers.
whenever I do a query that uses that document, and I happen to be online at the time?
Whenever you are online and a document is changed, you are notified in realtime.
Edit:
if one of these documents changes in the cloud (not on the smartphone), under what circumstances will the Firestore SDK update the document in the cache? Will it (a) update the document automatically, soon after the change on the cloud, or (b) update the document the next time the mobile app does a query that uses the document?
As long as you are in sync with those documents, it happens in the second moment the document is changed. What I mean through in sync is, that you are using in your code a get() call to get those documents or if you are listening in realtime for documents changes. Simply creating a reference to a document or a query without using a listener, you aren't notified in any way.
update the document the next time the mobile app does a query that uses the document?
Is not correct since you are online and in sync with the documents, the listener is triggered instantly.
Edit2:
In short, a listener is called everytime a change in a document occurs. "Next time the app does the query" is not correct sentence because when you create a query and attach a listener, you are always in sync with the database and when a document is changed you are immediately notified. It is not like, the documents are chainging in the database and when I query (at some point of time), I get the changes, no, you are always notified as the changes occur. That's the beauty of a realtime database, to be notified as the changes are happening :)
If you have the local cache enabled, Firestore automatically stores any documents it receives in that local cache.
This only happens when your app is requesting the documents by calling get() or onSnapshot. Firestore does not automatically request the documents on your behalf. So if your app doesn't request the document, the cache will not be updates.
I am understand that we have been charges for every read, write and delete operation, but my question are these operation also being counted in the firebase console?
Sometimes, we just wanna see the data through firebase console, and when click at the collection, it will automatically display the documents data and also sub-collection of the documents. Are we gonna get charged through activity in console too?
Are these operation also being counted in the firebase console?
Yes, all read, write or delete operations performed even from the cosole are counted.
Are we gonna get charged through activity in console too?
Yes.
In firestore pricing it given that pricing is according to number of reads.
So if I use a realtime listner on value which changes, say every second then I will charged for one read or for as much as data changes.
The billing for Firestore listeners is described on the Firestore pricing page:
Listening to query results
Cloud Firestore allows you to listen to the results of a query and get
realtime updates when the query results change.
When you listen to the results of a query, you are charged for a read
each time a document in the result set is added or updated. You are
also charged for a read when a document is removed from the result set
because the document has changed. (In contrast, when a document is
deleted, you are not charged for a read.)
Also, if the listener is disconnected for more than 30 minutes (for
example, if the user goes offline), you will be charged for reads as
if you had issued a brand-new query.
I would like some explanation about the pricing for the realtime listener.
On the Firestore documentation (https://firebase.google.com/docs/firestore/pricing) it's stated:
if the listener is disconnected for more than 30 minutes (for example,
if the user goes offline), you will be charged for reads as if you had
issued a brand-new query.
What about if there are no changes (neither locally, nor remotely). Do you still get charged for a read when you reconnect after 30 minutes?
In such case it seems a pricing burden comparing to the Firebase Realtime Database, where if you enable the offline caching, the listeners don't download any new data and hence you are not charged.
According to this, you don't get charged for reads that are fulfilled by the local cache.
However you would still get charged for 1 read for the query itself (regardless of the results).