Does reading (or writing) data from Realtime Database within a Firebase Cloud Function using transfer volume?
In my function almost the whole database is analyzed/read... can it happen, that the free download transfer limit will be exceeded?
Reading from the Realtime Database in any way is counted as a charged download. There is no exemption for traffic from Cloud Functions.
Related
I am pushing data to Firebase via Zapier and it lands in the Realtime Database. However, I am using Cloud Firestore. I've looked through a lot of documentation for both services but neither seems to have answers.. (there is information on migrating but I would like to keep both DBs).
How would I push each new or updated data entry from Realtime Database to Cloud Firestore?
There is nothing built into Firebase to automatically do this.
The fastest I can think of is to build a Cloud Function that listens for the Realtime Database writes, and then also sends them to Firestore.
Alternatively, doesn't Zapier allow you to have two zaps, one that writes to each database?
Currently I am using Firestore for my database and I have a users collection. Whenever a user document is created or updated to the users collection, a cloud function takes the user document and saves it in Elastisearch.
I am starting to be concerned about the scalability to this architecture. For example, suppose that several thousand cloud functions started writing documents to Elasticsearch at once, is Elasticsearch going to handle this load. Is there a better solution to this in Google cloud?
For example, can those cloud functions write the user documents in a queue and have cloud functions at the other end of the queue take a 100 documents and bulk write them to Elasticsearch.
I am new to Google cloud and would appreciate if you give me ideas, videos, and things to read.
Thanks
ElasticSearch has no limits on number of documents it can have per index but there are some limits such as maximum doc size and bulk writes mentioned in their documentation.
Maximum Document Size: 100KB [configurable in 7.7+]
Maximum Indexing Payload Size: 10MB
Bulk Indexing Maximum: 100 documents per batch
As far as I know, Google Cloud has no full text search API.
Talking of bulk writes, if realtime availability (data to be available immediately after adding) if not a concern, then you can store the new documents in Firestore along with a timestamp they were added and a boolean value if a document has been indexes in Elasticsearch.
Then instead of running a cloud function with onCreate trigger, you can run a scheduled cloud function every N minutes which will:
Query documents which have not been added in Elasticsearch
Make batches of 100 (for the 1000/batch limit)
Upload them to Elasticsearch
This way you are are more documents per cloud function run so that'll be a bit efficient but if you need your new data to be available immediately then this won't work.
Firebase Storage is using 490 MB, but no bucket has been initialized.
I am not able to track where this storage com from, but it is quite strange to check that Firebase is charging 0.10 USD for an empty Storage.
Where can I delete this storage and why firebase is charging for having no buckets?
Currently I am using Firebase Authentication, Firestore, Realtime Database, Hosting and Functions.
Every time you deploy to Cloud Functions, it takes some space in a new storage bucket. This is covered in the documentation FAQ about Cloud Functions billing for node 10.
You can delete those files over time if you want.
My recommendation is to use object lifecycles in the artifacts storage bucket! that way the files deletion is automated.
Follow this documentation:
https://cloud.google.com/storage/docs/lifecycle
(Firebase should be the one responsible for creating this rule!)
In my project I‘m using the realtime database as well as the Firestore database. My cloud functions triggers when a document is added to a specific collection inside my Firestore database. The function then writes to my realtime database to update some statistics. I noticed that even if no user is connected to my realtime database, the count of concurrent connections increases when the cloud function writes to the realtime database. Does this mean that my cloud functions count as concurrent connections and will decrease the amount of users that can connect to my realtime database at once as there is a limit of 100k concurrent connections?
You'll incur one connection for each server instance running your functions. It is not one per individual function invocation. So, if it only requires one server instance to run your function, then it will only cost one connection. When a server instance is automatically deallocated from your project, its connection will go away.
You will need an extremely busy set of functions in order to make a real dent in your 100K concurrent connection cap for Realtime Database. I wouldn't worry about it.
It seems odd to me that Firestore would charge me for read queries to locally cached data, but I can't find any clarification to the contrary in the Firestore Pricing document. If I force Firebase into offline mode and then perform reads on my locally cached data, am I still charged for each individual entity that I retrieve?
Second, offline users in my app write many small updates to a single entity. I want the changes to persist locally each time (in case they quit the app), but I only need eventually consistent saves to the cloud. When a user reconnects to the internet and Firestore flushes the local changes, will I be charged a single write request for the entity or one per update call that I made while offline?
Firestore could potentially fit my use case very well, but if offline reads and writes are charged at the same rate as online ones it would not be an affordable option.
As the offical documentation says,
Cloud Firestore supports offline data persistence. This feature caches a copy of the Cloud Firestore data that your app is actively using, so your app can access the data when the device is offline. You can write, read, listen to, and query the cached data. When the device comes back online, Cloud Firestore synchronizes any local changes made by your app to the data stored remotely in Cloud Firestore.
So, every client that is using a Firestore database and sets PersistenceEnabled to true, maintains it's own internal (local) version of the database. When data is inserted/updated, it is first written to this local version of the database. As a result, all writes to the database are added to a queue. This means that all the operations that where stored there will be commited on Firebase servers once you are back online. This also means that those operations will be seen as independent operations and not as a whole.
But remeber, don't use Firestore as an offline-only database. It is really designed as an online database that came work for short to intermediate periods of being disconnected. While offline it will keep queue of write operations. As this queue grows, local operations and app startup will slow down. Nothing major, but over time these may add up.
If Google Cloud Firestore priceing model does not fit your use case very well then use Firebase Realtime Database. As mentioned also in this post from the Firebase offical blog, one the reasons you still might want to use the Realtime Database is:
As we noted above, Cloud Firestore's pricing model means that applications that perform very large numbers of small reads and writes per second per client could be significantly more expensive than a similarly performing app in the Realtime Database.
So it's up to you which option you choose.
According to this If you want to work completely offline with Cloud Firestore you can disable network by :
FirebaseFirestore.getInstance().disableNetwork()
but firestore will cause client offline error for first user get request, that you must consider this error as empty response.