I am using Firebase cloud functions for a project, and some of those functions fetch data from a firebase database.
I'd like to speed up some of these queries by leveraging a LRU cache, but it's not clear whether this is possible with Firebase cloud functions.
Does anyone know if the Firebase cloud functions have access to any kind of cache / semi-persistent memory access? Any help others can offer on this question would be hugely helpful!
If you want to share any sort of persistent data between function invocations, you will have to use another product, and code your function to use that. Cloud Functions themselves only have immedaite access to the memory on the server instance that's running a particular invocation, and there could be many server instances all running functions at the same time.
If you're OK with maintaining a small local cache in memory on each instance, that's fine. But you will have problem with this if you allow the cache to get so large that a function can't do its work with more limited memory. You should also expect the cache to get reset whenever a server instances get deallocated, which happens outside of your control.
Since you're working in Google Cloud, consider using a product such as Memorystore to implement your cache.
Related
Going through this blog, I could see that my app would be better off using the Firebase cloud functions for Firebase Firestore rather than directly accessing Firestore using client-side SDK.
I can implement the Firestore READ operation using get(), WRITE operation using set() or update() &
DELETE using delete() methods. All these one-shot operations are fine.
Is possible to implement addSnapshotListener to fetch real-time updates? If yes, how?
While it technically is possible to use addSnapshotListener, there are not a lot of use-cases where this makes sense.
Cloud Functions are short-lived, a maximum of 9 minutes, so not well suited for scenarios where you need to listen to the database for longer times.
Typically, you'll want to instead define a separate Cloud Function that responds to the thing you'd otherwise want to listen to.
Realtime listeners are not compatible with the way Cloud Functions works. Functions can only run for a limited amount of time and are shut down after that timeout expires. Database listeners are indefinite, and will continue listening until they are removed. If you add a listener inside a function, the system will still shut it down after the function timeout expires.
It's unclear to me why you need a listener in a Cloud Functions, but it's almost certainly not the right thing to do, given the way functions work.
Writing a firebase/google cloud function, and need to store an environment value for use across multiple function calls. That value expires and needs to be re-fetched on occasion and updated in production.
I'm looking for a lightweight option for that. Seems all the advice I can find is that you need to spin up a VPC and create a dedicated Redis instance... or you need to create a cloud database and store it there... I just need to save a simple string, and it seems like an awful lot of infrastructure to do that.
One would think environment variables would work, but you can only set on the command line and they are only refreshed on deploy...
To store environment data, you can use the firebase
functions:config:set command.
To get environment data, you can use the functions.config() function.
See https://firebase.google.com/docs/functions/config-env.
So, is there a way to update/set a value in my code? I cannot rely on the command line to update it as it expires, like a cron to update and redeploy.
In Google Apps Script, for example, I'd just use the 'cache' helper service and store the value for a few hours. Any equivalent cache available to cloud functions without resorting to storing on GCS or in a database (it's a single, simple token string...)? Thanks.
Cloud Functions does not offer any form of shared environment variables between functions. You will need to look to an external source such as Cloud Secrets Manager, Cloud Storage or one of the databases. I use both Cloud Storage and Datastore for this feature. I am now looking into Cloud Secrets Manager as my software usually has secrets as well.
I am working on a project that, currently, is 100% Firebase. Ideally, given I'm fully Firebase, I'd like to stay with Firebase for a next task which is updating some of the records based on external API calls once per day.
I'm currently using Firebase Functions for triggered events, not using it for API calls, everything that happens in the functions is after a user does something, and doesn't respond back to any clients (only responds back to the database for updates).
Is Firestore Cloud Functions a good place to run something like this that could call an external API and then update as necessary? I saw the scheduled functions that require the Blaze plan, have considered it but not sure if there's another approach that's better built for this task.
Cloud Functions that trigger on Firestore events probably aren't what you're looking for. Firestore triggers only fire when something in your Cloud Firestore database has changed. That means you need something that's writing to some document in the database in order to get the code to run. Which means you need a way to schedule that operation.
No matter what kind of trigger you write, you will need to be on a billing plan in order to make external requests anyway. So even if you somehow managed to put together a solution that uses Firestore triggers, your project would still need to be on a billing plan.
This approach is perfectly okay - in fact, I am using the exact same approach in my project which has 100% Firebase back-end. The overall (Firebase) Cloud Functions gives flexibility in terms of invocation i.e. they can be invoked based on trigger (e.g. storage or database event) or can be called with the HTTP end-point. So, depending on your need you can either use Firestore trigger or database trigger or call an end-point.
Switching to Blaze plan is perfectly fine since otherwise we can't call an external end-point. I switched to Blaze plan just a few months back and didn't pay anything for that as my usage is within the free limit.
According to firestore documentation:
a transaction is a set of read and write operations on one or more documents.
Also:
Transactions will fail when the client is offline.
Now the limitation in firestore is that:
In Cloud Firestore, you can only update a single document about once per second, which might be too low for some high-traffic applications.
So using cloud functions and running transactions to increment/decrement counters when the traffic is high will fail.
So they have discussed to use the approach of distributed counters.
According to the algorithm of distrbuted counter:
create shards
choose a shard randomly
run a transaction to increment/decrement the counter
get all the shards and aggregate the
result to show the value of a counter
Scenerio:
consider you have a counter which is to be updated when a document is added and that counter is being displayed in the UI. Now for good UX, I cannot block the UI when network is offline. So I must allow creation/updation of documents even when client is offline and sync these changes once client is online so that everyone else listening to these changes receive the correct value of the counter.
Now transactions fail when the client is offline.
So my question for best user experience (even when offline) is:
Do you really require a transaction to increment a counter? I know
transactions ensure that writes are atomic and are either
successful/unsuccessful and prevent partial writes. But what's the
point when they fail offline? I was thinking maybe write them to local cache and sync it once the network is back online.
Should this be done via client sdks of via cloud functions?
Do you really require a transaction to increment a counter?
Definitely yes! Because we are creating apps that can be used in a multi user environment, transactions are mandatory, so we can provide consistent data.
But what's the point when they fail offline?
When there is a loss of network connectivity (there is no network connection on user device), transactions are not supported for offline use. This is because a transaction absolutely requires round trip communications with server in order to ensure that the code inside the transaction completes successfully. So, transactions can only execute when you are online.
Should this be done via client sdks of via cloud functions?
Please note, that the Firestore SDK for Android has a local cache that's enabled by default. According to the official documentation regarding Firestore offline persistence:
For Android and iOS, offline persistence is enabled by default. To disable persistence, set the PersistenceEnabled option to false.
So all read operations will come from cache if there are no updates on the server. So Firestore provides this feature for handle offline data.
You can also write a function in Cloud Function that will increment the counter while a new document is added or to decrement the conter while a document is deleted.
I also recommend you to take a look:
How to count the number of documents under a collection in Firestore?
So you may also consider using Firebase realtime database for that. Cloud Firestore and Firebase realtime database work very well together.
Edit:
It allows one to upvote the answer even when the device is offline. After the network is online, it syncs to the server and the counter is updated. Is there a way i can do this in firestore when the device is offline.
This is also happening by default. So if the user tries to add/delete documents while offline, every operation is added to a queue. Once the user regains the connection, every change that is made while offline, will be updated on Firebase servers. With other words, all queries will be commited on the server.
Cloud fnctions are triggered only when the change is received and that can only happen when the device is online.
Yes, that correct. Once the device regains the network connection, the document is added/deleted from the database, moment in which the function fires and increases/decreases the counter.
Edit2:
Suppose I have made around 100 operations offline, will that not put a load on the cloud functions when the device comes online? What's your thought on this?
When offline, pending writes that have not yet been synced to the server are held in a queue. If you do too many write operations without going online to sync them, that queue will grow fast and it will not slow down only the write operations it will also slow down your read operations. So I suggest use this database for its online capabilities.
Regarding Cloud Functions for those 100 offline operations, there will be no issues. Firebase servers work very well with concurent operations.
My Current Plan:
I'm currently creating an IOS App that will access/change java/python files that are stored in the Google Cloud Storage. Once confirmed the App will talk with App Engine that will have a Compute Engine VM receive files and compile them. Once compiled have the result returned back to the IOS App
Is there any better or easier method to achieve this task? Should I use firebase or Google Cloud Functions? Would it be any help
Currently, I'm lost how to design and have requests sent between many platforms.
It would also depend on what type of data processing you are doing to the files in Cloud Storage. Ideally you would want to avoid as many "hops" between services as possible. You could do everything via Cloud Functions and listen on GCS Triggers. You can think of Cloud Functions as a sudo App Engine Backend to use for quick request handling.
Use Cloud Functions to respond to events from Cloud Storage or Firebase Storage to process files immediately after upload
If you are already using Firebase, it would be better to stay within their ecosystem as much as possible. If you are doing bigger or more intensive data processing you might want to look at different options.
With more information and current pain points, we may be able to offer more insight.