I need to initialize potentially thousands of Firebase app instances server-side using firebase-admin with a service account and then listen to specific Realtime DB and Firestore events on these app instances. These instances have nothing to do with one another and the only access I have to them is through their service account credentials files.
My main concern is scaling because new project app instances will be initialized on the fly. It is important that the events listened to should not be duplicated, so I can not have multiple server instances initialize listen to the events of all the apps.
Ideally, I would love to have an auto-scaling serverless solution and to keep it inside the Google Cloud Platform family. If there was a way to work with the app instances, while they are auto-scaled behind the scenes - like a load balancer for a collection of app instances. I've looked at the Google Cloud Pub/Sub solution to potentially do something like this, but could not come up with something concrete.
The non-scaling solution I have so far, using cloud functions:
EDIT: After feedback regarding Cloud Functions
on startup of my project's cloud functions, initialize all the current app instances
have a cloud function listen to newly added service accounts and initialize a new app instance for it. Currently, I have no idea how many project instances I can keep in memory before it becomes a problem.
Start up a single Node server to initialize all the current app instances. As new projects are added, simply add a reference for it. Monitor the server closely to ensure it doesn't run out of memory. Scale vertically as far as possible. When it gets to the point where necessary, start up a second server instance, but manually distribute project instances between servers, so there are no duplicates.
Bonus question: are only the individual cloud functions scaled as needed? ie. what happens to other standard JS functions/objects created outside a cloud function but inside the cloud functions index.js file?
Any advice would be greatly appreciated. Happy to elaborate on anything.
Related
I am using Firebase cloud functions for a project, and some of those functions fetch data from a firebase database.
I'd like to speed up some of these queries by leveraging a LRU cache, but it's not clear whether this is possible with Firebase cloud functions.
Does anyone know if the Firebase cloud functions have access to any kind of cache / semi-persistent memory access? Any help others can offer on this question would be hugely helpful!
If you want to share any sort of persistent data between function invocations, you will have to use another product, and code your function to use that. Cloud Functions themselves only have immedaite access to the memory on the server instance that's running a particular invocation, and there could be many server instances all running functions at the same time.
If you're OK with maintaining a small local cache in memory on each instance, that's fine. But you will have problem with this if you allow the cache to get so large that a function can't do its work with more limited memory. You should also expect the cache to get reset whenever a server instances get deallocated, which happens outside of your control.
Since you're working in Google Cloud, consider using a product such as Memorystore to implement your cache.
So I was watching multiple tutorials about how to present data on an actual webpage using Cloud Firestore. The thing is, everyone was using an actual index.html file sitting in Public folder(instead of serving html content within node.js code) and tags which would mean that their program would use client-side JavaScript instead of node.js. But why? for what reason? According to Firebase tutorials and documentation, having an ACTUAL index.html sitting in Public folder is only for making static webpages, thus, if I'm making a complicated and dynamic webpage(which will also present Firestore data within the webpage), I should be using node.js right?
The Firebase SDKs for Cloud Firestore perform local persistence (caching) of doucments fetched from the database. The persistence is enabled by default for Android and iOS, and you can programmatically enable it for web (currently experimental). This local caching allows the client to avoid requesting documents from the server, which is obviously faster and cheaper than going through some API endpoint to request the data each time it's needed.
There is also the fact that the SDKs will push you realtime updates of data as it changes on the server, as long as you have a listener attached to some document or query of interest. You won't be able to duplicate this if you write it all in Cloud Functions. You will spend a tremendous amount of time trying to duplicate and scale this behavior using socket.io or something similar on a backend you control.
You could write the entire app to be driven through API endpoints that you create. There's nothing wrong with that, if it meets your needs. But you'll write more code, you'll sacrifice realtime updates, and it will likely be slower and more expensive than allowing the client SDKs to optimize for you.
My Current Plan:
I'm currently creating an IOS App that will access/change java/python files that are stored in the Google Cloud Storage. Once confirmed the App will talk with App Engine that will have a Compute Engine VM receive files and compile them. Once compiled have the result returned back to the IOS App
Is there any better or easier method to achieve this task? Should I use firebase or Google Cloud Functions? Would it be any help
Currently, I'm lost how to design and have requests sent between many platforms.
It would also depend on what type of data processing you are doing to the files in Cloud Storage. Ideally you would want to avoid as many "hops" between services as possible. You could do everything via Cloud Functions and listen on GCS Triggers. You can think of Cloud Functions as a sudo App Engine Backend to use for quick request handling.
Use Cloud Functions to respond to events from Cloud Storage or Firebase Storage to process files immediately after upload
If you are already using Firebase, it would be better to stay within their ecosystem as much as possible. If you are doing bigger or more intensive data processing you might want to look at different options.
With more information and current pain points, we may be able to offer more insight.
I'm trying to build a relatively simple application that has several different views, requires authentication, collects some user data, stores it in a database, and needs backend logic to manipulate that data and spit it back out to the user in a view.
The stack I've decided on is Vue for the frontend, with Express and Node for server side logic and Firebase for some of their services (namely auth, firestore, storage).
I haven't been able to find any examples of this stack (Vue, Express, Firebase) anywhere (I have however found Vue/Express or Vue/Firebase examples). My question is whether or not Express is obsolete here in that I can use Vue router to do my routing. Is the difference that one does the rendering server-side?
You could use Cloud Functions for Firebase as your backend and then limit your stack to Vue.js and Firebase.
From the doc:
Cloud Functions for Firebase lets you automatically run backend code
in response to events triggered by Firebase features and HTTPS
requests. Your code is stored in Google's cloud and runs in a managed
environment. There's no need to manage and scale your own servers.
For your specific need: "backend logic to manipulate that data and spit it back out to the user in a view." you have several possible approaches:
You manipulate the data with Cloud Functions (in the back end), write the results of this manipulation to the Real Time Database and setup some listeners in your Vue.js frontend (with the on() method) that will refresh your front end upon changes in the database
Use some HTTPS Cloud Functions that you call, from your Vue.js front-end, as REST API endpoints. You can use Axios for example. See the doc here.
One advantage of the first solution is that you can easily, by default, rely on the database security rules, while it would need some more extra work in the second case.
To avoid rewriting Firebase DB logic in various apps (iOS, Android, Web) we earlier used a service/middle layer to hold this logic. In this way the app never interacts with the DB directly.
However, in the new architecture with Firebase and Google Cloud Functions, would it be wise to route all DB calls through Cloud Functions or should this be done only selectively based on use case?
In almost all the examples I’ve seen so far, the app directly interacts with Firebase DB and the Cloud Functions are meant to only listen to certain events and used selectively. They are not meant to be a middle layer.
This approach would however lead to the need to duplicate DB logic in all apps. Can this duplication of code be avoided?
Yes. By moving certain functionality from your application code into Cloud Functions, you will only have to implement that logic once: in JavaScript. This is great for certain logic that you either don't want on the client (too big, too secret, too slow, etc).
But:
Each client will still need code to access the functionality in Cloud Functions. This can be as simple as a write through the Database SDK, but can also get quite involved.
The functionality will only be available if the user has a network connection. Unlike client-side functionality, it won't work when the user is disconnected/offline.