I have so far been very impressed with the firebase platform for hosting a client-side single page app and for data storage. However, I have one component that I don't know where to host...
I want to have a background process that aperiodically updates the database. The nature of when an update is needed is based on an external source and, although the general timeframe of when updates are available is known, the exact timing is not. My thinking was to have a background task running that has some smarts to determine when an update is needed, and then trigger an update at that time.
I don't know where I would host something like this. I considered running it in a loop in a firebase function, but due to pricing model being based on time, that would get very expensive, and functions are not suited for daemon-type processes. The actual "database update" would be suitable for a function, but not the triggering logic. Also, I have seen functions-cron which does offload the triggering logic, but since my updates are not truly periodic, it doesn't seem exactly appropriate. I haven't looked too much into AppEngine and how that relates to the firebase platform...so basically my question:
What are the options for "reasonably-priced" hosting an always-running background task?
Google App Engine - Standard is something you want to look at more. It is reasonably priced since what you are doing will likely fit into GAE-Std's free daily quota. In GAE-Std, you create a scheduled cron job: GAE will call you task as if it was an incoming web request.
See Firebase doc for integrating with GAE
See GAE doc for cron jobs
Related
We are building an app for our teams out in the field that they collect their daily information using Firebase. However one of our concerns is poor connectivity. We are looking to build an Online/Offline button they can click to essentially work offline for when things slow down. We've built a workflow in which we query all the relevant information from Firestore.
I wanted to know if there was a way to tell Firestore to work directly on the cache only and not try to hit the servers directly. I don't want Firestore attempting to make server calls until they enable online again.
You shouldn't need to do this. If you use realtime listeners, they will already first return the data from the local cache, and only then reach out to the server to check for updates.
If you are performing one-time reads, the SDK will by default try to reach the server first (since it has only one chance to give you a value). If you want it to only check the local cache, you can pass an argument to the get call to do so.
You can also disable the network completely, in which case the client will never call on the network and only serve from the local cache. I recommend reading about that and more in the documentation on using Firestore offline.
The firebase documentation includes a warning that if you use a kill switch to stop using firebase when a budget cap is exceeded, as follows:
Warning: This example removes Cloud Billing from your project,
shutting down all resources. Resources might not shut down gracefully,
and might be irretrievably deleted. There is no graceful recovery if
you disable Cloud Billing. You can re-enable Cloud Billing, but there
is no guarantee of service recovery and manual configuration is
required.
I'm trying to investigate what gets irretrievably deleted. Does the datastore get deleted when the kill switch is activated? Is there any opportunity to save data previously stored in cloud firestore, before the deletion takes place? Is there a way to download the database so that I can keep a back up in this scenario?
Please review the following reply from Firebase Team member(samstern) to gain more clarity on this:
these things are handled on a per-product basis and each product has different thresholds for quota overages and different procedures for what happens to inactive resources in a bad state.
For instance I know in Realtime Database if your DB is too big for the
free plan after you downgrade from a paid plan we will not delete
your data automatically. Instead you'll just be stopped from using the
database until you restore your billing.
However that statement clearly says that some products may delete data
if you pull Cloud Billing. It could be for technical reasons, it could
be for policy reasons.
If you want to turn off a product without risking this pulling your
billing account is NOT the best way to do this. It's the nuclear
option if you get into a bad situation and want to get out at all
costs. Instead you should use per-product APIs to shut down the
individual product or service and prevent future usage. This could
include doing things like flipping off APIs in the APIs console,
changing security rules to prevent all further writes, deleting Cloud
Functions, etc
The best source of information I've been able to uncover in answer to this particular question is a discussion on reddit which indicates that you can't recover access to your data, until you pay the bill (including blow out charges) - so maybe that buys some time, but if you don't pay, the project gets deleted. There may also be lost data for things happening at the time the kill switch activates.
Sorry if that has been answered - I couldn't find a complete answer - as there seems to be conflicting resources.
What I'm trying to achieve is the architecture where my backend 'communicates' in real time with millions of clients through Firebase DB (RTD or FS). Kind of like what's drawn here:
The architecture in a nutshell: millions of clients write 'events' to the Firebase DB, my java server listens to these 'events', processes them and writes 'responses' to the Firebase DB that are synced back to the clients.
The question: Is App Engine the best solution? Is it even suitable for the job?
On the one hand, in App Engine's documentation there's an example of it used that way:
https://cloud.google.com/solutions/mobile/mobile-firebase-app-engine-flexible
On the other hand, there are (seemingly) known issues with that approach:
1) App Engine instances awake on http requests, not on firebase events. https://stackoverflow.com/a/38357458/1806956
Jobs have a timeout, so even if we do a cron wakeup every minute, it doesn't ensure (or does it?) that the listener will keep living forever.
2) App Engine does not support the Firebase Admin SDK due to background threads? https://stackoverflow.com/a/45046605/1806956
3) App Engine limits the number of background threads. In a real app, we're talking about potential thousands of concurrent users, all throwing events...
Are the above issues not updated? Thank you...
I am using firebase functions for an Uberlike product. I can't get expected performance. Specially it takes a long time to load data from realtime-db. Up to 2-3 seconds for a read.
It's may be due to called start, which is discussed here. => Why is Cloud Functions for Firebase taking 25 seconds?
So I decided to move the functionality of these functions to a VM instance. Using firebase onWrite and admin SDK, a similar functionality can be achieved on a virtual machine.
Is it okay to do so? Will I get any scalability issue?
It is definitely possible to run similar code on your own hardware/VM. In fact that is how many of Firebase's own back-end processes ran, before Cloud Functions was available.
What you'll miss is the auto-scaling of Cloud Functions though. Your machine/VM will always be running, and has a limited capacity (how much it can handle). Unlike Firebase, it has a fixed capacity.
Cloud Functions on the other hand, scaled down to 0 when there are no request, and scales up to meet demand as needed. Whether that is needed for your use-case, only you can determine.
I'm building an app which uses user contributed content.
The contribution by each user should be available to all others in real time.
I was looking into firebase Realtime database for this.
However, when a user contributes content, there are quite heavy validations and calculations (read server side) to be done on the data before making it available to others.
Is it possible to have a server side validation in firebase ? Or should I look for alternatives ?
Initially, Firebase did not have a feature to implement server-side processing/calculations. All your processing had to be done on the client side.
Now, they've recently introduced a new feature called Cloud Functions For Firebase. Its a really useful new addition where you can write server-side code without the hassles of managing servers or instances. Read up more about it from the above link.
Also, this Youtube playlist by Jen Person is a great start. And, you can find examples similar to your use case here.