Hey so with my current feed database design, I am using Redis for the cache for super-fast reads, which are routed through my Google Cloud Functions. The Redis database handles all post data and timeline updates, which is great and all, but I forgot one of the most considerable caveats to this. Firebase Firestore only permits one document write per second, meaning that if I have a document that stores the post data (post_id, user_id, content, like_count), the like_count would be impossible to track with the possibility for many likes per second. Does anyone have any solutions to this?
You can shard your counter among multiple documents and query them in aggregate as needed.
You can also try Cloud Tasks queue to smooth out the write frequency. It will add considerable complexity to the system, but is really the only genericized way in GCP to manage the rate of some work. This might not work out the way you need, however.
If you use Cloud Tasks, your task will need to be configured with a rate limit, and it will have to deliver the document data to write to yet another function or other HTTP endpoint that will perform the write.
Related
i've been searching for what is the concurrent users limit for the cloud firestore spark plan but couldn't find it.
https://firebase.google.com/docs/firestore/quotas
It did said 1.000.000 concurrent users limit, but did not mention whether it is for the spark plan or the blaze plan. I've also tried searching answer elswhere, but did not find it answered specifically (with a source).
Help would be appreciated, thank you.
Per the Cloud Firestore pricing information (which Firebase uses):
When you use Firestore, you are charged for the following:
The number of documents you read, write, and delete.
The amount of storage that your database uses, including overhead for metadata and indexes.
The amount of network bandwidth that you use.
There is also no mention of any connection limits on Firebase's pricing page or the quotas documentation that you linked.
Unlike the Realtime Database, Cloud Firestore does not charge on a per-connection basis.
This video series also covers the ins and outs of Firebase products and is well worth sitting through.
Think of Cloud Firestore like a folder on your computer, which can contain thousands of little text files, similar to how documents in Cloud Firestore are stored. Users can update them with little chance of collision and grabbing a single document file would only require feeding 1s and 0s back to the requestor. This is why you are charged for network bandwidth rather than by individual connection.
In comparison, the RTDB was similar to one large JSON text file, with many people all trying to update it at once. Because parsing this text file on the server side was required to read and write data from it, it required compute resources to be able to do so. For this reason (among others), the number of connections the RTDB manager processes handled on behalf of spark plans were rate-limited to prevent abuse.
I know that I could implement a counter in my application but using an api would still be a cleaner solution - if one exists?
Basically, Firestore has Spark free tier limits (think 50,000 reads/day) that I don't want to exceed. So whenever my app was going to do firestore reads, I would like a way to simply ask firestore whether I'm over a certain number.
I'm also reading that Google intentionally got rid of Firebase spending limits.. which seems really sketchy... Impossible to set the Cloud Firebase daily spending limit
There is no such API as part of Firebase. The ways to monitor usage are documented here, but none of them is an API.
You might be able to get some data through the Cloud Monitoring API. But this API isn't made for client-side access though, so you'll have to wrap it yourself.
A final alternative would be to look at a service like https://firerun.io/ who automate a lot of this.
I am looking for clarification on how the actions triggered by cloud functions affect the cost of hosting an app with Firebase.
My situation:
I have a cloud function that is triggered when a post is made...that cloud function writes that post to all of the appropriate uid's on the /feed node mentioned below.
I am using Firestore to host a feed/follow system. It is setup with each user having their own feed at the following path
/feed/{uid}/posts
My question is, if a user has 1,000,000 followers...does that translate to 1,000,000 writes in the eyes of Firestore?
Does that mean that single post will cost $1.80 to distribute? Based on the cost of $0.18/100k writes.
I am just trying to see how this will scale with thousands of users posting dozens of posts...those distribution costs seem very expensive at scale for this situation.
EDIT
It looks like it does cost for every read/write in a cloud function base on the following video:
https://www.youtube.com/watch?time_continue=49&v=6NegFl9p_sE&feature=emb_logo
It depends on how you are writing to Firestore, but bottom line is that yes, you get billed for each read/write operation performed to Firestore as mentioned over at their documentation.
Additionally, you get billed per Cloud Function invocation as mentioned here, so you would need to keep this in mind if you want to keep your expenses low.
Hope you find this useful.
I'm using Firestore I have these questions, regarding how to user behavior will have an impact on app costs:
what's is more cost-effective:
To use a realtime form that saves in the database while the user typing in a web form
To save all the fields in the form at once using a firebase function
questions:
is it overkill to proxy with cloud functions? (just to avoid costs)
when the user types (realtime updates) is it considered as a new write to the database every time?
what's is more cost-effective:
To use a realtime form that saves in the database while the user typing in a web form
This is going to cost you a write for each time the form is save in realtime.
To save all the fields in the form at once using a firebase function
This is going to cost you a single write.
The difference in cost between the two should be obvious - multiple writes vs. a single write.
questions:
is it overkill to proxy with cloud functions? (just to avoid costs)
If you're proxying for no other reason than to save costs, it's overkill. The function invocation will cost you money, in addition to the document write, which will cost the same no matter where it originates.
when the user types (realtime updates) is it considered as a new write to the database every time?
As I said before, yes, it is.
The only real reason to send form submissions through a function is the ability to do deep, secure checking for validity of the form fields. Client side checks are not secure. You could use security rules to perform checks, but those are limited. If you need to make sure the form fields have stricly checked values, a Cloud Function might be your best choice. But it's not possible to tell given the information in your question.
There's no particular reason you need to use a function to save all at the same time -- at whatever point you would call the function, instead call a single update to the database. Using a function here is going to be strictly more expensive (assuming it provides no functionality other than the database write), since you incur the cost of the write and you incur the cost of a function execution.
Of course, its possible you have some other reason to call a cloud function to do the write beyond a simple proxy -- such as to ensure constraints that cannot be enforced by security rules alone. In that case, the cost may be worth the added functionality.
As for is it better to batch or write in real time, it will certainly be cheaper to write all at once, as you are charged for every document write to Firestore. More specifically, each set or update is charged as a single write. So, its definitely going to be less expensive to only write the document once for many fields, as opposed to write it in real time (or per field) as the user is entering data.
My users can create documents (let's say tasks) in a subcollection with a bunch of security rules checking for authentication, permissions and data validity. They can even select multiple tasks and copy them in the same collection.
Now, a regular user will likely create at most a hundred tasks at once, but what if someone with bad intentions manage to obtain my database credentials, authenticate and try to create a huge number of valid documents programmatically? This will result in Firestore scaling without problems and an unexpected surprise in my Firebase billing.
This is my first concern, but I'm also thinking about the possibility to limit a collection size for other reasons, and it would be at the same time a solution for the problem described.
I read about techniques to count documents in a collection described in the Firestore documentation, but I did not found a solution.
Keeping a counter on a doc field updated with a transaction in a cloud function would be inefficient in my case. Distributed counters increase the complexity of my data model a bit, and also I would not know how to properly read those counters in security rules for every task creation, and even if that would be an efficient solution.
Does anyone has suggestions?
I believe the way for a person to gain read/write access to your database would be to either to hack Google servers, in which case no one is safe and it doesn't really matter what you do, or to guess the exact name of your collections and documents.
As for the latter case, what I have done in my project is that for each collection and document I have used the name I wanted plus random 10-char Strings (including all kinds of chars and numbers. For example Users-x5NfaS1jCb) which kind of serve as independent, separate passwords every step of the way. This, at least, makes it difficult to guess the name of the collections and documents.
(Just like mentioned in the question) If using authentication does not cause any complications for you project, you can use it to further raise the security of your database by limiting access to users authenticating through your app only.
I guess (have never tried it) you can make use of Firebase Functions to limit the number of documents available in any given collection based on the criteria you want. This function will be invoked every time an event in created in the database.
If by "obtain my database credentials", you mean finding the username and password to your Firebase account, well it doesn't really matter what you do again. If they know what they are doing, they can take so many advantages that this particular issue will be the least of your problems.
All in all, if you ask me, your database is safe unless either someone guesses your collection and document names, or gains access to your Firebase account.
These are the only things I can think of for now. I'll try to update my answer later.