I'm having a time trying to wrap my head around what I thought was a simple concept.
I have an app where I sign up a user, allow that user to set a 'photoURL' to their 'user' information in the Firebase Auth system. This works. When the user creates a post in my app, I want to display the title, image and
'photoURL' of the creator.
Currently, I save the post:
-Post {
-id
-title
-image
-photoURL <- from current logged in user }
I also allow users to visit the posters page via routing /poster/'displayName'
So later, when a user updates their profile information like displayName or photoURL, do I need to go find all posts, comments, messages, replies and any other place that this user has a record and update the photoURL?
What I thought I would be able to do is say: (pseudo code)
get all posts =>
foreach(post)
post = {
title: post.title.val()
image: post.image.val()
avatar: firebase.database().ref().child('users' + post.key)
}
Everything I read says I need to store that photoURL in my own 'Users' table. If I do that, then none of the posts get updated unless I write a server call to do that every time there is a change. Problem is, if I have 100K users, and 10% of them change their photoURL, I then have to change it in posts, comments, replies and messages per user. If the average user has 100 posts, 4000 comments, 6000 replies, we're looking at about 10K places * 10K users that have to be updated and if the average server call is 137ms, then my costs are around $175 (costs)
The other option is to pull information from two tables and create a new object every time. This would lead to about double the server calls and time thus doubling my costs.
Is this the best approach for this? I thought this would be a lot easier to just get the user photo and display name.
Sorry for the epic long post but I'm trying to learn. Thanks all!
What you're describing is a typical issue when working with noSQL databases. On the one hand, data duplication makes your app and its queries run faster. On the other hand, if you want to change any that duplicated data, it can be problematic to find and replace all occurrences.
There's no "best" way to determine what to do. It's completely up to your particular case. It sounds like, if you have extreme amounts of data duplication that could be costly to update, it would be better to simply query the user record every time rather than to do the updates. But again, it's ultimately up to you.
Related
I have been using Firestore for a very long time. I am building an app now where scalability and keeping low costs is important. (I am using flutter)
My app has users, which have user profiles, also they can add friends and talk to them (like instagram or facebook).
I have a problem building this friends system.
My model for this friends system currently looks like this:
Users collection. Each document id = user id from auth, those docs contain data like name, username, profile picture, etc.
Friends collection. Each document id = user id from auth. For each user, those docs contain a field called: friends, which is an array with each of his friends user ids.
The model looks like:
Friends collection:
- uid:
- friends_list: [friend_uid1, friend_uid2, ...]
This is how my "backend" looks.
Now I want to show my user a list of his friends. How do I do that?
I want a list that looks like instagram, with a nice UI showing each of my user friend profile pic, name, last message, etc.
I can not find a straight forward way to do this with Firestore and queries.
Let's say I do it like this:
Get all my friends user ids in an array.
Get all their user documents using .get() for each document.
This is not doable in firestore cause it would eliminate all the querying power I have (such being able to query only for users with name "x"), I would have to fetch all users and do the query on my front-end (or in a cloud function, same thing, not scalable).
If I do this like:
Get all document using a query for all users in the Friends collection, where friends_list contains my user id.
Save from those documents only the documentID and fetch all the friends user data manually.
This comes with another problem. In Firestore there is no way of fetching a document without fetching all of its fields, so the first query which I use to get the ids only of my friends would actually give me their id + their friend list instead (cause when I query, it also gets the document id + the data), which is not good.
If I do it like:
When you add a friend, instead of just saving its uid, save its uid + data.
Now I can easily show my user his friends list nicely and do some querying on front-end.
The problem here is that now if one of my friends updates his profile photo, I need to update it in every document of all of his friends, which is very write expensive for just a little profile update.
There is also the problem of watching for more data, maybe I have another collection with Chats, and I want to show the last message of my chat with a friend, now I have to fetch the chat rooms too, which is more hard to query data that comes with all the problems that I mentioned before.
In conclusion: I don't see a good scalable way to do this kind of system within Firestore. It seems a simple system which any basic app should have, but I do not see how I can do it in a way that does not make lots of reads or read more data (or sensitive data) than it should.
What kind of model would you do for a friends system like this?
You're decribing a quintessential drawback of NoSQL Databases.
A similar example is actually given in the Get to Know Cloud Firestore series.
Like others have commented, the answer really depends on your application. And this is the assessment you'll have to do. Like which of the options is cheaper depending on the use case of the app.
For example, if you go with your third option and store the friend's user data that you'll need to populate the list. This means you'll have to implement measures to keep the integrity of the copied data whenever the user updates their information.
You can then look at the usage of your app and determine how often users change their information vs how often you would need to retrieve full users if you don't copy the data to find the cheapest method for your application.
I am using FireStore for my Flutter application. Users can post comments under some articles. They can also reply to other comments. Comments have the following structure
where ancestorsId is a list containing all the parent comments id.
Comments can be deleted by the poster or an admin.
When a comment is deleted, all the children comments should be deleted as well.
How can I do that with safety and at the lowest cost ? I have thought to the following so far:
Case 1: Using a Go server and Custom Claims
I can set user role as a custom claim. Then, when a user clicks on delete comment button, it sends a request to the server with the comment ID and the user Token ID. I can check if the user is admin with the token ID. If it is not the case, the server asks the comment data and check if comment userId and token user Id match. If one of those two conditions is true, I can get all comment children with a where request on comments collection, and delete all involved comments with a batch.
Problems:
Custom claims use token, that live for 1 hour. It could create troubles if a crazy admin starts deleting everything, because his admin token can be valid for up to 1 hour. But if I use a server, I think I can manage this.
I need to get comments before deleting them. It involves two operations and then the price is actually twice the price of deleting comments operations.
Case 2: Using FireStore rules
I could stick with a client only, and use FireStore rules instead of using a server. It means that I could no longer use custom claims, and I would have to store users role in a field in my users collection. I could use a delete rule like:
match /comments/{comment}{
allow delete: if request.auth != null && (request.auth.uid == request.resource.data.userId || isAdmin(request));
}
where isAdmin is a function that checks if the user is an admin.
Problems:
isAdmin will need to read a data from another document and thus cost money
This solution doesn't delete children comments. And the rule doesn't allow a user to request another user's comment deletion, unless he is admin.
What could be a solution to solve my issue at low cost without putting safety aside ?
It seems to me that you really only have one solution that works, as the second approach leaves orphaned documents in the database.
But if you do consider the second approach valid for your app, you're trading the cost of reading-and-then-deleting some documents for the cost of leaving them in the database. While the cost of keeping a document in the database is low, you'll end up paying it every month. Since the number of orphaned documents will keep growing, the storage cost for them will keep growing too. So while deleting them now may seem more expensive, it's just a one time cost.
If you're worried about the cost of running Cloud Functions, keep in mind there's a pretty decent free tier for those. Even if that tier is not enough to run your code in production, it should at least be enough to give you a feeling for what the cost is gonna be.
There are several articles (firestore and firebase realtime database) explaining how to build a user presence system but I cannot find a resource for a friend presence system.
A simple user presence system is not perfect for some applications such as chat apps where there are millions of users and each user wants to listen to only his/her friends. I've found similar questions:
exact same question on stackoverflow
exact same issue on github
Two ok solutions with a realtime database are: (solutions are from the above stackoverflow post)
Use many listeners (one for each friend) with a collection of users. Possibly have a cap on the number of friends to keep track of.
Each user has friends collections and whenever a user's status changes, his/her status changes wherever he/she shows up in some user's friends collection as well.
Is there a better way to do? What kind of databases do chat apps like discord, whatsapp and etc. use to build their friends presence system?
I came to two approaches that might be worth looking into. Note, that I have not tested how it will scale longer term as I just pushed to prod. First step, write a users presence on their user document (will need firebase, cloud functions, and cloud firestore per https://firebase.google.com/docs/firestore/solutions/presence).
Then take either approach:
Create an array field on your user documents (users> {userID}) called friends. Every time you add a friend add your id to this array, and vice versa. Then, on the client run a function like:
db.collection(users).where("friends", "array-contains", clientUserId).onSnapshot(...)
In doing so, all documents with friends field that contains the clientUserId will be listened to for real-time updates. For some reason, my team didn't approve of this design but it works. If anyone can share their opinion as to why I'd appreciate it
Create a friend sub-collection like so: users>{userID}>friends
. When you add a friend, add a document to your friend sub-collection with the id equal to your friends userID. When a user logs on, run a get query for all documents in this collection. Get the doc IDs and store into an array (call it friendIDs). Now for the tricky part. It'd be ideal if you can read use the in operator for unlimited comparison values because you can just run an onSnapshot as so:
this.unSubscribeFriends = db.collection(users).where(firebase.firestore.FieldPath.documentId(), "in", friendIDs).onSnapshot((querySnapshot) => {get presence data}). Since this onSnapshot is attached to this.unSubscribeFriends you just need to call this once to detach the listener:
componentWillUnmount() {
this.unSubscribeFriends && this.unSubscribeFriends()
}
Because a given users friends can definetely increase into the hundreds I had to create a new array called chunkedFriendsArray consisting of a chunked version of friendIDs (chunked as in every 10 string IDs I splice into a new array to bypass the in operator 10 comparison values limit). Thus, I had to map chunkedFriendsArray and set an onSnapshot like the one above for every array of a max length of 10 inside chunkedFriendsArray. The problem with this is that the all the listeners are attached to the same const (or this.unSubscribeFriends in my case). I have to call this.unSubscribeFriends as many times as chunkedArrays exist in chunkedFriendsArray:
componentWillUnmount() {
this.state.chunkedFriendsArray.forEach((doc) => {
this.unSubscribeFriends && this.unSubscribeFriends()
})
}
It feels weird having many listeners attached to the same const (method this.unSubscribeFriends) and calling the same exact one to stop listening to them. I'm sure this will lead to bugs in my production code.
There are other decentralize approaches but the two I listed are my best attempts at avoiding having a bunch of decentralized presence data.
So I have a Firestore database where i have all my posts in, I push them on page load into the store so I can have a fast navigation and only need to fetch once.
What I wanna do now is to use a persisted state so i dont need to refetch it if the user opens a new window or F5(reloads) the page.
The problem is im not sure how to check if new Posts are in the Firestore without querying all posts and I havent found any methods to do it in a healthy and Read efficient way.
There's no super easy way around it - at the end you have some data, and the server has another, you need to check for differences.
If you're only trying to figure out if there are new posts on the backend, which are not loaded on your frontend, then just get the date on your last post, and then ask Firebase for all posts after this date :)
Of course if you don't have posts, ask for everything.
Keep in mind you need to manually check if posts are deleted ;)
Realtime updates with the onSnapshot method can be used to keep local data in sync with the server. If you initially load it into vuex then subsequent changes on server side will be reflected automatically.
https://firebase.google.com/docs/firestore/query-data/listen
To share one set of data across tabs/windows you could look at something like this
https://github.com/xanf/vuex-shared-mutations
I am trying to wrap my head around how to structure my data for firebase realtime database. I read the docs and some other questions on SO finding following advices:
data should be as flat as possible
be expensive on writes to have cheap reads
avoid nesting data
duplication of data may be okay
Keeping this in mind let me describe my specific use case. Frist there is user with following attributes:
Firstname
Lastname
Profile picture (big)
Profile picture (small)
A user may create a story, that consist of following attributes:
User
Text
Timestamp
The visual representation of a story may look like this:
My question is, how would you associate user information (firstname, lastname, small profile picture) with a story?
What I thought about:
put a user_id in the story that contains the foreign id to the specific user. To load the story we would have to make two request to the database, one to get the story and one for the user.
{
user_id : 'XYZ',
text: 'foobar',
timestamp: ...
}
put firstname, lastname and small profile picture in the story. Only one request would be necessary to display the story. But we would have to update each user's story, when e.g. the profile picture changes.
{
user_id : 'XYZ',
firstname: 'sandra',
lastname: 'adams',
smallProfilePicutre: '...',
text: 'foobar',
timestamp: ...
}
So when there are few stories created and most of the time there are just reads, approach 1. would be expensive, because we pay for two reads to display a story. Approach 2. would be more cost efficient.
I would like to here your thoughts and ideas on this.
I'm with Jay here: you pretty much got all of it in your question already. Great summary of the practices we recommend when using Firebase Database.
Your questions boils down to: should I duplicate my user profile information into each story? Unfortunately there's no single answer for that.
Most developers I see will keep the profile information separate and just keep the user UID in the post as a unmanaged foreign key. This has the advantage of needing to update the user profile in only one place when it changes. The performance to read a single story is not too bad: the two reads are relatively fast, since they go over the same connection. When you're showing a list of stories, it is unexpectedly fast since Firebase pipelines the requests over its single connection.
But one of the first bigger implementation I helped with actually duplicated the user data over the stories. As you said: reading a story or list of stories is as fast as it can be in that case. When asked how they dealt with keeping the user information up to date in the stories (see strategies here), they admitted they didn't. In fact: they argued many good reasons why they needed the historical user information for each story.
In the end, it all depends on your use-case. You'll need to answer questions such as:
Do you need the historical information for each user?
Is it crucial that you show the up-to-date information for a user in older posts?
Can you come up with a good caching strategy for the user profiles in your client-side code?