I am trying to create an social app using flutter, in that app I have users and their posts, so in firestore I am creating 2 collections,
Users
Posts
So, in users it will have user data like email, display picture, bio etc. For identification I'm creating a key in posts which will have a reference to the user to whom the post belongs, like so,
Now, I while I ask a particular post I also want some of the user details like UserName and display picture so that I can show it in the Post in the UI like,
So, I want to use StreamBuilder as it will reflect any changes made, but I can't get user details if I'm using StreamBuilder.
How can I achieve this?
According to your data model, you need to make two queries, the first one to get the user ID and others to get the posts corresponding to the user.
final StreamController<List<Post>> _postsController =
StreamController<List<Post>>.broadcast();
Stream listenToPostsRealTime() {
// Register the handler for when the posts data changes
_postsCollectionReference.snapshots().listen((postsSnapshot) {
if (postsSnapshot.documents.isNotEmpty) {
var posts = postsSnapshot.documents
.map((snapshot) => Post.fromMap(snapshot.data, snapshot.documentID))
.where((mappedItem) => mappedItem.userId == FirebaseAuth.instance.currentUser().uid)
.toList();
// Add the posts onto the controller
_postsController.add(posts);
}
});
return _postsController.stream;
}
I would like to recommend you to visit this video: "How to structure your data" and this post, remember that you will be charged for every read to the database, so if you're going to gather information from the Users document every time you retrieve a Post, it would be a better idea to have duplicated information from the Users document inside your Posts document, this way you only have to query your Posts document. Remember that denormalizing data (like replicating it) on the NoSQL world is a common behaviour.
Your structure could be something similar to:
Posts:
createdAt: "June 25, 2020"
description: "New food"
email: "test#email.com"
image: "https://example.com/assets/image.jpg"
likes:
[updatedAt: "June 25,2020", user: likerUser]
user:
[ name: posterUser,
profile_picture: "https://example.com/assets/posterPicture.jpg",
profile_url: "https://example.com/profiles/userPoster"]
In order to keep your data synced between both documents I recommend you to check this answer
Related
Summary
How could I model my database in Firebase to keep, for example, reviews in a specific page updated with the users info, this is, if a user changes it's avatar or name, the reviews should also display the updated data of the user.
I've used MongoDB most of the time, with Mongoose, and I am now working on a mobile app with Firebase. In Mongo I would just store a ref to the user in the review, and populate the field to retrieve the data I wanted from the document. Is there something like this in Firebase, and is it even a good or acceptable practice?
Quick Questions
Is there something like ".populate()" in Firebase?
Should I model the documents as much as possible to have the data that will be used in the view, and avoid "joins"?
Example
We have a users collection, and a store collection with reviews in it.
As far as I've read, you should minimize the doc reads, and so we should model our data with the specific values we need for the view were they will be used, so that we only need to do one query.
For the sake of simplification, let's say:
User has a name, email, avatar
users: {
user_id_1: {
email: "user1#gmail.com",
name: "John Doe",
avatar: "some_firestore_url"
}
}
Should the store collection:
Have nested collection of reviews like this
stores: {
store_id_1: {
name: "Dat Cool Store!",
reviews: {
user_id_1: {
name: "John Doe",
avatar: "some_firestore_url",
text: "Great store love it!",
timestamp: "May 07, 2020 at 03:30"
}
}
}
}
The problem I see with this, is that unless we use a function that updates every field in every document with the new values there is no other way to update the data in name and avatar.
Have the user_id in a field and query for the user information after:
stores: {
store_id_1: {
name: "Dat Cool Store!",
reviews: {
review_id_1: {
user: "user_id_1",
text: "Great store love it!",
timestamp: "May 07, 2020 at 03:30"
}
}
}
}
This is the mimicking the way I would do in MongoDB.
Sorry if some of it sounds confusing or I didn't explain myself the best way, but it's 4 o'clock in the morning here and I'm just trying to get it right :)
How could I model my database in Firebase to keep, for example, reviews in a specific page updated with the user's info, this is, if a user changes its avatar or name, the reviews should also display the updated data of the user.
Without knowing the queries you intend to perform, it's hard to provide a viable schema. We are usually structuring a Firestore database according to the queries that we want to perform.
In Mongo I would just store a ref to the user in the review, and populate the field to retrieve the data I wanted from the document. Is there something like this in Firebase, and is it even a good or acceptable practice?
Yes, there is. According to the official documentation regarding Firestore supported data-types, a DocumentReference is one of them, meaning that you can store only a path to a document and not the entire document. In the NoSQL world, it's quite common to duplicate data, so to have the same data in more than one place. Again, without knowing the use-case of your app it's hard to say whether using normalization it's better than holding only a reference. For a better understanding, I recommend you read my answer from the following post:
What is denormalization in Firebase Cloud Firestore?
And to answer your questions:
Is there something like ".populate()" in Firebase?
If you only store a DocumentReference, it doesn't mean that the data of the document that the reference is pointing to will be auto-populated. No, you first need to get the reference from the document, and right after that, based on that reference, you have to perform another database call, to actually get the data from the referenced document.
Should I model the documents as much as possible to have the data that will be used in the view, and avoid "joins"?
Yes, you should only store the data that you actually need to be displayed in your views. Regarding a JOIN clause, there isn't something like this supported in Firestore. A query can only get documents in a single collection at a time. If you want to get, for example, data from two collections, you'll have at least two queries to perform.
Another solution would be to add a third collection with data already merged from both collections so you can perform a single query. This is already explained in the link above.
Some other information that might be useful is explained in my answer from the following post:
Efficiency of searching using whereArrayContains
Where you can find the best practice to save data into a document, collection, or subcollection.
For me, the way I would go ahead with structuring my json collection also depends on the size of data, I am trying to store in the collection.
Let's say the number of users if small and I only want to support a thousand users. So in that case, I can go with this structure.
{
"store_id_1": {
"name": "Dat Cool Store!",
"reviews": [
{
"user_id_1": {
"name": "John Doe",
"avatar": "some_firestore_url"
},
"text": "Great store love it!",
"timestamp": "May 07, 2020 at 03:30"
},
{
"user_id_2": {
"name": "John Doe 2",
"avatar": "some_firestore_url 2"
},
"text": "Great store love it! TWO",
"timestamp": "May 27, 2020 at 03:30"
}
]
}
}
So now, you can have all the user info embedded in the stores collection. This will reduce your reads too.
But in case you want to scale it, then, I would suggest only store the users metadata and then make another read from users collection.
Hope this helps!
I'm toying with several ideas for using Firestore for a social media feed. So far, the ideas I've had haven't panned out, so for this one I'm hoping to get the community's feedback.
The idea is to allow users to post information, or to record their activity, and to any user following/subscribed to that information, display it. The posts information would be in a root collection called posts.
The approaches, as far as I can tell, require roughly the same number of reads and writes.
One idea is to have within the users/{userId} have a field called posts which is an array of documentIds that I'm interested in pulling for the user. This would allow me to pull directly from posts and get the most up-to-date version of the data.
Another approach seems more Firebasey which is to store documents within users/{userId}/feeds that are copies of the posts themselves. I can use the same postID as the data in posts. Presumably, if I need to update the data for any review, I can use a group collection query to get all collections called feeds, where the docID is equal (or just create a field to do a proper "where", "==", docId).
Third approach is all about updating the list of people who should view the posts. This seems better as long as the list of posts is shorter than the lists of followers. Instead of maintaining all posts on every follower, you're maintaining all followers on each post. For every new follower, you need to update all posts.
This list would not be a user's own posts. Instead it would be a list of all the posts to show that user.
Three challengers:
users/{userId} with field called feed - an array of doc Ids that point to the global posts. Get that feed, get all docs by ID. Every array would need to be updated for every single follower each time a user has activity.
users (coll)
-> uid (doc)
-> uid.feed: postId1, postId2, postId3, ...] (field)
posts (coll)
-> postId (doc)
Query (pseudo):
doc(users/{uid}).get(doc)
feed = doc.feed
for postId in feed:
doc(posts/{postId}).get(doc)
users/{userId}/feed which has a copy of all posts that you would want this user to see. Every activity/post would need to be added to every relevant feed list.
users (coll)
-> uid (doc)
-> feed: (coll)
-> postId1 (doc)
-> postId2
-> postId3
posts (coll)
-> postId (doc)
Query (pseudo):
collection(users/{uid}/feed).get(docs)
for post in docs:
doc(posts/{post}).get(doc)
users/{userId}/feed which has a copy of all posts that you would want this user to see. Every activity/post would need to be added to every relevant feed list.
users (coll)
-> uid (doc)
posts (coll)
-> postId (doc)
-> postId.followers_array[followerId, followerId2, ...] (field)
Query (pseudo):
collection(posts).where(followers, 'array_contains', uid).get(docs)
Reads/Writes
1. Updating the Data
For the author user of every activity, find all users following that
user. Currently, the users are stored as documents in a collection, so this is followerNumber document reads. For each of the users, update their array by prepending the postId this would be followerNumber document writes.
1. Displaying the Data/Feed
For each fetch of the feed: get array from user document (1 doc read). For each postId, call, posts/{postId}
This would be numberOfPostsCalled document reads.
2. Updating the Data
For the author user of every activity, find all users following that
user. Currently, the users are stored as documents in a collection, so this is followerNumber document reads. For each of the users, add a new document with ID postId to users/{userId}/feed this would be followerNumber document writes.
2. Displaying the Data/Feed
For each fetch of the feed: get a certain number of posts from users/{userId}/feed
This would be numberOfPostsCalled document reads.
This second approach requires me to keep all of the documents up to date in the event of an edit. So despite this approach seeming more firebase-esque, the approach of holding a postId and fetching that directly seems slightly more logical.
3. Updating the Data
For every new follower, each post authored by the person being followed needs to be updated. The new follower is appended to an array called followers.
3. Displaying the Data
For each fetch of the feed: get a certain number of posts from posts where uid == viewerUid
Nice, when I talk about what is more optimal I really need a point or a quality attribute to compare, I' will assume you care about speed (not necessary performance) and costs.
This is how I would solve the problem, it involves several collections but my goal is 1 query only.
user (col)
{
"abc": {},
"qwe": {}
}
posts (col)
{
"123": {},
"456": {}
}
users_posts (col)
{
"abc": {
"posts_ids": ["123"]
}
}
So far so good, the problem is, I need to do several queries to get all the posts information... This is where cloud functions get into the game. You can create a 4th collection where you can pre-calculate your feed
users_dashboard
{
"abc": {
posts: [
{
id: "123", /.../
}, {
id: "456", /.../
}
]
}
}
The cloud function would look like this:
/* on your front end you can manage the add or delete ids from user posts */
export const calculateDashboard = functions.firestore.document(`users_posts/{doc}).onWrite(async(change, _context) {
const firestore = admin.firestore()
const dashboardRef = firestore.collection(`users_dashboard`)
const postRef = firestore.collection(`posts`)
const user = change.after.data()
const payload = []
for (const postId of user.posts_ids) {
const data = await postRef.doc(postId).get().then((doc) => doc.exists ? doc.data() : null)
payload.push(data)
}
// Maybe you want to exponse only certain props... you can do that here
return dashboardRef.doc(user.id).set(payload)
})
The doc max size is 1 MiB (1,048,576 bytes) that is plenty of data you can store in, so you can have like a lot of posts here. Let's talk about costs; I used to think firestore was more like to have several small docs but I've found in practice it works equally well with big size into a big amount of docs.
Now on your dashboard you only need query:
const dashboard = firestore.collection(`users_dashboard`).doc(userID).get()
This a very opinionated way to solve this problem. You could avoid using the users_posts, but maybe you dont want to trigger this process for other than posts related changes.
It looks like your second approach is best in this situation.. I don't really understand what #andresmijares was trying to do and he mentioned something like storing posts in a document which is not a good approach, imagine if you have more than 20K posts (which what I think a document can hold) then the document won't be able to store any more data.. a better approach is to store posts as a document inside a Collection (just like in your 2nd option).. So let's recall here what's the best approach.
1)_ You share a post in the (posts "Collection") and in users you're following's (Feed "Collection").. maybe this can be done with cloud function and let's not forget to aggregate (with cloud functions also) the number of posts that needs to appear in the user's profile.
2)_ You follow a user and get all of their posts from the (posts "Collection") into your (Feed "Collection") this way you get to see all of their posts on your feed.
with this approach, there will be a lot of writes once but the read will be fast.. and if your app is about reading more and writing less then there's nothing to worry about unless i'm wrong.
so I want to create a sort of social media application and use firestore as main database.
the goal is to create "facebook" news feed.
each user will have a list of friends and each user will be able to create posts.
each post can be modified to be visible to all the users of the application or just the user friends. so each user will be able to post posts to all his friends and to post posts to everyone in the application.
also, users can "save" posts they liked in the newsfeed.(LikedPosts subcollection)
USERS (collection)
DocumentID - userId gathered from Authentication uid
firstName: String
lastName: String
username: String
birthdate: Timestamp
accountCreationDate: Timestamp
FRIENDS (subcollection)
DocumentID - userId of the friend
username: String
LikedPosts (subcollection)
authorUserId: String
authorUsername: String
text: String
imagePath: String
POSTS (collection)
DocumentID - postId randomly generated
authorUserId: String
authorUsername: String
text: String
imagePath: String
likesCount: Number
forFriendsOnly:yes
LIKES (subcollection)
DocumentID - userID of the liker
username: String
now in the newsfeed for a user - How can I query for all the visible post (forFriendsOnly:no) and also to all the posts for friend only, that the current user is in the author friends subcollection.
also, if the user change his name, how can I change his name accordingly for all his previous posts, and all the save posts related to the user?(located in user likedpost subcollection)
I guess you were asking 2 questions.
First, Firestore recommends data duplication instead of joining query across collections. The way you designed the post and user has to rely on query concept in SQL.
It is still possible to achieve that, if you don't mind to have all the author's friend id as an array inside of that post document. Meanwhile, you have to sync author's friend array through trigger function when author add/delete friends.
I wouldn't really recommend this solution, because as a social platform, user's friends might be changing constantly, then you have to keep on updating all his post's friend array.
There is another solution, which is add one more subcollection under user as his visible "feeds". Then whenever an author creates a post, trigger function will write this post's summary to all his friends' visible "feeds" collection.
However, both above solutions are not perfect if you are concerned about accuracy, realtime, cost, etc. I guess that is the drawback we have to bear with. If you have to achieve the same thing as SQL, I guess the only option is using other solutions for query part, such as elastic search, mysql, neo4j, etc. PS: You can still wrap it with cloud functions.
Regards to your 2nd question, one way is not duplicate username if you think your user would change their name frequently. And always query username by user id from user collection. The other way is using trigger function to update the duplicated username when user change their names. I would recommend the second way, since user wouldn't change their names frequently.
Not necessarily related to your original question, but your LikedPosts subcollection likely needs a restructuring. If you can ensure uniqueness on your postId, then it should probably be something like:
LikedPosts (subcollection)
postId: Unique identifier for liked post
authorUserId: String
authorUsername: String
text: String
imagePath: String
The current structure only allows for one liked post, so you'll need to change it to be one document per liked post, or a document containing a list of all of the liked post ids.
Using the Firebase database I need to decide how to organize the user / post(e.g a tweet) relation. Two common reading tasks include:-
Showing an overview of a user's posts
Filtering and selecting specific posts based on their content
I currently use this:
posts
randproject1234jdfs
user_uid: randomUserName1234,
content: "example"
users
randomUserName1234
posts
project_id: randproject1234jdfs
nickname: "Mr Example"
This stores the same information twice:
In posts/randstring1234jdfs/user_uid , the value points to the user.
In users/1234/posts the project_id point to the post.
Would removing one of these lead to significantly slower data-reading (having to loop through the entire folder to see if the user_uid/project_id matches the given post/user)?
Or would it be better to organize the data in a different way altogether (e.g removing the user/posts split)?
You want to do the following:
Showing an overview of a user's posts
Filtering and selecting specific posts based on their content
You can do this then:
posts
randomid
user_uid: randomUserName1234,
content: "example"
posttitle: "English"
randomid2
user_uid: randomUserName1235,
content: "examples"
posttitle: "Math"
users
randomUserName1234 <-------------------- userid
email: email#gmail.com
nickname: "Mr Example"
randomUserName1235<--------------------anotheruserid
email: email#gmail.com
nickname: "Mr Awesome"
Since you want to show the user's post, then you can just query the node posts something like this: orderByChild(user_uid).equalTo(randomUserName1234) using that you can retrieve the content which are the posts I guess.
To retrieve posts of any user depending on content using the above u can do this:
orderByChild("posttitle").equalTo("English") and then retrieve the node content which I'am assuming they are the posts.
Remember denormalization is normal and it is important or you will have a hard time doing the queries. https://firebase.googleblog.com/2013/04/denormalizing-your-data-is-normal.html
The user node let it have the list of users in this app for example.
The posts node will be like above and it will be connected to the user by having this user_uid: randomUserName1235 as seen above.
I read a lot about nosql databases lately. I get that rule of thumb is to structure the data based on our view (of course, depends on the use case).
Now, let's say that we have a social app and the user has a profile but he also creates posts and we have to store them in the database.
So, I see some developers choose to do that like so:
Posts
-----UserID
-----------PostID
-----------------username: John
-----------------profileImage: https://...
-----------------posted_photo: https://...
This totally fits the structure base on the view. We'd go into posts and our userID and we could get all the data that our view needs. Now my question is, what happens when the user has made 100K posts and he decides to change his profile photo for example. All of his posts so far contain his old photo, so now, we have to write a method that would cycle through 100K of posts (or all of his posts in general) and update his photo. In 2 hours, he decides that "Nah, I don't like this photo, I'd change it back" and we have to make another 100K queries.
How is that (denormalized data) ok? Sure, its easier, its flat but then we have to make ridiculous amounts of queries to change a single profile photo. What's the way to handle this really?
I've done this storing user's data in a place and setting just the userID as post attribute.
posts:
userID:
postID:
userID: 'user1',
attachedImageURL: 'http:..',
message: 'hey',
reblogID: 'post4',
type: 'audio|poll|quote'
users:
user1:
name: 'john',
profileImage: 'http..'
It requires one more query to Firebase to retrieve user's profile data but it's a good way to solve this. It really depends on how you want to use those data.