Firebase Firestore, query a users friend's posts - firebase

I am looking create a social-media feed using Firebase. My data is structured like this:
users: {
uid: {
... // details
}
}
friends: {
uid: {
friends: { // sub collection
fuid: {
... // details
}
}
}
}`
posts: {
postId: {
postedBy: uid
... // details
}
}
Now I am trying to get the posts from all friends of the user, limit it to the most recent 10 posts, and then create a scrolling directive that queries the next set of 10 posts so that the user doesn't have to query and load posts^N for friends^N on the page load. But I'm not really sure how to query firebase in an effective manner like this, for the user's friends and then their posts.
I have the scrolling directive working, taken from Jeff Delaney's Infinite Scrolling Lesson on AngularFirebase.com. But it only handles the posts (boats in the tutorial) collection as a whole, without selectively querying within that collection (to check if the user is a friend).
The only solution that I could think of was to query all of the user's friends posts, store that in an array, and then chunk load the results in the DOM based on the last batch of posts that were loaded. This just seems like it could be really inefficient in the long-haul if the user has 100's of friends, with 100's of posts each.

If I get it right, you are duplicating the post for each user in the user's friend list right? I don't think it is a good idea if your app escalates... At this time, the cost for 100k doc writes is $0,18, so:
Imagine that a user of your app have 1000 friends. When he posts anything, you are making 1000 writes in the database. imagine that you have 1000 active users like him. You have just made 1.000.000 writes now and paid $1.80.
Now even worse: you probably have on each post, a duplicated field for user displayName and a profileImageUrl. Imagine that this user has 500 posts in his history and have just changed his profile picture. You will have to update one of the fields for each post on each of his 1000 friend's feed right? You will be doing 1000 * 500 = 500.000 writes just for updating the profileImageUrl! and if the user didn't like the photo? he tries 3 new photos and now in 10 minutes you had made 2.000.000 writes in the database. This means you will be charged $3.60. It may not seems too much, but pay attention that we're talking about 1 single user in a single moment. 1000 users changing profile picture 4 times in the same day and you are paying $3,600.00.
Take a look at this article: https://proandroiddev.com/working-with-firestore-building-a-simple-database-model-79a5ce2692cb#7709

I ended up solving this issue by leveraging Firebase Functions. I have two collections, one is called Posts and the other is called Feeds. When a user adds a post, it gets added to the Posts collection. When this happens, it triggers a Firebase Function, which then grabs the posting user's UID.
Once it has the UID, it queries another collection called Friends/UID/Friends and grabs all of their friend's UID's.
Once it has the UID's, it creates a batch add (in case the user has more than 500 friends), and then adds the post to their friend's Feeds/UID/Posts collection.
The reason I chose this route, was a number of reasons.
Firebase does not allow you to query with array lists (the user's friends).
I did not want to filter out posts from non-friends.
I did not want to download excessive data to the user's device.
I had to paginate the results in order from newest to oldest.
By using the above solution, I am now able to query the Feeds/UID/Posts/ collection, in a way that returns the next 10 results every time, without performance or data issues. The only limitation I have not been able to get around completely is it takes a few seconds to add the post to the user's personally feed, as the Function needs time to spin up. But this can be mitigated by increasing the memory allocation for that particular function.
I also do the above listed for posts that are edited and or deleted.

I think i have a solution for Firestore Social Feed queries. Not sure if it works but here it is;
A Friends collection keeps the friends UUID'S list as an array in a document. Every document in this collection is for a user. So when the user logs in we first have the friends list with a cloud function with "one read" right? All friends id's are in one document. And we also put a lastchecked time stamp to this document. Everytime we get friends array we record the date.
Now a cloud function can check all users posts one by one. As i understand latest IN queries allow an array up to 10 UUID's. So if user has 100 friend query will end in ten rounds. Now we have sth to serve.
Instead of directly serving the posts we create a collection for every user. We will put all this collected data to document but we slice it to days. Let's pretend we already have older posts in this usersfeed collection (every day as a document). So we had a last time check on our friends document. We query now -> last checked date. This way we only fetched unseen posts and sliced them daily (if they belong to more days ofcourse)
So while this happens on cloud function we already served the previous feed document. And when collection has new document firestore already listens and adds right? If the user scrolls down we get the previous days document. So every document will have more then one posts data as map / array.
This saves many read counts i guess.

Related

how to wrap my head around a FireStore query

I am new in Flutter - Firestore
I am learning flutter with firebase and creating a sample dating app
I have a list of users that I get in a stream and display it using List view
Firestore.instance.collection('users').snapshots()
I have learnt to filter this like so
.where((user) => user.age < settings.agemax && user.age > settings.agemin))
and all this works.
I also have a subcollection called shortlist (list of users that current user has shortlisted) that I get using,
Firestore.instance.collection('users').document(uid).collection('shortlist').snapshots()
Now I am trying to redefine my first query GetUsers with filters based on following
How do I exclude shortlisted users that I am fetching in a stream from all users stream
Similarly would also need to filter out "matched users" and "Blocked / declined" users as well !
I believe my question is how do I query Users Collection and exclude records with uid's that contained in a Shortlist subcollection. I am planning to use the same logic for matches and blocked !? am I on the right track ?
also ... do I need to refetch all records when a users shortlists/matches/blocks someone, as the stream would change or is there a way to remove that one record from the listview without rebuilding, may be I should separate this question in two.
If I understand correctly you are looking for the (just introduced) not-in operator, so I recommend also checking out this blog post.
I expect that this operator hasn't landed in the Flutter libraries yet, as that may take some time. I recommend checking the upcoming releases to see when it lands, or checking/filing an issue on the repo.
Until then, there's no way to exclude results from a query, so you will have to exclude the items from the stream of results in your application code.

Firestore(Firebase) store big list

I have one question for the store of big list in firestore (using flutter).
I have a database like this:
--DataBase
------Post
---------Comment
And each post/comment must store the list of user who liked/add fav.
In a first time i have create two list in my collection Post/Comment
But I know that the solution is not good, because the size of a document is limited
So I thought about a second solution, which consists in adding a collection "userData" in the Post / Comment collections and for each user create a document containing the data. Like this :
But the second solution bothers me. Because when the user loads the Post list and the comment List, for each post/comment I will do an additional query to find if the user already likes/add fav.
So if I have 50 comments per post and if the user looks at 100 posts, it generates 10,000 requests (100 * 50 * 2), and that per user.
I am afraid that the price of these requests does not cover the gains of the application.
So, is the second solution the recommended solution? Is there another solution ?
Thanks
I would use subcollections for this and simply not display the list of comments (or just show the latest comment) when the user is going through the list of posts. Once the user interacts with a post then paginate or create an infinite scroll of the comments collection using query-cursors

How can I make sure to fetch only the x most recent posts in DB?

I am trying to fiigure out how exacly I should make sure that I only fetch the most recent x posts from the DB.
My current fetch methods work as follows:
GetPeopleIFollowAnd loop over each UID
For each UID fetch his posts
By doing this however I cant efficiently (quickly) fetch only the most recent x posts. How can I do this?
Some possible ideas I have:
Create a new Node which, every time that one of your followers makes a post, they will be added beneath your UID with a timeStamp. So When I fetch I loop through this FollowersWhoCreatedRecently for each UID and only do so for the first 10
Problems I see with the above solution is that if a user were to have a millions of people following them this would be horribly slow when it comes to updating every single one of those million
I have found this which seems like it may be of use. How could I use this?
My DB structure for posts is as follows
Post
UID
postID
Media
media
image: URL
Based in a follower structure like:
WhoFollowsMeNode/UID/uid: true
I would have to, every time a user posts, loop through this list where for each user I would add the post to there timeLine... That seems undoable, and yet that what it seems they are doing here.
On Realtime Database, you can make use of limitToLast() method, you can then pass how many elements you want to bring, this method will fetch the last information that has been added in a node, since the documents are ordered with a timestamp, if you fetch them with limitToLast(10) you can get the last 10 posts of a user.
If you are working with a List of posts, you can invert the list when showing it to the user, doing this, you will see the most recent data first and the old data below.

How do I efficiently find if one set of nodes has elements contained in another in Firebase?

I am building a social media database schema, in which I have users, followers, tags and posts. To conform to the firebase model I have flattened the structure as suggested in the firebase documentation as seen below. The issue that I am struggling with is when a user selects a tag and sees a bunch of posts from the tagPosts table all related by tag returned, I would then like to show the posts created by the current users followers first.
In SQL this would be done with an inline query checking the users followers, against the posts returned by a specific tag.
However in firebase I am not sure how do this without downloading all the posts contained under the tagID node in tagPosts and checking through each post's creator against the node of Followers for the current user userID. This operation could easily grow out of hand for 100s of posts amongst 100s of users. Ive tried modeling off of this answer, How do I check if a firebase database value exists? and this article From SQL to Firebase — How to structure the DB for a social network app. Am I poorly structuring the data how do I fix this thank you so much.
`
Users-
-userID1
-misc. userData
-userID2
-misc. userData
Followers-
-userID1
-userIDOfFollower1
-userIDOfFollower2
Following-
-userID1
-userIDOfFollower1
-userIDOfFollower2
Posts-
-postID1
-userIDFromCreator
-misc. PostData
Tags-
-tagID1
-misc. TagData
TagsUsers
-tagID1
-userID1
-userID2
TagsPosts
-tagID1
-postID1
-postID2
Edits-Thank you Frank
In our storyboard flow we plan to have a user see a wall of tags determined by constantly updating popular score based on properties of the tag and where we predict the user may have interest. The user will then select a tag and see posts related to that tag, from those posts I would like to show the posts from a users followers before those of everyone else who’s post falls in the category of a specified tag.
I have considered two possibilities either I optimize on reads in which I would have to keep track of every time a users follower posts to a tag and record the tagID along with the postID in a node for every follower a user has who posted in a special node of FollowersTags which would have a structure of listing for each userID a list of users and the all the followers of a user posted to which would become 100s of writes for each post created directly proportional to the number of followers a friend has.
*creates a list of posts to a specific tag made by followers
FollowersTags
-userID1_tagID1(composite key)
-postID1
-postID2
-postID3
-postID4
-userID1_tagID2
-postID1
-postID2
-postID3
-postID4
Or I could optimize on writes as tried above, which presents us with our current predicament of having to perform a query 100s of times directly proportional to the number of posts in a tag.
Is there any way around these two options which of the two is the better approach.
Unfortunately I would not be able to predict the posts displayed to the user before they select a tag.
In the Firebase Realtime Database, I typically model the data in the database to what I show on the screen. So if you have a "wall" of recent, relevant posts for each user, consider modeling precisely that in your database: a list of recent, relevant posts (or post IDs) for each user.
UserWalls
userID1
"timestamp_or_push_id": "postId1"
"timestamp_or_push_id": "postId2"
userID2
"timestamp_or_push_id": "postId1"
"timestamp_or_push_id": "postId3"
While the problem of determining what to show remains the same, with this database model it's now a write-time problem, instead of a read-time problem.

Efficiently storing and retrieving likes

In my Firebase database I have posts and then authenticated users can "like" posts. How can I efficiently get the number of likes a post has received. I know using MongoDB I can add/remove the user's id to a list and then use a MongoDB function to get the length of it very quickly and set that equal to the likes amount, but I'm not suer how to do that using Firebase. I could also add/remove it to the list and increment a likeCount variable, but that seems like it would cause concurrency issues unless Firebase has a function for that. What functions can I call to best handle this and scale well? Thanks in advance!
You can do both things:
1) Create a votes node with the UID as key and a value to sum up all the votes.
post:{
//All the data
likes:{
$user_1:1,
$user_2:-1,
}
}
And then just get a SingleValue Event or a Value event(depending if you want to keep track of changes) and sum up all the children
2)You can use a transaction block and just save a value and increase or decrease it depending on the votes
(here is a link where you can find transactions for android,iOS or java)
https://firebase.google.com/docs/database/web/save-data#save_data_as_transactions
post:{
//All the data,
likes:2,
}
It really depends on how much information you want to store, and what the user can do once he/she already voted for some post,
I would recommend using both, to keep flexibility for the user to like (like in Facebook) so he can unlike something and use the transaction with number to keep it scalable.. so if a post gets 1,000,000 likes you don't have to count the 1,000,000 likes every time someone loads the post

Resources