How to structure a Social Network database structure like for example twitter where we can follow a users and get all their tweets in our timeline, i have already checked this Firestore - how to structure a feed and follow system
but the solutions in the post look flawed.
Firestore is different where it requires redundant data to access data efficiently, but suppose i am following 1000 people and if i need to get the posts of all those users by querying data for each 15 users i am following and using limit(10) method then orderBy(timeStamp) there may be unread posts between Queries, because we are getting the post using the last post timeStamp , how to structure the data for a social media app in Firestore
When modeling a use-case on a NoSQL database, you tend to optimize for the features of your application, and for frequent read-operations.
So in a social media application your main feature may be that the user sees the recent posts of everyone they follow. To optimize this operation for frequent reads, you'll want to store the posts that the each user should see in a document for that user. So when compared to Twitter, you'd pretty much have a document containing the twitter feed for each user. Or if there's too much data for a single document, you might want to put that in a collection. I often explain this as modeling the screens of your app in the database.
This is very different from the typical data model in a relational database, so it's normal that it takes time to get used to. For a good introduction, I recommend:
Reading NoSQL data modeling.
Watching Firebase for SQL developers, even though it's for the Realtime Database, it explains how to map common SQL concepts to Firebase's NoSQL model.
Watching Getting to know Cloud Firestore
To develop a social media app like Twitter. The Firestore queries are not enough.
Twitter generates a personalized timeline for every user.
This is where the cloud functions come into the picture.
You need a cloud function that monitors for new posts and copies them in their following user's timelines.
You don't need to copy the entire tweet data. You can just copy the tweet id and other fields which require ordering, like timestamp.
So when I query my timeline, I will get all the tweet ids.
Then I can just load the original tweet when the user is about to scroll.
Because the likes and dislikes should affect the original tweet.
Related
I have been using Firestore for a very long time. I am building an app now where scalability and keeping low costs is important. (I am using flutter)
My app has users, which have user profiles, also they can add friends and talk to them (like instagram or facebook).
I have a problem building this friends system.
My model for this friends system currently looks like this:
Users collection. Each document id = user id from auth, those docs contain data like name, username, profile picture, etc.
Friends collection. Each document id = user id from auth. For each user, those docs contain a field called: friends, which is an array with each of his friends user ids.
The model looks like:
Friends collection:
- uid:
- friends_list: [friend_uid1, friend_uid2, ...]
This is how my "backend" looks.
Now I want to show my user a list of his friends. How do I do that?
I want a list that looks like instagram, with a nice UI showing each of my user friend profile pic, name, last message, etc.
I can not find a straight forward way to do this with Firestore and queries.
Let's say I do it like this:
Get all my friends user ids in an array.
Get all their user documents using .get() for each document.
This is not doable in firestore cause it would eliminate all the querying power I have (such being able to query only for users with name "x"), I would have to fetch all users and do the query on my front-end (or in a cloud function, same thing, not scalable).
If I do this like:
Get all document using a query for all users in the Friends collection, where friends_list contains my user id.
Save from those documents only the documentID and fetch all the friends user data manually.
This comes with another problem. In Firestore there is no way of fetching a document without fetching all of its fields, so the first query which I use to get the ids only of my friends would actually give me their id + their friend list instead (cause when I query, it also gets the document id + the data), which is not good.
If I do it like:
When you add a friend, instead of just saving its uid, save its uid + data.
Now I can easily show my user his friends list nicely and do some querying on front-end.
The problem here is that now if one of my friends updates his profile photo, I need to update it in every document of all of his friends, which is very write expensive for just a little profile update.
There is also the problem of watching for more data, maybe I have another collection with Chats, and I want to show the last message of my chat with a friend, now I have to fetch the chat rooms too, which is more hard to query data that comes with all the problems that I mentioned before.
In conclusion: I don't see a good scalable way to do this kind of system within Firestore. It seems a simple system which any basic app should have, but I do not see how I can do it in a way that does not make lots of reads or read more data (or sensitive data) than it should.
What kind of model would you do for a friends system like this?
You're decribing a quintessential drawback of NoSQL Databases.
A similar example is actually given in the Get to Know Cloud Firestore series.
Like others have commented, the answer really depends on your application. And this is the assessment you'll have to do. Like which of the options is cheaper depending on the use case of the app.
For example, if you go with your third option and store the friend's user data that you'll need to populate the list. This means you'll have to implement measures to keep the integrity of the copied data whenever the user updates their information.
You can then look at the usage of your app and determine how often users change their information vs how often you would need to retrieve full users if you don't copy the data to find the cheapest method for your application.
We are building a social media web app using firebase and use firestore to store users and their posts.
When a user likes a post, we save it in posts/{postID}/likedBy/{userID} and also update totalLikes in the post document.
Let's say our app has 1 million daily users, and they all are liking viral posts very frequently.
Now, firebase says that a document cannot handle more than one write per second. However, we've seen that we can update the document several times per second, but they still don't recommend it.
My question is, what is the best way to store total post likes in firestore, if there's any. Or, should we use some other services?
EDIT: Firestore's distributed counters are made for exactly as suggested by the answer below.
Also, I want to query only those posts which are not liked by a user.
The way I can query this is if our documents inside posts collection contains Map of all the users who liked it, and then run a query where the map doesn't contain current userID. This approach isn't good because it limits the number of likes a post can get as the document size in firestore cannot exceed 1mb.
Another way can be to save the liked posts in the user's document, however by this, we'll not only loose the functionality to just fetch those posts which are not liked by user, it'll also limit the number of posts a user can like.
Third way can be to store the users who liked the post in a sub-collection, which will also loose the query functionality. Similar case would be with storing posts liked by a user in a sub-collection.
Now, either I've not enough knowledge of firestore(actually any other NoSQL database), or I'm thinking right but it's just that NoSQL isn't made for social media apps.
Let's say our app has 1 million daily users, and they all are liking
viral posts very frequently.
Now, firebase says that a document cannot handle more than one write
per second.
My question is, what is the best way to store total post likes in
firestore, if there's any. Or, should we use some other services?
This is the exact scenario for which Firebase recommends to use some distributed counters.
With distributed counterS, "each counter is a document with a subcollection of shards, and the value of the counter is the sum of the value of the shards."
"Write throughput increases linearly with the number of shards, so a distributed counter with 10 shards can handle 10x as many writes as a traditional counter." (traditional counter = counter in one document)
I'm building an app using flutter and firebase and was wondering what the best firestore database structure.
I want the ability for users to post messages and then search by both the content of the post and the posters username.
Does it make sense to create one collection for users with each document storing username and other info and a separate collection for the posts with each document containing the post and the username of the poster?
In the unlikely event where the number of posts exceeds a million or more, is there an additional cost of querying this kind of massive collection?
Would it make more sense to store each user's posts as a sub-collection under their user document? I believe this would require additional read operations to access each document's sub-collection. Would this be cheaper or more expensive if I end up getting a lot of traffic?
is there an additional cost of querying this kind of massive collection?
The cost and performance of reading from Firestore are purely based on the amount of data (number of documents and their size) you retrieve, and not in any way on the number of documents in the collection.
But what is limited in Firestore is the number of writes you can do to data that is "close to each other". That intentionally vague definition means that it's typically better for write scalability to spread the data over separate subcollections, if the data naturally lends itself to that (such as in your case).
To get a great introduction to Firestore, and to data modeling trade-offs, watch Getting to know Cloud Firestore.
I have a Firestore data structure and a document where all my followers can see the recentPosts of mine by querying the collection of documents based on the users field of the document where querying users name is present just like below.
my question is how to share a post of others to my followers, currently i am duplicating the shared post to my recentPostsand my seperate Collection of posts documents, but what if a user deletes the post and the post was shared by million users? i have to delete all the shared posts, is there a better solution?
Given your choice in data model, having to delete the duplicated posts is pretty much the normal solution. I also don't see this as problematic, given that:
You've already written the duplicate post to all these followers to begin with, so the delete is just another write.
Deletes and other writes are relatively uncommon in most applications. If not, consider whether you should really be duplicating the data to all followers.
You could choose to implement this with a global list of deleted posts, that each client then reads. But at that point you're making the code that reads data more complex to prevent writes, which is typically not the best approach when using NoSQL databases.
I have a collection, itemsCollection, which contains a very large amount of small itemDocs. Each itemDoc has a subcollection, statistics. Each itemDoc also has a field "owner" which indicates which user owns the itemDoc.
itemsCollection
itemDoc1
statistics
itemDoc2
statistics
itemDoc3
statistics
itemDoc4
statistics
...
I also have a collection, usersCollection, which contains basic user info.
usersCollection
user1
user2
user3
...
Since each itemDoc belongs to a specific user, it's necessary to display to each user which itemDocs they own. I have been using the query:
db.collection("itemsCollection").where("owner", "==", "user1")
I am wondering if this will scale effectively, i.e. whenever itemsCollection gets to be millions of records? If not, is the best solution to duplicate each itemDoc and its statistics subcollection as a subcollection in the user document, or should I be doing something else?
As Alex Dufter, the product manager from Firebase, explained in one of days at Firebase Dev Summit 2017 that Firestore was inspired in many ways by the feed-back that they had on the Firebase Realtime Database over the years. They faced two types of issues:
Data modelling and querying. Firebase Realtime Database cannot query over multiple properties because it ussaly involves duplication data or cliend-side filtering, which we all already know that is some kind of messy.
Realtime Database does not scale automatically.
With this new product, they say that you can now build an app and grow it to planetary scale without changing a single line of code. Cloud Firestore is also a NoSQL database that was build specifically for mobile and web app development. It's flexible to build all kinds of apps and scalable to grow to any size.
So because the new database was build knowing this iusses, duplication data is not nedeed anymore. So you will not have to worry about using that line of code, even if your data will grow to millions of records, it will scale automatically. But one thing you need to remember, if you will use multiple conditions, don't forget to use the indexes by simply adding them in the Firebase console. Here are two simple examples from the offical documentation:
citiesRef.whereEqualTo("state", "CO").whereEqualTo("name", "Denver");
citiesRef.whereEqualTo("state", "CA").whereLessThan("population", 1000000);