I am trying to make feed/timeline where a user can follow - Category,Album or another User. Every time a picture is added to Category,Album,User it should appear on the timeline. I am trying to model my database so it requires 1-2 get requests only.
One idea for the solution is fan-out structure, But how do i make the multi-path update in Firestore? How can i update all the followers timelines when someone uploads a photo ?
How do i structure the database when i cant query on sub-collections? Should i just make one collection which contains all user timeline posts as separate documents, which will be ridiculous amount of duplicated data.
Is there any other way instead of fan-out to structure a user timeline ?
But how do I make the multi-path update in Firestore?
The equivalent of Firebase Realtime Database's multi-path updates, are called batched writes in Cloud Firestore. You can read more in the documentation on batches writes.
Flat.
root
pictures
uid-abc123
url:"http://test.com/img1.jpg"
owner:useriduid,
created: 1529333679449
uid-abc1billion
url:"http://test.com/img1billion.jpg"
owner:useriduid,
created: 1529333679300
Querying and security rules are then easy as pie. You can add indexing and it's very scalable.
Re multipath writes, use batched writes.
Related
I am using both Firebase Database and Firestore in my app. I store users data like name, email, uid etc small details in documents of a collection as Users in firestore. It works perfectly. I made a node as Friends in firebase database to store friends list of a user. So whenever user open the app, it calls his information from Users from firestore and also his friends list from Friends from firebase database.
Now the thing is by this way it calls data from the Firestore and the Firebase database. So it means they are 2 requests/reads, one to Friends node and other to a document from Users collection. I think it would be better if i store friends list in Users document as an Array. So i will get only 1 read in Firestore. But i think that when the arrays of his friends list increases by 100+ elements. And also there are one or two more array lists like that. So will it take much time in retrieving a document from Users collection? or not? And which will be a better approach?
Here are the images of my current database structure as Users and Friends.
As per the Firestore usage and limits, the maximum size of a document is 1 MiB.
It means that as long as your user documents don't exceed the size limit, you can store friends data in arrays without a problem.
If you are planning to exceed the threshold, you may want to look for other options like creating subcollections to scale better as size of the subcollection doesn't affect the parent document's size in any way.
I built a chat app in flutter with firebase using mapping for each chat Text(only one doc was used in chat between 2 users). I observed that after I filled 1MB of data in doc, my mobile downloaded the chat history at 10-12 kbps from firebase.
Maybe the speed was a coincidence but I am sure that as your data grows in a single firestore doc, the mobile app does not bursty download the whole document simultaneously, instead it downloads at a much slower speed.
Please correct me If I am wrong.
The article about Best practices for Cloud Firestore states that we should keep the rate of write operations for an individual collection under 1,000 operations/second.
But at the same time, the Firebase team says in Choose a data structure that root-level collections "offer the most flexibility and scalability".
What if I have a root-level collection (e.g. "messages") which expects to have more than 1,000 write operations/second?
If you think at that limitation of 1,000 operations/second it's pretty much but if you find your self in a situation in which you need more than that, then you should consider changing your database schema to allow writes on multiple collections. So you should multiply the number of collections. Having a single collection of messages, in which every user can add messages doesn't sound as a good way to go since you can reach that limitation very soon. In this case you should split that collection into multiple other collections. A possible schema might be the one I have explained in the following video:
https://www.youtube.com/watch?v=u3KwKQddPoo
See, at the end of that video, there is collection named messages which in term contains a roomId document. This document contains a subcollection named roomMessages which contains as documents all messages from a chat room. In this case, there are no chances you can reach that limitation.
But at the same time, the Firebase team says in Choose a data structure that root-level collections "offer the most flexibility and scalability".
But also rememeber, Firestore can as quickly look up a collection at level 1 as it can at level 100, so you don't need to worry about that.
The limit of 1,000 ops/sec per collection only apply to realtime update, so as long as you don't have a snapshot listener this should be okay.
I asked the question on the Cloud Firestore Google Groups
The limit is 10,000 writes per second if no other limits apply first:
https://firebase.google.com/docs/firestore/quotas#writes_and_transactions
Also just keep in mind the best practices for scaling cloud firestore
I'm building an app using flutter and firebase and was wondering what the best firestore database structure.
I want the ability for users to post messages and then search by both the content of the post and the posters username.
Does it make sense to create one collection for users with each document storing username and other info and a separate collection for the posts with each document containing the post and the username of the poster?
In the unlikely event where the number of posts exceeds a million or more, is there an additional cost of querying this kind of massive collection?
Would it make more sense to store each user's posts as a sub-collection under their user document? I believe this would require additional read operations to access each document's sub-collection. Would this be cheaper or more expensive if I end up getting a lot of traffic?
is there an additional cost of querying this kind of massive collection?
The cost and performance of reading from Firestore are purely based on the amount of data (number of documents and their size) you retrieve, and not in any way on the number of documents in the collection.
But what is limited in Firestore is the number of writes you can do to data that is "close to each other". That intentionally vague definition means that it's typically better for write scalability to spread the data over separate subcollections, if the data naturally lends itself to that (such as in your case).
To get a great introduction to Firestore, and to data modeling trade-offs, watch Getting to know Cloud Firestore.
I am trying to figure out the best way to execute my cloud function for my firestore database, when data is being read.
I have a field on all of my documents with the timestamp of when the document was last used, this is used to delete documents that haven't been used in two days. The deletion is done by another cloud function.
I want to update this field, when the documents is being used AKA read from my db. What would be the best way to do this?
onWrite(), onCreate(), onUpdate() and onDelete() is not an option.
My database is used by a Android App written in Kotlin.
There are no triggers for reading data. That would not be scalable to provide. If you require a last read time, you will have to control access to your database via some middleware component that you write, and have all readers query that instead. It will be responsible for writing the last read time back to the database.
Bear in mind that Firestore documents can only be written about once every second, so if you have a lot of access to a document, you may lose data.
I have a collection, itemsCollection, which contains a very large amount of small itemDocs. Each itemDoc has a subcollection, statistics. Each itemDoc also has a field "owner" which indicates which user owns the itemDoc.
itemsCollection
itemDoc1
statistics
itemDoc2
statistics
itemDoc3
statistics
itemDoc4
statistics
...
I also have a collection, usersCollection, which contains basic user info.
usersCollection
user1
user2
user3
...
Since each itemDoc belongs to a specific user, it's necessary to display to each user which itemDocs they own. I have been using the query:
db.collection("itemsCollection").where("owner", "==", "user1")
I am wondering if this will scale effectively, i.e. whenever itemsCollection gets to be millions of records? If not, is the best solution to duplicate each itemDoc and its statistics subcollection as a subcollection in the user document, or should I be doing something else?
As Alex Dufter, the product manager from Firebase, explained in one of days at Firebase Dev Summit 2017 that Firestore was inspired in many ways by the feed-back that they had on the Firebase Realtime Database over the years. They faced two types of issues:
Data modelling and querying. Firebase Realtime Database cannot query over multiple properties because it ussaly involves duplication data or cliend-side filtering, which we all already know that is some kind of messy.
Realtime Database does not scale automatically.
With this new product, they say that you can now build an app and grow it to planetary scale without changing a single line of code. Cloud Firestore is also a NoSQL database that was build specifically for mobile and web app development. It's flexible to build all kinds of apps and scalable to grow to any size.
So because the new database was build knowing this iusses, duplication data is not nedeed anymore. So you will not have to worry about using that line of code, even if your data will grow to millions of records, it will scale automatically. But one thing you need to remember, if you will use multiple conditions, don't forget to use the indexes by simply adding them in the Firebase console. Here are two simple examples from the offical documentation:
citiesRef.whereEqualTo("state", "CO").whereEqualTo("name", "Denver");
citiesRef.whereEqualTo("state", "CA").whereLessThan("population", 1000000);