I have a subcollection for each doc in the users collection of my app. This subcollection stores docs that are related to the user, however they could just as well be saved to a master collection, each doc with an associated userId.
I chose this structure as it seemed the most obvious at the time but I can imagine it will make things harder down the road if I need to do database maintenance. E.g. If I wanted to clean up those docs, I would have to query each user and then each users docs, whereas if I had a master collection I could just query all docs.
That lead me to question what is the point of subcollections at all, if you can just associate those docs with an ID. Is it solely there so that you can expand if your doc becomes close to the 1MB limit?
Edit: October, 29th 2021:
To be clear about the following sentence that exists in the docs:
If you don't query based on the field with sequential values.
A timestamp just can not be considered consecutive. However, it still can be considered sequential. The same rules apply to alphabetical (Customer1, Customer2, Customer3, ...), or pretty much everything that can be treated as a predictably generated value.
Such sequential data in the Firestore indexes, it's most likely to be written in the physical proximity on the storage media, hence that limitation.
That being said, please note that Firestore uses a mechanism to map the documents to their corresponding locations. This means that if the values are not randomly distributed, the write operations will not be distributed correctly over the locations. That's the reason why that limitation exists.
Also note, that there is a physical limit on how much data you can write to such a location in a specific amount of time. Predictable key/values most likely will end up in the same location, which is actually bad. So there are more changes to reach the limitation.
Edit: July, 16th 2021:
Since this answer sounds a little old, I will try to add a few more advantages of using subcollections that I found over time:
Subcollections will always give you a more structured database schema, as you can always refer to a subcollection that is related only to a specific document. So you can nest only data that is related to a particular document.
As mention before, the maximum depth of a subcollection is 100. So an important feature here is that a Firestore Query is as fast at level 1, as it is at level 100. So there should be no concerns regarding depth. This feature is tested.
Queries in subcollections are indexed by default, as in the case of top-level collections.
In terms of speed, it doesn't really matter if you Query a top-level collection, a subcollection, or a collection group, the speed will always be the same, as long as the Query returns the same number of documents. This is happening because the Query performance depends on the number of documents you request and not on the number of documents you search. So querying a subcollection has the same effect as querying a top-level collection, no downsides at all.
When storing documents in a subcollection, please note that there is no need to storing the document ID as a field, as it is by default part of the reference. This means that you can store less data in the documents that exist in the subcollection. More important, if you would have saved the same data in a top-level collection, and you would have needed to create a Query with two whereEqualTo() calls + an orderBy() call, then an index would be required.
In terms of security, subcollections allow inheritance of security rules, which is useful because we can write less and less code to secure the database.
That's for the moment, if I found other benefits, I'll update the answer.
Let's take an example for that. Let's assume we have a database schema for a quiz app that looks like this:
Firestore-root
|
--- questions (collections)
|
--- questionId (document)
|
--- questionId: "LongQuestionIdOne"
|
--- title: "Question Title"
|
--- tags (collections)
|
--- tagIdOne (document)
| |
| --- tagId: "yR8iLzdBdylFkSzg1k4K"
| |
| --- tagName: "History"
| |
| --- //Other tag properties
|
--- tagIdTwo (document)
|
--- tagId: "tUjKPoq2dylFkSzg9cFg"
|
--- tagName: "Geography"
|
--- //Other tag properties
In which tags is a subcollection within questionId object. Let's create now the tags collection as a top-level collection like this:
Firestore-root
|
--- questions (collections)
| |
| --- questionId (document)
| |
| --- questionId: "LongQuestionIdOne"
| |
| --- title: "Question Title"
|
--- tags (collections)
|
--- tagIdOne (document)
| |
| --- tagId: "yR8iLzdBdylFkSzg1k4K"
| |
| --- tagName: "History"
| |
| --- questionId: "LongQuestionIdOne"
| |
| --- //Other tag properties
|
--- tagIdTwo (document)
|
--- tagId: "tUjKPoq2dylFkSzg9cFg"
|
--- tagName: "Geography"
|
--- questionId: "LongQuestionIdTwo"
|
--- //Other tag properties
The differences between this two approaches are:
If you want to query the database to get all tags of a particular question, using the first schema it's very easy because only a CollectionReference is needed (questions -> questionId -> tags). To achieve the same thing using the second schema, instead of a CollectionReference, a Query is needed, which means that you need to query the entire tags collection to get only the tags that correspond to a single question.
Using the first schema everything is more organised. Beside that, in Firestore Maximum depth of subcollections: 100. So you can take advantage of that.
As also #RenaudTarnec mentioned in his comment, queries in Cloud Firestore are shallow, they only get documents from the collection that the query is run against. There is no way to get documents from a top-level collection and other collections or subcollections in a single query. Firestore doesn't support queries across different collections in one go. A single query may only use properties of documents in a single collection. So there is no way you can get all the tags of all the questions using the first schema.
This technique is called database flatten and is a quite common practice when it comes to Firebase. So use this technique only if is needed. So in your case, if you only need to display the tags of a single question, use the first schema. If you want somehow to display all the tags of all questions, the second schema is recommended.
Is it solely there so that you can expand if your doc becomes close to the 1MB limit?
If you have a subcollection of objects within a document, please note that size of the subcollection it does not count in that 1 MiB limit. Only the data that is stored in the properties of the document is counted.
Edit Oct 01 2019:
According to #ShahoodulHassan comment:
So there is no way you can get all the tags of all the questions using the first schema?
Actually now there is, we can get all tags of all questions with the use of Firestore collection group query. One thing to note is that all the subcolletions must have the same name, for instance tags.
The single biggest advantage of sub-collections that I've found is that they have their own rate limit for writes because each sub-collection has its own index (assuming you don't have a collection group index). This probably isn't a concern for small applications but for medium/large scale apps it could be very important.
Imagine a chat application where each chat has a series of messages. You'll want to index messages by timestamp to show them in chronological order. The Firestore write limit for sequential values is 500/second, which is definitely within reach of a medium-sized app (especially if you consider the possibility of a rogue user scripting messages -- which is not currently easy to prevent with Security Rules)
// root collection
/messages {
chatId: string
timeSent: timestamp // the entire app would be limited to 500/second
}
// sub-collection
/chat/{chatId}/messages {
timeSent: timestamp // each chat could safely write up to 500/second
}
Surprised this hasn't been mentioned before, but sub-collections can (in some cases) help bypass the orderBy limitations:
You can't order your query by a field included in an equality (==) or in clause.
Suppose you want to get a users most recent 10 logins:
Top-Level:
//We can't use .orderBy after .where('==')
USER_LOGINS.where('userId', '==', {uid}).limit(10)
Sub-Collection:
//With a subcollection we can order and limit properly
USERS.doc({uid}).collection('LOGINS').orderBy('unixCreated', 'desc').limit(10);
Subcollections are also helpful in setting up security rules. Suppose you are building a chat app and have a user collection with a replies subcollection. You want other users to be able to add to the replies collection but want to give the user full rights to the user collection. If you have replies as an array of maps/objects in user collection, it severely limits the rules you can write against the user collection for the collection owner and other users to be able to add to the collection. Whereas, having it as its own subcollection makes writing security rules waaaaay easier.
Related
Coming from an RDMS background I am trying to understand NoSQL databases and planning a simple project that includes topics, posts & comments.
Topics have posts & posts have comments
I have found the following guide that suggests using the following top-level collections:
A users collection
A posts collection
A user-posts collection
A posts-comments collection
https://firebaseopensource.com/projects/firebase/quickstart-android/database/readme/
I fail to understand the benefits of (3) above as surely we can simply filter (2) based on the user, even 3 would still need to be filtered.
What is the logic of having comments as a top-level collection as opposed to having comments as a subcollection under posts? Is this not the better way to store hierarchical data?
In the NoSQL world, we are structuring a database according to the queries that we want to perform.
What is the logic of having comments as a top-level collection as opposed to having comments as a subcollection under posts?
None is better than the other. However, there are some differences:
What are the benefits of using a root collection in Firestore vs. a subcollection?
Is this not the better way to store hierarchical data?
There is no "perfect", "the best" or "the correct" solution for structuring a Cloud Firestore database. We always choose to create a structure for our database that satisfies our queries. So in your case, I would create a schema that looks like this:
Firestore-root
|
--- users (collection)
| |
| --- $uid (document)
| |
| --- //user fields.
|
--- posts (collection)
|
--- $postId (document)
|
--- uid: "veryLongUid"
|
--- //user fields.
|
--- comments (sub-collection)
|
--- $commentId (document)
|
--- uid: "veryLongUid"
|
--- //comment fields.
Using this schema you can:
Get all users.
Get all posts in the database.
Get all posts that correspond to only a particular user.
Get all comments of all posts, of all users in the database. Requires a collection group query.
Get all comments of all posts that correspond to a particular user. Requires a collection group query.
Get all comments of all users that correspond to a particular post.
Get all comments of a particular user that correspond to a particular post.
Am I missing something?
If you think that all the comments of a post might fit into 1 MiB maximum limitation, then you should consider adding all comments into an array. If not, I highly recommend you read the following approach:
How to reduce Firestore costs?
Where I have explained how can we store up to billions of comments and replies in Firestore.
Below is my database schema that stores a many-to-many relationship between a task and tag model. Google state that the maximum size that a document can be stored on Firestore is 1 MiB. If I continuously add tags to a task the document size would exceed that size limit.
Firestore-root
|
--- tasks (collection)
| |
| --- taskID (document)
| |
| --- title: "Go for a cycle"
| |
| --- completed: false
| |
| --- userID: "zaEh95kXJKapyVUqrPws58dyRIC3"
| |
| --- tagIDs: ["rWqTxB01TK9w8KRo2GHD"]
| |
| --- // Other task properties
|
--- tags (collection)
|
--- tagID (document)
|
--- title: "Health"
|
--- userID: "zaEh95kXJKapyVUqrPws58dyRIC3"
|
--- colour: "red"
|
--- // Other tag properties
A solution that I have found to work is to create a junction table, however every time I navigate to the detail view of a task I have to query the database to find those relationships which in return drives up billing costs. Can’t help but feel as though I am caught between a rock and a hard place.
Related / follow-up Q&As
What is the most efficient way to store tags in Firestore?
How to model a many-to-many relationship in Firestore
What is denormalization in Firebase Cloud Firestore?
when I tap to see the details of a task a query is sent to Firestore to retrieve the tags associated with it.
Since you store the data in two different collections, yes, two different queries are needed. One to get the tasks and the second one to get the corresponding tags data. But that's not bad.
However, there are some other options that you can take into consideration. The first one would be, instead of saving the IDs of tags into an array, just save the actual data, meaning the entire "Tag" object. Or at least the important data of the tag. This practice is called denormalization. If you're new to the NoSQL databases, please note that this practice is quite common when it comes to Firebase. Also bear in mind that when you are duplicating data, there is one thing that you should know about. In the same way, you are adding data, you need to maintain it. In other words, if you want to update/delete an item, you need to do it in every place that it exists.
When using the above solution, note that there are some limits when it comes to how much data you can put into a document. According to the official documentation regarding usage and limits:
Maximum size for a document: 1 MiB (1,048,576 bytes)
As you can see, you are limited to 1 MiB total of data in a single document. When we are talking about storing text (tag IDs), you can store pretty much. I doubt you'll reach the limitation but as your arrays get bigger, be careful about this constraint. A workaround for this would be to create another document and another document for storing the tags. But also note, that besides the number of reads, you are also charged with the bandwidth needed to download the documents.
So it's up to you to decide which solution works best for your application.
I have a root collection students which has student documents and each of this document has some sub-collections. for eg: marks and each mark document has again sub-collections and so on till depth 4.
Now If i remove a particular marks sub-collection, I want all of its nested sub collections to be deleted completely instead of hanging orphaned.
I read the following open github issue
https://github.com/firebase/firebase-admin-node/issues/361
and find out that deepDeleteCollection can be used only if we know that collection is a leaf sub-collection(reached max depth). but to get this information, we need to separately query by each document inside that sub collection, which doesn't makes sense in terms of performance.
What would be the best way to achieve deepCollectionDelete?
Please let me know In case I seem to miss something here.
Thanks in Advance.
but to get this information, we need to separately query by each document inside that sub collection, which doesn't makes sense in terms of performance.
Unlike in Firebase realtime database where to remove the whole structure within a particular node, you would have taken a reference and call removeValue() method, in Cloud Firestore this is not possible. In order to delete a document that contains a subcollection which in terms contains other documents with other subcollections, you need to find and remove all documents within subcollections from deeper hierarchy to higher hierarchy. For instance, let's assume you have a schema that looks like this:
Firestore-root
|
--- collectionOne
|
--- documentOne
| |
| --- subcollectionOne
| |
| --- documentTwo
| |
| --- subSubCollectionOne
| |
| --- //Documents
|
--- documentThree
|
--- subcollectionThree
|
--- // documents
To delete let's say documentOne, you need to get all documents within subSubCollectionOne and delete them and then find all documents within subcollectionOne and delete them and only at the end you should delete documentOne.
which doesn't makes sense in terms of performance.
This is not true. This process of deleting documents that exist in collections and subcollections works very well. You can delete the documents on client in smaller chunks or using a Cloud Function.
I'm looking for a proper way to structure Firestore database to handle multiple version histories of documents inside a single collection.
For example: I have a collection named offers which have multiple documents which correspond to multiple offers. For each of these documents, I'd like to have history of changes, something like changes on Google Docs.
Since documents support only adding fields directly or nesting another collection, here's a structure I had in mind:
collections: offers
- documents: offer1, (offer2, offer3, ...)
- fields populated with latest version of the offer content
- nested collection named history
- nested documents for each version (v1, v2, v3), which in turn have fields specifing state of each field in that version.
This seems a bit overly complicated since I have latest state and than nested collection for history. Can this be somehow in flat structure where latest item in array is the latest state, or something similar.
Also, history state is generated on a button click, so I don't need every possible change saved in a history, just snapshots when user saves it.
I'd like to use Firebase as my DB for this, as I need it some other things, so I'm not looking into different solutions for now.
Thanks!
EDIT: According to the Alex's answer, here's my another take on this.
Firestore-root
|
--- offers (collection)
|
--- offerID (document)
| (with fields populated )
| |
| --- history (collection) //last edited timestamp
| |
| --- historyId
| --- historyId
|
--- offerID (document)
(with fields populated with latest changes)
|
--- history (collection) //last edited timestamp
|
--- historyId
--- historyId
This way I can query whole offers collection and get array of offers together with latest status since it's on the same level as the collection itself. Then if I need specific content from history state, I can query history collection of specific offer and get it's history states. Does this make sense?
I'm not sure about denormalization as this seems like it solves my problem and avoids complication.
Once more, requirements are:
- being able to fetch all offers with latest state (works)
- being able to load specific history state (works)
Just every time I update history collection with new state, I overwrite the fields directly in offerID collection with the same, latest, state.
Am I missing something?
In my opinion, your above schema might work but you'll need to do some extra database calls, since Firestore queries are shallow. This means that Firestore queries can only get items from the collection that the query is run against. Firestore doesn't support queries across different collections. So there is no way in which you can get one document and the corresponding history versions that are hosted beneath a collection of that document in a single query.
A possible database structure that I can think of, would be to use a single collection like this:
Firestore-root
|
--- offerId (collection)
|
--- offerHistoryId (document)
| |
| --- //Offer details
|
--- offerHistoryId (document)
|
--- //Offer details
If you want to diplay all history versions of an offer, a single query is required. So you just need to attach a listener on offerId collection and get all offer objects (documents) in a single go.
However, if you only want to get the last version of an offer, then you should add under each offer object a timestamp property and query the database according to it descending. At the end just make a limit(1) call and that's it!
Edit:
According to your comment:
I need to get a list of all offers with their latest data
In this case you need to create a new collection named offers which will hold all the latest versions of your offers. Your new collection should look like this:
Firestore-root
|
--- offers (collection)
|
--- offerHistoryId (document)
| |
| --- date: //last edited timestamp
| |
| --- //Offer details
|
--- offerHistoryId (document)
|
--- date: //last edited timestamp
|
--- //Offer details
This practice is called denormalization and is a common practice when it comes to Firebase. If you are new to NoQSL databases, I recommend you see this video, Denormalization is normal with the Firebase Database for a better understanding. It is for Firebase realtime database but same rules apply to Cloud Firestore.
Also, when you are duplicating data, there is one thing that need to keep in mind. In the same way you are adding data, you need to maintain it. With other words, if you want to update/detele an item, you need to do it in every place that it exists.
In your particular case, when you want to create an offer you need to add it in two places, once in your offerId collection and once in your offers collection. Once a new history version of an offer is created, there is only one more operation that you need to do. As before, add the offerHistoryId document in your offerId collection, add the same object in your offers collection, but in this case you need to remove the older version of the offer from the offers collection.
I can think of it like this. Each offers document will have offerHistoryID as number.
You can have a separate root collection for versioned documents of offers(say offers_transactions).
Now write an update trigger cloud function on offers document which will have both after and before values of the document.
Before doing the doc update, you can write the before values into the offers_transactions along with timestamp and latest historyID.
Increment the offerHistoryID by 1 for that offer and update the doc with new values.
Now you can query the root collection offers_transactions for historic transactions based on your filters. This way you can keep your root collection offers cleaner.
Thoughts?
Here's a solution my team uses to leverage Google Cloud Functions to add every collection update to a dedicated "history" collection in Firestore (no command line necessary):
Identify path of document to watch: COLLECTION-NAME/{documentID} (or define a specific document to watch)
Create a new Cloud Function (1st gen because 2nd gen doesn't support Firestore triggers yet)
Set trigger as any Firestore "write" event watching the document path from Step 1.
In the Cloud Function's inline code editor, select the language of your choice (I'll use Python), and include google-cloud-firestore==2.6.0 in your requriements.txt file (or whatever the latest version is)
Finally, define your Cloud Function's code (be sure to import Firestore correctly!)
def hello_firestore(event, context):
resource_string = context.resource
# print out the resource string that triggered the function
print(f"Function triggered by change to: {resource_string}.")
# now print out the entire event object
print(str(event))
# now import firestore and add event to the 'history' collection
from google.cloud import firestore
db = firestore.Client(project="YOUR-PROJECT-ID")
newHistDoc = db.collection(u'history').add(event)
There is a firestore collection that stores recipes with a list of ingredients. We need to find recipes that contain at least one of the ingredients. How to implement it? This is possible with the firestore?
--- recipe1
|
--- ingredients: ["salt", "pepper", "sucar"]
--- recipe2
|
--- ingredients: ["pepper"]
--- recipe3
|
--- ingredients: ["salt", "pepper"]
How to choose a recipe in which there is either "pepper" OR "salt"?
This is possible with the firestore?
Yes it is.
We need to find recipes that contain at least one of the ingredients. How to implement it?
In order to implement this feature, you should use arrays. Please see below a database schema, that can help you achieve this:
Firestore-root
|
--- recipes (collection)
|
--- recipeId (document)
|
--- ingredients: ["salt", "pepper"] (array)
|
--- //other decipe details
In order to find all recipes that contain one of the ingredients, you should use a query that looks like this:
FirebaseFirestore rootRef = FirebaseFirestore.getInstance();
Query query = rootRef.collection("recipes").whereArrayContains("ingredients", "salt");
This is for Android but in the same way you can achieve this for other programming languages. See here more details.
But be aware of one thing, you cannot find all recipes that contain more than one of the ingredients, without making significant changes in your database. Firestore does not allow chained whereArrayContains() calls. So you can get all recipes that contain only one of the ingredients.
Edit: According to your comment:
How to choose a recipe in which there is either "pepper" OR "salt"?
You need to know that there is no OR clause in Firestore. According to my answer from this post, you should create two separate queries and merge the result cliend side.
Edit2:
100,000 recipes to combine on the client.
If this is the use-case of your app then you should query this way. Firestore scales massively.
But this is a fairly simple query!
It is in terms of SQL databases but this is how NoSQL databases work.
If you need to exclude recipes with the ingredient "sugar"?
According to the official documentation, beside the fact that there is no OR clause, there is also another query limitation:
Queries with a != clause. In this case, you should split the query into a greater-than query and a less-than query.
So this is the way in which you can solve this exclude or not equal situation.
Your link also does not solve the problem.
My link cannot solve a problem that cannot be solved in the way you want, it just indicates the constraints that Firestore official documentation provides. In that link, I also provide a workaround in the way the docs recommend. In this case, you should make your own attempt given the information in my answer and that post and ask another question if something else comes up.
The result - recipes with salt AND pepper. But not salt OR pepper!
No, if you create two separate queries and merge the result cliend side, you'll have the desired result. I've test it and it works pretty fine.
I know that there is no "OR". I therefore ask - is there a solution?
Yes it is, the solution I have provided you above, which is the simplest one. If you are not happy with that, you might also consider change your entire database structure so it can be organized according to a reverse look up. Your structure should look like this:
Firestore-root
|
--- salt_peper (collection)
| |
| --- recipeId (document)
| | |
| | --- //recipe with salt
| |
| --- recipeId (document)
| |
| --- //recipe with peper
|
--- salt_sugar (collection)
|
--- recipeId (document)
|
--- //recipe details
As you can see, this is another schema for your database. The first collection will provide you all recipes with salt OR peper.
This practice is called denormalization and is a common practice when it comes to Firebase. For a better understanding, I recommend you see this video, Denormalization is normal with the Firebase Database. It is for Firebase realtime database but same principle apply to Cloud Firestore.
Also, when you are duplicating data, there is one thing that need to keep in mind. In the same way you are adding data, you need to maintain it. With other words, if you want to update/detele an item, you need to do it in every place that it exists.
I also recommend you take a look at my answer from this post to see pro and cons regarding the tehnique above.