Right now I have a products collection where I store my products as documents like the following:
documentID:
title: STRING,
price: NUMBER,
images: ARRAY OF OBJECTS,
userImages: ARRAY OF OBJECTS,
thumbnail: STRING,
category: STRING
NOTE: My web app has approximately 1000 products.
I'm thinking about doing full text search on client side, while also saving on database reads, so I'm thinking about duplicating my data on Firestore and save a partial copy of all of my products into a single document to send that to the client so I can implement client full text search with that.
I would create the allProducts collection, with a single document with 1000 fields. Is this possible?
allProducts: collection
Contains a single document with the following fields:
Every field would contain a MAP (object) with product details.
document_1_ID: { // Same ID as the 'products' collection
title: STRING,
price: NUMBER,
category: STRING,
thumbnail
},
document_2_ID: {
title: STRING,
price: NUMBER,
category: STRING,
thumbnail
},
// AND SO ON...
NOTE: I would still keep the products collection intact.
QUESTION
Is it possible to have a single document with 1000 fields? What is the limit?
I'm looking into this, because since I'm performing client full text search, every user will need to have access to my whole database of products. And I don't want every user to read every single document that I have, because I imagine that the costs of that would not scale very well.
NOTE2: I know that the maximum size for a document is 1mb.
According to this document, in addition to the 1MB limit per document, there is a limit of index entries per document, which is 40,000. Because each field appears in 2 indexes (ascending and descending), the maximum number of fields is 20,000.
I made a Node.js program to test it and I can confirm that I can create 20,000 fields but I cannot create 20,001.
If you try to set more than 20,000 fields, you will get the exception:
INVALID_ARGUMENT: too many index entries for entity
// Setting 20001 here throws "INVALID_ARGUMENT: too many index entries for entity"
const indexPage = Array.from(Array(20000).keys()).reduce((acc, cur) => {
acc[`index-${cur}`] = cur;
return acc;
}, {});
await db.doc(`test/doc`).set(indexPage);
I would create the allProducts collection, with a single document with 1000 fields. Is this possible?
There isn't quite a fixed limitation for that. However, the documentation recommends having fewer than 100 fields per document:
Limit the number of fields per document: 100
So the problem isn't the fact that you duplicate data, the problem is that the documents have another limitation that you should care about. So you're also limited to how much data you can put into a document. According to the official documentation regarding usage and limits:
Maximum size for a document: 1 MiB (1,048,576 bytes)
As you can see, you are limited to 1 MiB total of data in a single document. When we are talking about storing text, you can store pretty much but as your documents get bigger, be careful about this limitation.
If you are storing a large amount of data in your documents and those documents should be updated by lots of admins, there is another limitation that you need to take care of. So you are limited to 1 write per second on every document. So if you have a situation in which the admins are trying to write/update products in that same document all at once, you might start to see some of these writes fail. So, be careful about this limitation too.
And the last limitation is for index entries per document. So if you decide to get over the first limitation, please note that the maximum limit is set to 40,000. Because each field has associated two indexes (ascending and descending), the max number of fields is 20,000.
Is it possible to have a single document with 1000 fields?
It is possible up to 40,000 properties but in your case with no benefits. I say that because every time you perform a query (get the document), only a single document will be returned. So there is no way you can implement a search algorithm in a single document and expect to get Product objects in return.
And I don't want every user to read every single document that I have, because I imagine that the costs of that would not scale very well.
Downloading an entire collection to search for fields client-side isn't practical at all and is also very costly. That's the reason why the official documentation recommends a third-party search service like Algolia.
For Android, please see my answer in the following post:
Is it possible to use Algolia query in FirestoreRecyclerOptions?
Firebase has a limit of 20k fields per document.
https://www.youtube.com/watch?v=o7d5Zeic63s
According to the documentation, there is no stated limit placed on the number of fields in a document. However, a document can only have up to 40,000 index entries, which will grow as documents contain more fields that are indexed by default.
Related
There is a way to batch get documents after you specified exactly all the file paths using
https://firestore.googleapis.com/v1/projects/projectName/databases/dbName/documents:batchGet.
And there is a way to get all documents and their fields under a collection by sending a GET to https://firestore.googleapis.com/v1/projects/projectName/databases/dbName/documents/collectionName
I ideally want to make a batch request to get document fields for all documents under an array of collections. Is there a way to do this without knowing the document names of every document I intend to get?
Example
I have a structure like projects/projectName/databases/dbName/documents/Inventory/productId/variantId/*location*
Each productId is a document, and under this it has a collection for each variant, and within that collection are documents for each location, that contains a field count.
For a basket, I want to get all inventory counts for all inventory locations, for each productId/variantId in that bas
It is not possible to get all documents based on an array of collection names.
You can use a collection group query and search all collections of a given name, but then you must know the path of each document you want to read.
Alternatively, you can get all documents under a specific path, but then you can't filter by ID anymore, and the collections have to be under a path - not an array.
Is there any way to get the size of a particular document in Firestore using flutter? I need the number of fields in a particular document.
I'm trying to get the total number of fields in a particular document , however i can retrieve the number of documents in a collection but not the number of fields in a document .
As an alternative, you could keep track of the number of fields of each document in a separate document. Suppose a document has two fields
document_1
name_field
age_field
then in a separate collection we keep track of the number of fields, using the same document id:
document_field_count
document_1
count: 2
There is no built-in operation to get the number of fields of a Firestore document. You will have to read the document and count its number of fields in your application code, by counting the number of elements in the snapshot.data() map.
I don't want to have automatic indexes created by firestore because I need to remove and add every five minutes 50-100 documents (each doc has +/-60 fields) to my subcollection. This causes of big volume for "Cloud Firestore Index Write Ops" (300k / day for only one user) and Cloud Storage. I don't need to sort, filtering that documents so I suppose I can turn off automatic indexes, right?
I know that I can add exemptions for fields, but I don't know how can I use it for documents in subcollections. What should I pass in Collection ID and Field path if the path for documents is like:
mainCollectionName/{id}/subcollectionName/{document=**}
and when should I select a collection checkbox and when collection group checkbox?
Unfortunately, it's not possible to disable indexes or create exemptions for documents to be indexed. As clarified in this similar post here, this cannot be achieved and there is even a limit of 200 exemptions of fields that can be done - you can check the limits here.
For your case, indeed, you would have to exempt the fields individually and besides that, to create the exemption, to set the collection you use its id and not the path. So, you would only need to set in the Collection ID field the subcollectionName and then the field to be exempted.
In addition to this, feel free to raise a Feature Request in Google's Issue Tracker, so they can check about implementing an exemption of documents in the future.
I was looking for a solution to Firestore's limitation of Sequential indexed fields which means the following from this doc.
"Sequential indexed fields" means any collection of documents that
contains a monotonically increasing or decreasing indexed field. In
many cases, this means a timestamp field, but any monotonically
increasing or decreasing field value can trigger the write limit of
500 writes per second.
As per the solution, I can add a shard field in my collection which will contain random value and create a composite index with the timestamp. I am trying to achieve this with the existing fields I have in my Document.
My document has the following fields:
{
users: string[],
createdDate: Firebase Timestamp
....
}
I already have a composite index created: users Arrays createdDate Descending. Also, I have created Exemptions for the fields field from Automatic index settings. The users field will contain a list of firebase auto-generated IDs so definitely its random. Now I am not sure whether the field users will do the job of field shard form the example doc. In this way we can avoid adding a new field and still increase the write rate. Can someone please help me with this?
While I don't have specific experience that says what you're trying to do definitely will or will not work the way you expect, I would assume that it works, based on the fact that the documentation says (emphasis mine):
Add a shard field alongside the timestamp field. Use 1..n distinct values for the shard field. This raises the write limit for the collection to 500*n, but you must aggregate n queries.
If each users array contains different and essentially random user IDs, then the array field values would be considered "distinct" (as two arrays are only equal if their elements are all equal to each other), and therefore suitable for sharding.
I have an app that returns a list of health foods. There will be approximately 10000-20000 foods (documents) in the product collection.
These foods are queried by multiple fields using arrayContains. This may be categories, subcategories and when the user searches in the search bar it is an arrayContains on the keywords array.
With so many products I plan to paginate the results of query as I get the documents. The issue is that I need to know the amount of results to display the total of results to the user.
I have read that for a query you are charged one read and then if you get the documents then they are further charged per document. Is there a way of getting the number of results for a query without getting all the documents.
I have seen this answer here:
Get size of the query in Firestore
But in this example they say to use a counter which doesn't seem practical as I am using a query on keyword when the user searches and I am using a mixture of categories, subcategories when the user filters.
Thanks
With so many products I plan to paginate the results of query as I get the documents.
That's a very good decision since getting 10000-20000 foods (documents) at once is not an option. Reason one is the cost, it will be quite expensive and second is that you'll get an OutOfMemoryError when trying to load such enormous amount of data.
The issue is that I need to know the amount of results to display the total of results to the user.
There is no way in Firestore so you can know in advance the size of the result set.
Is there a way of getting the number of results for a query without getting all the documents.
No, you have to page through all the results that are returned by your query to get the total size.
But in this example they say to use a counter which doesn't seem practical as I am using a query on keyword when the user searches
That's correct, that solution doesn't fit your needs since it solves the problem of storing the number of all documents in a collection and not the number of documents that are returned by a query. As far as I know, it's just not scalable to provide that information, in the way this cloud hosted, NoSQL, realtime database needs to "massively scale".
For any future lurker, a "solution" to this problem is to paginate results with a cursor until the query doesn't return any more documents. When the query snapshot is empty, return undefined for your cursor and handle from there:
const LIMIT = 100
const offset = req.query.offset
const results = db.collection(COLLECTION)
.offset(offset)
.limit(LIMIT)
.get()
const docs = results.docs.map(doc => doc.data())
res.status(200).send({
data: docs,
// Return the next offset or undefined if no docs are returned anymore.
offset: docs.length > 0 ? offset + LIMIT : undefined
})