Right now I have a products collection where I store my products as documents like the following:
documentID:
title: STRING,
price: NUMBER,
images: ARRAY OF OBJECTS,
userImages: ARRAY OF OBJECTS,
thumbnail: STRING,
category: STRING
NOTE: My web app has approximately 1000 products.
I'm thinking about doing full text search on client side, while also saving on database reads, so I'm thinking about duplicating my data on Firestore and save a partial copy of all of my products into a single document to send that to the client so I can implement client full text search with that.
I would create the allProducts collection, with a single document with 1000 fields. Is this possible?
allProducts: collection
Contains a single document with the following fields:
Every field would contain a MAP (object) with product details.
document_1_ID: { // Same ID as the 'products' collection
title: STRING,
price: NUMBER,
category: STRING,
thumbnail
},
document_2_ID: {
title: STRING,
price: NUMBER,
category: STRING,
thumbnail
},
// AND SO ON...
NOTE: I would still keep the products collection intact.
QUESTION
Is it possible to have a single document with 1000 fields? What is the limit?
I'm looking into this, because since I'm performing client full text search, every user will need to have access to my whole database of products. And I don't want every user to read every single document that I have, because I imagine that the costs of that would not scale very well.
NOTE2: I know that the maximum size for a document is 1mb.
According to this document, in addition to the 1MB limit per document, there is a limit of index entries per document, which is 40,000. Because each field appears in 2 indexes (ascending and descending), the maximum number of fields is 20,000.
I made a Node.js program to test it and I can confirm that I can create 20,000 fields but I cannot create 20,001.
If you try to set more than 20,000 fields, you will get the exception:
INVALID_ARGUMENT: too many index entries for entity
// Setting 20001 here throws "INVALID_ARGUMENT: too many index entries for entity"
const indexPage = Array.from(Array(20000).keys()).reduce((acc, cur) => {
acc[`index-${cur}`] = cur;
return acc;
}, {});
await db.doc(`test/doc`).set(indexPage);
I would create the allProducts collection, with a single document with 1000 fields. Is this possible?
There isn't quite a fixed limitation for that. However, the documentation recommends having fewer than 100 fields per document:
Limit the number of fields per document: 100
So the problem isn't the fact that you duplicate data, the problem is that the documents have another limitation that you should care about. So you're also limited to how much data you can put into a document. According to the official documentation regarding usage and limits:
Maximum size for a document: 1 MiB (1,048,576 bytes)
As you can see, you are limited to 1 MiB total of data in a single document. When we are talking about storing text, you can store pretty much but as your documents get bigger, be careful about this limitation.
If you are storing a large amount of data in your documents and those documents should be updated by lots of admins, there is another limitation that you need to take care of. So you are limited to 1 write per second on every document. So if you have a situation in which the admins are trying to write/update products in that same document all at once, you might start to see some of these writes fail. So, be careful about this limitation too.
And the last limitation is for index entries per document. So if you decide to get over the first limitation, please note that the maximum limit is set to 40,000. Because each field has associated two indexes (ascending and descending), the max number of fields is 20,000.
Is it possible to have a single document with 1000 fields?
It is possible up to 40,000 properties but in your case with no benefits. I say that because every time you perform a query (get the document), only a single document will be returned. So there is no way you can implement a search algorithm in a single document and expect to get Product objects in return.
And I don't want every user to read every single document that I have, because I imagine that the costs of that would not scale very well.
Downloading an entire collection to search for fields client-side isn't practical at all and is also very costly. That's the reason why the official documentation recommends a third-party search service like Algolia.
For Android, please see my answer in the following post:
Is it possible to use Algolia query in FirestoreRecyclerOptions?
Firebase has a limit of 20k fields per document.
https://www.youtube.com/watch?v=o7d5Zeic63s
According to the documentation, there is no stated limit placed on the number of fields in a document. However, a document can only have up to 40,000 index entries, which will grow as documents contain more fields that are indexed by default.
I'm trying to figure out if there's a reasonable way of doing this:
My problem:
Exceeding my daily quota for reads in firestore pretty fast.
My database and what I do:
My database looks like this (simplified):
sessions: { // collection
sessionId: { // document
users: { // collection
userId: { // document
id: string
items: { // collection
itemId: trackObject
}
}
}
}
}
Now I want to retrieve from one session, all users and their items. Most sessions have 2-3 users but some users have around 3000 items. I basically want to retrieve an array like this:
[
{
userId,
items: [
...items
],
},
...users
]
How I go about it currently:
So I get all users:
const usersRef = db.collection(`sessions/${sessionId}/users`);
const userSnapshots = await usersRef.get();
const userDocs = userSnapshots.docs;
Then for each user I retrieve their items:
(I use a for-loop which can be discussed but anyhow)
const user = userDocs[i].data();
const itemsRef = usersRef.collection(`${user.id}/items`);
const itemSnapshots = await itemRef.get();
const items = itemSnapshots.docs
Finally I retrieve the actual items through a map:
user.items = items.map(doc => doc.data());
return user;
My theory:
So it looks like if I do this on a session where a user has 3000 items, the code will perform 3000 read operations on firestore. After just 17 runs I eat up my 50000 operations a day.
This reasoning is somewhat based on this answer.
My question:
Is there any other way of doing this? Like getting all tracks in one read-call? Should I see if I can fit all the items into an array-key in the user-object instead of storing as a collection? Is the free version of firestore simply not designed for this many documents being retrieved in one go?
If you're trying to reduce the number of document reads, you'll need to reduce the number of documents that you need to read to implement your use-case.
For example, it is fairly unlike that a user of your app will want to read the details of all 3000 items. So you might want to limit how many items you initially read, and load the additional items only on demand.
Also consider if each item needs to be its own document, or whether you could combine all items of a user into a single document. For example, if you never query the individual items, there is no need to store them as separate documents.
Another thing to consider if whether you can combine common items into a single document. An example of this is, even if you keep the items in a separate subcollection, to keep the names and ids of the most recent 30 items for a user in the user's document. This allows you to easily show a user and their 30 most recent items. Doing this you're essentially pre-rendering those 30 items of each user, significantly reducing the number of documents you need to read.
To learn more on data modeling considerations, see:
Cloud Firestore Payments
Going over read-quota in firebase firestore
the video series Getting to know Cloud Firestore, specifically What is a NoSQL Database? How is Cloud Firestore structured? and How to Structure Your Data
this article on NoSQL data modeling
I've been working with Google Cloud Firestore. I'm about to import 13000+ records from a CSV to the firestore back-end. I'll be using this collection for look up and auto-completion purposes.
I'm curious and concerned to know if this is a good idea. Also, I'm looking for some suggestions on what techniques should I be using to make retrieval of this this data as efficient as possible. I'm working with Angular 5 and using AngularFire2 to connect with Firestore.
The document itself is really small such as:
{
address: {
state: "NSW"
street: "19 XYZ Road"
suburb: "Darling Point"
},
user: {
name: "ABC",
company: "Property Management Company"
}
file_no: "AB996"
}
Most of the searching would be based on file_no property of the document.
Update
I just imported all 13k+ records to Firestore. It is really efficient. However, I have one issue. After importing the records, I'm getting the message on my Firestore console that my daily limit for Read Operations is reached (0.05 of 0.05 Million Ops). I just wrote data and displayed those records in a Data Table. I used the following query:
this.propertyService
.getSnapshotChanges()
.subscribe(properties => {
this.properties = properties;
this.loadingIndicator = false;
});
getSnapshotChanges(): Observable < any > {
return this.afs.collection(this.propertiesCollection).snapshotChanges()
.map((actions) => {
return actions.map((snapshot) => {
const data = snapshot.payload.doc.data();
data.id = snapshot.payload.doc.id;
return data;
});
});
}
How dos this makes my reading limit exceed?
The number of documents in a collection is of no consequence when you use Cloud Firestore. That's actually one of its bigger perks: no matter how many documents are in a collection, the queries will take the same amount of time.
Say you add 130 document and (for sake of example) it takes 1 second to get 10 documents out of it. That's the performance you'll get no matter how many documents are in the collection. So with 1300 documents it will also take 1 second, with 13K it will take 1 second, and with 13M, it will also take 1 second.
The problem more developers run into is to make their use-cases fit within the API of Firestore. For example: the only way to search for strings is with a so-called prefix match, there is no support for full-text search. This means that you can search for Prop* and find Property Management Company, but not for *Man* to find it.
I have this simple struct:
{
"items": {
"item1": {
"likes": 4
},
"item2": {
"likes": 5
}
}
}
I enable offline mode:
Firebase.getDefaultConfig().setPersistenceEnabled(true);
Multiple users can increase the "likes" fields at the same time. How can I increase the likes filed by 1 in offline mode? If I get the likes value than set it with the increased value, what happens it the device reaches the server but another users before that have increased the same likes field? Will it overwite the value?
I can imagine my data structure is not so good because I'm new to NoSQL. I can add Firebase userids to the likes fields instead of a counter integer and summarize them to get the result. But this means lots of unneccessary data beacuse I don't need to know who liked I just need the number of likes.
Please help me to find out the best solution! Thank you!
Considering the following structure of the Firebase database:
root
game1
$playerUidA
score: 50
$playerUidB
score: 10
.....
game2
$playerUidC
score: 20
$playerUidD
score: 30
.....
game3
.....
I want to run a query that will return all game nodes, where the children of each game node (the players) will be sorted based on the score. The game node contains the UID of each player, and each UID node contains the score of the player. I am storing other data as well, however, for the sake of this example I will be only using the score.
Can I do that with a single query? Something like
rootRef.child("root").orderByChild("score")? Unfortunately that doesn't seem to work.
Or the only way to achieve that is by manually sorting the items on the client?
#Puf - Hope you'll answer that :)
Although this question is rather old there might be people (like me) stumbling over it. Especially because it is pretty intuitive to structure the database in a similar way the author of the question did, to create (for example) a leaderboard system for a game. Since the answer is a bit outdated I wanted to add some things.
Some time ago the devs added the possibility to order by deeply nested children! (See Ordering by a specified child key) To do so you basically have to do the exact same thing the author did and additionally use the first part of the answer given by #adolfosrs. Basically you have to do two things:
Work with .indexOn (as described by #adolfosrs)
Use the OrderByChild() command.
To make this work on the example given by #steliosf you would have to do the following:
First set the .indexOn in your database rules:
{
"rules": {
"$gameId": {
".indexOn": "score",
"$playerUid": {
...
}
}
}
}
Second use the Command the author of the question already used:
rootRef.Child("root").OrderByChild("score")
I would recommend that you always add a LimitToFirst() or LimitToLast() command to avoid that you pull the whole database which might be a lot of data (depending on the size of your database of course). To get for example the top 10 scores you could use:
rootRef.Child("root").OrderByChild("score").LimitToLast(10)
Since the data is ordered in ascending order you need to use LimitToLast().
If you want all the games sorted by the player score all you need to do is to work with your .indexOn rule.
{
"rules": {
"$gameId": {
".indexOn": "score",
"$playerUid": {
...
}
}
}
}
This will keep your data sorted in the database so when you retrieve it you will have all the data ready. But keep in mind that if you want to retrieve all the games and all the players it means you will be fetching the whole database. So you should look at the needs of your application and maybe rethink the structure.
Something that could help when scaling is to iterate over the games and retrieve a limited amount of users with firebase.database().ref(gameId).limitToLast(10).
Update
For your example you will have all the games with the following request:
firebase.database().ref().once('value', snap => {
//prints all players for each game sorted ascending.
snap.forEach(game => {
game.forEach(player => {
console.log(player.val());
});
});
});