I've been working with Google Cloud Firestore. I'm about to import 13000+ records from a CSV to the firestore back-end. I'll be using this collection for look up and auto-completion purposes.
I'm curious and concerned to know if this is a good idea. Also, I'm looking for some suggestions on what techniques should I be using to make retrieval of this this data as efficient as possible. I'm working with Angular 5 and using AngularFire2 to connect with Firestore.
The document itself is really small such as:
{
address: {
state: "NSW"
street: "19 XYZ Road"
suburb: "Darling Point"
},
user: {
name: "ABC",
company: "Property Management Company"
}
file_no: "AB996"
}
Most of the searching would be based on file_no property of the document.
Update
I just imported all 13k+ records to Firestore. It is really efficient. However, I have one issue. After importing the records, I'm getting the message on my Firestore console that my daily limit for Read Operations is reached (0.05 of 0.05 Million Ops). I just wrote data and displayed those records in a Data Table. I used the following query:
this.propertyService
.getSnapshotChanges()
.subscribe(properties => {
this.properties = properties;
this.loadingIndicator = false;
});
getSnapshotChanges(): Observable < any > {
return this.afs.collection(this.propertiesCollection).snapshotChanges()
.map((actions) => {
return actions.map((snapshot) => {
const data = snapshot.payload.doc.data();
data.id = snapshot.payload.doc.id;
return data;
});
});
}
How dos this makes my reading limit exceed?
The number of documents in a collection is of no consequence when you use Cloud Firestore. That's actually one of its bigger perks: no matter how many documents are in a collection, the queries will take the same amount of time.
Say you add 130 document and (for sake of example) it takes 1 second to get 10 documents out of it. That's the performance you'll get no matter how many documents are in the collection. So with 1300 documents it will also take 1 second, with 13K it will take 1 second, and with 13M, it will also take 1 second.
The problem more developers run into is to make their use-cases fit within the API of Firestore. For example: the only way to search for strings is with a so-called prefix match, there is no support for full-text search. This means that you can search for Prop* and find Property Management Company, but not for *Man* to find it.
Related
I'm trying to figure out if there's a reasonable way of doing this:
My problem:
Exceeding my daily quota for reads in firestore pretty fast.
My database and what I do:
My database looks like this (simplified):
sessions: { // collection
sessionId: { // document
users: { // collection
userId: { // document
id: string
items: { // collection
itemId: trackObject
}
}
}
}
}
Now I want to retrieve from one session, all users and their items. Most sessions have 2-3 users but some users have around 3000 items. I basically want to retrieve an array like this:
[
{
userId,
items: [
...items
],
},
...users
]
How I go about it currently:
So I get all users:
const usersRef = db.collection(`sessions/${sessionId}/users`);
const userSnapshots = await usersRef.get();
const userDocs = userSnapshots.docs;
Then for each user I retrieve their items:
(I use a for-loop which can be discussed but anyhow)
const user = userDocs[i].data();
const itemsRef = usersRef.collection(`${user.id}/items`);
const itemSnapshots = await itemRef.get();
const items = itemSnapshots.docs
Finally I retrieve the actual items through a map:
user.items = items.map(doc => doc.data());
return user;
My theory:
So it looks like if I do this on a session where a user has 3000 items, the code will perform 3000 read operations on firestore. After just 17 runs I eat up my 50000 operations a day.
This reasoning is somewhat based on this answer.
My question:
Is there any other way of doing this? Like getting all tracks in one read-call? Should I see if I can fit all the items into an array-key in the user-object instead of storing as a collection? Is the free version of firestore simply not designed for this many documents being retrieved in one go?
If you're trying to reduce the number of document reads, you'll need to reduce the number of documents that you need to read to implement your use-case.
For example, it is fairly unlike that a user of your app will want to read the details of all 3000 items. So you might want to limit how many items you initially read, and load the additional items only on demand.
Also consider if each item needs to be its own document, or whether you could combine all items of a user into a single document. For example, if you never query the individual items, there is no need to store them as separate documents.
Another thing to consider if whether you can combine common items into a single document. An example of this is, even if you keep the items in a separate subcollection, to keep the names and ids of the most recent 30 items for a user in the user's document. This allows you to easily show a user and their 30 most recent items. Doing this you're essentially pre-rendering those 30 items of each user, significantly reducing the number of documents you need to read.
To learn more on data modeling considerations, see:
Cloud Firestore Payments
Going over read-quota in firebase firestore
the video series Getting to know Cloud Firestore, specifically What is a NoSQL Database? How is Cloud Firestore structured? and How to Structure Your Data
this article on NoSQL data modeling
This question already has answers here:
Firestore: Query by item in array of document
(5 answers)
Closed 4 years ago.
I have read this https://firebase.google.com/docs/firestore/solutions/arrays but I still couldn't figure out how it solves the issue of querying not-hardcoded user-generated items in array.
Let's say I have a Party collection:
Party {
name: "Birthday party",
invitees: [56448, 869987, 230232, 202030] # user id's of invited people
}
How can I query only parties where I'm in the invitees array without making an index for each possible ID?
https://firebase.google.com/docs/firestore/solutions/arrays
Limitations
The solution shown above is a great way to simulate array-like structures in Cloud Firestore, but you should be aware of the following limitations:
Indexing limits - A single document can have only 20,000 properties in order to use Cloud Firestore built-in indexes. If your array-like data structure grows to tens of thousands of members, you may run into this limit.
So the short answer is no.
If I were using Firebase Database, I would create a cloud function to watch the party/invitees branch on creation and modification, then propagate the invitation to a branch on the userID profile. Because you can be invited and uninvited, the changeset contains old and new and you can remove invitations from people who are in the previous but not in the next.
I haven't explored Cloud Functions for Firestore (yet) but I suspect that you can do something similar.
If you follow the method described in the documentation you point to (i.e. https://firebase.google.com/docs/firestore/solutions/arrays) and instead of storing the users in an array you save an object like
Party {
name: "Birthday party",
invitees: {
"56448": true,
"869987": true,
"230232": true,
.....
}
}
you will be able to query like
var yourUserID = .....; //e.g. '869987'
db.collection('invitees')
.where('invitees.' + yourUserID, '==', true)
.get()
.then(() => {
// ...
});
And as the doc says "This technique relies on the fact that Cloud Firestore creates built-in indexes for all document fields, even fields in a nested map.", so no worries for maintaining the indexes.
I want to get the total number of the documents that exist in firestore.
I don't want to get the data only the total number of inside Products collection I have 200.000 items is that possible with Angular 4-5, not angular.js
Can someone expert tell me how I can achieve that ??
My code so far and is not work
get_total_messages() {
this.messages_collection = this.afs.collection<MessageEntity>('messages');
return this.messages_collection.snapshotChanges();
}
End this is how I try to get the data but is not what I want;
this.firebase_Service.get_total_messages().subscribe( data => {
console.log(data);
});
There is no API to get the count of the number of documents in a Firestore collection. This means that the only ways to get the count are:
Get all documents and count them client-side.
Store the count as a separate property and update that as you add/remove documents.
Both approaches are quite common in NoSQL databases, with the second of course being a lot more efficient as the number of documents grows.
Firebase provides a sample of using Cloud Functions to keep a counter. While this sample is written for the Firebase Realtime Database, it can easily be modified to work on Cloud Firestore too.
Firestore also provides documentation on running aggregation queries and running distributed counters. Both seem slightly more involved than the first sample I linked though.
this.firebase_Service.get_total_messages().subscribe( data=>this.totalnumber=data.length);
//now, you can get total number of messages
luckily , i've solved somehow using the code,
try this, and it works well .
this.db.collection('User').valueChanges()
.subscribe( result => {
console.log(result.length);
})
In the documentation, it states there is a maximum write rate of 1write/sec to any single document. Is this limit shallow or deep?
For instance, suppose I have the following structure:
users<collection>/{uid}<doc> : {
name: "whatever",
likes<collection>: {
shard_0<doc>: {
like_count: 5
},
shard_1<doc>: {
like_count: 4
},
...
shard_n<doc>: {
like_count: 4
}
}
}
Lets say my max write rate to any single like shard document is 1 write/2 seconds. Clearly as I increase the number of like shard documents, I will never go over the write limit for any one of them, however if the limit is deep and I sum them up, I will certainly go over the max write rate for the {uid} document.
As long as likes in your example is a subcollection, you're actually following the pattern we recommend for distributed counters in Cloud Firestore. If you were trying to represent the shards in a field (e.g. a list) on a single document, that wouldn't work out for the reason you identify.
My Firebase database is more than 800mb large and with more than 100.000 objects (news articles).
What I want to do is to fetch just the first 50 objects (most recent) and then to sort the objects got from the result of the first query according to child parameters.
So, for example, when the page is loaded, I need angularfire / firebase to load just first 50 objects and to stop loading the rest of objects in database. Then, I want to filter out just these 50 objects (articles) based on node category music.
So far, my first query seems to be fine (but if there is better way to ask firebase to load X objects and to stop, I would appreciate). But, the second part, I can’t figure it out because firebase throw an error.
The error is:
Query: Can't combine startAt(), endAt(), and limit(). Use limitToFirst() or limitToLast() instead
Here is my sample code:
var myArticlesRef = new Firebase(FIREBASE_URL + 'articles/');
var latestArticlesRef = myArticlesRef.limitToFirst(20); // is this the recommended way to ask firebase to stop
var latestArticlesOrder = latestArticlesRef.orderByChild('category').equalTo(‘Music’); // <- how to do something similar?
var latestArticlesInfo = $firebaseArray(latestArticlesOrder);
$scope.latestArticles = latestArticlesInfo;
console.log($scope.latestArticles);
This should work:
var query = myArticlesRef.orderByChild('category').equalTo(‘Music’).limitToFirst(20);
So you're asking Firebase to return the first 20 articles in the Music category.
While it is common to think of queries like this when coming from a relational/SQL mindset, I recommend that you consider this alternative data structure:
articles_by_category
music
article1: { }
article2: { }
article3: { }
...
technology
article4: { }
...
So instead of storing the articles in one big list, store them by category. That way to access the articles about music, you only have to do:
var query = ref.child('articles_by_category').child('music').limitToFirst(20);
With this approach the database doesn't have to execute any query and it can scale to a much higher number of users.
This is something you'll see regularly in a NoSQL world: you end up modeling your data for the way your application wants to query it. For a great introduction, see this article on NoSQL data modeling. Also read the Firebase documentation on data modeling and indexes.