I've been recently making use of Flutter and Firebase to build some Networking Apps similar to the likes of LinkedIn. One of the App's features is its ability to match your profile with other users registered in the database based on factors such as location and the type of work you do. To do that I've been using Firestore snapshots with some conditions applied. Two of these conditions ask Firebase to sort the users by 'Last Active' and to only read documents of users that have signed in within the past month. In order to declutter the results and decrease the number of documents reads that are requested at a time. This is one example of a firebase snapshot I'm using:
Firestore.instance
.collection('usersIsb')
.where('Set', isEqualTo: true)
.where('Account Type', isEqualTo: _tempType)
.where('Services List', arrayContainsAny: _tempServices)
.where('Location', isEqualTo: widget.userData.location)
.where('Last Active', isGreaterThanOrEqualTo: widget.dateLimit)
.orderBy('Last Active', descending: true)
.limit(10)
.snapshots()
Naturally, because of the complexity of the request, Firebase asks me to create an Index for it, I've done that, and everything seems to be working correctly without any noticeable slowdowns or issues. However, I have a couple of questions that I'd like answered:
Are these indexes real-time? As in updated every time a new user document is created?
How many Indexes can I have in one Firestore database? The indexing process sounds intensive so I'm assuming there are drawbacks.
Is this how it's supposed to be done in the first place? It feels like I'm doing something wrong...
Extra: These are the Indexes I currently have enabled in my Firestore database.
This is my first post on the platform so feel free to ask for more information if needed, any advice on how to achieve the objective more efficiently is also appreciated.
Are these indexes real-time? As in updated every time a new user document is created?
Yes
How many Indexes can I have in one Firestore database?
The documentation on index limits says:
Maximum number of composite indexes for a database: 200
Maximum number of single-field index exemptions for a database: 200
You should read through the entire documentation there to understand all the limits.
Is this how it's supposed to be done in the first place?
That's a matter of opinion. You can do whatever you want within the documented limits. If your indexes and queries work for your use cases, then that's fine.
Related
I am aware it would be very difficult to query by a value that does not exist in an array but is there a way to do this without doing exactly that?
Here is my scenario - I have a subscription based service where people can opt in and "follow" a specific artist. In my backend, this creates a subscription doc with their followerId, the id of the artist they want to follow (called artistId), and an array called pushed. The artist can add new releases, and then send each follower a notification of a specific song in the future. I would like to keep track of which follower has been pushed which release, and this done in the aforementioned pushed array. I need a way to find which followers have already been pushed a specific release and so...
I was thinking of combining two queries but I am not sure if it is possible. Something like:
db.collection('subscriptions').where('artistId', '==', artistId)
db.collection('subscriptions').where('artistId', '==', artistId).where('pushed', 'array-contains', releaseId)
And then take the intersection of both query results and subtract from the 1st query to get the followers that have not been pushed a specific release.
Is this possible? Or is there a better way?
There is no way to query Firestore for documents that don't have a certain field or value. It's not "very difficult", but simply not possible. To learn more on why that is, see:
Firestore get documents where value not in array?
Firestore: how to perform a query with inequality / not equals
The Get to know Cloud Firestore episode on how Firestore queries work.
Your workaround is possible, and technically not even very complex. The only thing to keep in mind is that you'll need to load all artists. So the performance will be linear to the number of artists you have. This may be fine for your app at the moment, but it's something to definitely do some measurements on.
Typically my workaround is to track not what releases were pushed to a user, but when the last push was sent to a user. Say that a release has a "notifications sent timestamp" and each user has a "last received notifications timestamp". By comparing the two you can query for users who haven't received a notification about a specific release yet.
The exact data model for this will depend on your exact use-case, and you might need to track multiple timestamps for each user (e.g. for each artist they follow). But I find that in general I can come up with a reasonable solution based on this model.
For Elasticsearch case you need to sync with your database and elasticsearch server. And also need to make firewall rules at your Google Cloud Platform, you need keep away the arbitrarily request to your server, since it may cause bandwith cost.
The not-in operator is now available in Firestore!
citiesRef.where('country', 'not-in', ['USA', 'Japan']);
See the docs for a full list of examples:
https://firebase.google.com/docs/firestore/query-data/queries#in_not-in_and_array-contains-any
citiesRef.where('country', 'not-in', [['USA']]);
Notice the double array around [['USA']]. You need this to filter out any docs that have 'USA' in the 'country' array.
Single array ['USA'] assumes that 'country' is a string.
I have several question about how does firebase count read in my flutter app,
i have an app which use multiple firestore snapshot as stream like this:
_firestore
.collection(salesOrderPath)
.where("unit", isEqualTo: 1)
.snapshots()
_firestore
.collection(salesOrderPath)
.where("status", isEqualTo: 2)
.snapshots()
This two stream contain some same document, does that same document counted twice or once?
2.If i have multiple where filter on my firestore snapshot like this:
_firestore
.collection(salesOrderPath)
.where("unit", isEqualTo: 1)//10 Document
.where("status", isEqualTo: 2)//4 Document
.orderBy('creationDate',descending: true)
.snapshots()
Would i be charged by 10 read or just 4?
Maybe not related, but i saw something called Network egress as limit in the firebase pricing, what is this network egress meaning actually is it for storage or for firestore?
How long does cache from firestore in our app last before we need to reread it again from firestore?
I am new at this and dont quite understand a lot of thing, thankyou so much for answering
I think you should definitely watch video on Firebase Guide regarding pricing (actually I suggest to watch all of them).
I don't think you will find better description of pricing.
I guess question#1 is 2 reads as there are 2 queries. Question#2 anwser is 4 reads. Question#3 - if you watch the video you will know that this is rather negligible cost.
Now point 4#. Unfortunately I don't know, but I found something that might be interesting for you here.
Firestore bills you for your result set. So if your whole query returns 10 documents from a million documents. You only get billed for those 10 documents.
Network Egress is:
Network traffic that begins inside of a network and
proceeds through its routers to a destination somewhere outside of the
network
So if you ask for one document, you get billed for that 1 read plus the 'work done' for firestore to get that document for you.
The cache does not have an explicit expiry date. It will expire if it needs to make more space for new data or there was a deletion server-side and your cache now needs to resync. The size of the cache is 40MB.
Like others have mentioned I highly recommend their series on youtube
I am aware it would be very difficult to query by a value that does not exist in an array but is there a way to do this without doing exactly that?
Here is my scenario - I have a subscription based service where people can opt in and "follow" a specific artist. In my backend, this creates a subscription doc with their followerId, the id of the artist they want to follow (called artistId), and an array called pushed. The artist can add new releases, and then send each follower a notification of a specific song in the future. I would like to keep track of which follower has been pushed which release, and this done in the aforementioned pushed array. I need a way to find which followers have already been pushed a specific release and so...
I was thinking of combining two queries but I am not sure if it is possible. Something like:
db.collection('subscriptions').where('artistId', '==', artistId)
db.collection('subscriptions').where('artistId', '==', artistId).where('pushed', 'array-contains', releaseId)
And then take the intersection of both query results and subtract from the 1st query to get the followers that have not been pushed a specific release.
Is this possible? Or is there a better way?
There is no way to query Firestore for documents that don't have a certain field or value. It's not "very difficult", but simply not possible. To learn more on why that is, see:
Firestore get documents where value not in array?
Firestore: how to perform a query with inequality / not equals
The Get to know Cloud Firestore episode on how Firestore queries work.
Your workaround is possible, and technically not even very complex. The only thing to keep in mind is that you'll need to load all artists. So the performance will be linear to the number of artists you have. This may be fine for your app at the moment, but it's something to definitely do some measurements on.
Typically my workaround is to track not what releases were pushed to a user, but when the last push was sent to a user. Say that a release has a "notifications sent timestamp" and each user has a "last received notifications timestamp". By comparing the two you can query for users who haven't received a notification about a specific release yet.
The exact data model for this will depend on your exact use-case, and you might need to track multiple timestamps for each user (e.g. for each artist they follow). But I find that in general I can come up with a reasonable solution based on this model.
For Elasticsearch case you need to sync with your database and elasticsearch server. And also need to make firewall rules at your Google Cloud Platform, you need keep away the arbitrarily request to your server, since it may cause bandwith cost.
The not-in operator is now available in Firestore!
citiesRef.where('country', 'not-in', ['USA', 'Japan']);
See the docs for a full list of examples:
https://firebase.google.com/docs/firestore/query-data/queries#in_not-in_and_array-contains-any
citiesRef.where('country', 'not-in', [['USA']]);
Notice the double array around [['USA']]. You need this to filter out any docs that have 'USA' in the 'country' array.
Single array ['USA'] assumes that 'country' is a string.
I have a collection of conversations, each conversation having a hidden<Map> where each participant is the key, having a boolean value, so I can see if he archived the conversation on his end or not. Therefore, the query looks like this:
store.conversations
.where( 'participants', 'array-contains', uid )
.where( `hidden.${uid}`, '==', false )
.orderBy( 'createdAt', 'desc' )
Problem rises when adding orderBy, which makes it a "range" query. So, given each document has a different set of keys in the hidden<Map>, Firestore is suggesting the following, which obviously wouldn't work:
participants Arrays
hidden.`48m6lKjwvKUOboAxlc0ppX2R7qF2` Ascending
createdAt Descending
How do I go around this? I guess flattening the Map would be a solution but, not most elegant. Any advice?
Firestore is suggesting the following, which obviously wouldn't work:
participants Arrays
hidden.`48m6lKjwvKUOboAxlc0ppX2R7qF2` Ascending
createdAt Descending
You can create such an index and it will work but the problem rises if your app becomes popular and you'll have you'll have a big number of users. This means that for every conversation you'll have to create an index and this is not such a good idea because when it comes to indexes, there are some limitations. According to the official documentation regarding Firestore usage and limits:
Maximum number of composite indexes for a database: 200
Number that can be reached very quickly.
I guess flattening the Map would be a solution
You're guessing right. This practice is also called denormalization and is a common practice when it comes to Firebase. If you are new to NoQSL databases, I recommend you see this video, Denormalization is normal with the Firebase Database for a better understanding. It is for Firebase realtime database but same rules apply to Cloud Firestore.
Also, when you are duplicating data, there is one thing that need to keep in mind. In the same way you are adding data, you need to maintain it. With other words, if you want to update/detele an item, you need to do it in every place that it exists.
For more information please also see my answer from the following post:
What is denormalization in Firebase Cloud Firestore?
So you can denormalize your database and create conversations without the need of creating indexes. For your use-case, you should consider augmenting your data structure to allow a reverse lookup by creating a new collection or subcollection named userConversations that can hold as documents all the conversations that a user has. For a simple query, there is no index needed.
I have two Firestore collections, Users and Posts. Below are simplified examples of what the typical document in each contains.
*Note that the document IDs in the friends subcollection are equal to the document ID of the corresponding user documents. Optionally, I could also add a uid field to the friends documents and/or the Users documents. Also, there is a reason not relevant to this question that we have friends as a subcollection to each user, but if need-be we change it into a unified root-level Friends collection.
This setup makes it very easy to query for posts, sorted chronologically, by any given user by simply looking for Posts documents whose owner field is equal to the document reference of that user.
I achieve this in iOS/Swift with the following, though we are building this app for iOS, Android, and web.
guard let uid = Auth.auth().currentUser?.uid else {
print("No UID")
return
}
let firestoreUserRef = firestore.collection("Users").document(uid)
firestorePostsQuery = firestore.collection("Posts").whereField("owner", isEqualTo: firestoreUserRef).order(by: "timestamp", descending: true).limit(to: 25)
My question is how to query Posts documents that have owner values contained in the user's friends subcollection, sorted chronologically. In other words, how to get the posts belonging to the user's friends, sorted chronologically.
For a real-world example, consider Twitter, where a given user's feed is populated by all tweets that have an owner property whose value is contained in the user's following list, sorted chronologically.
Now, I know from the documentation that Firestore does not support logical OR queries, so I can't just chain all of the friends together. Even if I could, that doesn't really seem like an optimal approach for anyone with more than a small handful of friends.
The only option I can think of is to create a separate query for each friend. There are several problems with this, however. The first being the challenges presenting (in a smooth manner) the results from many asynchronous fetches. The second being that I can't merge the data into chronological order without re-sorting the set manually on the client every time one of the query snapshots is updated (i.e., real-time update).
Is it possible to build the query I am describing, or am I going to have to go this less-than optimal approach? This seems like a fairly common query use-case, so I'll be surprised if there is not a way to do this.
The sort chronologically is easy provided you are using a Unix timestamp, e.g. 1547608677790 using the .orderBy method. However, that leaves you with a potential mountain of queries to iterate through (one per friend).
So, I think you want to re-think the data store schema.
Take advantage of Cloud Functions for Firebase Triggers. When a new post is written, have a cloud function calculate who all should see it. Each user could have an array-type property containing all unread-posts, read-posts, etc.
Something like that would be fast and least taxing.