I'm trying to perform a filter by pattern over a Firestore collection. For exemple, in my Firestore database I have a brand called adidas. The user would have an search input, where typing "adi", "adid", "adida" or "adidas" returns the adidas document. I pointed out several solutions to do this :
1. Get all documents and perform a front-end filter
var brands = db.collection("brands");
filteredBrands = brands.filter((br) => br.name.includes("pattern"));
This solution is obviously not an option due to the Firestore pricing. Moreover it could be quite long to perform the request if the number of documents is high.
2. Use of Elasticsearch or Algolia
This could be interesting. However I think this is a bit overkill to add these solutions' support for only a pattern search, and also this can quickly become expensive.
3. Custom searchName field at object creation
So I had this solution : at document creation, create a field with an array of possible search patterns:
{
...
"name":"adidas",
"searchNames":[
"adi",
"adida",
"adidas"
],
...
}
so that the document could be accessed with :
filteredBrands = db.collection("brands").where("searchNames", "array-contains", "pattern");
So I had several questions:
What do you think about the pertinence and the efficiency of this 3rd solution? How far do you think this could be better than using a third party solution as Elasticsearch or Algolia?
Do you have any other idea for performing pattern filter over a firestore collection?
IMHO, the first solution is definitely not an option. Downloading an entire collection to search for fields client-side isn't practical at all and is also very costly.
The second option is the best option considering the fact that will help you enable full-text search in your entire Cloud Firestore database. It's up to you to decide if it is worth using it or not.
What do you think about the pertinence and the efficiency of this 3rd solution?
Regarding the third solution, it might work but it implies that you create an array of possible search patterns even if the brand name is very long. As I see in your schema, you are adding the possible search patterns starting from the 3rd letter, which means that if someone is searching for ad, no result will be found. The downside of this solution is the fact that if you have a brand named Asics Tiger and the user is searching for Tig or Tige, you'll end up having again no results.
Do you have any other ideas for performing pattern filters over a Firestore collection?
If you are interested to get results only from a single word and using as a pattern the staring letters of the brand, I recommend you a better solution which is using a query that looks like this:
var brands = db.collection("brands");
brands.orderBy("name").startAt(searchName).endAt(searchName + "\uf8ff")
In this case, a search like a or ad will work perfectly fine. Besides that, there will be no need to create any other arrays. So there will be less document writing.
I have also written an article called:
How to filter Firestore data cheaper?
That might also help.
Related
I am working on small app the allows users to browse items based on various filters they select in the view.
After looking though, the firebase documentation I realised that the sort of compound query that I'm trying to create is not possible since Firestore only supports a single "IN" operator per query. To get around this the docs says to use multiple separate queries and then merge the results on the client side.
https://firebase.google.com/docs/firestore/query-data/queries#query_limitations
Cloud Firestore provides limited support for logical OR queries. The in, and array-contains-any operators support a logical OR of up to 10 equality (==) or array-contains conditions on a single field. For other cases, create a separate query for each OR condition and merge the query results in your app.
I can see how this would work normally but what if I only wanted to show the user ten results per page. How would I implement pagination into this since I don't want to be sending lots of results back to the user each time?
My first thought would be to paginate each separate query and then merge them but then if I'm only getting a small sample back from the db I'm not sure how I would compare and merge them with the other queries on the client side.
Any help would be much appreciated since I'm hoping I don't have to move away from firestore and start over in an SQL db.
Say you want to show 10 results on a page. You will need to get 10 results for each of the subqueries, and then merge the results client-side. You will be overreading quite a bit of data, but that's unfortunately unavoidable in such an implementation.
The (preferred) alternative is usually to find a data model that allows you to implement the use-case with a single query. It is impossible to say generically how to do that, but it typically involves adding a field for the OR condition.
Say you want to get all results where either "fieldA" is "Red" or "fieldB" is "Blue". By adding a field "fieldA_is_Red_or_fieldB_is_Blue", you could then perform a single query on that field. This may seem horribly contrived in this example, but in many use-cases it is more reasonable and may be a good way to implement your OR use-case with a single query.
You could just create a complex where
Take a look at the where property in https://www.npmjs.com/package/firebase-firestore-helper
Disclaimer: I am the creator of this library. It helps to manipulate objects in Firebase Firestore (and adds Cache)
Enjoy!
I wanted to get some community consensus on how to achieve the following with the Firebase JS SDK (e.g., in React):
Suppose I have a collection users and I wanted to paginate users that do not have document IDs matching a subset of IDs (O(100-1000)). This subset of excluded IDs is dynamic based on the authenticated user.
It seems the not in query only supports up to 10 entries, so this is out of the question.
It also seems it's not possible to fetch all document IDs and filter on the client side, at least not in the 'firebase' JS SDK.
The only workaround I can think of is to have a document that keeps an array of all users document IDs, pull that document locally and perform the filtering/pagination logic locally. The limitation here is that a document can be at most 1MB, so realistically the single document can store at most O(10K) IDs.
Firestore has a special bunch of methods for pagination which may be useful for you. Those are called "query cursors".
You can use them to define the start point startAt() or startAfter() and to define an end point endAt() or endBefore(). Additionally, if needed, those can be combined with limit method.
I strongly encourage you to check this tutorial. Here you can find a quick video explaining the matter and lot of examples in all popular languages.
I'm working on a library app, and am using Firestore with the following (simplified) two collections books and wishes:
Book
- locationIds[] # Libraries where the book is in stock
Wish
- userId # User who has wishlisted a book
- bookId # Book that was wishlisted
The challenge: I would like to be able to make a query which gets a list of all Book IDs which have been wishlisted by a user AND are currently available in a library.
I can imagine two ways to solve this:
APPROACH 1
Copy the locationIds[] array to each Wish, containing the IDs of every location having a copy of that book.
My query would then be (pseudocode):
collection('wishes')
.where('userId' equals myUserId)
.where('locationIds' contains myLocationId)
But I expect my Wishes collection to be pretty large, and I don't like the idea of having to update the locationIds[] of all (maybe thousands) of wishes whenever a book's location changes.
APPROACH 2
Add a wishers[] array to each Book, containing the IDs of every user who has wishlisted it.
Then the query would look something like:
collection('books')
.where('locationIds' contains myLocationId)
.where('wishers' contains myUserId)
The problem with this is that the wishers array for a particular book may grow pretty huge (I'd like to support thousands of wishes on each book), and then this becomes a mess.
Help needed
In my opinion, neither of these approaches are ideal. If I had to pick one, I will probably go with Approach 1 simply because I don't want my Book object to contain such a huge array.
I'm sure I'm not the first person to come across this sort of problem, is there a better way?
You could try dividing the query in two different requests. For instance, in pseudocode:
wishes = db.collection('wishes').where('userId', '==', myUserId)
book_ids = [wish.bookId for wish in wishes]
books = db.collection('books').where('bookId', 'in', book_ids)
result = [book.bookId for book in books if book.locationIds]
Notice that this is just an example, this code probably doesn't work, since I haven't tested it and the keywork in just supports 10 values. But you get the idea. A good idea would be adding the length of the locationIds or whether it's empty or not in a separate attribute so you could omit the last iteration querying the books with:
books = db.collection('books').where('bookId', 'in', book_ids).where('hasLocations', '==', True)
Although you would still have to iterate to only get the bookId.
Also, you should avoid using arrays in Firestore since it doesn't have native support for them, as explained in their blog.
Is it mandatory to use NoSQL? Maybe you could do this M:M relation better in SQL. Bear in mind that I'm no database expert though.
I'm trying to add a feature in my app to search users by their name in Firebase, so I've tried to make a query that returns all the users where their name contains the search.
For example, if we have 5 users and their names are: "Bernard, Francois, Connard, Antoine, Penard" and the search is "nar", the query would return "Bernard, Connard, Penard". Does someone have an idea how to do the query? Or else a better idea to find the users?
Unfortunately, Firebase does not offer substring-match functionality. There are some hacks available with good thought on data structures and a bit of code work if you only want PREFIX based matches, but there is no operator for INFIX matches.
Your best option is to acknowledge that Firebase is an excellent database, but not an excellent search engine. There is nothing wrong with this - it does its actual job very well. When you need search, you will have the best results "pairing" it with an actual search engine such as ElasticSearch, Algolia, CloudSearch, etc.
I need to find all documents that contain a substring in any field,
i.e. documents:
{
a:"asd",
b:100,
c:
{
z:9
}
},
{
v:"asdfg",
p:"a100"
}
So when I search for "asd" or for "10", I need to get both documents.
I don't know what fields can be in these documents, so I cannot just search by each of them one by one.
Searching across all fields across all documents would mean having to scan through every DocumentDB, since this type of query doesn't take advantage of any indices.
I would suggest taking a look at a full text search product (e.g. Azure Search) rather than a database (e.g. DocumentDB) for this kind of scenario.
Can you elaborate on your use-case and what you're trying to accomplish? Feel free to ping me (andrl {at} microsoft) - I'd be more than happy to provide guidance.
Without knowing more about your use case, Azure Search (with suggestions enabled) seems to be more appropriate. Search suggestions is what allows azure search to deliver result hits based on substrings.