Search for data in Firestore DB: Queries [duplicate] - firebase

This question already has answers here:
Firestore Search - LIKE
(1 answer)
Google Firestore: Query on substring of a property value (text search)
(25 answers)
Flutter: Search through text data from Firestore
(2 answers)
Closed 2 months ago.
I'm trying to search the 'author_name' field in my firestore database that 'CONTAINS' any of the keywords
For example,
fields": {
"Title": {
"stringValue": "Fire in the gate"
},
"author_name": {
"stringValue": "Stacey"
},
"story": {
"stringValue": "fire was in the gate run run run"
}
}
in the given entry, if the term "Stace"`` or ``"Sta"`` or "cey" is entered it, should return the data that has the author_name field which CONTAINS the given word.
I'm trying to implement this using the where() clause in firebase which helps in building complex queries.
Is there any other way to implement this?
Note:
I'm using a low-code tool which mandates usage of REST API. I can't use any custom code in the app to implement this search. So, the search should be done on the firestore itself

Cloud Firestore doesn't natively support full-text search and instead recommends making use of a third-party indexing service that specializes in text search for NoSQL databases like Cloud Firestore.
These indexing services can be connected with Firebase using Cloud Functions for Firebase and each service recommended by the documentation has a pre-built extension that you can deploy to connect that service.
Note: Implementing full-text search will require the enabling of billing on your Firebase project and require connecting to services that also require billing. Most of the options listed below have free tiers which should cover light usage. At minimum, you will be charged a few cents a month by Google Cloud for hosting the containers that store your Cloud Functions. It is your responsibility to estimate your expected usage to determine the relevant billing costs and plan appropriately.
The recommended services (at the time of writing) include:
Algolia (setup guide, extension)
Elastic (setup guide, extension)
Typesense (setup guide, extension)
To deploy support for each service, you can either refer to their respective links above or view the code samples to define your own implementation.
Because you want to expose this search capability as an API, you will also want to deploy a HTTPS Cloud Function to handle the client request and forward it over to the indexing service you selected.

Related

Why does Google Firebase Cloud Firestore has "write", "update" security rules, while we should not allow users to write directly on database?

I'm implementing an application that using Google Firebase Cloud Firestore. Because my application has a lot of small write requests, it will be very costly if I use Firebase Cloud Functions in the middle. Therefore, I asked a question on Should we allow users to write to database directly. All responses said that I should not do so. Then, why does Google Firebase Cloud Firestore has "write", "update" security rules? Users should not write to the database anyhow.
EDIT (responding to DIGI's comment & answer):
From the answer, it seems like we can use the firebase rule instead. Then, how should we correctly use them?
For example, my app is recording user's location when they turn on the map & update it to the database so that user's friends can see it in real-time. How should I ensure that the data is in the form {longtitude: double, latitude: double, timeStamp: TimeStamp}, and the user doesn't change any other few in the document?
This is a common misconception from standard MySQL databases and similar where a server acts as a layer of logic and sits between the client and database.
Firebase still has this layer of logic in a simplified version known as security rules which allow basic read and write operations from the client. The design is by shifting computing power from the server backend onto the client instead, saving in server computing costs on their backend by distributing that requirement to users' devices.
The concerns listed in SE are from people who are unaware of the limitations and restrictions you can place inside rules and your project to control what can be requested with firebase and your app.
To clarify, Firebase does not hide your database keys, they are easily accessible and yes, a user can theoretically make a client using your database backend. but so long as rules are in place, conditions defined, Cors configuration, and app origin settings are enforced, you can prevent all of this.
For multiple writes, Realtime is ideal since it can handle data faster and more cost-effective than Firestore writes.
So my example will respond with the intent of realtime database per your edit.
{
“rules”: {
“location”: {
“$uid”: {
“.validate”: “newData.hasChildren(['longtitude', 'latitude', 'timestamp']) &&
newData.child('longtitude').isNumber() &&
newData.child('latitude').isNumber() &&
newData.child('timestamp').val() <= now”,
}
}
}
}
I highly suggest getting familiar with some core concepts
https://medium.com/#juliomacr/10-firebase-realtime-database-rule-templates-d4894a118a98
Full documentation here: https://firebase.google.com/docs/reference/security/database

Retrieving data from firebase by the dialog flow inline editor fulfillment

I am just start learning how to write fulfillment to read a data stored in the firebase as a first step.
Actually, I stored the same data in the firebase realtime database
and the firebase cloud database just to try from which one i must read my data, so I am just stuck how to retrieve it by the inline editor
My data is just a JSON object String names.
Note : form where i should start to learn Node.js for dialogflow fulfillment to do basic operation as storing and retrieving ?
You can use the firebase-admin library for node.js.
The inline editor is just Firebase Cloud Functions under the covers.
Your fulfillment code needs to run "somewhere" in the cloud - you'll need an HTTPS URL that you will provide, so this is called a "webhook". Firebase Cloud Functions are a convenient place to do this - they provide a HTTPS endpoint and good scalability, but you can run your fulfillment webhook anywhere on the public Internet.
Dialogflow provides an easy way to use Firebase Cloud Functions, by providing the inline code editor. This uses Firebase Cloud Functions to do the work, but hides the URL from you, so it is one fewer thing you need to deal with.
There are a number of good places to get started, but one is using Google's Codelabs for the Assistant make sure you also have looked at Google Docs for Actions on Google, which links to other resources as well.

Does Firestore in "Datastore mode" have any advantages "Native mode"

Google Cloud Firestore is going to replace the legacy Google Cloud Datastore soon. One then has the choice between using Cloud Firestore in "native mode" or in "datastore mode". The former allows access to Firestore through the usual Firestore SDK while the latter allows usage of the old Cloud Datastore SDK (which has no Web/Mobile APIs).
I am not yet familiar with Firestore. My question is: Apart from porting things to a new API, are there actually any things that can not be done with Firestore in "native mode" which could be done with the old Cloud Datastore (or its replacement: Firestore in "datastore mode")? Or any other advantage of using "datastore mode" (like costs, for example)?
If not, then it seems there is actually no advantage of using Firestore in "datastore mode" other than compatibility for older code using the old Cloud Datastore.
Am I right in my assumption that Firestore "datastore mode" has absolutely no advantage besides being able to use the legacy Datastore API (at the cost of not being able to use the newer and probably more feature-rich Firestore APIs, including mobile and web APIs)?
Cost.
Firestore in Datastore mode supports key-only and projection queries just like the original datastore. That means that the result set of those queries counts towards "Cloud Firestore Small Operations" which are free.
We accumulate billions of those small operations per day and not having projection queries would effectively 10-fold our datastore cost which would be unbearable.
As this feature is not available in Firestore native mode - and also with the strong consistency added in - we expected it to be less performant then in the original datastore, but in our tests this was not the case. Firestore in datastore mode consistently performed about twice as fast for our application across all types of operations.
There are advantages of using "datastore mode", even for a new project.
I am evaluating the two modes of firestore "Datastore mode" and "Native mode" for a migration project from MySQL.
On one hand, I consider using the "Datastore mode" for one back-end global repository because:
Server side only
Strong performance expectation on search queries across all entities
Query and sort on several properties altogether
Structured data model with one root kind and few second level kinds
A lot of write with limited transaction requirement, a huge number of read with projection within an entity group
On the other hand, the "Native mode" seems to fit some requirements for a user facing specific application because:
Web, iOS, Android, API interface with bi-directional sync
Several occasionally connected use cases
Few large polymorphic objects to sync and to persist
Mostly simple query on one properties (parent object)
Though, there is one reason which advocate for Datastore mode for the second project
Multi-tenancy with namespace
There are also common needs fulfilled by both mode, which support the decision to migration to NoSQL technologies
Scalability
No admin
Availability
Speed of development
The items 2 to 5 and 10 are based on features specific for the Datastore Mode, not possible in Native Mode. The items 6 to 9 are specific to Native Mode.
Update : March, 21st 2019
Six months after the evaluation described in my first answer, my team is using both Firestore (native) mode and Datastore mode.
2 projects based on Firestore. We are using a lot the concepts of collection, sub-collection and documents and the undelying segregation of data.
We also have implemented listeners in iOS and Android apps for sub-collections selected accordung to strong business and security rules, which is not possible with Datastore.
9 projects based on Datastore. For three 3 of them, we are are using a lot the concepts of namespace and kind. We also use the global indexing of kind's properties and the projection of properties server-side, which is not possible with Firestore.
PS: we are considering to open-source our python library for fast development of a common API running either on Firestore and on Datastore.
According to official documentation, although Cloud Firestore is backwards compatible with Cloud Datastore, the new data model, real-time updates, and mobile and web client library features are not.
Cloud Firestore in Datastore mode uses Cloud Datastore system behavior but accesses Cloud Firestore's storage layer, removing the following Cloud Datastore limitations:
Eventual consistency, all Cloud Datastore queries become strongly
consistent.
Transactions are no longer limited to 25 entity groups.
Writes to an entity group are no longer limited to 1 per second.
Datastore mode disables Cloud Firestore features that are not compatible with Cloud Datastore:
The project will accept Cloud Datastore API requests and deny Cloud Firestore API requests.
The project will use Cloud Datastore indexes instead of Cloud Firestore indexes.
You can use Cloud Datastore client libraries with this project but not Cloud Firestore client libraries.
Cloud Firestore real-time capabilities will not be available.
In the GCP console, the database will use the Cloud Datastore viewer.
Because Firestore in Datastore mode is optimized for server use cases and for App Engine, we recommend using Firestore in Datastore mode for databases that will be used primarily by App Engine apps. Firestore in Native mode is most useful for mobile and real-time notification use cases.
https://cloud.google.com/appengine/docs/flexible/go/using-cloud-datastore
When looking further into the docs after reading this post, I came across this official GCP page. It explains in plain English that Firestore "Datastore mode" recommended for primarily server-side use cases.
This answers my question perfectly, because my use case is only server-side. Note: I don't have any first hand experience with Firestore yet, but I appreciate the answers that do.

Google Firestore a subset or superset of Google Cloud Datastore?

Google announced Firestore, the new document datastore on the block.
I have been developing an application using Google Cloud Datastore for over six months now and after reading the blog, I feel Firestore seems to be a better choice.
The concept of the alternate collection-document-subcollection looks excellent to me because while designing schema for datastore I was aware I will be unable to query nested fields. Now with firestore subcollections, I get full query capabilities which is a game changer for me (I can get maximum data with minimum queries).
As a counter argument, the flowchart suggests me to use datastore because I do not have any mobile clients.
Will it be a good idea to use Firestore just like Datastore ?
(I will conveniently ignore the mobile client/realtime updates/syncing features!)
Update 2 (01/31/19)
As of today, Cloud Firestore is no longer in Beta and is Generally Available:
https://cloud.google.com/blog/products/databases/announcing-cloud-firestore-general-availability-and-updates
This means that Cloud Datastore is no longer an option for new projects (you can keep using it on existing projects). New projects that want to use the Datastore API can use Cloud Firestore in Datastore mode.
Update 1
As you we have noticed, we've expanded Cloud Firestore since this question was posted.
This means Cloud Firestore now has 2 modes:
The original launch was 'Native mode'
The new launch adds 'Datastore mode'
'Datastore mode' is the 3rd gen of Cloud Datastore. 1st was called Master/Slave Datastore, 2nd was High-Replication Datastore (HRD) that was rebranded as Cloud Datastore in 2013.
The below answer is still largely relevant since both modes are currently mutually exclusive, so you need to pick one or the other.
The main differences are the improves of Cloud Firestore in Datastore mode over Cloud Datastore. The biggest ones are:
Write through-put per entity group now unbounded (was 1 write/second)
Transactions no longer limited to 25 entity groups
All queries now strongly consistent.
Also note Cloud Firestore regardless of mode is beta, so the new Service-Level Agreement (SLA) doesn't go into effect until the product reaches General Availability (GA).
Original Answer
Cloud Datastore (CD) and Cloud Firestore (CF) are similar, however different in significant ways.
CF is mobile-centric with direct from mobile client functionality with the Firebase SDKs and Rules functionality. CD is server-centric with a wider range of server client libraries, as well as some mature frameworks on App Engine Standard that bundle in memcache functionality.
CF has a newer storage layer that is strongly consistency in the same way as Cloud Spanner, however, it's still in beta without an SLA. CD's storage layer is only strongly consistent within entity-groups and eventually consistent across entity-groups, however, it is GA with a 99.95% SLA for the Multi-Region locations.
CF is only available in the US Multi-Region at this time. CD is available Cloud across a dozen locations including places in the Americas, Europe, Asia, and Australia.
CF during beta has a guideline limit of 2500 writes/second while we build experience monitoring and tuning the system prior to GA, whereas CD will happily handle >1M writes/second (please reach out to your account rep first though).
CF and CD's set of query capabilities are overlapping but not the same. Overall CD has a broader set of query capabilities we haven't built in CF yet, so you'd have more flexibility in CD.
Overall, I'd consider this list to see if any of the differences make or break what you're trying to build then pick the DB that fits closest to your needs.
Firestore is the 3rd generation architecture and replacement for Datastore, essentially available in 2 modes: Native mode and Datastore mode.
Documentation regarding the choices: https://cloud.google.com/datastore/docs/firestore-or-datastore
Video overview: https://www.youtube.com/watch?v=SYG-BgXoJFQ
I'd say that Datastore is now a subset of Firestore:
Cloud Firestore is the next major version of Cloud Datastore and a re-branding of the product.
See Choosing between Cloud Firestore and Cloud Datastore
Cloud Firestore can operate in "Datastore mode", making it backwards- compatible with Cloud Datastore. Some time after Cloud Firestore is released for general availability, Google will begin contacting owners of existing Cloud Datastore databases to schedule an automatic upgrade to Cloud Firestore in Datastore mode. See auto upgrade
Google documentation says:
Firestore is the new version of Datastore and removes several
Datastore limitations.
I think cloud firestore also has nodejs client and its not centric to mobile. Actually thats the difference between Firebase realtime database which was mobile-centri vs Cloud Firestore which is anything centric.

MongoDB in Azure Cosmos DB

I was wondering if MongoDB is fully supported in Azure Cosmos DB through the MongoDB API https://learn.microsoft.com/es-es/azure/cosmos-db/mongodb-introduction
I have read that the aggregation pipeline, map-reduce and the full-text indexes is not fully integrated. Does anyone have further information about it? Would you use MongoDB in Azure Cosmos DB considering its current status?
Cosmos DB implements MongoDB wire protocol and many customers already use MongoDB API in production. Aggregation pipeline is in private preview and you can enable it by emailing askcosmosmongoapi#microsoft.com. Map-reduce functionality is mostly covered by aggregation pipiline. Full-text search is partially available through Azure Search, which can index MongoDB collections and $regex operator within MongoDB API covers less complex text search. You can find some other feature requests and their status at https://feedback.azure.com/forums/263030-azure-cosmos-db/category/321994-mongodb-api
Cosmos DB's MongoDB layer implements a large subset of native MongoDB functionality. Specifics of supported features are published here.
You mentioned aggregation pipeline: As of November 2017, this is now supported.
Regarding "current status" of the Cosmos DB MongoDB API: It's a production database with SLA. You'll need to make your own decision on whether to use it, based on feature set and your app's needs.
You can activate aggregation pipeline through Azure portal by going to Preview Features menu.

Resources