I want to know how wso2 api manager integrates solr; why the search logic relies on solr instead of the database, and sometimes the records obtained by getallApis are not consistent with the number of records in the database table AM_API.
As the documentation states WSO2 API Manager has Apache Solr based indexing for API documentation content. It provides both the API Publisher and Developer Portal a full-text search facility to search through the API documentation, and find the documents and related APIs.
I would recommend you to check out an excellent technical answer of #jpountz. Event though, the comparison made between postgresql vs lucene (underlying library is used by solr), it will give you some insight on why it's been chosen search engine (Lucene in the case wso2) over databases.
Related
Our ASP.NET Core app logs trace messages to App Insights. We need to be able to query them and filter by some customDimentions. However, I have found 3 APIs and am not sure which one to use:
App Insights REST API
Azure Log Analytics REST API
Azure Data Explorer .NET SDK (Preview)
Firstly, I don't understand the relationships between these options. I thought that App Insights persisted its data to Log Analytics; but if that's the case I would expect to only be able to query through Log Analytics.
Regardless, I just need to know which is the best to use and I wish that documentation were clearer. My instinct says to use the App Insights API, since we only need data from App Insights and not from other sources.
The difference between #1 and #2 is mostly historical and converging.
Application Insights existed as a product before log analytics, and were based on different underlying database technologies
Both Application Insights and Log Analytics converged to use the same underlying database, based on ADX (Azure Data Explorer), and the same exact REST API service to query either. So while your #1 and #2 links are different, they point to effectively the same service backend by the same team, but the pathing/semantics are subtly different where the service looks depending on the inbound request.
both AI and LA introduce the concept of multi-tenancy and a specific set of tables/schema on top of their azure resources. They effectively hide the entire database from you, and make it look like one giant database.
there is now the possibility (suggested) to even have your Application Insights data placed in a Log Analytics Workspace:
https://learn.microsoft.com/en-us/azure/azure-monitor/app/create-workspace-resource
this lets you put the data for multiple AI applications/components into the SAME log analytics workspace, to simplify query across different apps, etc
Think of ADX as any other kind of database offering. If you create an ADX cluster instance, you have to create database, manage schema, manage users, etc. AI and LA do all that for you. So in your question above, the third link to ADX SDK would be used to talk to an ADX cluster/database directly. I don't believe you can use it to directly talk to any AI/LA resources, but there are ways to enable an ADX cluster to query AI/LA data:
https://learn.microsoft.com/en-us/azure/data-explorer/query-monitor-data
And ways to have a LA/AI query also join with an ADX cluster using the adx keyword in your query:
https://learn.microsoft.com/en-us/azure/azure-monitor/logs/azure-monitor-data-explorer-proxy
My team uses CosmosDb to store data.
For our use case some of this data needs to be searchable.
Currently there are some filters in the Gremlin that has been implemented in CosmosDb so far, but not enough to suit our needs, which are mainly search in text.
This would be implemented to make a fuzzy search for a vertex, say, a person, where both name, email and company name would be included in the text.
In https://github.com/Azure/azure-documentdb-dotnet/issues/413 there was some talk of some string filters, but there has been no updates for a while.
My question is would it be better to use Azure Search for this use case?
We could add a step in the pipeline that would synchronize our data to an Azure Search service upon doing CRUD, but this would mean slower CRUD as well as data duplication, and the consumer of our api would have to use a search endpoint to get an id, and then do an additional lookup afterwards to get any related data.
If you can expose the data you want to make searchable to Azure Search using a "vanilla" (non-Gremlin) SQL API query, consider using Azure Search indexers for Cosmos DB. However, for simple string matching searches Azure Search may be an overkill - use it if you need more sophisticated searches (natural language-aware in many languages, custom tokenization, custom scoring, etc.).
If you need a tighter integration between Cosmos DB Graph API and Azure Search, vote for this UserVoice suggestion.
Firebase's realtime database SDK has callback methods built in which allow the client to listen for database changes. (e.g. as per https://firebase.google.com/docs/database/android/start/)
Does Azure have any functionality like that in either the SQL Database or DocumentDB products? If not, why not? And what is the best way to achieve this functionality using Azure?
Thanks.
Edit context: I'm building a cross-platform mobile app using Xamarin.
Microsoft just announced a new feature in DocDb called Change Feed which sounds like what you are looking for. It is enabled by default and included in the cost of DocDb.
From their announcement: "With Change Feed support, DocumentDB provides a sorted list of documents within a DocumentDB collection in the order in which they were modified. This feed can be used to listen for modifications to data within the collection and perform actions..."
Documentation on how to work with the change feed can be found here: https://learn.microsoft.com/en-us/azure/documentdb/documentdb-change-feed
Azure's SQL Database service now has Extended Events (currently in Preview). More info here.
DocumentDB does not have events you can subscribe to. You would need to build your own mechanism for taking action when modifying your database contents.
EDIT: Cosmos DB (formerly called DocumentDB, when this answer was originally posted) provides a change feed you can subscribe to.
As for "best way to achieve this functionality" - there is no best way. It's whatever you decide to build, based on your app, chosen frameworks, etc.
Google offers Datastore for the GCP which is the "C" in CQRS. But, where is the "Q" (Query)?
Datastore has, maybe justifiably, two very big limitations to making it a viable query system:
No subtext searching
No sorting on fields unless they've been filtered on
Google Cloud offers an "App Engine Search API", but this is not an endpoint API. It is a library API. Meaning, one must manage the infrastructure around it.
With Datastore, one of the advantages is that I can manage access via rules in a "server-less" way, so that my client app can directly deal with the Datastore (I.e "server-less").
With search API, I must write an application to index my documents using one of four supported languages, expose end points, manage scalability... Defeating the purpose of using server-less services like Datastore if I have to do all this manually anyway.
Does Google Cloud offer something more of a compliment to Datastore in the way of searching, filtering, ordering in a server-less way? Will they?
With Datastore, one of the advantages is that I can manage access via rules in a "server-less" way, so that my client app can directly deal with the Datastore (I.e "server-less").
I do not think Datastore's security model is powerful enough for direct remote client (e.g. web browser) access. Specifically even with IAM for Datastore it is missing record-level permissions. The Firebase Realtime Database with its security rules would seem more appropriate.
With search API, I must write an application to index my documents using one of four supported languages, expose end points, manage scalability
Similarly you can't do direct remote client access to the search API. However the Search API is designed to work with AppEngine, which manage individual machines/instances for you and implements autoscaling based on request rate and response latencies.
Perhaps to get a truely 'serverless' setup you could combine Firebase and Algolia as described here?
Is it possible expose and sell google cloud endpoint as an Api? I have created a simple but useful cloud endpoint. I want paid customers to access it directly as an api. How will I create a client-Id or API key dynamically for such clients, etc? For example, google also sells search service as API, where any user can go generate its own API key and Secret, and start using google search service.
Right now, no, or at least not without a lot of work.
The current product was designed with the "same party" use case as the primary goal (the API producer and consumer are the same). There are a number of things that would need to be added to the product to enable the kind of use you're describing. First and foremost on that list would be some kind of API consumer dashboard (like the one Google offers developers for consumers of its APIs).
Endpoints is built on the same API infrastructure as the rest of Google's APIs, and Google does offer this feature on some of its APIs. That may give you a sense of where the product is headed in future iterations.