I have setup APIM (2.5.0)and Analytics. I have some users and few of the APIs created by the user.
For instance, one of the user onboarded one of the free API which is providing the weather information.
Another user subscribed it from store and started consuming it. At the end of the day, there were around 20 hits from the user to consume weather API.
But, among this 20 hits, 2 hits didnot fetch any result from the weather API URL (URL which we configure in Production and Sandbox URL of the publisher) as the connection was down but the as the APIM was running and allowed the user to consume, the logging continued and produced 20 hits for the user but technically it should be 18 hits only ( as 2 hits were not completed).
How to overcome this case, where the APIs are external and they onboarded in WSO2 APIM (using swagger or manually created) and logging is maintained so that TOTAL_REQUEST_COUNT should come out to be same of the positive hits/response.
Any information on this topic would be helpful.
Thanks
If you are maintaining TOTAL_REQUEST_COUNT it means the request you are getting which is 20 & it's correct. There should be another parameter you should have which will be TOTAL_RESPONSE_COUNT it will maintain the response count. So that you will get to know how many request are coming & how many got responded. Also if you want to see successful response count, have something `TOTAL_SUCCESS_RESPONSE_COUNT.
If you are having backend services in tomcat, then always make sure that thread count in WSO2 should be always greater than thread count of tomcat. WSO2 threads can be configured in <WSO2AM_HOME>/repository/conf/nhttp.properties. Look at last couple of lines in this file.
Also, you can change your retry policy by making changes into API publisher :9443/publisher by editing existing API.
Related
Is there a way to fetch (via iOS) the current number of Firestore API requests an app has used for the day / week / billing cycle?
I'm trying to see if I can fetch this value to display it to the user. I know some database services offer this as a service in order to determine how many requests are remaining.
There is no such client API available. You are always free to file a feature request.
I am running WSO2 APIM 2.1.0 , I am setting Tier Availability to Unlimited . Still getting 429 too many request error while doing load testing.
You need to make changes as per given in document.
If you have custom throttling policies, it has some issues with WSO2-2.1.0. Better to get upgraded to wso2-2.6.0.
There are many possible reasons for this kind of issue.
Are you trying throttling scenario on unauthenticated API? When you are going to add a new Subscription level throttling tier in the admin portal, you could see the existing list of subscription tiers in the Subscription Tier List. In this list, you will find a tier named Unauthenticated which has a request quota of 500. This is a subscription tier that automatically applied when the authentication type of your resources is 'None'. That is used when you can invoke APIs without tokens. And this tier is not visible in the Throttling tier list of the application.
Please check the Traffic Manager node connectivity.
Check whether any other throttling tier like application, API, resource level are published with the API.
Check the unit time while you are getting the throttled out response. Invoke the API after unit time getting passed. If the unit time is a larger value, then you should restart the server and try again.
Check the throttling configurations in the gateway, traffic manager, and the publisher.
Wanting to validate my ARM template was deployed ok and to get an understanding of the telemetry options...
Under what circumstances do the following get logged to Log Analytics?
DataPlaneRequests
MongoRequests
QueryRuntimeStatistics
Metrics
From what I can tell arduously in the last few days connecting in different ways.
DataPlaneRequests are logged for:
SQL API calls
Table API calls even when the account was setup for SQL API
Graph API calls against an account setup for Graph API
Table API calls against an account setup for Table API
MongoRequests are logged for:
Mongo requests even when the account was setup for SQL API
However I haven't been able to see anything for QueryRuntimeStastics (even when turning on PopulateQueryMetrics) nor have I seen any AzureMetrics appear?
Thanks Alex for spending time and trying out different options of logging for Azure Cosmos DB.
There are primarily two types of monitoring paths for Azure Cosmos DB.
Metrics: These are low latency (<5 min) and aggregated metrics which are exposed on Azure Monitor API for consumption. THese metrics are primarily used for diagnosis of the app for any live site issues.
Logs: These are raw request logs coming at 2hours+ latency and are used for customer for primarily audit scenarios to understand who accessed the data.
Depending on your need you can choose either of the approaches.
DataPlaneRequests by default shows all the requests across all the API's and Mongo Requests only show Mongo specific calls. Please note Mongo requests would also be seen in Data Plane requests.
Metrics would not be see in Log Analytics due to a knowwn which our partner team is fixing.
Let me know if you have any further questions here.
I'm using the Java client library's MetricServiceClient for getting StackDriver timeseries. I am authenticating using a user oauth token (this user has access to multiple projects), but there seems to be some kind of global quota across multiple projects because when I fetch only one or two projects at a time I have no throttling, but when I fetch four or five different projects at a time, I start getting throttled with errors like the following:
io.grpc.StatusRuntimeException: RESOURCE_EXHAUSTED: Insufficient tokens for quota 'DefaultGroup' and limit 'USER-100s' of service 'monitoring.googleapis.com' for consumer 'project_number:764086051850'.
I have confirmed this by alternating which projects are being fetched so that I can say it is not any single project -- they all start to get rate limited. Another strange thing is, that project_number in the error message doesn't correspond to any project I am fetching, or even have access to -- it is meaningless to me.
This appears to be the quota for # of requests per 100 seconds, but I have that set to 10,000 on all projects and I'm not doing nearly that many requests, as the quota historical chart in the web console confirms.
Is there really some global quota that applies across multiple projects and if so, is there some way to work around it? It is much simpler to me to have a single user with access to multiple projects instead of having to make service account tokens for them all.
The token quota you're hitting is for users using Application Default Credentials (which uses a shared gcloud project for billing), while it exists to get users up and run quickly but it's not recommended for actual production use. Therefore using a proper service account bound to the user's project is highly recommended and the solution to the issue.
I'm trying to get started implementing Web Push in one of my apps. In the examples I have found, the client's endpoint URL is generally stored in memory with a comment saying something like:
In production you would store this in your database...
Since only registered users of my app can/will get push notifications, my plan was to store the endpoint URL in the user's meta data in my database. So far, so good.
The problem comes when I want to allow the same user to receive notifications on multiple devices. In theory, I will just add a new endpoint to the database for each device the user subscribes with. However, in testing I have noticed that endpoints change with each subscription/unsubscription on the same device. So, if a user subscribes/unsubscribes several times in a row on the same device, I wind up with several endpoints saved for that user (all but one of which are bad).
From what I have read, there is no reliable way to be notified when a user unsubscribes or an endpoint is otherwise invalidated. So, how can I tell if I should remove an old endpoint before adding a new one?
What's to stop a user from effectively mounting a denial of service attack by filling my db with endpoints through repeated subscription/unsubscription?
That's more meant as a joke (I can obvioulsy limit the total endpoints for a given user), but the problem I see is that when it comes time to send a notification, I will blast notification services with hundreds of notifications for invalid endpoints.
I want the subscribe logic on my server to be:
Check if we already have an endpoint saved for this user/device combo
If not add it, if yes, update it
The problem is that I can't figure out how to reliably do #1.
I will just add a new endpoint to the database for each device the user subscribes with
The best approach is to have a table like this:
endpoint | user_id
add an unique constraint (or a primary key) on the endpoint: you don't want to associate the same browser to multiple users, because it's a mess (if an endpoint is already present but it has a different user_id, just update the user_id associated to it)
user_id is a foreign key that points to your users table
if a user subscribes/unsubscribes several times in a row on the same device, I wind up with several endpoints saved for that user (all but one of which are bad).
Yes, unfortunately the push API has a wild unsubscription mechanism and you have to deal with it.
The endpoints can expire or can be invalid (or even malicious, like android.chromlum.info). You need to detect failures (using the HTTP status code, timeouts, etc.) when you try to send the push message from your application server. Then, for some kind of failures (permanent failures, like expiration) you need to delete the endpoint.
What's to stop a user from effectively mounting a denial of service attack by filling my db with endpoints through repeated subscription/unsubscription?
As I described above, you need to properly delete the invalid endpoints, once you realize that they are expired or invalid. Basically they will produce at most one invalid request. Moreover, if you have high throughput, it takes only a few seconds for your server to make requests for thousands of endpoints.
My suggestions are based on a lot of experiments and thinking done when I was developing Pushpad.
Another way is to have a keep alive field on you server and have your service worker update it whenever it receives a push notification. Then regularly purge endpoints which haven't been responded to recently.