This is one part of my request. And I have 725 those kind of requests for each day in 2 year span.
I am getting analytics for 30 days traffic for certain dataset I am creating.
When I try to query the analytics for all 725 datasets I get quota error Requests per user per 100 seconds even though I put time.pause(2) before each request.
Is there something else I can do to avoid hitting the API quota?
{
"reportRequests":[
{
"viewId":"104649158",
"dateRanges":[
{
"startDate":"2017-12-01",
"endDate":"2017-12-31"
}
],
"metrics":[
{
"expression":"ga:pageviews"
},
{
"expression":"ga:uniquePageviews"
},
{
"expression":"ga:pageviewsPerSession"
},
{
"expression":"ga:timeOnPage"
},
{
"expression":"ga:avgTimeOnPage"
},
{
"expression":"ga:entrances"
},
{
"expression":"ga:entranceRate"
},
{
"expression":"ga:exitRate"
},
{
"expression":"ga:exits"
}
],
"dimensions":[
{
"name":"ga:pagePathLevel2"
}
],
"dimensionFilterClauses":[
{
"filters":[
{
"dimensionName":"ga:pagePathLevel2",
"operator":"REGEXP",
"expressions":[
"23708|23707|23706|23705|23704|23703|23702|23701|23700|23699|23698|23697|23696|23695|23694|23693|23692"
]
}
]
}
]
}
]
}
1) You should increase the user quota to 1000 requests (if not done already) by going into your Coogle Cloud Console -> Top-left Menu -> APIs & Services -> Analytics Reporting API -> Quota:
https://console.cloud.google.com/apis/api/analyticsreporting.googleapis.com/quotas
2) You could increase the time range and use the ga:yearMonth dimension to still get your monthly breakdown. However you might face sampling issues: since your query is "custom" (you use a filter + dimension), sampling will apply if for the given time range the total number of sessions at property level exceeds 500K (regardless of how many are actually included in the response). In this case there is no absolute answer, you have to find the time ranges that suit you best. samplesReadCounts / samplingSpaceSizes will help you detect sampling, and if required you will need to handle pagination.
While it is correct that you can request a quota increase. Which will increase the total number of requests that you can make this is still limited.
In the API Console, there is a similar quota referred to as Requests per 100 seconds per user. By default, it is set to 100 requests per 100 seconds per user and can be adjusted to a maximum value of 1,000. But the number of requests to the API is restricted to a maximum of 10 requests per second per user.
more info
requests per user per 100 seconds
Is a user based quota it is linked to the maximum of 10 requests per second per user quota (10 queries per second (QPS) per IP address). This is basically flood protection. It prevents a single user from making to many requests against the api and there by making it hard for the rest of use to use the the api.
What you need to understand first is that 100 requests per user per second is very subjective. When you run your request there is really no way for you to know what server your request will run on if your the only one running on that server then its possible you could kick off 100 request in 10 seconds and then be blocked for the next 90 seconds.
quotaUser
The second thing you need to know is that user based normally means ip based so if these request may be going against different views but if its all running from the same Ip address this can cause some confusion and it assumes you are the same user. To get around that you can use an alternate parameter called quota User which you can send a random string to this with every request and it can help it wont completely reduce it google tends to catch on to what you are doing eventually.
quotaUser An arbitrary string that uniquely identifies a user.
Lets you enforce per-user quotas from a server-side application even in cases when the user's IP address is unknown. This can occur, for example, with applications that run cron jobs on App Engine on a user's behalf.
You can choose any arbitrary string that uniquely identifies a user, but it is limited to 40 characters.
Learn more about Capping API usage.
Implementing exponential backoff
Google normally recommends that you implement something called exponential backoff this basically means that you try a request if it fails then you wait a few seconds and try again if that fails then you wait twice as long as you waited before and then try again you do this about 10 times and normally you are able to get though.
If you are using one of the official google client libraries most of them have exponential backoff already implemented
flood buster
I while ago i wrote an article about something i called flood buster its a way of keeping track of how fast i was going to try and prevent the user quota error the code is in C# you may find it useful flood buster
not really an issue
while getting these errors may be ugly it doesn't really matter you should just make the request again. Google does not count this error against you unless you do it constantly for hours at a time.
2000 requests per project 100 s
You need to remember that the number of requests your project can make In total per 100 seconds is 2000. This can not be increased.
So if you have two users each eating up 1000 requests per 100 seconds your going to hit the project based quota and there is nothing you can do about that. Allowing a single user to eat all your quota is IMO not a very good idea unless this is a single user application.
Related
I have stackdriver alerts/incidents on metrics like cloud run revision request latencies.
If there were a few calls a long time ago that had high latency, but there have not been any new requests since then which had a low latency, the incident will be permanently firing. This is because when there are no new requests coming in, there are no data points for the metric.
Is there a way to automatically stop an incident from firing when there are no recent data points for the underlying metrics? Or is there an alternative way to have alerts on high request latencies in cloud run that automatically switches off the alarm again when no new requests are coming that have a high latency?
The solution of https://stackoverflow.com/a/63997540/6473907 does not work as-is, because the google cloud run built-in metric for the request count does not go to zero when there are no more requests coming in. Instead, it just stops providing any data points. The solution for us was to create a custom logs-based metric that counts the log entries written for every request by cloud run, because the logs-based metric does indeed go to zero, then combine it with the AND_WITH_MATCHING_RESOURCE as described in https://stackoverflow.com/a/63997540/6473907
The chart compares the request count as obtained from the google pre-defined metric run.googleapis.com/request_count (in violet) with the metric generated by a custom logs-based metric (in blue). Only the latter goes to zero when no more requests are coming in.
Edit: This solution will not work because the request count stops being sent to Stackdriver instead of dropping to zero. As explained in the other (more correct) answer, the solution is to create a logs-based metric for the requests, and this will properly drop to zero when there are no additional requests.
This behaviour is documented in the alerting docs:
If measurements are missing (for example, if there are no HTTP
requests for a couple of minutes), the policy uses the last recorded
value to evaluate conditions.
There are a few recommendations in there to mitigate this issue, but all the suggestions assume you're actually collecting metrics, not your situation where there are no metrics at all (because you stopped receiving requests).
This is probably by design: even if you are not receiving additional requests, you might still want to check why all the latest requests had this increased latency.
To work around this feature, you could try to use multiple conditions in your alert policy:
One condition related to the latency: if latency > X
One condition related to the existence of requests: if request count > 1
If you combine those with AND_WITH_MATCHING_RESOURCE, it should only trigger if there's high latency and there are requests. The incident should be resolved when one of the 2 conditions are not met. Even if no new metrics are ingested related to the latency (so the alerting policy still thinks the latency is high), the request count will stop matching after the duration period specified.
When using https://www.linkedin.com/countserv/count/share?format=json&url= to access an article's sharecount, is there an api daily limit?
We noticed that the time it was taking to retrieve count data was taking as much as 20 seconds on our production server. We added logic to cache the number of counts, and the 20 second delay stopped the next day. We are left wondering though what the limit might be (we can't seem to find it in your documentation).
I wanted to know if there is a way in IIS to limit calls per user? Say A user can only make 100 calls per minute. If a user "foo" makes 100 calls in less than one minute they are stopped from making calls there on.
but if user "bar" is making < 100 calls per minute he should not be blocked
Any way to set that up in IIS?
It is the web application itself who should set such limits,
https://www.nuget.org/packages/MvcThrottle/
As far as I know no. But you can do bitrate throttling:
http://www.iis.net/downloads/microsoft/bit-rate-throttling
Maybe that will help.
We have a very active web page which has lots of ajax and regular updates via jquery. It can load a huge amount of data (< 100k per minute) every user in peak situations and we had 2,000 people online during the last peak.
What we would like to do is count the number of concurrent users. If over 500 (and not a registered user) then bad luck, hit the road!
Has anyone got a class or some other process? Our server recycles every hour so I am thinking of an application level variable that adds one to the current count if successful (gold users are exempt from the test but are added to the quota so we may have 600 users).
Has anyone else played with this idea?
TIA
Just some ideas...
application.lock()
application('visitors') = application('visitors') + 1
application.unlock()
You should stress-test this solutions up to the numbers you want to allow. It will probably work is my fair guess.
Consider counting the ajax url page instead, that gives a more accurate estimate of the load. When going for session's you will not know when I've left. Counting via the Ajax line gives a more accurate number of visitors.
Just suggestion: in GLOBAL.ASA on Session OnStart you could increase count of running Sessions in some global (Application) variable.
Do not forget to decrease it in GLOBAL.ASA on Session OnEnd
As I am not coming close to 100000 queries per day I am assuming that Google is referring to the Freebase 10 requests per second per user limit. (I am passing in my Goggle Key)
If i am running a query that crosses multiple Freebase domains is that considered more than one request? Or is a single query considered one request regardless of it size?
thanks
Scott
Yes, it sounds like you're exceeding the per/second rate limit. You'll need to introduce some delays in your application so that you don't exceed the limit. The rate limit only applies to HTTP requests so you can query as much data as you like as long as it fits in one request.