Every morning between 8 and 9 am CET my site shows this error:
Error Over Quota
This application is temporarily over its serving quota. Please try again later.
although billing is enabled and the site has been running for many months now. On the old billing status page, there's a setting for Maximum Daily Budget (Set this to handle peak traffic and to buffer against sudden traffic surges.) which is set to $0.00, but even if I change that to e.g. $10.00, it still shows the 503 error, so it seems this has nothing to do with it.
The new billing page looks like this:
It happens every morning between 7 and 8 am CET, so that would be around midnight PST which would at least indicate that it still might have something to do with billing?
Here's how the external monitoring system shows the outages, i.e. multiple outages every morning and then no problems for the rest of the day.
The Google Developers Console Overview page Errors by status code also shows the 503 error in green.
If I look at the monitoring logs most of the 503 errors between 7 and 9am for the following pages:
/
/wp-cron.php
And then there are some 500 errors, e.g. at 8 am
08:07:08.092 ... [26/Nov/2014:23:07:08 -0800] "GET / HTTP/1.1" 500 0 - "Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)" "www.coworking-radolfzell.de" ms=921 cpu_ms=1042 loading_request=1 exit_code=108 instance=00c61b117c9ad93b9da1f63314065ff8f4188095 app_engine_release=1.9.16
08:07:08.092 This request caused a new process to be started for your application, and thus caused your application code to be loaded for the first time. This request may thus take longer and use more CPU than a typical request for your application.
08:07:08.092 Process terminated due to exceeding quotas.
The (new) quota details page says Quotas are reset every 24 hours. Next reset: 1 hours and now it's 8:30 am CET. All of the listed resources are rated Okay.
If I go to the old Quota Detailsp page, though, I see that my requests' Frontend Instance Hourshave reached 100% and are rated Limited.
So I went back to the new console under Compute / App Engine / Settings which showed Your budget today is $10.00. Effective tomorrow, you will be using only free quota. which I now increased to a daily budget of USD 5.
My site now still shows the 503 error, but hopefully this will be the solution. I should be able to tell in 24 hours, shouldn't I?
If anyone from Google App Engine is reading this, there might be some inconsistencies between the old and new panels that should probably be fixed.
My problem has been solved by increasing the daily budget vom 0 to 5, although it took a couple of hours to take effect. We haven't had any notifications from the monitoring system, so it seems to be all fine now.
Related
Yesterday, about 1500 PST, my connection to the translation API began emitting errors about exceeding:
Error 8: RESOURCE EXHAUSTED: Quota Error
Quota exceeded for quota metric ‘v2 and v3 general model characters’ and limit ‘v2 and v3 general model characters per day’.
I have googled online, and found this can happen, and that it resets at 0000 hours PST. This time has come and gone, and second day now, the same Error 8 is arriving when attempting to access the service.
I have also signed in to my console, and looked at my quotas. None of them are over-quota, or have they been in the last 7 days. (I’m no where near the limits)
I had a spike (bug) where the code got into a bad state and spammed the API retrying the request, and I’ve found an fixed this issue. This was yesterday, so I don’t know why it’s still giving errors.
OK so I found the answer. My credit card had expired, and the billing was overdue. Unfortunately this is not obvious in any of the errors, or many of the pages the google search results point to (check quotas etc)…hopefully this helps someone else who searches.
Hi this was working for me a couple of weeks ago, however i keep getting this now.
Quota exceeded for quota group 'AnalyticsDefaultGroup' and limit 'CLIENT_PROJECT-1d'
error 429
As far as i am aware this should reset midnight pacific time. But it does not seem to have done that.
Even on the test account I get this error, I know I have resolved the issue that was causing it but I don't know why I still get this error.
Has it blacklisted the viewid? or IP? How can I get more info so I can resolve this issue.
I have been testing an application I am developing using the webapi, and I have started to get the following error message:
GCSP: Hello error: [1010] The Gracenote ODP 15822 [Name: *registered-name*]
[App: *registered-app*] application has reached is daily lookup limit with
Gracenote. You may try again tomorrow or may contact Gracenote support at
support#gracenote.com.
[Gracenote Error: <ERR>]
The application I am developing is looking up track details and cover artwork for songs being streamed from Mood/Pandora for Business service. It is making approximately one call for each song, so something like 15 searches per hour on average. I may have done more during testing, but not a lot more.
Once completed, I would expect this service to make fewer than 500 searches per day per location, and for it initially to be used at 4 locations.
What are the lookup limits I am running into?
What are my options to get a higher lookup limit?
Thanks
My app has been working fine since I changed my free account to a Pay-As-You-Go account but yesterday I started receiving the Access denied due to invalid subscription key error
I haven't changed anything. What can be the source of this error?
Microsoft CRIS servers were experiencing down time starting from ~ 10 pm GMT (according to my observation) on Wednesday 04/19 till 9 am GMT Thursday 14/20. I've experienced problems with log in, deployment access. But some of their servers went back and I was able to log in console. However all my deployments weren't accessible including newly created one. So, as their system went down, you might have problems accessing your deployment. Even if error description is related to invalid subscription key, it might be that issue. It's a new service and I still see 500 Internal errors in console often. If issue is still reproducing, I recommend you to contact their support team crservice#microsoft.com. They are very helpful. However they respond during European working hours.
I have experienced similar issues when we hit the API limits. It would not be a bad idea to check if either the number of requests per second or the volume has been reached ( in case of bootstrap keys it is 1000 hits)
I have an ASP.NET MVC website that gets about 6500 hits a day, on a shared hosting platform at Server Intellect. I keep seeing app restarts in the logs and I cannot figure out why.
I've read Scott Gu's article here: http://weblogs.asp.net/scottgu/archive/2005/12/14/433194.aspx
and implemented the technique, and here's what shows up in my log:
Application Shutdown:
_shutDownMessage=HostingEnvironment initiated shutdown
HostingEnvironment caused shutdown
_shutDownStack=at
System.Environment.GetStackTrace(Exception e, Boolean needFileInfo) at
System.Environment.get_StackTrace() at
System.Web.Hosting.HostingEnvironment.InitiateShutdownInternal() at
System.Web.Hosting.HostingEnvironment.InitiateShutdown() at
System.Web.Hosting.PipelineRuntime.StopProcessing()
It seems to occur about every five minutes.
Are there any other ways to debug this?
UPDATE: Here are the application pool settings mentioned by Softion:
CPU
Limit : 0
Limit Action : no action
Limit Interval : 5 Minutes
Process Model
Idle Timeout : 20 Minutes
Ping Maximum Response Time : 90 Seconds
Startup Time Limit : 90 Seconds
Rapid-Fail Protection
Enabled : True
Failure Interval : 5 Minutes
Recycling
Private Memory Limit : 100 MB
Regular Time Interval : 1740 Minutes (29 Hours)
Request Limit : 0
Specific Times : none
Virtual Memory Limit : 0
You can easily grab the reason of the shutdown by HostingEnvironment.
You read Scott Gu article, but you missed its comments.
var shutdownReason = HostingEnvironment.ShutdownReason;
If the reason is HostingEnvironment, check the IIS application pool parameters controlling recycling. I've put a red dot near each one. Check the description in the bottom help box in your own copy for full info.
You can ask your provider to give you the applicationHost.config file where all these parameters are set. They find it in C:\Windows\System32\inetsrv\config. I'm sure you can also get them using some .NET api.
For 6500 hits a day, which is a very low hit rate, i'm betting the "Idle time-out" is set to 5mn.
Update (moved comments to here //jgauffin)
CPU Limit 0 = disabled.
Process Model Idle Timeout : 20 Minutes (20mn without a request recycles your app).
Rapid-Fail Protection enabled (5mn). You need to know the maximum failures count. If your app throws more than this exception count in 5mn it will we recycled.
Private Memory Limit : 100 MB. Yes you should profile, this is a low limit.
Regular Time Interval : 1740 Minutes (29 Hours): it will recycle every 29h.
Request Limit : 0 (disabled).
Virtual Memory Limit : 0 (disabled).
Rapid-Fail Protection enabled (5mn). You need the maximum failures count. If your app throws more than this exception count in 5mn it recycles. If it recycles every 5mn this should be the thing to check. There should be 0 unhandled exception in secondary worker threads. Wrap your code into a try catch there.
re update:
The settings asked to the provider help, but is way better to ask for information on the reason of the restarts like I mentioned on my original answer i.e. the actual log entries of the restarts like I mentioned on my orig answer. From those you can know specifically what was triggered, I've seen happen one hitting different limits.
You really have to:
profile your application with a
realistic amount of test data
My money is on hitting resource limits set by your hosting provider.
Before going crazy with optimization without a target, contact your provider and ask them to give you information on the restarts.
Typical recycles:
idle x amount of time / like 15 mins
more than x amount of memory / like 200 MB
more than x % processor over y time / like 70 over 1 minute
a daily recycle
Once you know the case, you have to find out what's taking those resources. For this you have to profile your application with a realistic amount of test data. Knowing if it is memory or processor can help on knowing what to look for.
Is IIS set to recycle the app pool frequently?
Is there some kind of runaway memory leak in the app pool?
It requires a bit of know how on what your app does here's a list of things that can cause the app to restart/reset or even shut down
StackOverflowException
OutOfMemoryException
Any unhandled exception that crashes a thread
CodeContracts use Environment.FailFast when a contract violation occurs
Exceptions are quite easy to track if you can reproduce the issue with a debugger attached you can go into Visual Studio and enable all exceptions when they are thrown not caught by user code. It will sometimes reveal intresting stuff that otherwise is hidden away.