I have been working on an application that provides some web services to a client. The MVC application is built on .Net 4.5 framework. Everything worked fine until we looked at the response statistics our client had generated, where we could see we weren’t responding 100%. The statistics showed response drop to 90%-95% every other day.
Inspecting the logs
The first thing I did was to check our application log where I couldn’t find any exceptions logged. We log all (unhandled) exceptions in our application, which would normally lead to an error HTTP status code, so finding no such exception made me think that the request has not reached the application at all. The next step was to check the IIS access logs and there I could not find any 500 errors. I was under the impression that something is going on with the network which is causing 500's.
Upon further investigation, I realized those are being generated by IIS server. I found this by setting up Failed Request Tracing on IIS and after a while I found following error logged:
ModuleName - ManagedPipelineHandler Notification -
EXECUTE_REQUEST_HANDLER HttpStatus - 500 HttpReason - Error Retrieving
Data HttpSubStatus - 0 ErrorCode -The operation completed
successfully. (0x0)
I've looked at various stackoverflow posts but nothing really helped me. I guess it is because of no of requests we make per minute around 200 requests per minute). But I get response for 90-95% requests.
Attached detailed error log here
I figured out what the error is. .Net servers don't receive these requests in the first place, these are getting lost in the network. There is an oracle gateway which is buggy and killing few requests.
Related
We have been getting 503 Service Unavailable responses recently using the HERE API autosuggest endpoint. When it occurs, it can last for hours. Below is a typical request we get a 503 response from:
https://places.api.here.com/places/v1/autosuggest?q=Bradford&at=52.8,-1.4&addressFilter=countryCode=GB&result_types=address,place&app_code=########&app_id=###########
We have been using this service for a long while now and majority of the time the endpoint responds as expected. However, we are using it in a production environment, so any downtime (especially hours) is not good. Can we do anything when 503 occurs (E.g. calling a different endpoint)?
Thank you for any help or suggestions
Unfortunately no you cannot, outside of ensuring your application itself responds correctly to the error. You can also bookmark our status page (https://status.here.com/status) for more information when something goes wrong. I'm sorry the API was flakey for you recently, but hopefully this is just a temporary issue and doesn't return.
I have a couple Azure VMs behind a Basic Load Balancer with an HTTP URL Health Probe for the Backend Pool. To mark a server down, that URL returns Status Code 503 (Service Unavailable), but when I call that page from those VMs, the Status Code returned is 403. That still has the desired effect, I suppose, of marking the server down - but I dont understand why the code I set has changed.
This is from an ASP.NET web forms application on the VMs. I look at developer tools in the browser, and from my local machine or from a Dev server on our local network, that page returns Status Code 503, but calling that page from the VMs in Azure, the Status Code is 403.
Here's where I set the Status Code in that page:
Response.Clear()
Response.StatusCode = 503
Response.Flush()
I suppose I should mention that my local is a Windows 10 box, and the server VM is Windows Server 2016. Both are running IIS 10. The application is compiled with .NET Framework 4.6.
Here's the dev tools from my localhost:
Here's the dev tools from the server in azure:
Why the change? Anything I can do to stop this behavior?
So today I tried enabling Failed Request Tracing, but either something wasnt set up correctly, or the error was being handled elsewhere, and didnt result in any failed requests being logged.
Since I wasnt getting any failed requests logged, I opened up Process Monitor and could see that immediately after the call to my Health Probe page, I was getting a call to my custom HTTP Error page. That page must have been what was giving the 403 (dont know why, b/c that page works correctly for other HTTP Errors with a friendly error message and logging of the error to my custom error tracking solution).
I was going to change the Status Code to see if there was something special with the 503 that I was setting that was handled differently in IIS, but that got me thinking about how I was setting the status code...
In my research today, I saw this page https://www.leansentry.com/HowTo/AspNet-Response-Flush-Poor-Performance which cautions against using Response.Flush(). The code that I had implemented was in the Page_PreRender method, so there's not really a need to Flush there anyway.
I removed the Response.Flush and of course, my troubles went away.
The Health Probe page no longer triggers an Error from the Azure VM, and therefore, the status code that I get in my client browser is the 503 that I set in code.
So I guess this case is closed. I will need to figure out why the HTTP Error page was throwing a 403 instead of returning the friendly error message, but that should be easy enough...
We've recently had an issue where our application/website was displaying "The service is unavailable" message to users. Please note it wasn't the typical message where it displays the 503 error code that IIS typically generates...the only text on the page was "The service is unavailable" (it did not show any error codes or any additional text).
We stopped the application pool and started it back up, and the site began to work again. It was down for approximately 15 minutes.
We've gone through all of the log files and interestingly enough there was some traffic going to the server during the 15 minutes we were seeing the "The services is unavailable" message, without any 503 attempts being logged, no windows event errors, etc.
Does anyone have any ideas how to isolate the issue or find a log file that would display what happened?
I am trying to solve an issue with the Report Viewer control displaying the message:
The request failed with HTTP status 401: Unauthorized.
rather than a report. This message is shown in the location where the report would be displayed. The Share Point parts of the page are displayed normally.
This occurs in a Report Viewer control on a SharePoint 2013 page.
Some other reports on other pages show up just fine. In Report Server, the browsing permissions on these reports are the same.
A testing environment with the same configuration is working correctly. I could not find any differences in the web.config on the web server and in the RSReportServer.config file on the report server.
I http traffic from my browser, using the Network tab in the IE developer tools and saw something remarkable. POST Requests from javascript to the report server control are like:
/Reserved.ReportViewerWebControl.axd?OpType=SessionKeepAlive&ControlID=597f9e1be96e4291b847ce06da66940d
and receive a HTTP 500 status, whereas the response body is just two bytes:
OK
The report that works correctly, issues similar requests, but just receives HTTP 200 status, also with OK as response body.
On my testing environment, thing are different, again. The report issues three requests to ReportViewerWebControl.axd. The first get a HTTP 401 status. The next request adds an Authentication header with an NTLM token. This also gets an 401 status. The third adds another NTLM value and gets a 200 status.
This looks like some challenge mechanism and I found an MSDN article Error: The request failed with HTTP status 401: Unauthorized when you open a report in Microsoft Dynamics CRM 4.0 and tested with added to my RSReportServer.config file. This did not help and this key is also absent in my testing environment.
I also enabled FailedRequest logging to get more insight in the Internal server error, return by the requests to ReportViewerWebControl.axd. This logging is very elaborate, but I could not find anything pointing to a cause of this error.
Under request summary it reads
188. view trace
Warning -MODULE_SET_RESPONSE_ERROR_STATUS
ModuleName ManagedPipelineHandler
Notification EXECUTE_REQUEST_HANDLER
HttpStatus 500
HttpReason Internal Server Error
HttpSubStatus 0
ErrorCode The operation completed successfully. (0x0)
ConfigExceptionInfo
and under Authentication & Authorization
No. Event
60. view trace -AUTH_START AuthTypeSupported MapCliCert
61. view trace -AUTH_END
64. view trace -AUTH_START AuthTypeSupported Basic
65. view trace -AUTH_END
68. view trace -AUTH_START AuthTypeSupported NT
70. view trace -AUTH_END
91. view trace -FILTER_AUTH_COMPLETE_START
92. view trace -FILTER_AUTH_COMPLETE_END
I do not understand what all of that means, but it is the
ErrorCode The operation completed successfully.
looks like a contradiction.
Does anyone have suggestions on next steps to tackle this problem?
I have an api application which are givin me quite a headache. So in desperation I'm posting my problem here.
Short version
In short I need to know how I can debug my application to figure out what is causing the 500 Internal Server Error, I've tried to look into the Event log, but it doesn't tell me much besides which requests have failed, nothing about why they failed. So I honestly have no clue where to go from here.
The API is an ASP.Net MVC project and is hosted on an IIS 8.5 Windows Server 2012 R2
try sending a get request and reading the response. To get an increased level of error info, enable remote errors
https://technet.microsoft.com/en-us/library/bb684665.aspx