I am looking for an approach to debugging this scenario. I have verified in Fiddler that there is no HTTP response at all. To be clear, as I understand it a controller method should not simply hang, there is no exception. I have verified the lack of response in Fiddler. The method returns a valid object, verified by stepping through the code to the final return statement.
This is different from the original question in that the controller method is hit, and was not before. The reason for this is explained in the original question. ASP.NET Web Api. Controller not hit. No response at all. Approaches to diagnose?
UPDATE
I am now seeing this behaviour, even though the request completes the handler and returns 200
ExtensionlessUrlHandler and "Recursion too deep; the stack overflowed"
1506. -GENERAL_REQUEST_END
BytesSent
6069
BytesReceived
436
HttpStatus
200
HttpSubStatus
0
From near the end
ErrorDescription
Internal Server Error
0 ms
Warning
1170. -MODULE_SET_RESPONSE_ERROR_STATUS
ModuleName
ManagedPipelineHandler
Notification
EXECUTE_REQUEST_HANDLER
HttpStatus
500
HttpReason
Internal Server Error
HttpSubStatus
0
ErrorCode
Recursion too deep; the stack overflowed.
(0x800703e9)
This turned out to be a crashed instance of RabbitMQ in combination with OWin middleware that was trying to use that instance (to log exceptions such as being unable to connect to the MQ instance; or rather attempting to log them by sending them to.. the MQ instance) and was thus swallowing exceptions in a recursive fashion. The stack overflow was caused by re-entering these middleware instances endlessly. The logging middleware was throwing exceptions because it could not log and the exception handling middleware was handling those exceptions by sending them to the logging middleware. Interesting stuff.
In addition to re-booting to cure the crashed and inaccessible RabbitMQ (restarting the service was not enough) the problem was still not resolved (different symptoms as described above) unless the nuget package MassTransit.RabbitMQ 3.3.2 (old version) and the dependencies (including RabbitMQ.Client) which this exact version brings with it, were installed, rather than the latest versions.
I hope this will help someone.
Related
I have built a fairly simple API, which is running on IIS on my local machine. It works fine. I then put an Ocelot gateway in front of it, which also works fine. I now have a Blazor app that is supposed to get its data from this API, but it is just failing. When this app runs using its own API it runs fine, so I know there is nothing wrong with my Blazor code. Here is the line that fails using the Ocelot gateway:
customers = await Http.GetFromJsonAsync<Customer[]>("http://gateway.oracle.local/customers");
If I just paste that URL into my browser, it returns the data fine.
Here is the error I am getting:
crit: Microsoft.AspNetCore.Components.WebAssembly.Rendering.WebAssemblyRenderer[100]
Unhandled exception rendering component: TypeError: Failed to fetch
System.Net.Http.HttpRequestException: TypeError: Failed to fetch
---> System.Runtime.InteropServices.JavaScript.JSException: TypeError: Failed to fetch
at System.Net.Http.BrowserHttpHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.BrowserHttpHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
at System.Net.Http.Json.HttpClientJsonExtensions.<GetFromJsonAsyncCore>d__13`1[[OracleWCON_V2.RODataAccess.Models.Customer[], OracleWCON_V2.RODataAccess, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null]].MoveNext()
at OracleWCON_V2.Client.Pages.Customers.ListCustomers.OnInitializedAsync() in D:\source\repos\OracleWCON_V2_220512\Client\Pages\Customers\ListCustomers.razor.cs:line 27
at Microsoft.AspNetCore.Components.ComponentBase.RunInitAndSetParametersAsync()
at Microsoft.AspNetCore.Components.RenderTree.Renderer.GetErrorHandledTask(Task taskToHandle, ComponentState owningComponentState)
I get exactly the same error if I use the URL that goes directly to the API without the Ocelot gateway. Just to re-iterate, if I use either of those URLs directly in my browser, I get the results I am expecting.
The fact it is a type error would lead me to think one of the fields in my Customer class was incorrect for the returned data, but I copied the class from the API, so I don't see how that could be true, but I stand to be corrected.
Any advice gratefully received.
This issue was caused because my API was running on HTTP, whilst my App was using HTTPS. This caused my browser to block the app as it had mixed content, some HTTP and some HTTPS. I still don't have this working due to certificate issues, but at least I got to the bottom of the original problem.
Well, this is an exceptional behavior, one needed for our use-case.
We are using handler interceptor to "increment active request count" at "preHandle" method. On "afterCompletion" method we decrement the active request counter. So far so good.
Sync calls are straight forward. But in case of async there is a 2nd request (DispatcherType: ASYNC) which is being used to decrement the counter while the main request (DispatcherType: REQUEST) is used to increment the counter. We check the dispatcher type to avoid double increment.So far so good.
The problem happening in case of some problematic clients, which disconnects after firing the request. In such a case while the main request enters the server but before the async thread starts the client disconnect itself (like close the browser).In this case the 2nd request (DispatcherType: ASYNC) not getting created at all.
This situation leaving the counter increased (by the main request).
For our use-case, we have to decrement the counter no matter what if it is incremented for a request.
Looking forward to your help/suggestion here. Thanks in advance.
Other details:
Spring Boot application
Spring framework: 4.3.4.RELEASE
Tomcat : 7
Using RestController
Handler Interceptor: AsyncHandlerInterceptor
In the asynchronous mode we use ResponseBodyEmitter to send data to client
Log in server:
Exception in thread "pool-87-thread-1" java.lang.IllegalStateException: The request associated with the **AsyncContext has already completed processing**.
at org.apache.catalina.core.AsyncContextImpl.check(AsyncContextImpl.java:497)
at org.apache.catalina.core.AsyncContextImpl.getRequest(AsyncContextImpl.java:209)
at org.apache.catalina.core.AsyncContextImpl.dispatch(AsyncContextImpl.java:198)
at org.apache.catalina.core.AsyncContextImpl.dispatch(AsyncContextImpl.java:170)
at org.apache.catalina.core.AsyncContextImpl.dispatch(AsyncContextImpl.java:164)
at org.springframework.web.context.request.async.StandardServletAsyncWebRequest.dispatch(StandardServletAsyncWebRequest.java:123)
at org.springframework.web.context.request.async.WebAsyncManager.setConcurrentResultAndDispatch(WebAsyncManager.java:353)
at org.springframework.web.context.request.async.WebAsyncManager.access$200(WebAsyncManager.java:58)
at org.springframework.web.context.request.async.WebAsyncManager$7.handleResult(WebAsyncManager.java:416)
at org.springframework.web.context.request.async.DeferredResult.setResultInternal(DeferredResult.java:199)
at org.springframework.web.context.request.async.DeferredResult.setErrorResult(DeferredResult.java:214)
at org.springframework.web.servlet.mvc.method.annotation.ResponseBodyEmitterReturnValueHandler$HttpMessageConvertingHandler.completeWithError(ResponseBodyEmitterReturnValueHandler.java:219)
at org.springframework.web.servlet.mvc.method.annotation.ResponseBodyEmitter.completeWithError(ResponseBodyEmitter.java:204)
at org.springframework.web.servlet.mvc.method.annotation.ResponseBodyEmitter.sendInternal(ResponseBodyEmitter.java:169)
at org.springframework.web.servlet.mvc.method.annotation.ResponseBodyEmitter.send(ResponseBodyEmitter.java:159)
at net.atpco.pipeline.common.post.KryoResponseEmitterPostBox.send(KryoResponseEmitterPostBox.java:48)
at net.atpco.pipeline.common.post.KryoResponseEmitterPostBox.lambda$0(KryoResponseEmitterPostBox.java:37)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Updates:
On further research I found this in spring documentation... "Note that HandlerInterceptor implementations may need to do work when an async request times out or completes with a network error. For such cases the Servlet container does not dispatch and therefore the postHandle and afterCompletion methods will not be invoked. Instead, interceptors can register to track an asynchronous request through the registerCallbackInterceptor and registerDeferredResultInterceptor methods on WebAsyncManager. This can be done proactively on every request from preHandle regardless of whether async request processing will start." This deferredResultInterceptor seems like solving the issue.
I have an ASP.NET Web API web service which throws a SerializationException in certain circumstances. The problem is that I'm unable to trap and log this exception server-side -- the only place it shows up is in the body of the HTTP response to the client.
I registered an ExceptionFilterAttribute as described in Exception Handling in ASP.NET Web API and verified that it works properly when I throw an exception within my controller. Unfortunately the SerializationException is being thrown during the response (after the controller) and appears to be completely swallowed up by ASP.NET. I also tried hooking Application_Error() in Global.asax.cs but it didn't show up there either.
How can I catch SerializationException exceptions during the Web API response?
If, instead of returning an object, you use the ApiController.CreateResponse() method and return a HttpResponseMessage you can then do response.Content.LoadIntoBufferAsync().Wait() and that will force the serialization to happen whilst you are still in the action and therefore can catch the exception.
BTW, Serialization of responses actually happens at the host layers(in HttpControllerHandler, when hosted in IIS and in HttpSelfhostServer, when hosted in SelfHost) which is way below the stack and not immediately after the response is returned from an action.
WebAPI Stack Poster: http://www.asp.net/posters/web-api/ASP.NET-Web-API-Poster-grayscale.pdf
That said, I am not able to come up with a straight forward way to achieve this. This is cumbersome, but may be override the default Xml and Json formatter's WriteToStreamAsync methods and try-catch-log any exceptions?
Alternatively, you can enable Web API Tracing which would log the exceptions happening during serialization. But yeah, if you do not know for the requests which cause the serialization errors, then you might want to enable tracing all the time which i am not sure is something you might want to do.
You can catch all Web API exceptions by registering an implementation of IExceptionHandler.
See Web API Global Error Handling
...there are a number of cases that exception filters can’t handle. For example:
Exceptions thrown from controller constructors.
Exceptions thrown from message handlers.
Exceptions thrown during routing.
Exceptions thrown during response content serialization .
One thing not mentioned in that article is that your IExceptionHandler must be registered, either by GlobalConfiguration.Configuration.Services.Add(...) or via an IoC container configured to be used by DependencyResolver.
We have an ASP.Net website that connects to an external web service, and it abruptly stopped working a few days ago.
The basic code is this:
Try
request = New ExternalWebService.ProcessRequestService
' Error occurs here:
response = request.processCommand(parameters)
' Do some other stuff with response here
Catch webEx As System.Net.WebException
' This triggers and generic error gets displayed
Catch ex As Exception
Finally
End Try
The System.Net.WebException gets triggered with a SendFailure exception, and returns this stack trace:
The underlying connection was closed: An unexpected error occurred on a send. -
at System.Web.Services.Protocols.WebClientProtocol.GetWebResponse(WebRequest request)
at System.Web.Services.Protocols.HttpWebClientProtocol.GetWebResponse(WebRequest request)
at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters)
at ExternalWebService.ProcessRequestService.processCommand(parameters)
When I try to debug this problem on my development machine, it works fine and no exception occurs.
There has been no recent updates to our webserver, and I can connect to the wsdl service from the web server with FireFox (I cannot connect with IE however, but that might be related to another issues we have with IE on the web server)
I've tried all the resolutions posted for Error Message #2 here, including setting KeepAlive = False and ServicePointManager.SecurityProtocol = SecurityProtocolType.Ssl3, however none have worked to fix the issue.
Our webserver is running Windows Server 2003 and IIS6.0
What can cause this error to occur on one machine, but not another?
Edit: Here's the Trace information:
System.Web.Services.Asmx Information: 0 : Calling XmlSerializer [Create XmlSerializer]
Method: System.Xml.Serialization.XmlSerializer#1669721556::FromMappings(System.Xml.Serialization.XmlMapping[]#4002219=[4], ExternalWebService.ProcessRequestService#93032588=ExternalWebService.ProcessRequestService)
Caller: System.Web.Services.Protocols.SoapClientType#42808772::.ctor(ExternalWebService.ProcessRequestService#93032588=ExternalWebService.ProcessRequestService)
ProcessId=2796
LogicalOperationStack=
ThreadId=14
DateTime=2012-12-21T17:02:39.2826250Z
Timestamp=61755143048799345
System.Web.Services.Asmx Information: 0 : Return from XmlSerializer [Create XmlSerializer]
Caller: System.Web.Services.Protocols.SoapClientType#42808772::.ctor(ExternalWebService.ProcessRequestService#93032588=ExternalWebService.ProcessRequestService)
ProcessId=2796
LogicalOperationStack=
ThreadId=14
DateTime=2012-12-21T17:02:39.6576250Z
Timestamp=61755144323396760
System.Web.Services.Asmx Information: 0 : Calling XmlSerializer [Write Request]
Method: Microsoft.Xml.Serialization.GeneratedAssembly.ArrayOfObjectSerializer#1231684::Serialize(System.Xml.XmlTextWriter#8023888=.., System.Object[]#19695157=[40], (null), (null))
Caller: ExternalWebService.ProcessRequestService#60553900::Serialize()
ProcessId=2796
LogicalOperationStack=
ThreadId=14
DateTime=2012-12-21T17:02:39.7357500Z
Timestamp=61755144598320131
System.Web.Services.Asmx Information: 0 : Return from XmlSerializer [Write Request]
Caller: ExternalWebService.ProcessRequestService#60553900::Serialize()
ProcessId=2796
LogicalOperationStack=
ThreadId=14
DateTime=2012-12-21T17:02:39.7357500Z
Timestamp=61755144621621529
System.Web.Services.Asmx Information: 0 : Calling WebRequest.GetResponse
Method: System.Net.HttpWebRequest#2800407::GetResponse()
Caller: ExternalWebService.ProcessRequestService#60553900::GetWebResponse()
ProcessId=2796
LogicalOperationStack=
ThreadId=14
DateTime=2012-12-21T17:02:39.7357500Z
Timestamp=61755144622114240
Apparently in my particular case, it was because the web service vendor had increased their security, and our web server didn't have the correct library needed to decipher the connection.
Our only clue was a single System event viewer entry with an ID of 36874 and a source of Schannel a few days back on the same day our site stopped working that said:
An SSL connection request was received from a remote client
application, but none of the cipher suites supported by the client
application are supported by the server. The SSL connection request
has failed.
However if you're debugging this problem yourself, this link was very useful since it gave some simplified descriptions of the error which pointed me in the right direction, that my problem was related to not being able to connect to their web service.
The underlying connection was closed: An unexpected error occurred on a send
This problem occurs when the client computer cannot send an HTTP
request. The client computer cannot send the HTTP request because the
connection has been closed or is unavailable. Lookup the error in
this article http://support.microsoft.com/kb/915599.
and the linked Microsoft support page was also fairly detailed about some different resolutions you could try to fix the most common causes of this problem.
Just an idea, don't know if it's the good one : the same thing happened few months ago on one of our server, because someone (security team ? :/) changed the write permission on the "%SYSTEMROOT%\Temp" folder. Since that action, XMLSerializer couldn't generate the temporary assembly used to serialize/deserialize.
From http://www.hanselman.com/blog/ChangingWhereXmlSerializerOutputsTemporaryAssemblies.aspx
:
When using the XmlSerializer from ASP.NET there are permissions issues
can can be solved by granting the user account read/write permissions
on the %SYSTEMROOT%\Temp folder
So perhaps you can check that the account used by your application pool has write permissions on the "%SYSTEMROOT%\Temp" folder...
Good luck...
I'm getting this error on my web service:
An error occurred while communicating with the remote host. The error code is 0x80070001
The weird thing is that the CPU and memory usage on the server is not high.
The detailed error message:
Error Message:
An error occurred while communicating with the remote host. The error code is 0x80070001.
Stack Trace:
at System.Web.Hosting.IIS7WorkerRequest.RaiseCommunicationError(Int32 result, Boolean throwOnDisconnect)<br />
at System.Web.Hosting.IIS7WorkerRequest.ReadEntityCoreSync(Byte[] buffer, Int32 offset, Int32 size)<br />
at System.Web.Hosting.IIS7WorkerRequest.ReadEntityBody(Byte[] buffer, Int32 size) <br />
at System.Web.HttpRequest.GetEntireRawContent()<br />
at System.Web.HttpRequest.get_InputStream()<br />
at Service.Upload() in c:\inetpub\path\Service.aspx.cs:line 213
After Google Search:
Thread on Microsoft Forums and I disable the offloading of network card and nothing change :(
Question on Stack OverFlow No answers :(
How can I fix this problem?
More information
Inner Exception Message: The handle is invalid. (Exception from HRESULT: 0x80070006 (E_HANDLE))
Very Important
This is only happened with a web application, but if I used a web site, nothing happened (no error).
And after a lot of research I think this error is caused by the web.config file.
I receive this error on my site also. I've determined the agent that generally causes this problem and it is a webcrawler like the one used by Google to index pages.
Anyway, I believe the answer is either Cause 1 or Cause 2:
Cause 1
This problem occurs because the client makes an HTTP request by using an HTTP method that does not comply with the HTTP specifications.
Cause 2
This problem occurs because a client makes an HTTP request by sending the POST method to a page that is configured to be handled by the StaticFile handler. For example, a client sends the POST method to a static HTML page. However, pages that are configured for the StaticFile handler do not support the POST method.
Check out the microsoft link for more explanation and possible resolution: http://support.microsoft.com/kb/942051
Close any object which is not in use (close when you are done using an object and initialize everything to null until you NEED to use it.)
This is because of an internal memory leakage in your code. As a result, it is not able to create a new object in memory.