Well, this is an exceptional behavior, one needed for our use-case.
We are using handler interceptor to "increment active request count" at "preHandle" method. On "afterCompletion" method we decrement the active request counter. So far so good.
Sync calls are straight forward. But in case of async there is a 2nd request (DispatcherType: ASYNC) which is being used to decrement the counter while the main request (DispatcherType: REQUEST) is used to increment the counter. We check the dispatcher type to avoid double increment.So far so good.
The problem happening in case of some problematic clients, which disconnects after firing the request. In such a case while the main request enters the server but before the async thread starts the client disconnect itself (like close the browser).In this case the 2nd request (DispatcherType: ASYNC) not getting created at all.
This situation leaving the counter increased (by the main request).
For our use-case, we have to decrement the counter no matter what if it is incremented for a request.
Looking forward to your help/suggestion here. Thanks in advance.
Other details:
Spring Boot application
Spring framework: 4.3.4.RELEASE
Tomcat : 7
Using RestController
Handler Interceptor: AsyncHandlerInterceptor
In the asynchronous mode we use ResponseBodyEmitter to send data to client
Log in server:
Exception in thread "pool-87-thread-1" java.lang.IllegalStateException: The request associated with the **AsyncContext has already completed processing**.
at org.apache.catalina.core.AsyncContextImpl.check(AsyncContextImpl.java:497)
at org.apache.catalina.core.AsyncContextImpl.getRequest(AsyncContextImpl.java:209)
at org.apache.catalina.core.AsyncContextImpl.dispatch(AsyncContextImpl.java:198)
at org.apache.catalina.core.AsyncContextImpl.dispatch(AsyncContextImpl.java:170)
at org.apache.catalina.core.AsyncContextImpl.dispatch(AsyncContextImpl.java:164)
at org.springframework.web.context.request.async.StandardServletAsyncWebRequest.dispatch(StandardServletAsyncWebRequest.java:123)
at org.springframework.web.context.request.async.WebAsyncManager.setConcurrentResultAndDispatch(WebAsyncManager.java:353)
at org.springframework.web.context.request.async.WebAsyncManager.access$200(WebAsyncManager.java:58)
at org.springframework.web.context.request.async.WebAsyncManager$7.handleResult(WebAsyncManager.java:416)
at org.springframework.web.context.request.async.DeferredResult.setResultInternal(DeferredResult.java:199)
at org.springframework.web.context.request.async.DeferredResult.setErrorResult(DeferredResult.java:214)
at org.springframework.web.servlet.mvc.method.annotation.ResponseBodyEmitterReturnValueHandler$HttpMessageConvertingHandler.completeWithError(ResponseBodyEmitterReturnValueHandler.java:219)
at org.springframework.web.servlet.mvc.method.annotation.ResponseBodyEmitter.completeWithError(ResponseBodyEmitter.java:204)
at org.springframework.web.servlet.mvc.method.annotation.ResponseBodyEmitter.sendInternal(ResponseBodyEmitter.java:169)
at org.springframework.web.servlet.mvc.method.annotation.ResponseBodyEmitter.send(ResponseBodyEmitter.java:159)
at net.atpco.pipeline.common.post.KryoResponseEmitterPostBox.send(KryoResponseEmitterPostBox.java:48)
at net.atpco.pipeline.common.post.KryoResponseEmitterPostBox.lambda$0(KryoResponseEmitterPostBox.java:37)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Updates:
On further research I found this in spring documentation... "Note that HandlerInterceptor implementations may need to do work when an async request times out or completes with a network error. For such cases the Servlet container does not dispatch and therefore the postHandle and afterCompletion methods will not be invoked. Instead, interceptors can register to track an asynchronous request through the registerCallbackInterceptor and registerDeferredResultInterceptor methods on WebAsyncManager. This can be done proactively on every request from preHandle regardless of whether async request processing will start." This deferredResultInterceptor seems like solving the issue.
Related
I have a ReST service I'm connecting to with a Jersey async client. I'm using InvocationCallback to handle responses as callbacks. All is well unless I bounce the server during heavy load from my client service. After a couple of bounces, I find that, beneath the covers, jersey client has registered internal callback handlers for messages that the server will never respond to (because it's been restarted and has lost any indication of an outstanding request).
I would have expected Jersey to (internally) poll its registered async handler list looking for "stale" handlers and cleaning them up with timeout exceptions to the registered invocation callback handlers periodically, but these unanswered handlers never get cleaned up.
What am I doing wrong? This has GOT to be a known issue in Jersey.
I'm building my client like this:
ClientBuilder builder = ClientBuilder.newBuilder();
this.client = builder
.executorService(es)
.register(JsonProcessingFeature.class)
.register(LoggingFeature.class)
.property(LoggingFeature.LOGGING_FEATURE_LOGGER_NAME_CLIENT, log.getName())
.connectTimeout(TIMEOUT_MILLIS, TimeUnit.MILLISECONDS)
.readTimeout(TIMEOUT_MILLIS, TimeUnit.MILLISECONDS)
.withConfig(new ClientConfig().connectorProvider(new JdkConnectorProvider()))
.build();
I'm struggling to find any good examples on how to implement error handling with Spring WebFlux.
The use case I want to handle is notifying HTTP clients that a stream has terminated unexpectedly. What I have found it that with the out of the box behaviour, when a stream is terminated, for example by raising a RuntimeException after x items have been processed, is handled too gracefully! The client is flushed all items up until the exception is raised, and then the connection is closed. As far as the client is concerned the request was successful. The following code shows how this has been setup:
public Mono<ServerResponse> getItems(ServerRequest request) {
Counter counter = new Counter(0);
return ServerResponse
.ok()
.contentType(MediaType.APPLICATION_STREAM_JSON)
.body(operations.find(query, Document.class, "myCollection")
.map(it -> {
counter.increment();
if(counter.getCount() > 500) {
throw new RuntimeException("an error has occurred");
}
return it;
}), Document.class);
}
What is the recommended way to handle the error and notify the HTTP client that the stream terminated unexpectedly?
It really depends on how you'd like to communicate that failure to the client. Should the client display some specific error message? Should the client reconnect automatically?
If this is a "business error" that doesn't prevent you from writing to the stream, you could communicate that failure using a specific event type (look at the Server Sent Events spec).
Spring WebFlux supports ServerSentEvent<T>, which allows you to control various fields such as event, id, comment and data (the actual data). Using an Flux::onErrorMap operator, you could write a specific ServerSentEvent that has an "error" event type (look at the ServerSentEvent.builder() for more).
But this is not transparent to the client, as you'd have to subscribe to specific events and change your JavaScript code otherwise you may display error messages as regular messages.
I am looking for an approach to debugging this scenario. I have verified in Fiddler that there is no HTTP response at all. To be clear, as I understand it a controller method should not simply hang, there is no exception. I have verified the lack of response in Fiddler. The method returns a valid object, verified by stepping through the code to the final return statement.
This is different from the original question in that the controller method is hit, and was not before. The reason for this is explained in the original question. ASP.NET Web Api. Controller not hit. No response at all. Approaches to diagnose?
UPDATE
I am now seeing this behaviour, even though the request completes the handler and returns 200
ExtensionlessUrlHandler and "Recursion too deep; the stack overflowed"
1506. -GENERAL_REQUEST_END
BytesSent
6069
BytesReceived
436
HttpStatus
200
HttpSubStatus
0
From near the end
ErrorDescription
Internal Server Error
0 ms
Warning
1170. -MODULE_SET_RESPONSE_ERROR_STATUS
ModuleName
ManagedPipelineHandler
Notification
EXECUTE_REQUEST_HANDLER
HttpStatus
500
HttpReason
Internal Server Error
HttpSubStatus
0
ErrorCode
Recursion too deep; the stack overflowed.
(0x800703e9)
This turned out to be a crashed instance of RabbitMQ in combination with OWin middleware that was trying to use that instance (to log exceptions such as being unable to connect to the MQ instance; or rather attempting to log them by sending them to.. the MQ instance) and was thus swallowing exceptions in a recursive fashion. The stack overflow was caused by re-entering these middleware instances endlessly. The logging middleware was throwing exceptions because it could not log and the exception handling middleware was handling those exceptions by sending them to the logging middleware. Interesting stuff.
In addition to re-booting to cure the crashed and inaccessible RabbitMQ (restarting the service was not enough) the problem was still not resolved (different symptoms as described above) unless the nuget package MassTransit.RabbitMQ 3.3.2 (old version) and the dependencies (including RabbitMQ.Client) which this exact version brings with it, were installed, rather than the latest versions.
I hope this will help someone.
I am new to Spring integration.
If I have a request coming in with batch payload(json array)
and I use splitter to split it into jsonobject,
and then I do validation.
If some of the validation failed and throw exception into error channel.
How can I make a response to client indicating some of the jsObject failed
and some works?
not sure handler at errorChannel gonna help since the validation result comes async into errorChannel.
And if I call the gateway like this, how can I construct a validation Result for the whole payload with validation status for each jsObject inside?
Future<validationResult> r = gateway.send(...)
(gateway just forward the request to following endpoint right away)
You have to take a look into Aggregator EIP: http://www.enterpriseintegrationpatterns.com/patterns/messaging/Aggregator.html, http://docs.spring.io/spring-integration/reference/html/messaging-routing-chapter.html#aggregator.
So, all your objects are send for the validation and their results (good or bad) send to the <aggregator> to build a single validationResult for the reply to that gateway.
I have an ASP.NET Web API web service which throws a SerializationException in certain circumstances. The problem is that I'm unable to trap and log this exception server-side -- the only place it shows up is in the body of the HTTP response to the client.
I registered an ExceptionFilterAttribute as described in Exception Handling in ASP.NET Web API and verified that it works properly when I throw an exception within my controller. Unfortunately the SerializationException is being thrown during the response (after the controller) and appears to be completely swallowed up by ASP.NET. I also tried hooking Application_Error() in Global.asax.cs but it didn't show up there either.
How can I catch SerializationException exceptions during the Web API response?
If, instead of returning an object, you use the ApiController.CreateResponse() method and return a HttpResponseMessage you can then do response.Content.LoadIntoBufferAsync().Wait() and that will force the serialization to happen whilst you are still in the action and therefore can catch the exception.
BTW, Serialization of responses actually happens at the host layers(in HttpControllerHandler, when hosted in IIS and in HttpSelfhostServer, when hosted in SelfHost) which is way below the stack and not immediately after the response is returned from an action.
WebAPI Stack Poster: http://www.asp.net/posters/web-api/ASP.NET-Web-API-Poster-grayscale.pdf
That said, I am not able to come up with a straight forward way to achieve this. This is cumbersome, but may be override the default Xml and Json formatter's WriteToStreamAsync methods and try-catch-log any exceptions?
Alternatively, you can enable Web API Tracing which would log the exceptions happening during serialization. But yeah, if you do not know for the requests which cause the serialization errors, then you might want to enable tracing all the time which i am not sure is something you might want to do.
You can catch all Web API exceptions by registering an implementation of IExceptionHandler.
See Web API Global Error Handling
...there are a number of cases that exception filters can’t handle. For example:
Exceptions thrown from controller constructors.
Exceptions thrown from message handlers.
Exceptions thrown during routing.
Exceptions thrown during response content serialization .
One thing not mentioned in that article is that your IExceptionHandler must be registered, either by GlobalConfiguration.Configuration.Services.Add(...) or via an IoC container configured to be used by DependencyResolver.