Our ASP.NET Web API 2 application is consumed by mobile applications. Using Azure Application Insights we detected many responses with response code 500(Internal server error) without any Exception associated with them in Application Insights. During debugging session we did not encounter any Exception thrown.
We had a suspicion that this could be caused by client disconnects. After implementing a .net application simulating disconnects, we could replicate this issue even on a clean webapi2 project.
After further investigation we found out, that the result code 500(after the client is disconnected) occurs only when specific conditions are met. The request need to be cancelled on a non GET http operation and before ExecuteRequestHandler asp.net event is reached. On GET requests we cannot replicate this issue, neither on requests which entered or passed the ExecuteRequestHandler event.
Our goal is to filter out client disconnects from logs and focus on real issues.
This issue may be related to ASP.NET Web API OperationCanceledException when browser cancels the request, however the accepted solution does not work, because the disconnect occurs before any DelegatingHandler is reached. We would not use the mentioned solution, because client disconnects is not a Server issue but rather a client. For example in netcore the cancelled requests have response code 0 in logs. Anyway, the client will not see the result nor the result code because it is already gone.
Other possibly related question.
We are using the latest version Microsoft.AspNet.WebApi v5.2.7 and the investigation take place only on a development machine with IISExpress.
UPDATE
Including the minimal code to reproduce on clean webapi2 project.
Global.asax.cs
public class WebApiApplication : System.Web.HttpApplication
{
protected void Application_Start()
{
GlobalConfiguration.Configure(WebApiConfig.Register);
}
protected void Application_EndRequest(object sender, EventArgs e)
{
var context = HttpContext.Current;
Debug.WriteLine($"{context.Response.StatusCode}:{context.Response.SubStatusCode}:{context.Request.Path}:{context.Request.Url.AbsoluteUri}");
}
}
HomeController.cs
[RoutePrefix("api/foo")]
public class HomeController : ApiController
{
[Route("bar")]
[HttpPut]
public async Task<IHttpActionResult> Put()
{
await Task.Delay(200);
return Ok();
}
}
PUT http://localhost:56952/api/foo/bar
On average 1 of 5 cancelled requests end with 500 when cancelled after 10ms. The response code is accessible from server logs Application Insights or Output Window in VS when logged. The client will not receive the response code, because the connection is closed before any response code is returned.
UPDATE 2
Telemetry from Application Insights
I am considering this as a bug in .Net Framework. I have written a workaround to end the currently executing request when the specified criterion are met:
Cancellation is requested from the client
Client is not connected
Handling should occur only before the ExecuteRequestHandler event
public class CancellationHttpModule : IHttpModule
{
public void Init(HttpApplication application)
{
application.OnExecuteRequestStep(OnExecuteRequestStep);
}
private void OnExecuteRequestStep(HttpContextBase context, Action step)
{
if (context.Response.StatusCode != 0
&& context.Response.ClientDisconnectedToken.IsCancellationRequested
&& !context.Response.IsClientConnected
&& (int)context.CurrentNotification < (int)RequestNotification.ExecuteRequestHandler)
{
context.Response.StatusCode = 0;
context.ApplicationInstance.CompleteRequest();
}
step();
}
public void Dispose()
{
}
}
After the handling takes places, status code is changed to zero(just like in .net core) and skipping all remaining events in ASP.NET pipeline to EndRequest event. I have tested this implementation on production for a month and did not find requests resulting into 500 without a corresponding Exception record in logs.
Related
I have a xamarin android app that makes requests to an api hosted on .net core (on IIS on Windows Server). Initial requests ALWAYS take a long time to load (presumably because of some warmup process). How do I ensure that the API is ready to go by the time the user needs to make a request?
Do I just make rapid async get/post requests on app startup? This seems in-efficient...
You need to use health check for your API:
public class ExampleHealthCheck : IHealthCheck
{
public ExampleHealthCheck()
{
// Use dependency injection (DI) to supply any required services to the
// "warmed up" check.
}
public Task<HealthCheckResult> CheckHealthAsync(
HealthCheckContext context,
CancellationToken cancellationToken = default(CancellationToken))
{
// Execute "warmed up" check logic here.
var healthCheckResultHealthy = true;
if (healthCheckResultHealthy)
{
return Task.FromResult(
HealthCheckResult.Healthy("The check indicates a healthy result."));
}
return Task.FromResult(
HealthCheckResult.Unhealthy("The check indicates an unhealthy result."));
}
}
Add your service to health check services:
public void ConfigureServices(IServiceCollection services)
{
services.AddHealthChecks()
.AddCheck<ExampleHealthCheck>("example_health_check");
}
In Startup.Configure, call UseHealthChecks in the processing pipeline with the endpoint URL:
public void Configure(IApplicationBuilder app, IHostingEnvironment env)
{
app.UseHealthChecks("/health");
}
Link to documentation: https://learn.microsoft.com/ru-ru/aspnet/core/host-and-deploy/health-checks?view=aspnetcore-2.2
I have a requirement to start a process on the server that may run for several minutes, so I was thinking of exposing the following hub method:-
public async Task Start()
{
await Task.Run(() => _myService.Start());
}
There would also be a Stop() method that allows a client to stop the running process, probably via a cancellation token. I've also omitted code that prevents it from being started if already running, error handling, etc.
Additionally, the long-running process will be collecting data which it needs to periodically broadcast back to the client(s), so I was wondering about using an event - something like this:-
public async Task Start()
{
_myService.AfterDataCollected += AfterDataCollectedHandler;
await Task.Run(() => _myService.Start());
_myService.AfterDataCollected -= AfterDataCollectedHandler;
}
private void AfterDataCollectedHandler(object sender, MyDataEventArgs e)
{
Clients.All.SendData(e.Data);
}
Is this an acceptable solution or is there a "better" way?
You don't need to use SignalR to start the work, you can use the applications already existing framework / design / API for this and only use SignalR for the pub sub part.
I did this for my current customers project, a user starts a work and all tabs belonging to that user is updated using signalr, I used a out sun library called SignalR.EventAggregatorProxy to abstract the domain from SignalR. Disclaimer : I'm the author of said library
http://andersmalmgren.com/2014/05/27/client-server-event-aggregation-with-signalr/
edit: Using the .NET client your code would look something like this
public class MyViewModel : IHandle<WorkProgress>
{
public MyViewModel(IEventAggregator eventAggregator)
{
eventAggregator.Subscribe(this);
}
public void Handle(WorkProgress message)
{
//Act on work progress
}
}
I've developed a sample SignalR application based on ASP.NET 4.5 & Owin, and I've hosted that app on IIS 7.5.
Everything is working fine, but how can I handle exceptions in Owin?
Consider the following code:
[HubName("SampleHub")]
public class SampleHub : Hub
{
public SampleHub()
{
throw new InvalidOperationException("?!");
}
}
This exception won't call Application_Error (and this is my problem).
Where can I get all exceptions from Owin for logging and debugging purposes similarly to Application_Error?
I'm not interested in something like this:
app.UseErrorPage(new ErrorPageOptions()
{
ShowCookies = true,
ShowEnvironment = true,
ShowExceptionDetails = true,
ShowHeaders = true,
ShowQuery = true,
ShowSourceCode = true
});
This is totally useless for advanced scenarios, something like ASP.NET Web API and ASP.NET MVC.
Action filters with OnException method for override purposes is much better.
If you want exception handling specifically for SignalR Hubs, OWIN middleware is not the way to go.
To illustrate just one reason why, suppose that SignalR is using its WebSocket transport when an exception is thrown from inside a Hub method. In this case, SignalR will not close the WebSocket connection. Instead SignalR will write a JSON encoded message directly to the socket to indicate to the client that an exception was thrown. There is no easy way using OWIN middleware to trigger any sort of event when this happens outside of possibly wrapping the entire OWIN WebSocket Extension which I would strongly advise against.
Fortunately SignalR provides its own Hub Pipeline which is perfectly suited for your scenario.
using System;
using System.Diagnostics;
using Microsoft.AspNet.SignalR.Hubs;
public class MyErrorModule : HubPipelineModule
{
protected override void OnIncomingError(ExceptionContext exceptionContext, IHubIncomingInvokerContext invokerContext)
{
MethodDescriptor method = invokerContext.MethodDescriptor;
Debug.WriteLine("{0}.{1}({2}) threw the following uncaught exception: {3}",
method.Hub.Name,
method.Name,
String.Join(", ", invokerContext.Args),
exceptionContext.Error);
}
}
You can use the ExceptionContext for more than just logging. For example you can set ExceptionContext.Error to a different exception which will change the exception the client receives.
You can even suppress the exception by setting ExceptionContext.Error to null or by setting ExceptonContext.Result. If you do this, It will appear to the client that the Hub method returned the value you found in ExceptonContext.Result instead of throwing.
A while back a wrote another SO answer about how you can call a single client callback for every exception thrown by a Hub method: SignalR exception logging?
There is also MSDN documentation for HubPipelineModules: http://msdn.microsoft.com/en-us/library/microsoft.aspnet.signalr.hubs.hubpipelinemodule(v=vs.118).aspx
The answer by #halter73 is great for errors thrown inside hubs, but it doesn't catch errors thrown during their creation.
I was getting the exception:
System.InvalidOperationException: 'foobarhub' Hub could not be resolved.
The server was returning an HTML page for this exception, but I needed it in JSON format for better integration with my Angular app, so based on this answer I implemented an OwinMiddleware to catch exceptions and change the output format. You could use this for logging errors instead.
public class GlobalExceptionMiddleware : OwinMiddleware
{
public GlobalExceptionMiddleware(OwinMiddleware next)
: base(next)
{
}
public override async Task Invoke(IOwinContext context)
{
try
{
await Next.Invoke(context);
}
catch (Exception ex)
{
context.Response.ContentType = "application/json";
context.Response.StatusCode = 500;
await context.Response.WriteAsync(JsonConvert.SerializeObject(ex));
}
}
}
Add the registration in OwinStartup.cs, just remember to place it before the MapSignalR method call:
public class OwinStartup
{
public void Configuration(IAppBuilder app)
{
app.Use<GlobalExceptionMiddleware>(); // must come before MapSignalR()
app.MapSignalR();
}
}
Our ASP.NET app sends multiple Ajax requests in parallel on the same session. We also read/write to HttpSessionState during some of those requests. What I WANT is for all the concurrent requests to execute in parallel for performance reasons. What I get is they are serialized by ASP.NET. I experimented with configuring enableSessionState="ReadOnly", but this breaks our Forms Authentication.
Is there a way to get both session state AND concurrency in one session? Do I need to use a custom SessionState or Provider? Any samples of this out there?
PS I'm not worried about thread safety when accessing the SessionState - I can do that programmatically.
From MSDN (link):
However, if two concurrent requests are made for the same session (by using the same SessionID value), the first request gets exclusive access to the session information. The second request executes only after the first request is finished.
So at least for those AJAX calls that require write access to the Session you are out of luck with the default providers.
Not sure if you could get around this using a custom provider.
You can achieve parallell execution for those AJAX calls that do not need access to the session by blocking the ASP.NET_SessionId cookie in an HttpModule. See my answer to this question.
Edit: In order to make this answer more self reliant, I add a slightly modified version of the HttpModule and a bit of discussion (further down). Here's the module code that you can use to prevent Session state from serializing your Ajax calls:
using System;
using System.Web;
namespace TestModule
{
public class TestPreventCookie : IHttpModule
{
public void Dispose()
{
}
public void Init(HttpApplication application)
{
application.BeginRequest +=
(new EventHandler(this.Application_BeginRequest));
application.PostAcquireRequestState +=
(new EventHandler(this.Application_PostAcquireRequestState));
}
private void Application_BeginRequest(Object source, EventArgs e)
{
//prevent session cookie from reaching the service
HttpApplication application = (HttpApplication)source;
HttpContext context = application.Context;
if (BlockCookie(context))
{
context.Request.Cookies.Remove("ASP.NET_SessionId");
}
}
private void Application_PostAcquireRequestState(Object source, EventArgs e)
{
HttpApplication application = (HttpApplication)source;
HttpContext context = application.Context;
if (BlockCookie(context))
{
var s = context.Session;
if (s != null)
s.Abandon();
}
}
private bool BlockCookie(HttpContext context)
{
// put code here that determines if the session cookie should be blocked
// this could be based on the path of the current request for instance
// only block the cookie when you *know* that access to the Session is not needed
}
}
}
The idea behind this module is that using some criteria based on the project requirements, we remove the ASP.NET_SessionId cookie from the current context (note, we don't expire it on the client).
This means that further on in the request pipeline, the server will create a new session. In order to prevent this newly created session from destroying the existing ASP.NET_SessionId cookie on the client, we abandon it immediately after it has been created.
The end result is that each request that is "intercepted" by the module will execute as if it had no session.
I am developing custom HTTP server with netty 3.3.1.
I need to implement something like this
HTTP Server receives request
HTTP Server parses it and invokes HTTP request as a client to other machine
HTTP Server waits for the response of request sent in (2)
HTTP Server sends response to request from (1) based on what had received in (3)
It means that client request (2) has to behave as synchronous.
What I wrote is based on HttpSnoopClient example but it does not work, because I receive
java.lang.IllegalStateException:
await*() in I/O thread causes a dead lock or sudden performance drop. Use addListener() instead or call await*() from a different thread.
I've refactored the code from the example mentioned above and now it looks more less like this (starting from line 7f of HttpSnoopClient):
ChannelFuture future = bootstrap.connect(new InetSocketAddress(host, port));
future.addListener(new ChannelFutureListener() {
public void operationComplete(ChannelFuture future) {
if (!future.isSuccess()) {
System.err.println("Cannot connect");
future.getCause().printStackTrace();
bootstrap.releaseExternalResources();
return;
}
System.err.println("Connected");
Channel channel = future.getChannel();
// Send the HTTP request.
channel.write(request);
channel.close();
// Wait for the server to close the connection.
channel.getCloseFuture().addListener(new ChannelFutureListener() {
public void operationComplete(ChannelFuture future) {
System.err.println("Disconnected");
bootstrap.releaseExternalResources(); // DOES NOT WORK?
}
});
}
});
}
}
The run() command from the above example is invoked in the messageReceived function of my herver handler.
So it became asynchronous and avoid await* functions. Request is invoked properly. But - for uknown reason for me - the line
bootstrap.releaseExternalResources(); // DOES NOT WORK?
does not work. It throws an exception saying that I cannot kill the thread I am currently using (which sounds reasonable, but still does not give me an answer how to do that in a different way).
I am also not sure is this a correct approach?
Maybe you can recommend a tutorial of such event programming techniques in netty? How to deal - in general - with a few asynchronous requests that has to be invoked in specified order and wait for each other?
Thank you,
If you really want to release the bootstrap on close you can do it like this:
channel.getCloseFuture().addListener(new ChannelFutureListener() {
public void operationComplete(ChannelFuture future) {
System.err.println("Disconnected");
new Thread(new Runnable() {
public void run() {
bootstrap.releaseExternalResources();
}
}).start();
}
});