We are running an Owin applications on IIS 8.5 on Win 2012 R2 behind a loadbalancer. One some occations, requests to certain URLs hang indefinitely. If the user selects cancel in the browser, and reloads the page, everything is OK and the server responds quickly.
In IIS manager, we can see the requests hanging:
The hang seems to occur inside the Owin pipeline. We are running .NET 4.5.2 and the latest Owin packages.
Here's the code for the /api/whoami endpoint:
[Authorize]
public class WhoAmIController : ApiController
{
[Route("whoami")]
[HttpGet]
public IHttpActionResult Whoami()
{
return Json(ClaimsPrincipal.Current.Identity.Name);
}
}
An observation is that requests to this endpoint hangs in the AuthenticateRequest stage in the IIS pipeline, not PreExecuteRequestHandler which is the default IIS stage for the OWIN pipeline. Our only authentication method is cookie authentication, which executes in the AuthenticateRequest stage.
Any ideas?
Edit: adjusted screenshot
The problem was that we had code executing on the IIS event PreSendRequestHeaders, which apperently is bad according to
http://www.asp.net/aspnet/overview/web-development-best-practices/what-not-to-do-in-aspnet,-and-what-to-do-instead#presend
Our intention was to adjust HTTP headers on the way out on all request. The fix was to move the code to the BeginRequest event.
Have you tried adding .ConfigureAwait(false) to your async calls?
Doing so will make the continuation continue on the "correct" thread.
Related
In an asp.net MVC 5 project I'm using a katana owin based middlewere to handle the authentication. Inside the Start.cs file I have the Startup class with the Configuration method.
Is there a way to get the full URL of the request inside the Configuration method? I need to get the last part of it to be stored in a cookie
public void Configuration(IAppBuilder app) {
app.UseCookieAuthentication(new CookieAuthenticationOptions { ... }
// something here to get the full URL
// other authentication code here
}
Startup runs outside of the request-cycle. In fact, it only runs once, and then multiple successive URLs can be serviced before it ever runs again (when AppPool recycles, server restarts, etc.)
Long and short, even if you could access the URL, it wouldn't do you any good because it would simply be the first random URL that was accessed, which may or may not be applicable to whatever you're trying to do here.
I'm using SignalR to push updates out to connected web clients. I listen to the disconnected event in order to know when I should start my reconnection logic
$.connection.hub.disconnected(function() {
// Initiate my own reconnection logic
});
The SignalR hub is hosted in in IIS (along with my site)
[assembly: OwinStartup(typeof(Startup))]
namespace MyNamespace.SignalR
{
public class Startup
{
public void Configuration(IAppBuilder app)
{
app.MapSignalR();
}
}
}
Upon connection, the client calls a server method to join a group
public class MyHub : Hub
{
public void JoinGroup(string groupName)
{
Groups.Add(Context.ConnectionId, groupName);
}
}
And then I push messages to this group:
context.Clients.Group(groupName).sendMessage();
If I manually recycle the application pool in IIS, SignalR starts trying to reconnect and I eventually receive a disconnected event on the client side if it fails (after the timeout).
However, my problem is that if I manually restart the website in IIS, I do not receive any disconnected event at all, and I can't see in the logs that SignalR has detected any connection problem at all. How can I detect that I have been disconnected?
I know I should probably persist the group connections somehow, since that is saved in memory I guess. But that shouldn't affect the initial problem that the client receives no notification of the disconnection? Shouldn't the client side signalr code throw some kind of exception/event?
disconnected fires first when the built in logic for reconnection have timed out. You also need to listen to the recconect event, something like i did here
https://github.com/AndersMalmgren/SignalR.EventAggregatorProxy/blob/ReconnectOnClosed/SignalR.EventAggregatorProxy.Client.JS/jquery.signalR.eventAggregator.js#L157
So I finally found out what the problem is and how to solve it. Some background though:
At the moment we manually release new versions of our application by going to "Basic settings..." under the website in IIS and changing the "Physical Path" from C:\websites\version1 to C:\websites\version2. Apparently this gives the same behavior as doing a restart of the website in IIS (not a hard reset, not stopping the website, not recycling the app pool) and according to this: "does NOT shut the site down, it merely removes the Http.sys binding for that port". And no matter how long we wait, the connected clients never receive any kind of indication that they should reconnect.
So the solution is to recycle the application pool after each release. The "lost" clients will receive disconnected events and reconnect to the new version of the site.
I have run into an issue which can be replicated in the following way (you need IIS8 so must be on Windows 8+ or Windows Server 2012 R2+):
Create a new website in IIS Manager, say TestWs on port 8881, pointing to a new folder, say C:\temp\testws, and add the following Web.config file in there
<?xml version="1.0"?>
<configuration>
<system.web>
<compilation targetFramework="4.5"/>
<httpRuntime targetFramework="4.5"/>
</system.web>
</configuration>
Now add the following WsHandler.ashx file in the same folder
<%# WebHandler Language="C#" Class="WsHandler" %>
using System;
using System.Threading;
using System.Web;
public class WsHandler : IHttpHandler
{
public void ProcessRequest(HttpContext context)
{
context.AcceptWebSocketRequest(async webSocketContext =>
{
while (true)
{
await webSocketContext.WebSocket.ReceiveAsync(new ArraySegment<byte>(new byte[1024]), CancellationToken.None);
}
});
}
public bool IsReusable { get { return true; } }
}
Then create a websocket from within the developer toolbar in your browser like so
var ws = new WebSocket("ws://localhost:8881/wshandler.ashx");
ws.onclose = function() { console.log('closed'); };
In task manager you will see there is a w3wp.exe process for this application, if you kill it the client get the onclose event raised and the closed text will be printed.
However if you create a websocket as described above and go to IIS manager and recycle the application pool, the websocket will not be closed, and there will now be two w3wp.exe processes.
Closing the web socket ws.close(); or refreshing the browser will cause the original w3wp.exe process to be shut down.
It seems the presence of the open websocket is causing IIS to be unable to recycle the app pool correctly.
Can anyone figure out what to change in my code or what to change in IIS to get this to work?
As far as I know, while a WebSocket is open, IIS won't tear down the app domain, so you see this behaviour exhibited.
The best I can suggest is that you do some cross process signalling to force the old instance to shutdown. You could achieve this with an EventWaitHandle:
Create a named EventWaitHandle in your web application, and signal it at startup.
On a separate thread, wait on the wait handle
When it is signalled, call HostingEnvironment.InitiateShutdown to force any running old instance to shutdown.
Try setting "Shutdown Time Limit" to 1 second (App Pool > Advanced Settings > Process Model) [PS: I don't have IIS8. I'm checking the properties in IIS7.]
This property defines the time given to worker process to finish processing requests and shutdown. If the worker process exceeds the shutdown time limit, it is terminated.
I can see default value in IIS7 is 90 seconds. If that's the same value in IIS8 too, then it might be giving that much time to earlier worker process to finish it's work. After 90 seconds (1.5 mins) it will terminate that process and your web socket will get closed. If you change it to 1 second it will terminate the earlier worker process will get terminated almost instantly (as soon as you recycle app pool) and you will get the expected behavior.
As I had the same problem, here's the solution I figured out:
In your IHttpHandler you should have an static object which inherits IStopListeningRegisteredObject. Then use HostingEnvironment.RegisterObject(this) to get notified (via StopListening) when the Application Pool is recyled.
You'll also need a CancellationTokenSource (static, too), which you'll hand over in ReceiveAsync. In StopListening() you can then use Cancel() of the CancellationTokenSource to end the await. Then catch the OperationCanceledException and call Abort() on the socket.
Don't forget the Dispose() after the Cancel() or the App-Pool will still wait.
I have a weird behavior going on with SignalR, I have a hub defined like so:
public class NotificationHub : Hub
{
//Empty
}
on my js I have the following:
function blabla {
// bla bla
$.connection.NotificationHub.client.AppendNewStory = function (story) {
// appends a new story, long jquery code
};
$.connection.hub.start().done(function () {
_ConnectionStarted = true; // there must be some built in way
});
}
I call js from a class on my mvc project
public SomeClass
{
private IHubContext _context;
public SomeClass()
{
_context = GlobalHost.ConnectionManager.GetHubContext<NotificationHub>();
}
public void Notify(SomeData ...)
{
_context.Clients.User(userId).AppendNewStory(data);
}
}
Problem has two symptoms:
Server to client class fail intermittently (can't figure out a scenario).
My web server seems to get VERY SLOW, subsequent ajax calls and regular webrequests timeout.
Using fiddler I found the following:
SignalR/hubs call succeeds.
SignalR negotial call succeeds.
connect with WebSockets transport failed with HTTP 504!
subsequent connect calls try foreverFrame and long polling, both succeed with 200.
a poll request is issued and NEVER RETURNS! after that everything becomes slow or hangs.
Something that might aid the debugging, server hangs if i'm opening 2 tabs or more. a single tab seems ok.
I'm also using IE 11 and chrome, same problem except chrome tries serverSentEvents instead of foreverFrame.
Any ideas on how to debug this issue, see why I get 504? or why the server hangs? I have Windows 8.1 Update 1, IIS 8, SignalR 2.0, .NET 4.5.1.
Did you install WebSockets in Windows 8.1? Do this under add/remove features. I had similar random issues connecting until I did this. I also had to enable the following setting in the web.config:
<add key="aspnet:UseTaskFriendlySynchronizationContext" value="true"/>
Is there anything in between your server and browser. Some proxies have trouble with web sockets and server sent events. Try choosing the transports in the javascript. I saw similar things with Firefox over OpenIG. OpenIG didn't like server sent events and just hung.
One possibility that you'll want to investigate is whether you're running into a limit on the number of connections allowed to IIS. I'm not sure whether websocket connections are included in that limit, but normal connections - and a long polling connection will consume one of those - are typically limited to "3" on a the "Basic" version of Windows 8 (and are limited to 10 on the Professional and other versions). I'm guessing that your long polling connection(s) are consuming all the available connections, and that's what is causing everything else to time out.
See here for more details:
http://www.jpelectron.com/sample/WWW%20and%20HTML/IIS-%20OS%20Version%20Limits.htm
I'd either upgrade to Professional (if you're already at "Professional", then discount this answer), or install Websockets and see if that helps things.
Ok thanx to everyone who answered, you all helped debug the issue, unfortunately nothing worked.
problem turned out I needed to set the targetFramework attribute in web.config:
<httpRuntime targetFramework="4.5.1">
Everything worked smoothly afterwards.
I am in the process of investigating to convert our web application to the web farm. So I started with web garden, by converting the 'Maximum Worker Process = 3'. Following is the simplified version of my problem.
Following is my Static Object class.
public static class MyStaticObject
{
public static string MyProperty {get;set;}
}
Then on Page Load I initialized the Static Objects as follows -
MyStaticObject.MyProperty = "My Static Property";
Then using asp.net ajax [WebMethod] create the ajax method on my web page
[WebMethod()]
public static string getStaticProperty()
{
return MyStaticObject.MyProperty;
}
// Then I call this Ajax method using Javascript and set the return to the text box.
This test is not working as expected. Following are my assumptions and wrong outcome from the test.
I thought when we set virtual directory to be web garden, then each request to the virtual directory is handled by different process in web garden, so my next few request to the server, should return null as I have initialized the static objects for one working process. But even if I click the ajax button for 20 times in row (means 20 requests) even then the static objects return me value.
Am i right in assuming on restarting the IIS should kill all the static objects.
Static objects are not shared in web gardens/web farms.
I am surprised by the behaviour of IIS, static objects and web garden.
Is I am assuming wrong or my way of testing is wrong.
Thanks.
Your assumptions about the way static objects are managed in AppPools / web gardens is correct.
However, your assumption about the way that web requests are distributed is not. HTTP requests are round-robined by the http.sys driver to IIS worker processes only when a new TCP connection is established, not when a new request arrives. Since keepalives are enabled by default, even though you made 20 requests in a row, they probably were all served by the same IIS worker process.
You can have IIS disable keepalives for testing purposes from the HTTP Response Headers section in IIS Manager, under Set Common Headers. That should cause your browser to open a new connection for each request.
To test with keepalives enabled, you can use the Web Capacity Analysis Tool (WCAT), available with the IIS 6 Resource Kit, to generate a multi-threaded load that accesses both IIS processes in your web garden.