SignalR - Issues restarting SignalR in self-hosted process - signalr

Question

I've got SignalR running successfully in a self-hosted process (currently a WinForm host). Clients can connect with their browsers and interact perfectly. The problem comes when I need to restart the game realm without killing the host process entirely. After disposing of the IDisposable that WebApp.Start() returns, I then reload all my business data and call WebApp.Start() again. It seems to work, no exceptions or warnings are bubbled up, but client connections from browsers fail, no matter if new browser windows or tabs. I turned on SignalR tracing in javascript and am capturing traffic through Fiddler as well. The /negotiate call works, but then the /connect call for WebSockets times out after 5 seconds, then fails through all the other SignalR fallbacks. If I kill the host process completely and restart it, everything works great again. Manual restarts by killing the entire process every time aren't feasible. I turned on server-side logging and here's the behavior I'm seeing:
SignalR.Transports.TransportHeartBeat Information: 0 : Connection a9f279c9-9dcb-4cc2-806b-dd95e96561b4 is New.
SignalR.Transports.TransportHeartBeat Verbose: 0 : KeepAlive(a9f279c9-9dcb-4cc2-806b-dd95e96561b4)
SignalR.Transports.TransportHeartBeat Verbose: 0 : KeepAlive(a9f279c9-9dcb-4cc2-806b-dd95e96561b4)
SignalR.Transports.TransportHeartBeat Verbose: 0 : KeepAlive(a9f279c9-9dcb-4cc2-806b-dd95e96561b4)
SignalR.Transports.TransportHeartBeat Verbose: 0 : KeepAlive(a9f279c9-9dcb-4cc2-806b-dd95e96561b4)
SignalR.Transports.TransportHeartBeat Verbose: 0 : KeepAlive(a9f279c9-9dcb-4cc2-806b-dd95e96561b4)
SignalR.Transports.TransportHeartBeat Verbose: 0 : KeepAlive(a9f279c9-9dcb-4cc2-806b-dd95e96561b4)
SignalR.Transports.TransportHeartBeat Verbose: 0 : KeepAlive(a9f279c9-9dcb-4cc2-806b-dd95e96561b4)
SignalR.Transports.TransportHeartBeat Verbose: 0 : KeepAlive(a9f279c9-9dcb-4cc2-806b-dd95e96561b4)
SignalR.Transports.TransportHeartBeat Verbose: 0 : KeepAlive(a9f279c9-9dcb-4cc2-806b-dd95e96561b4)
SignalR.Transports.TransportHeartBeat Information: 0 : Dispose(). Closing all connections
SignalR.Transports.WebSocketTransport Error: 0 : OnError(a9f279c9-9dcb-4cc2-806b-dd95e96561b4, System.Net.WebSockets.WebSocketException (0x80004005): An internal WebSocket error occurred. Please see the innerException, if present, for more details. ---> System.Net.HttpListenerException (0x80004005): The I/O operation has been aborted because of either a thread exit or an application request
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Net.WebSockets.WebSocketHttpListenerDuplexStream.<ReadAsyncCore>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
at System.Net.WebSockets.WebSocketBase.WebSocketOperation.<Process>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Net.WebSockets.WebSocketBase.<ReceiveAsyncCore>d__1.MoveNext()
at System.Net.WebSockets.WebSocketBase.ThrowIfConvertibleException(String methodName, Exception exception, CancellationToken cancellationToken, Boolean aborted)
at System.Net.WebSockets.WebSocketBase.<ReceiveAsyncCore>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.AspNet.SignalR.WebSockets.WebSocketMessageReader.<ReadMessageAsync>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.AspNet.SignalR.WebSockets.WebSocketHandler.<ProcessWebSocketRequestAsync>d__e.MoveNext())
SignalR.Transports.WebSocketTransport Information: 0 : End(a9f279c9-9dcb-4cc2-806b-dd95e96561b4)
SignalR.Transports.WebSocketTransport Information: 0 : CloseSocket(a9f279c9-9dcb-4cc2-806b-dd95e96561b4)
SignalR.Transports.WebSocketTransport Verbose: 0 : Cancel(a9f279c9-9dcb-4cc2-806b-dd95e96561b4)
SignalR.Transports.WebSocketTransport Verbose: 0 : DrainWrites(a9f279c9-9dcb-4cc2-806b-dd95e96561b4)
SignalR.Transports.WebSocketTransport Information: 0 : CompleteRequest (a9f279c9-9dcb-4cc2-806b-dd95e96561b4)
SignalR.Transports.TransportHeartBeat Verbose: 0 : Connection a9f279c9-9dcb-4cc2-806b-dd95e96561b4 exists. Closing previous connection.
SignalR.Transports.TransportHeartBeat Verbose: 0 : Connection a9f279c9-9dcb-4cc2-806b-dd95e96561b4 exists. Closing previous connection.
SignalR.Transports.WebSocketTransport Information: 0 : End(a9f279c9-9dcb-4cc2-806b-dd95e96561b4)
SignalR.Transports.WebSocketTransport Verbose: 0 : Cancel(a9f279c9-9dcb-4cc2-806b-dd95e96561b4)
SignalR.Transports.WebSocketTransport Information: 0 : CloseSocket(a9f279c9-9dcb-4cc2-806b-dd95e96561b4)
SignalR.Transports.WebSocketTransport Verbose: 0 : DrainWrites(a9f279c9-9dcb-4cc2-806b-dd95e96561b4)
SignalR.Transports.WebSocketTransport Information: 0 : CompleteRequest (a9f279c9-9dcb-4cc2-806b-dd95e96561b4)
SignalR.Transports.WebSocketTransport Information: 0 : CloseSocket(a9f279c9-9dcb-4cc2-806b-dd95e96561b4)
SignalR.Transports.TransportHeartBeat Verbose: 0 : Connection a9f279c9-9dcb-4cc2-806b-dd95e96561b4 exists. Closing previous connection.
SignalR.Transports.WebSocketTransport Information: 0 : End(a9f279c9-9dcb-4cc2-806b-dd95e96561b4)
SignalR.Transports.WebSocketTransport Verbose: 0 : Cancel(a9f279c9-9dcb-4cc2-806b-dd95e96561b4)
SignalR.Transports.WebSocketTransport Verbose: 0 : DrainWrites(a9f279c9-9dcb-4cc2-806b-dd95e96561b4)
SignalR.Transports.WebSocketTransport Information: 0 : CompleteRequest (a9f279c9-9dcb-4cc2-806b-dd95e96561b4)
SignalR.Transports.TransportHeartBeat Information: 0 : Connection 263abf1d-aa32-41aa-af16-d64c530cd001 is New.
SignalR.Transports.WebSocketTransport Information: 0 : CloseSocket(263abf1d-aa32-41aa-af16-d64c530cd001)
SignalR.Transports.TransportHeartBeat Verbose: 0 : Connection 263abf1d-aa32-41aa-af16-d64c530cd001 exists. Closing previous connection.
SignalR.Transports.WebSocketTransport Information: 0 : End(263abf1d-aa32-41aa-af16-d64c530cd001)
SignalR.Transports.WebSocketTransport Verbose: 0 : Cancel(263abf1d-aa32-41aa-af16-d64c530cd001)
SignalR.Transports.WebSocketTransport Verbose: 0 : DrainWrites(263abf1d-aa32-41aa-af16-d64c530cd001)
SignalR.Transports.WebSocketTransport Information: 0 : CompleteRequest (263abf1d-aa32-41aa-af16-d64c530cd001)
SignalR.Transports.TransportHeartBeat Verbose: 0 : Connection 263abf1d-aa32-41aa-af16-d64c530cd001 exists. Closing previous connection.
SignalR.Transports.ServerSentEventsTransport Information: 0 : End(263abf1d-aa32-41aa-af16-d64c530cd001)
SignalR.Transports.ServerSentEventsTransport Verbose: 0 : Cancel(263abf1d-aa32-41aa-af16-d64c530cd001)
SignalR.Transports.ServerSentEventsTransport Verbose: 0 : DrainWrites(263abf1d-aa32-41aa-af16-d64c530cd001)
SignalR.Transports.ServerSentEventsTransport Information: 0 : CompleteRequest (263abf1d-aa32-41aa-af16-d64c530cd001)
SignalR.Transports.WebSocketTransport Information: 0 : CloseSocket(a9f279c9-9dcb-4cc2-806b-dd95e96561b4)
SignalR.Transports.TransportHeartBeat Verbose: 0 : Connection a9f279c9-9dcb-4cc2-806b-dd95e96561b4 exists. Closing previous connection.
SignalR.Transports.WebSocketTransport Information: 0 : End(a9f279c9-9dcb-4cc2-806b-dd95e96561b4)
SignalR.Transports.WebSocketTransport Verbose: 0 : Cancel(a9f279c9-9dcb-4cc2-806b-dd95e96561b4)
SignalR.Transports.WebSocketTransport Verbose: 0 : DrainWrites(a9f279c9-9dcb-4cc2-806b-dd95e96561b4)
SignalR.Transports.WebSocketTransport Information: 0 : CompleteRequest (a9f279c9-9dcb-4cc2-806b-dd95e96561b4)
SignalR.Transports.WebSocketTransport Information: 0 : CloseSocket(a9f279c9-9dcb-4cc2-806b-dd95e96561b4)
SignalR.Transports.TransportHeartBeat Verbose: 0 : Connection a9f279c9-9dcb-4cc2-806b-dd95e96561b4 exists. Closing previous connection.
SignalR.Transports.WebSocketTransport Information: 0 : End(a9f279c9-9dcb-4cc2-806b-dd95e96561b4)
Note during disposal an error occurs, and then it goes into this state where it just keeps repeating that pattern forever, for both existing connections and new connections.
Any help appreciated.
Update:
Someone requested to share the code for the restart. It's basically a long running process, so at the core of my engine I've got a while loop that loops until I tell it to stop.
protected void MainLoop()
{
//load all data from the db
using (Microsoft.Owin.Hosting.WebApp.Start(dbRealm.SignalRUrl))
{
while (!this.stopping)
{
try
{
this.ProcessFrame();
}
catch (Exception exc)
{
//todo: log error
throw exc;
}
Thread.Sleep(1); //yield
}
}
//cleanup/shut down stuff
}
That is run on it's own thread via this code:
public void Start()
{
//threading/message stuffs
if (!this.running)
{
this.running = true;
ThreadStart ts = new ThreadStart(this.MainLoop);
this.mainThread = new Thread(ts);
this.mainThread.Start();
}
else
{
//message
}
}
Here's my Startup class:
class Startup
{
public void Configuration(IAppBuilder app)
{
app.UseCors(CorsOptions.AllowAll);
app.MapSignalR("/signalr", new HubConfiguration { EnableDetailedErrors = true, Resolver = new DefaultDependencyResolver() });
app.RunSignalR();
}
}
Update #2:
I may not have been clear enough in the OP. I did enable SignalR tracing on the client-side in addition to logging traffic with Fiddler. Here's the output in the browser:
[08:03:37] SignalR: Auto detected cross domain url.
[08:03:37] SignalR: Client subscribed to hub 'webmudhub'.
[08:03:37] SignalR: Negotiating with 'http://localhost:8080/negotiate?clientProtocol=1.5&connectionData=%5B%7B%22name%22%3A%22webmudhub%22%7D%5D'.
[08:03:37] SignalR: webSockets transport starting.
[08:03:37] SignalR: Connecting to websocket endpoint 'ws://localhost:8080/connect?transport=webSockets&clientProtocol=1.5&connectionToken=AQAAANCMnd8BFdERjHoAwE%2FCl%2BsBAAAAecyNVO0bfE6%2BclQisGhh8QAAAAACAAAAAAAQZgAAAAEAACAAAACo51RwDaqYAC1N2%2FkMiQkuFhbVLfrH%2FfJEfdz9TV9F7AAAAAAOgAAAAAIAACAAAADrjqx8pyEMjZ%2BCkUHeZJcdwXH2LaSe1Qlh9XA7fo84MTAAAAB7mClxFPbknblDmZ8a14Thoa6FcLY7%2BS6fJpAXpeO8AaB8y5n4iCD6DotnQH07UDNAAAAA9eV4FLk1hr83h8NDy%2BWwKvGF%2FGn%2F00AOmn%2BMEfvcS5fa2xO%2F0Vd1sjd6TpJdTpePmakv1uOjTekP6FFYXVTU7g%3D%3D&connectionData=%5B%7B%22name%22%3A%22webmudhub%22%7D%5D&tid=0'.
[08:03:37] SignalR: Websocket opened.
[08:03:42] SignalR: webSockets transport timed out when trying to connect.
[08:03:42] SignalR: Closing the Websocket.
[08:03:42] SignalR: webSockets transport failed to connect. Attempting to fall back.
[08:03:42] SignalR: serverSentEvents transport starting.
[08:03:42] SignalR: Attempting to connect to SSE endpoint 'http://localhost:8080/connect?transport=serverSentEvents&clientProtocol=1.5…XVTU7g%3D%3D&connectionData=%5B%7B%22name%22%3A%22webmudhub%22%7D%5D&tid=3'.
[08:03:42] SignalR: EventSource connected.
[08:03:47] SignalR: serverSentEvents transport timed out when trying to connect.
[08:03:47] SignalR: EventSource calling close().
[08:03:47] SignalR: serverSentEvents transport failed to connect. Attempting to fall back.
[08:03:47] SignalR: longPolling transport starting.
[08:03:48] SignalR: Opening long polling request to 'http://localhost:8080/connect?transport=longPolling&clientProtocol=1.5&conn…kP6FFYXVTU7g%3D%3D&connectionData=%5B%7B%22name%22%3A%22webmudhub%22%7D%5D'.
[08:03:52] SignalR: longPolling transport timed out when trying to connect.
[08:03:52] SignalR: Aborted xhr request.
[08:03:52] SignalR: longPolling transport failed to connect. Attempting to fall back.
[08:03:52] SignalR: Fallback transports exhausted.
[08:03:52] SignalR: Stopping connection.
[08:03:52] SignalR: Fired ajax abort async = true.
Note this is on a brand new connection from a new browser connection after calling WebApp.Start() the second time after the first one has been disposed. It feels like something might not be properly disposing down in SignalR, causing some sort of remnants to cause issues.
Here's another server-side log. I started the server, stopped it (disposing of SignalR), and started it back up (making a new WebApp.Start() call). It should at this point accept new connections. Instead I see this:
SignalR.Transports.TransportHeartBeat Information: 0 : Connection 92c4d8da-aeb5-4056-b274-625a83a366b1 is New.
SignalR.Transports.WebSocketTransport Information: 0 : CloseSocket(92c4d8da-aeb5-4056-b274-625a83a366b1)
SignalR.Transports.TransportHeartBeat Verbose: 0 : 92c4d8da-aeb5-4056-b274-625a83a366b1 is dead
SignalR.Transports.TransportHeartBeat Verbose: 0 : Connection 92c4d8da-aeb5-4056-b274-625a83a366b1 exists. Closing previous connection.
SignalR.Transports.WebSocketTransport Information: 0 : End(92c4d8da-aeb5-4056-b274-625a83a366b1)
SignalR.Transports.WebSocketTransport Verbose: 0 : Cancel(92c4d8da-aeb5-4056-b274-625a83a366b1)
SignalR.Transports.WebSocketTransport Verbose: 0 : DrainWrites(92c4d8da-aeb5-4056-b274-625a83a366b1)
SignalR.Transports.WebSocketTransport Information: 0 : CompleteRequest (92c4d8da-aeb5-4056-b274-625a83a366b1)
SignalR.Transports.TransportHeartBeat Verbose: 0 : KeepAlive(92c4d8da-aeb5-4056-b274-625a83a366b1)
SignalR.Transports.TransportHeartBeat Verbose: 0 : Connection 92c4d8da-aeb5-4056-b274-625a83a366b1 exists. Closing previous connection.
SignalR.Transports.ServerSentEventsTransport Information: 0 : End(92c4d8da-aeb5-4056-b274-625a83a366b1)
SignalR.Transports.ServerSentEventsTransport Verbose: 0 : Cancel(92c4d8da-aeb5-4056-b274-625a83a366b1)
SignalR.Transports.ServerSentEventsTransport Verbose: 0 : DrainWrites(92c4d8da-aeb5-4056-b274-625a83a366b1)
SignalR.Transports.ServerSentEventsTransport Information: 0 : CompleteRequest (92c4d8da-aeb5-4056-b274-625a83a366b1)
SignalR.Transports.TransportHeartBeat Verbose: 0 : KeepAlive(92c4d8da-aeb5-4056-b274-625a83a366b1)
SignalR.Transports.TransportHeartBeat Verbose: 0 : KeepAlive(92c4d8da-aeb5-4056-b274-625a83a366b1)
SignalR.Transports.TransportHeartBeat Verbose: 0 : KeepAlive(92c4d8da-aeb5-4056-b274-625a83a366b1)
Once again we see the new connection and then 5 seconds later the CloseSocket() call is made for some reason, perhaps because it thinks it is dead?
I'm basically trying to kind of follow this pattern:
https://weblog.west-wind.com/posts/2013/Sep/04/SelfHosting-SignalR-in-a-Windows-Service
I know that's a Windows Service and mine is hosted in a WinForm app, so maybe that's the difference? I'm wondering if the Windows Service creates a new AppDomain on each request to start the service. As I said, if I completely kill the WinForm app and restart it, everything works again.
It seems to me I should be able to Dispose of SignalR, and then Start it back up with another call to WebApp.Start(), but the second call to WebApp.Start doesn't error out, but it also doesn't properly accept any new incoming connections. They time out. This is obviously conjecture on my part, but perhaps a singleton pattern or similar down inside SignalR that isn't properly being disposed of in its IDisposable implementation? I chose WinForms for easy logging/notifications to each realm's respective WinForm host during development since I'm going to be running multiple instances (each with its own SignalR instance on its own port) on each Windows server.