I'm trying to integrate signalr with an existing asp.net forms web app.
After initially connecting successfully and the server side then calling back to the client js function, signalr seems to have trouble maintaining a connection. I'm developing on a windows 7 machine so the 10 connections limit makes this somewhat challenging to debug. I have however, seen what appears to be the same issue when the website is deployed to a 2003 Enterprise Ed. Server so I don't think I'm seeing a connection limit issue (I stand ready to be corrected though)
Looking in fiddler, I do eventually get a 200 for the connection request but the only JSON I get back as this:
{"C":"B,0|7,4|8,0|9,0","T":1,"M":[]}
I have no idea what this represents. Initially when the connection is successful I get this (which includes the data payload I expect):
"C": "B,0|BK,1|BL,0|BM,0",
"M": [{
"H": "notifyHub",
"M": "notificationReceived",
"A": ["[{\"TransitionNotificationId\":527,\"AuthorizationJobId\":53,\"TransitionType\":2,\"IsWorkShop\":true},
{\"TransitionNotificationId\":528,\"AuthorizationJobId\":53,\"TransitionType\":12,\"IsWorkShop\":true},
{\"TransitionNotificationId\":580,\"AuthorizationJobId\":61,\"TransitionType\":2,\"IsWorkShop\":true}]"]
}]
If I could interpret the JSON in the 'failed' request properly I'd have an idea of where to look for the problem.
Cheers in advance.
T:1 means you got a connection timeout. When using longpolling the connection will timeout every 120 seconds (by default). This is because most load balancers/proxies will kill idle connections after sometime. The other transports send a keep alive to stop this from happening.
As for the rest of the payload:
C: Cursor
M: Messages
H: Hubname
M: Method name
A: Method args
T: Timeout
D: Disconnect
Related
I have fundumental question about how async requests work at top level.
Imagin if we have a top level route called HomePage(). This route is an async route and within this route we call to 10 different APIs before sending the response(image it takes like 5 seconds, remember this is an example to understand the concept and these numbers are for learning purposes). All of these api requests are awaited. So the request handler just releases the thread hanlding this request and goes to handle other requests until the response for these apis come back. So lets add this constraint. Our network card can handle only 1 connection and that one is held open till the response for the request to HomePage is ready. Therefor we cannot make any other requests to the server so whats the difference if this whole thing was sync from the beggining. We cannot drop the connection to the first request to HomePage because if that's the case then how are we ever going to send back the response for that request and we cannot handle new requests because the connection is kept open.
I suspect that my problem is how the reponse is sent back on top level async routes.
Can anybody give a deep dive explaination on how these requests are handled that can take more requests and still send back the response(because if it can send back a response the connection HAS TO HAVE KEPT ALIVE). Examples would be much appreciated.
So lets add this constraint. Our network card can handle only 1 connection
That constraint cannot exist. Network cards handle packets, not connections. Connections are a virtual construct that exist in the host computer.
Can anybody give a deep dive explaination on how these requests are handled that can take more requests and still send back the response(because if it can send back a response the connection HAS TO HAVE KEPT ALIVE).
Of course the connection is kept alive. The top-level async method will return the thread to the thread pool, where it is available to handle any other requests.
If you have some artificial constraint on your web app that prevents it from having more than one connection, then there won't be any other requests to handle, and the thread pool threads will do nothing.
I have a very simple netty app which serves both as server and a client.
Client uses channel.writeAndFlush() to send request to server and then blocks on monitor.wait().
In client's InboundAdapter in channelRead() I find the appropriate monitor and do monitor.notify() to let the requesting client thread to proceed working on the server's reply.
On the server in ChannelHandler's channelRead() I do the following:
To limit the amount of requests being processed I submit a task which does the real work as a new task to existing EventLoop: ctx.executor().submit(new Task()). I that task I do heavy IO operations and after that I writeAndFlush() results back to client.
Here is my pipeline setup:
new ObjectEncoder(),
new ObjectDecoder(LibConstants.Search.MAX_REQUIEST_SIZE, ClassResolvers.cacheDisabled(null))
Here is the bootstrap config:
new ServerBootstrap()
.channel(NioServerSocketChannel.class)
.option(ChannelOption.SO_BACKLOG, 1000)
.option(ChannelOption.SO_KEEPALIVE, true)
I have 2 problems:
Rather often I get io.netty.handler.codec.DecoderException: java.io.UTFDataFormatException on the client when receiving a reply from server. I cannot find any obvious reason for this. Since my pipeline setup is so simple.
A reply from the server just wouldn't appear on the client. I the logs I see a successful flush on the server but the reply never arrived at the client. This is very hard to deal with since my app is very latency sensitive. Any timeout I would set will kill my user experience.
This all happens over a VPN network so there is a possibility that VPN device misbehaves in some weird way but I hoping that TCP would handle any sort of packet loss/corruption which can happen in the channel.
Any advice or experience you can share will be very appreciated!
We are using application request routing with 5 servers running IIS 7.5 and have just recently implemented a messaging system using SignalR in our application.
The SignalR connections are working as we expect (with the only drawback being that a message sent from one server doesn't get activated on the other 4).
The problem(?) we are having is that the Response Time of some requests on IIS that are shown in the load balancer (ARR) are coming up as 2-3 minutes sometimes, which I am assuming are because of connections using something like long-polling.
Our ARR is set to load balance using lowest response time, but it seems like this metric will be completely incorrect because of these SignalR connections. Is there any way to fix these connections so they don't get used in the ARR calculation for response time? Are we stuck having to move the SignalR messages to a separate server to avoid this type of thing (which admittedly would solve other things as well)?
I think the article below will help you. Try setting the "response buffer threshold" to 0 in ARR
http://matthewmanela.com/blog/using-signalr-in-an-arr-cluster/
My understanding of the (JavaScript) hub client is that if a connection is lost, it enters a 'Reconnecting...' phase which attempts to reconnect. If it can't do so, it will enter a 'Disconnected' state which is where it'll stay until asked to start again.
How long is the 'Reconnecting...' phase meant to last before it gives up? I've read 40 seconds before, but my client seems to take much less time - about 10, maybe less. [EDIT: Nevermind this part, I had configured a 10 disconnect on the server as a test... and forgot. I understand this is set by the server during the negotiate. Makes sense!] ... I'd prefer to have the client continually retry until it is told to abort - can this be done, and would it cause issues?
Another question; during the Reconnecting... phase, if I attempt to call a hub method (again, in JS) it never seems to complete. I'm using the returned Deferred to check for 'done' and 'fail' events, but neither seems to get called. Is this by design?
Thanks.
You can definitely have it continually reconnect.
Handle the disconnected event on the client and call connection.start:
$.connection.hub.disconnected(function() {
setTimeout(function() {
$.connection.hub.start();
}, 5000); // Re-start connection after 5 seconds
});
The only issues this would cause is that you could potentially be triggering infinite requests to a server that isn't there for client machines. This becomes even more troublesome when you introduce the mobile market into the situation (drains battery like crazy).
When you attempt to call a hub method while reconnecting SignalR will try to send your command. Since there are 2 channels, one for receiving data and one for sending, (for all transports except web sockets) in some cases it can still be possible to send requests while your offline. Therefore SignalR does not know if a request fails until the browser tells it that it could not successfully make the request.
Hope this helps!
I might have a clue... Touching the Web.config produces an appPool Recycle, meaning that a new worker process will be created for new requests while the existing process will continue for a while until the remaining requests end or the timeout is reached. Request that do not end in the timeout period are terminated.
Signalr client reconnects to the new process while the long running task is running in the old process, so when on the long running task you do
GlobalHost.ConnectionManager.GetHubContext<ForceHub>();
you actually get a reference for "old" hub while the client is connected to the "new" hub.
That's why the test preformed by Wasp worked: he was making a new request to publish on the signalr hub that was processed in the newly created worker process.
You could try to configure a singalr backplane (https://www.asp.net/signalr/overview/performance/scaleout-in-signalr), it’s really easy to configure it using Sql Server (https://www.asp.net/signalr/overview/performance/scaleout-with-sql-server). The backplane should be capable of connect the two worker processes and hopefully you will get the notification on the client.
If this is the problem, notifications generated by new requests will work even without the backplane. Notice that the real purpose of the backplane is to scale out signalr, this is, to connect a farm of WebServers between them.
Also keep in mind that running long-running task inside IIS is as task hard to achieve as, among other things, IIS does regular appPool recycles and has timeout limits for the requests to execute. I recommend that you read the following post: http://www.hanselman.com/blog/HowToRunBackgroundTasksInASPNET.aspx
“If you think you can just write a background task yourself, it's likely you'll get it wrong. I'm not impugning your skills, I'm just saying it's subtle. Plus, why should you have to?”
Hope this helps
I'm using SignalR 0.5.3 with hubs and I'm explicitely setting transport to long polling like this:
$.connection.hub.start({ transport: 'longPolling' }, function () {
console.log('connected');
});
with configuration like this (in global.asax.cs Application_Start method):
GlobalHost.DependencyResolver.UseRedis(server, port, password, pubsubDB, "FooBar");
GlobalHost.Configuration.DisconnectTimeout = TimeSpan.FromSeconds(2);
GlobalHost.Configuration.KeepAlive = TimeSpan.FromSeconds(15);
However the long polling doesn't seem to be working neither on development (IIS express) nor on production (IIS 7.5) environment. Connection seems to be made properly, however the long poll request is always timed out (after ~2 minutes) and reconnect happens afterwards. Logs from IIS are here. Response from first timed out request:
{"MessageId":"3636","Messages":[],"Disconnect":false,"TimedOut":true,"TransportData":{"Groups":["NotificationHub.56DDB6692001Ex"],"LongPollDelay":0}}
Timed out reconnect responses looks like this:
{"MessageId":"3641","Messages":[],"Disconnect":false,"TimedOut":true,"TransportData":{"Groups":["NotificationHub.56DDB6692001Ex"],"LongPollDelay":0}}
I would appreciate any help regarding this issue. Thanks.
Edit
If reconnect means the beginning of a new long poll cycle why it is initiated after ~2 minutes when KeepAlive setting in global.asax.cs is set to 15 seconds? Problem with this is that I have a reverse proxy in front of IIS which timeouts keep-alive requests after 25 seconds therefore I get 504 response when this reverse proxy timeout is reached.
Take a look at this post: How signalr works internally. The way long pulling works is after a set time the connection will timeout or receive a response and repull (reconnect)
Keep alive functionality is disabled for long polling. Seems that ConnectionTimeout is used instead.
This setting represents the amount of time to leave a transport
connection open and waiting for a response before closing it and
opening a new connection. The default value is 110 seconds.
https://learn.microsoft.com/en-us/aspnet/signalr/overview/guide-to-the-api/handling-connection-lifetime-events#timeoutkeepalive
If the request is timed out and server is not sending any data, but you expect it to send, maybe it is some issue on the server side that you don't yet see.