These are the configuration items the signalr server has
GlobalHost.Configuration.DisconnectTimeout = new TimeSpan(0, 0, 60);
GlobalHost.Configuration.KeepAlive = new TimeSpan(0, 0, 5);
There has never been any inactivity in production for more than 3 years with this setup. It is a self hosted signalr server running as a windows service, and there are evidences that both the client application's and the server's processes are running after the timeout exception that is received in the client. ie No crashes. The client is another windows service running on the same machine as the server. The exception is
System.TimeoutException: The client has been inactive since 26/03/2018 07:00:12 and it has exceeded the inactivity timeout of 00:01:10. Stopping the connection.
Given the above are all that needed to conclude that the keep-alive messages should be sent every 5 seconds by the client, This is totally unexpected, Is this a known bug . as per the docs, the keep alive timeout is not more than 1/3 of the disconnection timeout. The other puzzling information from the error message is that the inactivity timeout is 00:01:10 which does not match up with anything (neither of default or the configured timeouts).
Am I missing something obvious ?
Related
I have a .Net Application(REST API) which accesses SQL Server Database. Sometimes When there are many API requests at the same time, I get Connection Timeout exception due to which all the request fails. This is not always the case and happens only when there are many requests hitting the server at the same time.
Exception Details:
System.Data.SqlClient.SqlException (0x80131904): Connection Timeout
Expired. The timeout period elapsed while attempting to consume the
pre-login handshake acknowledgement. This could be because the
pre-login handshake failed or the server was unable to respond back in
time. The duration spent while attempting to connect to this server
was - [Pre-Login] initialization=4; handshake=14992; --->
System.ComponentModel.Win32Exception (0x80004005): The wait operation
timed out.
Any help here will be highly appreciated.
From this (upvoted) comment
in this case may be few reasons:
1. You are using SQL Server Express
2. Dead lock between your queries
3. Queries executed more then standard timeout- 30 sec
You should first exclude N1, as the server Edition has nothing to do with connection timeout that is imposted by client
Then exclude N2 as the deadlocks are resolved automatically by server deadlock monitor that controls for deadlocks every 5 seconds or even less when the deadlocks are frequent.
Than exclude N3 as connection timeout has nothing to do with execution timeout. And your timeout is CONNECTION timeout.
You should increase connection timeout on your client or troubleshoot your server, what happens there that your server becomes unresponsive.
I have an WebAPI service using ODP.NET to make connections to several oracle databases. Normally the web service would be hit several times a second and will never have long periods on inactivity. In our test site however, we did not use it for 2-3 days. This morning, we hit the service and got "connection request timeout" exceptions from ODP.NET, suggesting that the connection pool was out of available connections. We are closing the connections after use. The service was working fine before the period, but today the very first query got the timeout exception. Our app pool in IIS is configured to never reset.
My question then is, what can cause the connection pool to fill with bad connections after a period of inactivity, where these connections are not cleaned up in the usual 3 minute cycle? It only happened to 2 out of the 3 of our databases, and Validate Connection=true is set for all of them.
EDIT
So after talking to the DBA, there is some different between a connection/session being killed manually or by timeout and the database server severing the TCP connections. In this case, the TCP connection was severed as part of a regular backup (why is not important for this). I guess this happens when the whole database server goes offline at once. The basis of the question still applies I think though: why is ODP.NET unable to cleanup severed connections overtime? There is a performance counter that refers to "Stasis" connections, could those connections be stuck in that state? I would think that it should be able to see that a connection is no longer active (Validate Connection=True), kill it and not return it to the pool.
Granted, this problem can be solved by just resetting the app pool everything the database goes down. I would still like to configure ODP.NET connection pooling to be more fault tolerant.
I have run into this same issue, and the only solution I have found is to use the Connection Lifetime connection string parameter in conjunction with Validate Connection.
In my particular case, the connection timeout was set at the server and the connections in the pool would timeout, but not be sniped out of the pool, resulting in errors.
Setting both the Connection Lifetime and the Validate Connection parameters has resolved the issue.
Make sure the Connection Lifetime value that you choose is less than the server connection inactivity timeout.
The recommended solution is to use ODP.NET Fast Connection Failover (FCF). FCF will automatically remove invalid connections from the pool such that you don't need to use Validate Connection, Connection Lifetime, nor clear the pool.
To use FCF, set "HA events=true", use connection pooling, and have your DBA set up Fast Application Notification (FAN) on the server side. FAN is what alerts the ODP.NET pool when a DB service or node goes down or rebooted. Upon receiving the message, ODP.NET knows which connections to remove from the pool and removes them, leaving all other valid connections untouched.
Something else is going on here. Min Pool Size and some of the other settings help when the connection is severed from things like DBA configured idle timeouts and firewall tcp idle timeouts, 'connection request timeout' occurs when created a new connection.
This could be simple network problem. There could be something interfering with dns resolution of the servers. Another case is not having fully qualified entries in tnsnames. I've been bit by the latter a couple of times.
The other issue is the one you've already recognized - full pool.
Double check that you don't have a connection leak somewhere. A missing .Close is one thing but if you're not using a 'using' statement, a try/finally is required as an unhandled exception could be thrown prior to the .Close.
I would use perfmon to monitor some of the connection statistics to start - NumberOfPooledConnections, NumberOfActiveConnections, etc:
I have noticed that SignalR does not recognize disconnection events at the transport layer in some circumstances. If the transport is disconnected gracefully, either through dropping my VPN connection to the server, or issuing an ipconfig /release the appropriate events are fired. If I am connected to the VPN and I either unplug the network cable or turn my wireless off, SignalR does not recognize we are disconnected until the VPN connection times out. That is mostly acceptable because it does eventually realize it. However if I am not using a VPN and I simply unplug the network cable or turn off the wireless, SignalR never recognizes a disconnect has occurred. This is using SignalR 2.0.1 over WebSockets in a non CORS environment. If I enable logging in the hub I see no events logged in the console after waiting for over 5 minutes. One thing that bothers me that I do see in the log during start up is:
SignalR: Now monitoring keep alive with a warning timeout of 1600000 and a connection lost timeout of 2400000.
Is this my problem? I have tried manually setting my GlobalHost.Configuration.DisconnectTimeout to 30 seconds but it does not change any behavior in this scenario, or alter that log statement. What else might I be overlooking here?
Edit: I noticed in fiddler that my negotiate response has a disconnect timeout of 3600 and a keepalive of 2400 and that trywebsockets is false. This particular server is 2008 R2, which I do not believe supports Web Sockets. Does that mean long polling would be used? I don't see any long polling requests going out in fiddler or the console.
The timeout settings were my problem. I'm not sure why I originally did not see a change in the logging output when I adjusted the setting, but I went back to a sample app and saw things change there and now things are behaving properly.
It should be noted that the default settings for SignalR produce the following statement in logging output and that this is measured in milliseconds.
SignalR: Now monitoring keep alive with a warning timeout of 13333.333333333332 and a connection lost timeout of 20000
This is obvious when you read the information under Transport Disconnect Scenarios on the following page which says: The default keepalive timeout warning period is 2/3 of the keepalive timeout. The keepalive timeout is 20 seconds, so the warning occurs at about 13 seconds.
I'm using SignalR 0.5.3 with hubs and I'm explicitely setting transport to long polling like this:
$.connection.hub.start({ transport: 'longPolling' }, function () {
console.log('connected');
});
with configuration like this (in global.asax.cs Application_Start method):
GlobalHost.DependencyResolver.UseRedis(server, port, password, pubsubDB, "FooBar");
GlobalHost.Configuration.DisconnectTimeout = TimeSpan.FromSeconds(2);
GlobalHost.Configuration.KeepAlive = TimeSpan.FromSeconds(15);
However the long polling doesn't seem to be working neither on development (IIS express) nor on production (IIS 7.5) environment. Connection seems to be made properly, however the long poll request is always timed out (after ~2 minutes) and reconnect happens afterwards. Logs from IIS are here. Response from first timed out request:
{"MessageId":"3636","Messages":[],"Disconnect":false,"TimedOut":true,"TransportData":{"Groups":["NotificationHub.56DDB6692001Ex"],"LongPollDelay":0}}
Timed out reconnect responses looks like this:
{"MessageId":"3641","Messages":[],"Disconnect":false,"TimedOut":true,"TransportData":{"Groups":["NotificationHub.56DDB6692001Ex"],"LongPollDelay":0}}
I would appreciate any help regarding this issue. Thanks.
Edit
If reconnect means the beginning of a new long poll cycle why it is initiated after ~2 minutes when KeepAlive setting in global.asax.cs is set to 15 seconds? Problem with this is that I have a reverse proxy in front of IIS which timeouts keep-alive requests after 25 seconds therefore I get 504 response when this reverse proxy timeout is reached.
Take a look at this post: How signalr works internally. The way long pulling works is after a set time the connection will timeout or receive a response and repull (reconnect)
Keep alive functionality is disabled for long polling. Seems that ConnectionTimeout is used instead.
This setting represents the amount of time to leave a transport
connection open and waiting for a response before closing it and
opening a new connection. The default value is 110 seconds.
https://learn.microsoft.com/en-us/aspnet/signalr/overview/guide-to-the-api/handling-connection-lifetime-events#timeoutkeepalive
If the request is timed out and server is not sending any data, but you expect it to send, maybe it is some issue on the server side that you don't yet see.
asp.net 2.0 using odp.net 10.1.04 to connect to Oracle 10g.
Our web application has once a day this problem: ODP-1000 "Connection request timed out".
As we know, this is a memory leak problem, caused due open connections that don't get closed and disposed.
We have double checked our code and all connections after open are closed and disposed. All odp.net objects like OracleCommand, Oracledatareader are disposed after used.
On web.config, this is our connection string:
<add key="ConnectionString" value="user id=xxxx;data source=XXXX;password=xxxxx;Pooling=True;Connection Timeout=15;Incr Pool Size=5;Decr Pool Size=3;Max Pool Size=300;Min Pool Size=10;Validate Connection=true;"/>
We have increased Connection Timeout to 60.
As we can't find in our code any unclosed connection, is there any way to close or destroy inactive Oracle sessions when conn.Open gets error "Connection request timed out"?
According to some information:
http://rainmanalex.wordpress.com/2008/11/13/odpnet-connection-string-attributes/
"The regulator thread is spawned every 3 minutes and closes up to Decr Pool Size amount of pooled connections if they are not used. "
Instead of waiting 3 minutes, is there any way to force close an inactive connection when getting error "Connection request timed out"?
Something like:
If conn.Open gets error "Connection request timed out"
Then
Destroy inactive Oracle session
Try again conn.Open
End If