WCF stop responding intermitently - asp.net

Question

I have a ASP.Net web site that get all business logic from a WCF hosting site (IIS). Once in a while, it looks like the backend WCF frozen and that makes web front-end stop responding. I have to recycle both application pools to make it work again.
It happens more often lately, could be because we have more & more customers using the site. Before, it happens once every month, now it happens once every week. Maybe more.
We DO Close connection after each SVC call.
Error message received in Event Log after I recycle the pool
Message: System.ServiceModel.CommunicationException: An error occurred while receiving the HTTP response to .........../.....BusinessServices.svc. This could be due to the service endpoint binding not using the HTTP protocol. This could also be due to an HTTP request context being aborted by the server (possibly due to the service shutting down). See server logs for more details. ---> System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a receive. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
Web.config of the web app:
<bindings>
<wsHttpBinding>
<binding name="WSHttpBinding_IBusinessServices" closeTimeout="00:31:00" openTimeout="00:31:00"
receiveTimeout="00:10:00" sendTimeout="00:31:00"
bypassProxyOnLocal="false" transactionFlow="false"
hostNameComparisonMode="StrongWildcard" maxBufferPoolSize="524288"
maxReceivedMessageSize="2147483647" messageEncoding="Text"
textEncoding="utf-8" useDefaultWebProxy="true" allowCookies="false">
<readerQuotas maxDepth="999" maxStringContentLength="2147483647" maxArrayLength="1638400"
maxBytesPerRead="4096" maxNameTableCharCount="16384" />
<reliableSession ordered="true" inactivityTimeout="00:10:00" enabled="false" />
<security mode="Message">
<transport clientCredentialType="Windows" proxyCredentialType="None" realm="" />
<message clientCredentialType="Windows" negotiateServiceCredential="true" algorithmSuite="Default" />
</security>
</binding>
</wsHttpBinding>
</bindings>
<client>
<endpoint address="http://localhost:8890/MyBusinessServices.svc" binding="wsHttpBinding"
bindingConfiguration="WSHttpBinding_IBusinessServices"
contract="MyBusinessServices.IBusinessServices"
name="WSHttpBinding_IBusinessServices" />
</client>
Web.config of the WCF service, with throttling set to 1500
.....
<behavior name="AppServiceBehaviors">
<serviceMetadata httpGetEnabled="true" />
<serviceDebug includeExceptionDetailInFaults="true" />
<dataContractSerializer maxItemsInObjectGraph="2147483647" />
<serviceTimeouts transactionTimeout="00:15:00" />
<serviceThrottling maxConcurrentCalls="1500" maxConcurrentSessions="1500"
maxConcurrentInstances="2147483647" />
</behavior>
<behavior name="">
<serviceMetadata httpGetEnabled="true" />
<serviceDebug includeExceptionDetailInFaults="true" />
<dataContractSerializer maxItemsInObjectGraph="2147483647" />
</behavior>
</serviceBehaviors>
</behaviors>
This problem is intermittent but it's driving me crazy. Any ideas/suggestions is appreciated.
Eric

Answer 1

The issue you are facing right now can be due to many other factors that needs a lot more description than what we currently have.
The thrown exception is a kind of generic exception that may point to many scenarios possible. For example, if you have a load balancer in front of your service servers and one of them has an error in runtime but the other does not. Another example is the sql server is creating an exception at server side, but your wrapping exceptions may not send the related information to your client. So you cannot know exactly what is going on.
I have a few suggestions for you to check:
I can see that you have expressed that you are closing connection after each call. But you need to make sure that the calls that receive an exception is also closed. So your svc calls should look like this:
try
{
if (this.client != null)
{
IClientChannel channel = this.client as IClientChannel;
if (channel.State == CommunicationState.Faulted)
{
channel.Abort();
}
else
{
channel.Close();
}
}
}
finally
{
this.client = null;
}
Other than that, you may also want to check your Application Pool settings on IIS. You can make sure from event logs, that you are not forced to recycle/shutdown with Maximum Failures setting. (In advanced app pool settings - rapidFailProtectionMaxCrashes)

Answer 2

My 2 cents for you,
Service abort - error due to fatal error occurred inside,
Make sure while debugging WCF in VS IDE in Dev, keep the Exceptions(Ctrl D,E), check all "unhandled exception" columns. This enables IDE to break the application code line exactly, inspite of the CATCHes..this will unearth any bugs.
Please do let us know, if any findings here...
Good Luck,
HydTechie