ASP.NET MVC 3 WCF not working on load balancing servers - asp.net

Question

I searched the internet but couldn't find any scenarios like the one I'm having. The scenario is I have an ASP.NET MVC 3 Reporting application and stand-alone WCF hosted services. I am doing a bin deploy for ASP.NET MVC 3. Everything works great on my local environment and dev server. I am using a service reference in the ASP.NET MVC project to call into the client side proxy.
However, when I deploy to the clients test environment which is load balancing (ASP.NET and WCF layer both live on same load balancing servers), it is failing. If I run my local build and point my endpoints to their dev wcf services it works fine. It only fails when trying to launch the application from the test URL itself. I am getting the following error (edited the endpoint and service call information:
"System.ServiceModel.CommunicationException: An error occurred while
receiving the HTTP response to http:[testserver]. This could be due to
the service endpoint binding not using the HTTP protocol. This could
also be due to an HTTP request context being aborted by the server
(possibly due to the service shutting down). See server logs for more
details. ---> System.Net.WebException: The underlying connection was
closed: An unexpected error occurred on a receive. --->
System.IO.IOException: Unable to read data from the transport
connection: An existing connection was forcibly closed by the remote
host. ---> System.Net.Sockets.SocketException: An existing connection
was forcibly closed by the remote host at
System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset,
Int32 size) --- End of inner exception stack trace --- at
System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset,
Int32 size) at System.Net.PooledStream.Read(Byte[] buffer, Int32
offset, Int32 size) at System.Net.Connection.SyncRead(HttpWebRequest
request, Boolean userRetrievedStream, Boolean probeRead) --- End of
inner exception stack trace --- at
System.Net.HttpWebRequest.GetResponse() at
System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan
timeout) --- End of inner exception stack trace --- Server stack
trace: at
System.ServiceModel.Channels.HttpChannelUtilities.ProcessGetResponseWebException(WebException
webException, HttpWebRequest request, HttpAbortReason abortReason) at
System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan
timeout) at
System.ServiceModel.Channels.RequestChannel.Request(Message message,
TimeSpan timeout) at
System.ServiceModel.Channels.ServiceChannel.Call(String action,
Boolean oneway, ProxyOperationRuntime operation, Object[] ins,
Object[] outs, TimeSpan timeout) at
System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage
methodCall, ProxyOperationRuntime operation) at
System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage
message) Exception rethrown at [0]: at
System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage
reqMsg, IMessage retMsg) at
System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData&
msgData, Int32 type) at
ReportingApp.UserClientProxy.IUser.GetUsersBetween(DateTime start,
DateTime end) at
ReportingApp.Controllers.HomeController.PopulateResultsTable(ReportingViewModel
vm) at ReportingApp.Controllers.HomeController.Index() at
lambda_method(Closure , ControllerBase , Object[] ) at
System.Web.Mvc.ReflectedActionDescriptor.Execute(ControllerContext
controllerContext, IDictionary2 parameters) at
System.Web.Mvc.ControllerActionInvoker.InvokeActionMethod(ControllerContext
controllerContext, ActionDescriptor actionDescriptor, IDictionary2
parameters) at
System.Web.Mvc.ControllerActionInvoker.<>c_DisplayClass15.b_12()
at
System.Web.Mvc.ControllerActionInvoker.InvokeActionMethodFilter(IActionFilter
filter, ActionExecutingContext preContext, Func1 continuation) at
System.Web.Mvc.ControllerActionInvoker.InvokeActionMethodWithFilters(ControllerContext
controllerContext, IList1 filters, ActionDescriptor actionDescriptor,
IDictionary`2 parameters) at
System.Web.Mvc.ControllerActionInvoker.InvokeAction(ControllerContext
controllerContext, String actionName)"
My ASP.NET wcf configuration is as follows:
<bindings>
<basicHttpBinding>
<binding name="BasicHttpBinding_IUser" closeTimeout="00:10:00"
openTimeout="00:10:00" receiveTimeout="00:10:00" sendTimeout="00:10:00"
allowCookies="false" bypassProxyOnLocal="false" hostNameComparisonMode="StrongWildcard"
maxBufferSize="5242880" maxBufferPoolSize="5242880" maxReceivedMessageSize="5242880"
messageEncoding="Text" textEncoding="utf-8" transferMode="Buffered"
useDefaultWebProxy="true">
<readerQuotas maxDepth="5242880" maxStringContentLength="5242880" maxArrayLength="5242880"
maxBytesPerRead="5242880" maxNameTableCharCount="5242880" />
<security mode="None" >
<transport clientCredentialType="None" proxyCredentialType="None" realm=""/>
<message clientCredentialType="UserName"/>
</security>
</binding>
</basicHttpBinding>
</bindings>
<client>
<endpoint address="[testEndPoint]" binding="basicHttpBinding"
bindingConfiguration="BasicHttpBinding_IUser" contract="UserClientProxy.IUser"
name="BasicHttpBinding_IUser" />
My service web config looks like this:
<behaviors>
<serviceBehaviors>
<behavior>
<!-- To avoid disclosing metadata information, set the value below to false and remove the metadata endpoint above before deployment -->
<serviceMetadata httpGetEnabled="true"/>
<!-- To receive exception details in faults for debugging purposes, set the value below to true. Set to false before deployment to avoid disclosing exception information -->
<serviceDebug includeExceptionDetailInFaults="false"/>
<!--Doubled from value again to resolve issues with the reporting process.-->
<dataContractSerializer maxItemsInObjectGraph="524288" />
</behavior>
</serviceBehaviors>
</behaviors>
<bindings>
<basicHttpBinding>
<binding maxBufferSize="5242880" maxBufferPoolSize="5242880" maxReceivedMessageSize="5242880">
<readerQuotas maxDepth="5242880" maxStringContentLength="5242880" maxArrayLength="5242880"
maxBytesPerRead="5242880" maxNameTableCharCount="5242880" />
</binding>
</basicHttpBinding>
</bindings>
<serviceHostingEnvironment multipleSiteBindingsEnabled="true" />
</system.serviceModel>
<system.webServer>
<modules runAllManagedModulesForAllRequests="true"/>
</system.webServer>
Any idea what might be causing this issue? Could it be a permission issue or a configuration issue? I am confused as it works from my local build pointing to the clients test environment endpoint.
Thank you in advance!

Answer 1

I know this answer is late, but we had a similar issue with my Rest services, they worked fine on our development servers, then they stopped working when we deployed them to a load balanced pool and they made a call to another service that was also on the Load Balancing Pool.
We were able to get the call to work once we turned on Source Address Translation. Our Web admin had to change the Translation setting setting in the F5 pool to SNAT Pool. This link has more information for your network person.
https://support.f5.com/kb/en-us/products/lc_9_x/manuals/product/lc_config_guide_10_1/lc_addrtrans.html
So this might help in the case of when a service that was working fine, is now failing when calling something that is load balanced.