Sending WCF messages being delayed under load - networking

Question

When sending messages from a self hosted WCF service to many clients (about 10 or so), sometimes messages are being delayed significantly longer than I'd expect (several seconds to send to a client on local network). Does anyone have an idea why this would be and how to fix it?
Some background: the application is a stock ticker style service. It receives messages from a 3rd party server and re-publishes them to clients that connect to the service. It's very important that messages are published as quickly as possible, and in most cases the time between receiving a message and publishing it to all clients is less than 50ms (it's so quick it approaches the resolution of DateTime.Now).
Over the past few weeks, we've been monitoring some occasions when messages are delayed by 2 or 3 seconds. A few days ago, we got a big spike and messages were being delayed by 40-60 seconds. Messages are not being dropped as far as I can tell (unless the entire connection is dropped). The delays does not appear to be specific to any one client; it affects all clients (including ones on the local network).
I send messages to the clients by spamming the ThreadPool. As quickly as messages arrive I call BeginInvoke() once per message per client. The theory being that if any one client is slow to receive a message (because it's on dialup and downloading updates or something) that it won't impact other clients. That isn't what I'm observing though; it appears that all clients (including ones on the local network) are impacted by the delay by a similar duration.
The volume of messages I'm dealing with is 100-400 per second. Messages contain a string, a guid, a date and, depending on the message type, 10-30 integers. I've observed them using Wireshark as being less than 1kB each. We have 10-20 clients connected at any one time.
The WCF server is being hosted in a Windows service on a Windows 2003 Web Edition Server. I'm using the NetTCP binding with SSL/TLS encryption enabled and a custom username / password authentication. It has a 4Mbit internet connection, dual core CPU and 1GB ram and is dedicated to this application. The service is set to ConcurrencyMode.Multiple. The service process, even under high load, rarely exceeds 20% CPU usage.
So far, I've tweaked various WCF configuration options such as:
serviceBehaviors/serviceThrottling/maxConcurrentSessions (currently 102)
serviceBehaviors/serviceThrottling/maxConcurrentCalls (currently 64)
bindings/netTcpBinding/binding/maxConnections (currently 100)
bindings/netTcpBinding/binding/listenBacklog (currently 100)
bindings/netTcpBinding/binding/sendTimeout (currently 45s, although I've tried it as high as 3 minutes)
It appears to me like the messages are being queued inside WCF once some threshold is reached (hence why I've being increasing the throttling limits). But to affect all clients it would need to max out all outgoing connections with one or two slow clients. Does anyone know if this is true of the WCF internals?
I can also improve efficiency by coalescing incoming messages when I send them to the client. However, I suspect there's something underlying going on and coalescing won't fix the problem in the long term.
WCF Config (with company names changed):
<system.serviceModel>
<host>
<baseAddresses>
<add baseAddress="net.tcp://localhost:8100/Publisher"/>
</baseAddresses>
</host>
<endpoint address="ThePublisher"
binding="netTcpBinding"
bindingConfiguration="Tcp"
contract="Company.Product.Server.Publisher.IPublisher" />
</behavior>
Code used to send messages:
Private Sub HandleDataBackground(ByVal sender As Object, ByVal e As Timers.ElapsedEventArgs)
If Me._FeedDataQueue.Count > 0 Then
' Dequeue any items received in last 50ms.
While True
Dim dataAndReceivedTime As DataWithReceivedTimeArg
SyncLock Me._FeedDataQueue
If Me._FeedDataQueue.Count = 0 Then Exit While
dataAndReceivedTime = Me._FeedDataQueue.Dequeue()
End SyncLock
' Publish data to all clients.
Me.SendDataToClients(dataAndReceivedTime)
End While
End If
End Sub
Private Sub SendDataToClients(ByVal data As DataWithReceivedTimeArg)
Dim clientsToReceive As IEnumerable(Of ClientInformation)
SyncLock Me._ClientInformation
clientsToReceive = Me._ClientInformation.Values.Where(Function(c) Contract.CollectionContains(c.ContractSubscriptions, data.Data.Contract) AndAlso c.IsUsable).ToList()
End SyncLock
For Each clientInfo In clientsToReceive
Dim futureChangeMethod As New InvokeClientCallbackDelegate(Of DataItem)(AddressOf Me.InvokeClientCallback)
futureChangeMethod.BeginInvoke(clientInfo, data.Data, AddressOf Me.SendDataToClient)
Next
End Sub
Private Sub SendDataToClient(ByVal callback As IFusionIndicatorClientCallback, ByVal data As DataItem)
' Send
callback.ReceiveData(data)
End Sub
Private Sub InvokeClientCallback(Of DataT)(ByVal client As ClientInformation, ByVal data As DataT, ByVal method As InvokeClientCallbackMethodDelegate(Of DataT))
Try
' Send
If client.IsUsable Then
method(client.CallbackObject, data)
client.LastContact = DateTime.Now
Else
' Make sure the callback channel has been removed.
SyncLock Me._ClientInformation
Me._ClientInformation.Remove(client.SessionId)
End SyncLock
End If
Catch ex As CommunicationException
....
Catch ex As ObjectDisposedException
....
Catch ex As TimeoutException
....
Catch ex As Exception
....
End Try
End Sub
A sample of one of the message types:
<DataContract(), KnownType(GetType(DateTimeOffset)), KnownType(GetType(DataItemDepth)), KnownType(GetType(DataItemDepthDetail)), KnownType(GetType(DataItemHistory))> _
Public MustInherit Class DataItem
Implements ICloneable
Protected _Contract As String
Protected _MessageId As Guid
Protected _TradeDate As DateTime
<DataMember()> _
Public Property Contract() As String
...
End Property
<DataMember()> _
Public Property MessageId() As Guid
...
End Property
<DataMember()> _
Public Property TradeDate() As DateTime
...
End Property
Public MustOverride Function Clone() As Object Implements System.ICloneable.Clone
End Class
<DataContract()> _
Public Class DataItemDepth
Inherits DataItem
Protected _VolumnPriceDetail As IList(Of DataItemDepthItem)
<DataMember()> _
Public Property VolumnPriceDetail() As IList(Of DataItemDepthItem)
...
End Property
Public Overrides Function Clone() As Object
...
End Function
End Class
<DataContract()> _
Public Class DataItemDepthItem
Protected _Volume As Int32
Protected _Price As Int32
Protected _BidOrAsk As BidOrAsk ' BidOrAsk is an Int32 enum
Protected _Level As Int32
<DataMember()> _
Public Property Volume() As Int32
...
End Property
<DataMember()> _
Public Property Price() As Int32
...
End Property
<DataMember()> _
Public Property BidOrAsk() As BidOrAsk ' BidOrAsk is an Int32 enum
...
End Property
<DataMember()> _
Public Property Level() As Int32
...
End Property
End Class

Answer 1

After a long support request with Microsoft support, we managed to identify the issue.
Calling WCF channel methods using Begin/End Invoke delegate pattern actually turns into synchronous calls, not asynchronous.
The correct way to asynchronously call WCF methods is by any way except async delegates, which may include the thread pool, raw threads or WCF async callbacks.
In the end I used WCF async callbacks (which can be applied to a callback interface, although I couldn't find specific examples of that).
The following link makes this more explicit:
https://learn.microsoft.com/en-us/archive/blogs/drnick/begininvoke-bugs