Mosquitto client consuming is slowing down and data gap increasing - .net-core

In a system with multiple devices, diagnostic data is published to separate topics for each device to Mosquitto Broker. A client application tries to consume data by subscribing to these topics separately for each device. The client receives 5 or 6 record per device per minute.
My problem is, after worker application started, the data received from Mosquitto Boroker starts to have delays on the client side. These delays start from 1 second, as time progresses, it lags behind by 1 day and 2 days during the time it is on. After stopping borker, the client continues to consume data.
Any ideas on what causes these delays?
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
string dataHost = _configuration.GetValue<string>("DataHost");
mqttClient = new MqttClient(dataHost);
mqttClient.MqttMsgPublishReceived += MqttClient_MqttMsgPublishReceived;
var list = _configuration.GetValue<string>("DeviceList").Split(',').ToList();
foreach (var deviceId in list)
{
mqttClient.Subscribe(
new string[] { $"{deviceId}/#" },
new byte[] { MqttMsgBase.QOS_LEVEL_AT_LEAST_ONCE });
}
mqttClient.ConnectionClosed += MqttClient_ConnectionClosed;
mqttClient.Connect("client", "username", "pass", false, 10);
}
private void MqttClient_MqttMsgPublishReceived(object sender, MqttMsgPublishEventArgs e)
{
try
{
var message = Encoding.UTF8.GetString(e.Message);
var messageArray = e.Topic.Split('/');
if (IsValidJson(message))
{
ItemValue itemValue = JsonConvert.DeserializeObject<ItemValue>(message);
_diagnosticRepository.InsertAsync(new DiagnosticDTO
{
value = messageArray[4].ToString(),
message = message,
messageTime = DateTime.Now,
deviceTime = itemValue.datetime,
sequenceNo = itemValue.sequenceNo,
});
}
}
catch (Exception ex)
{
_logger.LogError(ex.Message, "Consumer Insert Error");
}

Related

Method in event handlers are not called in SignalR after some time

On starting, the application with SignalR 2 implementation works as expected. But after sometime (it can be after 5 minutes or 30 minutes or 10 minutes), the events are not called anymore. But the Server and Client are still connected. I know it is connected because it the trace level of the client for SignalR, it can still get the commands coming from the Server.
But my messages from my client is still being received by the server and its action executed so I know that the Server - Client connection is still good.
Also, I know that it is still connected because I always check the status of the connection.
This is how I implement the SignalR
public void ConnectAsync()
{
WriteToLogs("Connecting to Server...");
var dic = new Dictionary<string, string> { { "UserName", _signalrSettings.ApplicationName } };
_hubConnection?.Dispose();
_hubConnection = new HubConnection(_signalrSettings.ServerUri, dic);
_hubConnection.StateChanged += Connection_StateChanged;
_hubProxy = _hubConnection.CreateHubProxy(_signalrSettings.HubProxy);
try
{
_hubProxy.On<string, string>("Initial", (a, b) => Initial());
_hubConnection.Start().ContinueWith((s) =>
{
if (s.IsCompleted)
{
_isCompleted = true;
}
else if (s.IsCanceled)
{
_isCompleted = false;
}
else if (s.IsFaulted)
{
_isCompleted = false;
}
_hubProxy.On<string, string>("NewMessage", (userName, message) => NewMessage(userName, message)); // THE "NewMessage" is not fired after some time.
WriteToLogs( $#"Connection status is {_hubConnection.State}");
}).Wait();
}
catch (Exception e)
{
var trace = new StackTrace(e, true);
var reflectedType = trace.GetFrame(0).GetMethod().ReflectedType;
if (reflectedType != null)
{
_messageOperation.WriteErrors($#"Error {e.StackTrace} at {trace}");
}
}
}

How can I read byte array coming from server in UWP app?

I have a client-server app. The client is HoloLens 2 and the server is unity running on PC. I have created a socket using HoloLens but I can't read the byte array on HoloLens. In the connect method _inputStream and _outputStream are System.IO.Streams.
PC client uses System.Net.Sockets.TcpClient, System.Net.Sockets.NetworkStream but UWP (HoloLens) uses Windows.Networking.Sockets.StreamSocket.
In the ReceiveCallback I have an issue with int byteLenght = _inputStream.EndRead(result); When I tried this code on HoloLens I got the exception: "System.ArgumentException: The specified AsyncResult does not correspond to any outstanding IO operation". But I tested it with PC client (System.Net.Socket) and everything works fine.
Btw, I gave all related permissions.
How can I read byte array in UWP app?
Update:
#if UNITY_EDITOR
private void ConnectUWP(string host, string port)
#else
private async void ConnectUWP(string host, string port)
#endif
{
#if UNITY_EDITOR
errorStatus = "UWP TCP client used in Unity!";
#else
try
{
socket = new Windows.Networking.Sockets.StreamSocket();
Windows.Networking.HostName serverHost = new Windows.Networking.HostName(host);
_receivedData = new Packet();
_receiveBuffer = new byte[DataBufferSize];
await socket.ConnectAsync(serverHost, port);
successStatus = "Connected!";
_outputStream = socket.OutputStream.AsStreamForWrite();
//_streamWriter = new StreamWriter(_outputStream) { AutoFlush = true };
_inputStream = socket.InputStream.AsStreamForRead();
Task.Run(() => StreamToBytes(socket.InputStream));
}
catch (Exception e)
{
errorStatus = e.ToString();
Debugger.GetComponent<TextMeshPro>().text += "\n Exception: " + errorStatus;
}
#endif
#if !UNITY_EDITOR
public async Task StreamToBytes(Windows.Storage.Streams.IInputStream stream)
{
using (var reader1 = new Windows.Storage.Streams.DataReader(stream))
{
reader1.InputStreamOptions = Windows.Storage.Streams.InputStreamOptions.ReadAhead;
reader1.UnicodeEncoding = Windows.Storage.Streams.UnicodeEncoding.Utf8;
reader1.ByteOrder = Windows.Storage.Streams.ByteOrder.LittleEndian;
//InBuffer is always 256,
//even if there is more data waiting. If I put a task.delay in it will always return 25
try
{
await reader1.LoadAsync(256);
}
catch (Exception ex)
{
//..
}
while (reader1.UnconsumedBufferLength > 0)
{
var bytes1 = new byte[reader1.UnconsumedBufferLength];
reader1.ReadBytes(bytes1);
_receivedData.Reset(await HandleData(bytes1));
await reader1.LoadAsync(256);
}
reader1.DetachStream();
}
}
private async Task<bool> HandleData(byte[] data)
{
int packetLength = 0;
_receivedData.SetBytes(data);
if (_receivedData.UnreadLength() >= 4)
{
// If client's received data contains a packet
packetLength = _receivedData.ReadInt();
if (packetLength <= 0)
{
// If packet contains no data
return true; // Reset receivedData instance to allow it to be reused
}
}
while (packetLength > 0 && packetLength <= _receivedData.UnreadLength())
{
// While packet contains data AND packet data length doesn't exceed the length of the packet we're reading
byte[] packetBytes = _receivedData.ReadBytes(packetLength);
ThreadManager.ExecuteOnMainThread(() =>
{
using (Packet packet = new Packet(packetBytes))
{
int packetId = packet.ReadInt();
_packetHandlers[packetId](packet); // Call appropriate method to handle the packet
}
});
packetLength = 0; // Reset packet length
if (_receivedData.UnreadLength() >= 4)
{
// If client's received data contains another packet
packetLength = _receivedData.ReadInt();
if (packetLength <= 0)
{
// If packet contains no data
return true; // Reset receivedData instance to allow it to be reused
}
}
}
if (packetLength <= 1)
{
return true; // Reset receivedData instance to allow it to be reused
}
return false;
}
#endif
When I tried this code on HoloLens I got the exception: "System.ArgumentException: The specified AsyncResult does not correspond to any outstanding IO operation".
Please refer to BeginRead method document, In the .NET Framework 4 and earlier versions, you have to use methods such as BeginRead and EndRead to implement asynchronous I/O operations. These methods are still available in the .NET Framework 4.5 to support legacy code; however, the new async methods, such as ReadAsync, WriteAsync, CopyToAsync, and FlushAsync, help you implement asynchronous I/O operations more easily. I suppose this old api could not compatible with HoloLens device, so please try to use ReadAsync to replace like the following
byte[] bytes = await StreamToBytes(_inputStream);
public async Task<byte[]> StreamToBytes(Stream stream)
{
byte[] bytes = new byte[stream.Length];
await stream.ReadAsync(bytes, 0, bytes.Length);
stream.Seek(0, SeekOrigin.Begin);
return bytes;
}
Or follow the official's processing that use DataReader to read ReadBytes, for more code please refer streamsocket.
Update
using (var reader1 = new DataReader(_inputStream)
{
reader1.InputStreamOptions = Windows.Storage.Streams.InputStreamOptions.Partial;
reader1.UnicodeEncoding = Windows.Storage.Streams.UnicodeEncoding.Utf8;
reader1.ByteOrder = Windows.Storage.Streams.ByteOrder.LittleEndian;
await reader1.LoadAsync(256); //InBuffer is always 256,
//even if there is more data waiting. If I put a task.delay in it will always return 25
while (reader1.UnconsumedBufferLength > 0)
{
var bytes1 = new byte[reader1.UnconsumedBufferLength];
reader1.ReadBytes(bytes1);
await reader1.LoadAsync(256);
}
reader1.DetachStream();
}
I tried everything for TCP but TCP is nearly impossible for HoloLens with UWP app. So I tried UDP and it works perfectly (https://github.com/mbaytas/HoloLensUDP). I hope Microsoft put a TCP example for HoloLens 1 and 2 in near future. Thanks Nico for your help.

No need to Dynamodb client while consuming data from Kinesis using kcl

I am using kcl api version 2 ,and dont want to use Dynamodb Client for storing the records .
private static final Logger LOG = LoggerFactory.getLogger(DisplayConsumerApplication.class);
public static void main(String... args) {
KinesisAsyncClient kinesisClient = KinesisAsyncClient.builder().credentialsProvider(ProfileCredentialsProvider.create())
.region(Region.of("US-EAST-1")).build();
//DynamoDbAsyncClient dynamoClient =
// DynamoDbAsyncClient.builder().credentialsProvider(ProfileCredentialsProvider.
// create()) .region(Region.of("US-EAST-1")).build();
CloudWatchAsyncClient cloudWatchClient = CloudWatchAsyncClient.builder().credentialsProvider(ProfileCredentialsProvider.create())
.region(Region.of("US-EAST-1")).build();
ConfigsBuilder configsBuilder = new ConfigsBuilder("Sample","Sample", kinesisClient,null,
cloudWatchClient, UUID.randomUUID().toString(), new DisplayConsumerFactory());
Scheduler scheduler = new Scheduler(configsBuilder.checkpointConfig(), configsBuilder.coordinatorConfig(),
configsBuilder.leaseManagementConfig(), configsBuilder.lifecycleConfig(), configsBuilder.metricsConfig(),
configsBuilder.processorConfig(), configsBuilder.retrievalConfig());
Thread schedulerThread = new Thread(scheduler);
schedulerThread.setDaemon(true);
schedulerThread.start();
System.out.println("Press enter to shutdown");
BufferedReader reader = new BufferedReader(new InputStreamReader(System.in));
try {
reader.readLine();
} catch (IOException ioex) {
LOG.error("Caught exception while waiting for confirm. Shutting down", ioex);
}
Future<Boolean> gracefulShutdownFuture = scheduler.startGracefulShutdown();
LOG.info("Waiting up to 20 seconds for shutdown to complete.");
try {
gracefulShutdownFuture.get(20, TimeUnit.SECONDS);
} catch (InterruptedException e) {
LOG.info("Interrupted while waiting for graceful shutdown. Continuing.");
} catch (ExecutionException e) {
LOG.error("Exception while executing graceful shutdown.", e);
} catch (TimeoutException e) {
LOG.error("Timeout while waiting for shutdown. Scheduler may not have exited.");
}
LOG.info("Completed, shutting down now.");
}
}
As you can see I commented initialize of DynamodbClient ,but in that method it is manadatory to pass the object of Dynamoclient .So I passed as null ,but getting null pointer exception ,Could you please share your idea how I can use Scheduler without dynamodb client ?

C# detect TCP client disconnection

I am working on TCP multithread server with C# window application form, and I am trying to detect if the machine of client is shutdown and disconnects from the server. I have read some posts and have some ideas:
How to determine if the tcp is connected or not?
Instantly detect client disconnection from server socket
But I'm not sure where to call the function IsConnected
My code is like the following:
public BindingList<Tablet> tabletList = new BindingList<Tablet>();
private Socket socket_Server = null;
private Thread myThread = null;
private Socket socket_Connect = null;
private Dictionary<string, Socket> dic = new Dictionary<string, Socket> { };
private string RemoteEndPoint;
socket_Server = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
IPAddress ServerIP = IPAddress.Parse("192.168.2.146");
IPEndPoint point = new IPEndPoint(ServerIP, portNum);
socket_Server.Bind(point);
socket_Server.Listen(50);
myThread = new Thread(Listen_Disp);
myThread.IsBackground = true;
myThread.Start();
Console.WriteLine("Server start");
private void Listen_Disp()
{
try
{
while (true)
{
//This is not working
for (int i = 0; i < tabletList.Count; i++)
{
if (!SocketConnected(dic[tabletList[i].ip]))
{
Console.WriteLine(RemoteEndPoint + "disconnected");
}
}
try
{
socket_Connect = socket_Server.Accept();
RemoteEndPoint = socket_Connect.RemoteEndPoint.ToString();
Console.WriteLine(RemoteEndPoint + " is connected");
dic.Add(RemoteEndPoint, socket_Connect);
Tablet newTablet = new Tablet();
newTablet.ip = RemoteEndPoint;
newTablet.status = "Online";
tabletList.Add(newTablet);
}
catch (Exception ex)
{
Console.WriteLine(ex.ToString());
}
}
Console.WriteLine("end of while");
}
catch (Exception ex)
{
Console.WriteLine(ex.ToString());
}
}
static class SocketExtensions
{
public static bool IsConnected(this Socket socket)
{
try
{
return !(socket.Poll(1, SelectMode.SelectRead) && socket.Available == 0);
}
catch (SocketException) { return false; }
}
}
Thanks for help.
There's a lot of misinformation on that subject, and some of it is present in the questions you linked. The only reliable way to detect a TCP disconnection is to try to write to the connection. A read timeout can also indicate a dropped connection, but it may mean a lot of other things too, such as a stuck server. An EOS condition when reading indicates a graceful disconnect. The IsConnected() method and friends only give you a history of what you have done to this socket: they don't give you the current state of the connection. They can't, because, absent a pending write request, there is no state to
know. TCP doesn't maintain anything akin to a dial tone.

Wcf + asp.net + multiple threads

I have the following scenario:
A Service that does nothing but sleeps for the amount of time the WPF specifies through the WebRequest onject:
public class WorkerSvc : IWorkerSvc
{
#region IWorkerSvc Members
public void DoWork(int timeToSleep)
{
Trace.WriteLine(DateTime.Now.ToString() + "\tInside Stress Service with TimeToSleep: " + timeToSleep.ToString());
if (timeToSleep == 0)
return;
Thread.Sleep(timeToSleep * 1000);
Trace.WriteLine(DateTime.Now.ToString() + "\tThe Thread woke up.");
}
Then an aspx page (separate project) that calls into the service:
protected void Page_Load(object sender, EventArgs e)
{
System.Diagnostics.Trace.WriteLine(DateTime.Now.ToString() + " \tInside default stress web site page load;");
using (WorkerService.WorkerSvcClient client = new StressWebSite.WorkerService.WorkerSvcClient())
{
System.Diagnostics.Trace.WriteLine(DateTime.Now.ToString() + " \tCreated a new client of Stress WCF Service;");
var rowdata = Request.QueryString["SleepValue"];
System.Diagnostics.Trace.WriteLine(DateTime.Now.ToString() + " \tDetected Sleep Value in the Request: " + rowdata);
int realData = 0;
if (!string.IsNullOrEmpty(rowdata))
{
if (int.TryParse(rowdata, out realData))
{
System.Diagnostics.Trace.WriteLine(DateTime.Now.ToString() + " \tBefore Dowork() with SleepValue: " + realData);
client.DoWork(realData);
System.Diagnostics.Trace.WriteLine(DateTime.Now.ToString() + " \tAfter DoWork() with SleepValue: " + realData);
Response.Write(realData.ToString());
//Response.End();
}
}
}
}
A WPF Form That spanws a number of threads which essentially post some data to an aspx page:
private void PerformStress()
{
WebRequest request = WebRequest.Create(string.Concat(this.txtStressPageUrl.Text, "?", "SleepValue", "=", this.txtSleepValue.Text));
Trace.WriteLine("Created Web Request");
request.Credentials = CredentialCache.DefaultCredentials;
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Stream dataStream = response.GetResponseStream();
StreamReader reader = new StreamReader(dataStream);
string responseFromServer = reader.ReadToEnd();
//return responseFromServer;
}
private void button3_Click(object sender, RoutedEventArgs e)
{
int stressThreadsCount = 0;
if (!int.TryParse(this.txtStressThreadCount.Text, out stressThreadsCount))
MessageBox.Show("Enter number of threads to stress");
ThreadStart start = delegate()
{
DispatcherOperation op = Dispatcher.BeginInvoke(DispatcherPriority.Normal, new Action(PerformStress));
DispatcherOperationStatus status = op.Status;
while (status != DispatcherOperationStatus.Completed)
{
status = op.Wait(TimeSpan.FromMilliseconds(1000));
if (status == DispatcherOperationStatus.Aborted)
{
//
}
}
};
for (int i = 0; i < stressThreadsCount; i++)
{
var t = new Thread(start);
//this.runningThreads.Add(t);
t.Start();
//t.Join();
}
}
The problem is that it looks like eventhough I spawn several threads from the WPF side, to post to the aspx page, all the threads are processed in a serial manner, meaning each thread is waiting for the service to return.
I run this set up under cassiny.
I did not specify the WCF service's behavior to be a singleton.
Please help to ID the issue - I'd like to simulate multiple requests to the IIS and a page that calls into WCF to prove a concept.
Singleton will only affect how many objects will be created to service requests. By default, though, WCF services will only process a single request at a time. To change that, you should decorate your service implementation with
[ServiceBehavior(ConcurrencyMode = ConcurrencyMode.Multiple)]
For more info, see here: http://msdn.microsoft.com/en-us/library/system.servicemodel.concurrencymode.aspx

Resources