i'am using a DEALER-ROUTER pattern to make a sort of load balancing between client(s) and workers.
I'am wondering how to set a timeout between the client and the server to avoid infinite waits.
I tried to set the timeout parameter of the TryReceiveMultipartMessage method but it has no effect. I forced my server to reply after 10s and even setting a timeout to 1s I recieve the response.
My client code is :
public void Main(string Args[])
{
using(ctx = NetMQContext.Create())
{
using(client = ctx.CreateDealerSocket())
{
client.Options.Identity = Encoding.Unicode.GetBytes(Guid.NewGuid().ToString());
client.Connect("tcp://127.0.0.1:5556");
client.ReceiveReady += Client_ReceiveReady;
poller.AddSocket(client);
poller.PollTillCancelled();
}
}
void Client_ReceiveReady(object sender, NetMQSocketEventArgs e)
{
NetMQMessage clientMessage = new NetMQMessage();
TimeSpan timeout = TimeSpan.FromMilliseconds(1000);
if (e.Socket.TryReceiveMultipartMessage(timeout, ref clientMessage))
{
...
}
}
How can I set a Timeout on recieving message from a DEALER socket?
Thank you for your help.
Related
I'm working on a basic (non DB) connection pool which allows only 1 connection to be created per project.
The connection pool supports an async-task/threaded environment and therefor I have made use of a semaphore instead of a regular Lock.
I wrote a test, below, which is meant to stress test the connection pool.
The code works but under higher loads, the semaphore throws the following error
I can overcome this error by decreasing the load.
For example, increasing the _waitTimeMs to a higher number (i.e. 50ms or 100ms or 1000ms) or decreasing _numberOfTasks (i.e. to 5 or 3).
I should also mention that sometimes, it manages to run higher load tests without errors.
Is there a mistake or misconception in my code and/or use of semaphores?
using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.Threading;
using System.Threading.Tasks;
internal class Program
{
static int _numberOfTasks = 50;
static int _waitTimeMs = 1;
static SemaphoreSlim _dictLock = new SemaphoreSlim(1, 1);
static ConcurrentDictionary<string, bool> _pool = new ConcurrentDictionary<string, bool>();
/// <summary>
/// Only 1 connection allowed per project.
/// We reuse connections if available in pool, otherwise we create 1 new connection.
/// </summary>
static async Task<string> GetConnection(string projId)
{
try
{
// Enter sema lock to prevent more than 1 connection
// from being added for the same project.
if (await _dictLock.WaitAsync(_waitTimeMs))
{
// Try retrieve connection from pool
if (_pool.TryGetValue(projId, out bool value))
{
if (value == false)
return "Exists but not connected yet.";
else
return "Success, exists and connected.";
}
// Else add connection to pool
else
{
_pool.TryAdd(projId, false);
// Simulate delay in establishing new connection
await Task.Delay(2);
_pool.TryUpdate(projId, true, false);
return "Created new connection successfully & added to pool.";
}
}
// Report failure to acquire lock in time.
else
return "Server busy. Please try again later.";
}
catch (Exception ex)
{
return "Error " + ex.Message;
}
finally
{
// Ensure our lock is released.
_dictLock.Release();
}
}
static async Task Main(string[] args)
{
if (true)
{
// Create a collection of the same tasks
List<Task> tasks = new List<Task>();
for (int i = 0; i < _numberOfTasks; i++)
{
// Each task will try to get an existing or create new connection to Project1
var t = new Task(async () => { Console.WriteLine(await GetConnection("Project1")); });
tasks.Add(t);
}
// Execute these tasks in parallel.
Parallel.ForEach<Task>(tasks, (t) => { t.Start(); });
Task.WaitAll(tasks.ToArray());
Console.WriteLine("Done");
Console.Read();
}
}
}
Is there a mistake or misconception in my code and/or use of semaphores?
There's a bug in your code, yes. If the WaitAsync returns false (indicating that the semaphore was not taken), then the semaphore is still released in the finally block.
If you must use a timeout with WaitAsync (which is highly unusual and questionable), then your code should only call Release if the semaphore was actually taken.
I have an application that uses Spring Integration to send messages to a vendor application over TCP and receive and process responses. The vendor sends messages without a length header or an message-ending token and the message contains carriage returns so I have implemented a custom deserializer. The messages are sent as XML strings so I have to process the input stream, looking for a specific closing tag to know when the message is complete. The application works as expected until the vendor application is restarted or a port switch occurs on my application, at which time the CPU usage on my application spikes and the application becomes unresponsive. The application throws a SocketException: o.s.integration.handler.LoggingHandler : org.springframework.messaging.MessagingException: Send Failed; nested exception is java.net.SocketException: Connection or outbound has closed when the socket closes. I have set the SocketTimeout to be 1 minute.
Here is the connection factory implementation:
#Bean
public AbstractClientConnectionFactory tcpConnectionFactory() {
TcpNetClientConnectionFactory factory = new TcpNetClientConnectionFactory(this.serverIp,
Integer.parseInt(this.port));
return getAbstractClientConnectionFactory(factory, keyStoreName, trustStoreName,
keyStorePassword, trustStorePassword, hostVerify);
}
private AbstractClientConnectionFactory getAbstractClientConnectionFactory(
TcpNetClientConnectionFactory factory, String keyStoreName, String trustStoreName,
String keyStorePassword, String trustStorePassword, boolean hostVerify) {
TcpSSLContextSupport sslContextSupport = new DefaultTcpSSLContextSupport(keyStoreName,
trustStoreName, keyStorePassword, trustStorePassword);
DefaultTcpNetSSLSocketFactorySupport tcpSocketFactorySupport =
new DefaultTcpNetSSLSocketFactorySupport(sslContextSupport);
factory.setTcpSocketFactorySupport(tcpSocketFactorySupport);
factory.setTcpSocketSupport(new DefaultTcpSocketSupport(hostVerify));
factory.setDeserializer(new MessageSerializerDeserializer());
factory.setSerializer(new MessageSerializerDeserializer());
factory.setSoKeepAlive(true);
factory.setSoTimeout(60000);
return factory;
}
Here is the deserialize method:
private String readUntil(InputStream inputStream) throws IOException {
ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
String s = "";
byte[] closingTag = CLOSING_MESSAGE_TAG.getBytes(ASCII);
try {
Integer bite;
while (true) {
bite = inputStream.read();
byteArrayOutputStream.write(bite);
byte[] bytes = byteArrayOutputStream.toByteArray();
int start = bytes.length - closingTag.length;
if (start > closingTag.length) {
byte[] subarray = Arrays.copyOfRange(bytes, start, bytes.length);
if (Arrays.equals(subarray, closingTag)) {
s = new String(bytes, ASCII);
break;
}
}
}
} catch (SocketTimeoutException e) {
logger.error("Expected SocketTimeoutException thrown");
} catch (Exception e) {
logger.error("Exception thrown when deserializing message {}", s);
throw e;
}
return s;
}
Any help in identifying the cause of the CPU spike or a suggested fix would be greatly appreciated.
EDIT #1
Adding serialize method.
#Override
public void serialize(String string, OutputStream outputStream) throws IOException {
if (StringUtils.isNotEmpty(string) && StringUtils.startsWith(string, OPENING_MESSAGE_TAG) &&
StringUtils.endsWith(string, CLOSING_MESSAGE_TAG)) {
outputStream.write(string.getBytes(UTF8));
outputStream.flush();
}
}
the inbound-channel-adapter uses the ConnectionFactory
<int-ip:tcp-inbound-channel-adapter id="tcpInboundChannelAdapter"
channel="inboundReceivingChannel"
connection-factory="tcpConnectionFactory"
error-channel="errorChannel"
/>
EDIT #2
Outbound Channel Adapter
<int-ip:tcp-outbound-channel-adapter
id="tcpOutboundChannelAdapter"
channel="sendToTcpChannel"
connection-factory="tcpConnectionFactory"/>
Edit #3
We have added in the throw for the Exception and are still seeing the CPU spike, although it is not as dramatic. Could we still be receiving bytes from socket in the inputStream.read() method? The metrics seem to indicate that the read method is consuming server resources.
#Artem Bilan Thank you for your continued feedback on this. My server metrics seem to indicate that they deserialize method is what is consuming the CPU. I was thinking that the SendFailed error occurs because of the vendor restarting their application.
Thus far, I have been unable to replicate this issue other than in production. The only exception I can find in production logs is the SocketException mentioned above.
Thank you.
I have strange problem with my Game written in Unity (Mono). I have login functionality and then, after successfull login i'm sending some kind of keep-alive requests to ensure token is updated (I'm sending it every 30s).
The problem is, that after some time (sometimes it's 1 hour, sometimes 2.5 hours) all my requests have timeout status.
To be sure about my connection status I made some checks in code: I'm making simple GET to http://google.com and to main website of my API (not an API call. Just website GET). When I got timeout on API next time i figured out that:
After 1 timeout I have it always. Restarting application helps
I'm getting timeouts on API call and on GET request for base API website
Google is still responding with status 200 (no timeouts here)
Implementation:
On beginning i've been using RestSharp to handle requests, but the problem occured and there was a decision to throw RestSharp away and now we're using classic WebClient
class BetterWebClient : WebClient
{
private WebRequest _Request = null;
public TimeSpan? Timeout { get; set; }
protected override WebRequest GetWebRequest(Uri address)
{
this._Request = base.GetWebRequest(address);
if ( Timeout.HasValue )
{
_Request.Timeout = (int)Timeout.Value.TotalMilliseconds;
}
else
{
_Request.Timeout = 10*1000; //10s
}
if (this._Request is HttpWebRequest)
{
((HttpWebRequest)this._Request).AllowAutoRedirect = true;
}
return this._Request;
}
}
My HandleRequest function (which also calls Google and API website) looks like this:
public static void HandleRequest<TR>(string url, RestResponse<TR> executeGetRequest) where TR : new()
{
using (BetterWebClient w = new BetterWebClient() {Timeout = TimeSpan.FromSeconds(2)})
{
try
{
string downloadString = w.DownloadString("http://www.google.com");
Debug.Log("Google request: OK");
}
catch ( Exception )
{
Debug.LogError("Google request failed");
}
try
{
string downloadString = w.DownloadString("myAPIwebsite");
Debug.Log("WorldOfRescue.com request: OK");
}
catch ( Exception )
{
Debug.LogError("WorldOfRescue.com request failed");
}
}
var client = new BetterWebClient() {Timeout = TimeSpan.FromSeconds(5)};
client.Headers.Add("Accept", "application/json");
client.Headers.Add("Content-Type", "application/json");
try
{
string downloadString = client.DownloadString(url);
Debug.Log("Request for "+url+ " completed. Response: " + downloadString);
}
catch ( WebException e )
{
Debug.Log(e.Status);
throw;
}
catch ( Exception e )
{
Debug.Log(e.ToString());
throw;
}
finally
{
client.Dispose();
}
}
Do you have any idea why it happens? It looks like something is blocking me to send request to specific website, but in same time other website is working fine.
I am trying to throttle outgoing http requests using Jersey Client. Since I am running is a Vertx Verticle I created a special RateLimiter class to handle throttling.
My goal is to prevent HTTP calls from being made at a greater rate than 1 per second. the idea is that a submitted callable will run using the single threaded ExecutorService so that I can block that single thread in order to guarantee that these tasks are not handled in a greater rate.
Basically the only public method in this class is "call" :
public <T> Observable<T> call(Callable<Observable<T>> action) {
return Observable.create(subscriber -> {
Observable<Observable<T>> observed =
Observable.from(executor.submit(() -> {
return action.call();
})
).doOnError(throwable -> {
logger.error(throwable);
}
);
observed.subscribe(t -> {
try {
Thread.sleep(1000);
t.subscribe(data -> {
try {
subscriber.onNext(data);
} catch (Throwable e) {
subscriber.onError(e);
}
subscriber.onCompleted();
});
} catch (Exception e) {
logger.error(e);
}
});
});
}
this is my current implementation which uses 1 second sleep no matter how much time has passed since the previous call. initially I tried using a ScheduledExecutorService and calculate the delay time so that I will submit requests exactly at the rate of 1 per second. however, in both cases it often fails to meet the rate restrictions and I get two requests submitted immediately one after the other.
My assumption is that somewhere the requests is being handed to a different executing queue which is being polled by a different thread continuously, so that if for some reason that thread was busy and two requests exist in the queue at the same time, they will be executed sequentially but with no delays.
Any Ideas how to resolve this? maybe a different approach?
I would go with simple Vertx event bus and a queue, from which you poll every second:
public static void main(String[] args) {
Vertx vertx = Vertx.vertx();
vertx.deployVerticle(new DebounceVerticle(), (r) -> {
// Ok, verticle is ready!
// Request to send 10 events in 1 second
for (int i = 0; i < 10; i++) {
vertx.eventBus().publish("call", UUID.randomUUID().toString());
}
});
}
private static class DebounceVerticle extends AbstractVerticle {
HttpClient client;
#Override
public void start() {
client = vertx.createHttpClient();
BlockingQueue<String> queue = new LinkedBlockingQueue<>();
vertx.eventBus().consumer("call", (payload) -> {
String message = (String) payload.body();
queue.add(message);
System.out.println(String.format("I got %s but I don't know when it will be executed", message));
});
vertx.setPeriodic(1000, (l) -> {
String message = queue.poll();
if (message != null) {
System.out.println(String.format("I'm finally sending %s", message));
//Do your client magic
}
});
}
}
Just prepend web service call with guava RateLimiter. Here's an example in RxJava which shows how events every 500ms are throttled to be once per second.
Function<Long, Long> throttlingFunction = new Function<Long, Long>() {
private RateLimiter limiter = RateLimiter.create(1.0);
public Long apply(Long t) throws Exception {
limiter.acquire();
return t;
}
};
Observable.interval(500, TimeUnit.MILLISECONDS)
.map(throttlingFunction)
.subscribe(new Consumer<Long>() {
public void accept(Long t) throws Exception {
System.out.println(t);
}
});
Also in vert.x all the blocking stuff is supposed to be run with the help of executeBlocking.
We are using SignalR to push messages from server to client. Some of the things we are using server broadcast for:
Live notifications
Updating changes of shared data
Chat like functionality
One of our devs started with the StockTicker example, and we expanded it to push all of our different message types. Here is our general scheme:
private void RunJobs()
{
_jobs = GetAllJobs();
while (true)
{
bool workDone = false;
for (int i = 0; i < _jobs.Count; i++)
{
var j = _jobs.ElementAt(i);
bool workToDo = j.MessageAvailable();
workDone = workDone || workToDo;
if (workToDo)
{
var message = j.GetMessage();
_threadPool.QueueWorkItem(ProcessJob, j, message);
}
}
if (!workDone)
{
Thread.Sleep(_sleepTime);
}
}
}
/// <summary>
/// Method called by threads to process queued up Work Item (ISignalRJob)
/// </summary>
/// <param name="job">Job to run.</param>
private void ProcessJob(ISignalRJob job, QueueMessage message)
{
try
{
job.ProcessMessage(message);
}
catch (Exception e)
{
//handle exception
}
}
As each job processes, it performs an operation like:
protected override void ProcessMessage(QueueMessage message)
{
var nqm = JsonConvert.DeserializeObject<NotificationQueueMessage>(message.Body);
var notification = webService.GetNotification(notification.Id);
foreach(var userConnectionId in GetUserConnectionIds(nqm.UserId)){
_signalRConnectionContext.Clients.Client(userConnectionId).pushNotification(notification);
}
}
In a thread, monitor a series of queues for messages. If a message turns up, pop the message off the queue, and start a new thread to process the message (ProcessJob). The jobs will then do any service calls / db calls necessary to build the client message, then push the message to the client.
The service seems to work, but periodically the client will stop receiving the messages, although I have verified they are being sent from the server. Is it possible that pushing to a client connection in multiple threads is putting it in a bad state?
Should I perhaps be returning the result of the QueueMessage processing to the main SignalR thread, and return them all synchronously?