I am writing an integration webservice which will consume various webservices from a couple different backend systems. I want to be able to parallelize non-dependent service calls and be able to cancel requests that take too long (since I have an SLA to meet).
to aid in parallel backend calls, I am using the ASYNC client apis (generated by wsimport using the client-side jax-ws binding alteration files)
the issue I am having is that when I try to cancel a request, the Response<> appropriately marks the request as canceled, however the actual request is not really canceled. apparently some part of the JAX-WS runtime actually submits a com.sun.xml.ws.api.pipe.Fiber to the run queue which is what actually does the request. the cancel on the Result<> does not prevent these PIPEs from running on the queue and making the request.
has anyone run into this issue or a similar issue before?
My code looks like this:
List<Response<QuerySubscriberResponse>> resps = new ArrayList<Response<QuerySubscriberResponse>>();
for (int i = 0; i < 10; i++) {
resps.add(FPPort.querySubscriberAsync(req));
}
for (int i = 0; i < 10; i++) {
logger.info("Waiting for " + i);
try {
QuerySubscriberResponse re = resps.get(i).get(1,
TimeUnit.SECONDS); // execution time for this request is 15 seconds, so we should always get a TimeoutException
logger.info("Got: "
+ new Marshaller().marshalDocumentToString(re));
} catch (TimeoutException e) {
logger.error(e);
logger.error("Cancelled: " + resps.get(i).cancel(true));
try {
logger.info("Waiting for my timed out thing to finish -- technically I've canceled it");
QuerySubscriberResponse re = resps.get(i).get(); // this causes a CancelledExceptio as we would expect
logger.info("Finished waiting for the canceled req");
} catch (Exception e1) {
e1.printStackTrace();
}
} catch (Exception e) {
logger.error(e);
} finally {
logger.info("");
logger.info("");
}
}
I would expect that all of these requests would end up being cancelled, however in reality they all continue to execute and only return when the backend finally decides to send us a response.
as it turns out this was indeed a bug in the jax-ws implementation. Oracle has issued a Patch (RHEL) against wls 10.3.3 to address this issue.
Related
I'm running into a problem sending massive requests to a .NET Core web service. I'm using a SemaphoreSlim to limit the number of simultaneous requests. When I get a 10061 error (the web service has refused the connection), I want to dial back the number of simultaneous requests. My idea at the moment is to de-reference the SemaphoreSlim and create another:
await this.semaphoreSlim.WaitAsync().ConfigureAwait(false);
counter++;
Uri uri = new Uri($"{api}/{keyProperty}", UriKind.Relative);
string rowVersion = string.Empty;
try
{
HttpResponseMessage getResponse = await this.httpClient.GetAsync(uri).ConfigureAwait(false);
if (getResponse.IsSuccessStatusCode)
{
using (HttpContent httpContent = getResponse.Content)
{
JObject currentObject = JObject.Parse(await httpContent.ReadAsStringAsync().ConfigureAwait(false));
rowVersion = currentObject.Value<string>("rowVersion");
}
}
}
catch (HttpRequestException httpRequestException)
{
SocketException socketException = httpRequestException.InnerException as SocketException;
if (socketException != null && socketException.ErrorCode == PutHandler.ConnectionRefused)
{
this.semaphoreSlim = new SemaphoreSlim(counter * 90 / 100, counter * 90 / 100);
}
}
}
finally
{
this.semaphoreSlim.Release();
}
If I do this, what will happen to the other tasks that are waiting on the Semaphore that I just de-referenced? My guess is that nothing will happen until the object is garbage collected and disposed.
A SemaphoreSlim (just like any other object in .NET) will exist as long as there are references to it.
However, there is a bug in your code: the SemaphoreSlim being released is this.semaphoreSlim, and if this.semaphoreSlim is changed between being acquired and being released, then the code will release a different semaphore than the one that was acquired. To avoid this problem, copy this.semaphoreSlim into a local variable at the beginning of your method, and acquire and release that local variable.
More broadly, there's a difficult in the attempted solution. If you start 1000 tasks, they will all reference the old semaphore and ignore the updated this.sempahoreSlim. So you'd need a separate solution. For example, you could define a disposable "token" which is permission to call the API. Then have an asynchronous collection of these tokens (e.g., a Channel). This gives you full control over how many tokens are released at once.
I have an infinitely running process that pushes events from a server to subscribed SignalR clients. There may be long periods where no events take place on the server.
Currently, the process all works fine -- for a short period of time-- but eventually, the client stops responding to events pushed by the server. I can see the events taking place on the server-side, but the client becomes unaware of the event. I am assuming this symptom means some timeout period has been reached and the client has unsubscribed from the Hub.
I added some code to reconnect if the connection was dropped, and that has helped, but the client still eventually stops seeing new events. I know there are many different timeout values that can be adjusted, but it's all pretty confusing to me and not sure if I should even be tinkering with them.
try
{
myHubConnection = new HubConnectionBuilder()
.WithUrl(hubURL, HttpTransportType.WebSockets)
.AddMessagePackProtocol()
.AddJsonProtocol(options =>
{
options.PayloadSerializerSettings.ContractResolver = new DefaultContractResolver();
})
.Build();
// Client method that can be called by server
myHubConnection.On<string>("ReceiveInfo", json =>
{
// Action performed when method called by server
pub.ShowInfo(json);
});
try
{
// connect to Hub
await myHubConnection.StartAsync();
msg = "Connected to Hub";
}
catch (Exception ex)
{
appLog.WriteError(ex.Message);
msg = "Error: " + ex.Message;
}
// Reconnect lost Hub connection
myHubConnection.Closed += async (error) =>
{
try
{
await Task.Delay(new Random().Next(0, 5) * 1000);
await myHubConnection.StartAsync();
msg = "Reconnected to Hub";
appLog.WriteWarning(msg);
}
catch (Exception ex)
{
appLog.WriteError(ex.Message);
msg = "Error: " + ex.Message;
}
};
This all works as expected for a while, then stops without errors. Is there something I can do to (1) ensure the client NEVER unsubscribes, and (2) if the connection is lost (network outage for example) ensures the client resubscribes to the events. This client must NEVER timeout or give up trying to reconnect if required.
It seems that Ignite(2.0) Messaging's send function works in sync mode, it will be blocked be the listener. And below is my testing code.
ignite.message().localListen("TEST", (nodeId, Msg) -> {
try {
Thread.sleep(500);
} catch (Exception ex) {
}
return true;
});
for (int i = 0; i < 100; i++) {
ignite.message().send("TEST", "Hello World");
}
It cost about 50 seconds to send 100 messages, and it is almost equals the sleep time 500 ms * 100. seems the send function in sync mode not in async mode.
Does anybody know how to change the send function in async mode?
Thanks in advance.
Seems async listener invocation was missed while adding new API, but you still have two options:
Use deprecated withAsync(), unless sendAsync() will be added.
Pass your own Executor in predicate, if you always return true, for example.
I've just opened a ticket for that IGNITE-5570
It seems that you're testing within one node. In this case there is no message sent and listener is invoked synchronously. Network communication is asynchronous in Ignite, so if you do the testing on two nodes, you should not see such behavior.
I have setup a SignalR hub which has the following method:
public void SomeFunction(int SomeID)
{
try
{
Thread.Sleep(600000);
Clients.Caller.sendComplete("Complete");
}
catch (Exception ex)
{
// Exception Handling
}
finally
{
// Some Actions
}
m_Logger.Trace("*****Trying To Exit*****");
}
The issue I am having is that SignalR initiates and defaults to Server Sent Events and then hangs. Even though the function/method exits minutes later (10 minutes) the method is initiated again ( > 3 minutes) even when the sendComplete and hub.stop() methods are initiated/called on the client prior. Should the user stay on the page the initial "/send?" request stays open indefinitely. Any assistance is greatly appreciated.
To avoid blocking the method for so long, you could use a Taskand call the client method asynchronously.
public void SomeFunction(Int32 id)
{
var connectionId = this.Context.ConnectionId;
Task.Delay(600000).ContinueWith(t =>
{
var message = String.Format("The operation has completed. The ID was: {0}.", id);
var context = GlobalHost.ConnectionManager.GetHubContext<SomeHub>();
context.Clients.Client(connectionId).SendComplete(message);
});
}
Hubs are created when request arrives and destroyed after response is sent down the wire, so in the continuation task, you need to create a new context for yourself to be able to work with a client by their connection identifier, since the original hub instance will no longer be around to provide you with the Clients method.
Also note that you can leverage the nicer syntax that uses async and await keywords for describing asynchronous program flow. See examples at The ASP.NET Site's SignalR Hubs API Guide.
Is there a way to specify the wait time of retrying a message for a particular exception?
E.g. If object is in SomethingInProgress status, throws an SomethignInProgressException and I want to the message to be retry after 40m. Or is it more appropriate to raise a SomethingInProgressEvent and use bus.defer?
This is part of the reason why Rebus does not have the concept of second-level retries - I've simply not seen any way that this function could be created in a way that was generic and still flexible enough.
To answer your question shortly: No, there's no (built-in) way of varying the time between retries for a particular exception. In fact, there's no way to configure a wait time between retries at all - failing messages will be retried as fast as possibly, and then moved to the error queue if they keep failing to avoid "clogging up the pipes".
In your case, I suggest you do something like this:
public void Handle(MyMessage message) {
var headers = MessageContext.GetCurrent().Headers;
var deliveryAttempt = headers.ContainsKey("attempt_no")
? Convert.ToInt(headers["attempt_no"])
: 0;
try {
DoWhateverWithThe(message);
} catch(OneKindOfException e) {
if (deliveryAttempt > 5) {
bus.Advanced.Routing.ForwardCurrentMessage("error");
return;
}
bus.AttachHeader(message, "attempt_no", deliveryAttempt + 1);
bus.Defer(TimeSpan.FromSeconds(20), message);
} catch(AnotherKindOfException e) {
if (deliveryAttempt > 5) {
bus.Advanced.Routing.ForwardCurrentMessage("error");
return;
}
bus.AttachHeader(message, "attempt_no", deliveryAttempt + 1);
bus.Defer(TimeSpan.FromMinutes(2), message);
}
}
which I just wrote off the top of my head without being 100% certain that it actually compiles ... but the gist of it is that we track how many delivery attempts we've made in a custom header on the message, bus.Deferring the message an appropriate time span for each failed delivery attempt, immediately forwarding the message to the error queue when our max # of delivery attempts has been exceeded.
I hope that makes sense :)
A more recent example of how to do this is:
public async Task Handle(IFailed<MyMessage> message)
{
var maxAttempts = 10;
var optionalHeaders = new Dictionary<string, string>();
if (message.Headers != null && message.Headers.ContainsKey("attemptNumber"))
{
// increment the attempt number
var attemptNumber = int.Parse(message.Headers["attemptNumber"]);
attemptNumber++;
optionalHeaders.Add("attemptNumber", attemptNumber.ToString());
if (attemptNumber > maxAttempts)
{
// log I give up message, message will move to dead queue
return;
}
}
else
optionalHeaders.Add("attemptNumber", "1");
// if message failed to process, defer processing for 5 minutes and try again
await Bus.Defer(TimeSpan.FromMinutes(5), message.Message, optionalHeaders);
}