Trying to create a non retry-able listener, on deserializataion error, it should just print the suspect message as string and move on.
However, when explicitly setting the DefaultErrorHandler (in an effort to see msg/payload body), it goes into a retry loop. Without setting it, it just prints the exception msg (expected string but got null) and moves on.
I've tried setting Backoff with 0 retries with no luck. Also I'm still unable to see the contents of the suspect message.
#Bean
public ConcurrentKafkaListenerContainerFactory<String, GenericRecord> kafkaListenerCAFMessageContainerFactory() {
ConcurrentKafkaListenerContainerFactory<String, GenericRecord> factory = new ConcurrentKafkaListenerContainerFactory<>();
factory.setConsumerFactory(getCafKafkaConfig());
factory.getContainerProperties().setAckMode(ContainerProperties.AckMode.MANUAL);
factory.setConcurrency(consumerConcurrency);
DefaultErrorHandler defaultErrorHandler = new DefaultErrorHandler((record, exception) -> {
// trying to print the payload that doesn't serialize
LOGGER.error(exception.getMessage() + record.value().toString()); // but record.value() is always null
});
#org.springframework.kafka.annotation.KafkaListener(topics = "gcrs_process_events", containerFactory = "kafkaListenerCAFMessageContainerFactory")
public void listenCafMsgs(ConsumerRecord<String, GenericRecord> record, Acknowledgment ack) {...}
The value has to be null so that the deserializer is type safe - for example, if you are using the JsonDeserializer to convert byte[] to a Foo - it is not type safe if it returns a record with a byte[] payload.
See the cause of the ListenerExecutionFailedException - it is a DeserializationException which has the data field.
/**
* Get the data that failed deserialization (value or key).
* #return the data.
*/
public byte[] getData() {
return this.data; // NOSONAR array reference
}
Related
We are using the Rebus framework in our application and we currently have some issues adding additional headers before the messages are handled by our handlers.
In my startup:
}).Events(e =>
{
e.BeforeMessageSent += rebusEventHandler.RebusBeforeMessageSent;
e.BeforeMessageHandled += rebusEventHandler.RebusBeforeMessageHandled;
})
Catching the event:
public void RebusBeforeMessageHandled(IBus bus, Dictionary<string, string> headers, object message, IncomingStepContext context, MessageHandledEventHandlerArgs args)
{
… we are fetching mocking headers here
foreach (var mockingHeader in mockingHeaders)
{
headers.TryAdd(mockingHeader.Key, mockingHeader.Value); // values added here are gone in the next step
}
headers["my-test"] = "test"; //added for testing (also not saved)
}
When I look at the messageContext.headers in the next handler, the headers I set in the BeforeMessageHandled event are gone again. The my-test header is also gone. So it seems that the modified headers are not saved.
We also use the BeforeMessageSent event and those headers are saved as expected.
public void RebusBeforeMessageSent(IBus bus, Dictionary<string, string> headers, object message, OutgoingStepContext context)
{
//…
var messageIdentity = _mijnIdentityContext.CurrentIdentity;
var messagingIdentityJson = JsonConvert.SerializeObject(messageIdentity);
headers.Add("x-mijn -identity", messagingIdentityJson);
}
Am I correct that both events are supposed to allow us to modify (and save) the headers?
The problem is that messageContext.Headers refers to the headers of the transport message, which is the (Dictionary<string, string>, byte[]) tuple as represented by the type TransportMessage that Rebus uses to carry the incoming message.
During the execution of the incoming message pipeline, the message goes through several steps, and one of them is deserialization, which results in a new copy of the message – the "logical message" – which comes in the form of Message, which is basically a (Dictionary<string, string>, object) tuple.
The headers of Message can be accessed via messageContext.Message.Headers, so I bet you can find your header there. 🙂
I have an application that uses Spring Integration to send messages to a vendor application over TCP and receive and process responses. The vendor sends messages without a length header or an message-ending token and the message contains carriage returns so I have implemented a custom deserializer. The messages are sent as XML strings so I have to process the input stream, looking for a specific closing tag to know when the message is complete. The application works as expected until the vendor application is restarted or a port switch occurs on my application, at which time the CPU usage on my application spikes and the application becomes unresponsive. The application throws a SocketException: o.s.integration.handler.LoggingHandler : org.springframework.messaging.MessagingException: Send Failed; nested exception is java.net.SocketException: Connection or outbound has closed when the socket closes. I have set the SocketTimeout to be 1 minute.
Here is the connection factory implementation:
#Bean
public AbstractClientConnectionFactory tcpConnectionFactory() {
TcpNetClientConnectionFactory factory = new TcpNetClientConnectionFactory(this.serverIp,
Integer.parseInt(this.port));
return getAbstractClientConnectionFactory(factory, keyStoreName, trustStoreName,
keyStorePassword, trustStorePassword, hostVerify);
}
private AbstractClientConnectionFactory getAbstractClientConnectionFactory(
TcpNetClientConnectionFactory factory, String keyStoreName, String trustStoreName,
String keyStorePassword, String trustStorePassword, boolean hostVerify) {
TcpSSLContextSupport sslContextSupport = new DefaultTcpSSLContextSupport(keyStoreName,
trustStoreName, keyStorePassword, trustStorePassword);
DefaultTcpNetSSLSocketFactorySupport tcpSocketFactorySupport =
new DefaultTcpNetSSLSocketFactorySupport(sslContextSupport);
factory.setTcpSocketFactorySupport(tcpSocketFactorySupport);
factory.setTcpSocketSupport(new DefaultTcpSocketSupport(hostVerify));
factory.setDeserializer(new MessageSerializerDeserializer());
factory.setSerializer(new MessageSerializerDeserializer());
factory.setSoKeepAlive(true);
factory.setSoTimeout(60000);
return factory;
}
Here is the deserialize method:
private String readUntil(InputStream inputStream) throws IOException {
ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
String s = "";
byte[] closingTag = CLOSING_MESSAGE_TAG.getBytes(ASCII);
try {
Integer bite;
while (true) {
bite = inputStream.read();
byteArrayOutputStream.write(bite);
byte[] bytes = byteArrayOutputStream.toByteArray();
int start = bytes.length - closingTag.length;
if (start > closingTag.length) {
byte[] subarray = Arrays.copyOfRange(bytes, start, bytes.length);
if (Arrays.equals(subarray, closingTag)) {
s = new String(bytes, ASCII);
break;
}
}
}
} catch (SocketTimeoutException e) {
logger.error("Expected SocketTimeoutException thrown");
} catch (Exception e) {
logger.error("Exception thrown when deserializing message {}", s);
throw e;
}
return s;
}
Any help in identifying the cause of the CPU spike or a suggested fix would be greatly appreciated.
EDIT #1
Adding serialize method.
#Override
public void serialize(String string, OutputStream outputStream) throws IOException {
if (StringUtils.isNotEmpty(string) && StringUtils.startsWith(string, OPENING_MESSAGE_TAG) &&
StringUtils.endsWith(string, CLOSING_MESSAGE_TAG)) {
outputStream.write(string.getBytes(UTF8));
outputStream.flush();
}
}
the inbound-channel-adapter uses the ConnectionFactory
<int-ip:tcp-inbound-channel-adapter id="tcpInboundChannelAdapter"
channel="inboundReceivingChannel"
connection-factory="tcpConnectionFactory"
error-channel="errorChannel"
/>
EDIT #2
Outbound Channel Adapter
<int-ip:tcp-outbound-channel-adapter
id="tcpOutboundChannelAdapter"
channel="sendToTcpChannel"
connection-factory="tcpConnectionFactory"/>
Edit #3
We have added in the throw for the Exception and are still seeing the CPU spike, although it is not as dramatic. Could we still be receiving bytes from socket in the inputStream.read() method? The metrics seem to indicate that the read method is consuming server resources.
#Artem Bilan Thank you for your continued feedback on this. My server metrics seem to indicate that they deserialize method is what is consuming the CPU. I was thinking that the SendFailed error occurs because of the vendor restarting their application.
Thus far, I have been unable to replicate this issue other than in production. The only exception I can find in production logs is the SocketException mentioned above.
Thank you.
Taken from: https://learn.microsoft.com/en-us/azure/cosmos-db/create-graph-dotnet
Am getting a exception on the .wait() part:
NullReferenceException: Object reference not set to an instance of an object.
at Gremlin.Net.Driver.Connection.ReceiveAsync[T]()
at Gremlin.Net.Driver.Connection.SubmitAsync[T](RequestMessage requestMessage)
at Gremlin.Net.Driver.ProxyConnection.SubmitAsync[T](RequestMessage requestMessage)
at Gremlin.Net.Driver.GremlinClient.SubmitAsync[T](RequestMessage requestMessage)
at Gremlin.Net.Driver.GremlinClientExtensions.SubmitAsync[T](IGremlinClient gremlinClient, String requestScript, Dictionary`2 bindings)
Code:
private static string database = "db";
private static string collection = "col";
private static string hostname = "grem-test.gremlin.cosmosdb.azure.com";
public void test()
{
var gremlinServer = new GremlinServer(hostname, 443, enableSsl: true,
username: "/dbs/" + database + "/colls/" + collection,
password: authKey);
var gremlinClient = new GremlinClient(gremlinServer);
var grem = "g.V()";
var t = gremlinClient.SubmitAsync<dynamic>(grem);
t.Wait();
foreach (var result in t.Result)
{
// The vertex results are formed as dictionaries with a nested dictionary for their properties
string output = JsonConvert.SerializeObject(result);
Console.WriteLine(String.Format("\tResult:\n\t{0}", output));
}
It should be:
var task = gremlinClient.SubmitAsync<dynamic>(grem);
task.Wait();
Taken from Gremlin C# Samples:
// Create async task to execute the Gremlin query.
var task = gremlinClient.SubmitAsync<dynamic>(query.Value);
task.Wait();
I started with the sample application where it uses:
private static Task<ResultSet<dynamic>> SubmitRequest(GremlinClient gremlinClient, string query)
{
try
{
return gremlinClient.SubmitAsync<dynamic>(query);
}
catch (ResponseException e)
{
// They have extra stuff here for the request information that isn't relevant
throw;
}
}
I expanded from there and never had any issues besides the occasional exception from trying to run a query while another one was still working. I can only assume that running the queries this way works better than directly calling SubmitAsync().
One other thing I would recommend is double-checking your values for the server parameters just in case.
I have an action in which I make three parallel HTTP calls (to other services), then I merge the contents of the responses into one document and finally I send it back to the client.
This is a working sample of the code:
#Inject
WSClient wsc;
public CompletionStage<Result> getUrlData() throws Exception {
List<CompletionStage<WSResponse>> stages = new ArrayList<>();
stages.add(wsc.url("http://jsonplaceholder.typicode.com/posts/1").get());
stages.add(wsc.url("http://jsonplaceholder.typicode.com/posts/2").get());
stages.add(wsc.url("http://jsonplaceholder.typicode.com/posts/3").get());
return Futures
.sequence(stages)
.thenApply(responses -> {
StringBuilder builder = new StringBuilder("[");
responses.stream().forEach(response -> builder.append(response.getBody()).append(","));
builder.deleteCharAt(builder.length()-1).append("]");
return ok(builder.toString());
})
.exceptionally(ex -> ok("{\"error\": \"An error has occurred\"}"));
If one of services is not available (you can simulate this behavior modifying the domain name of one of the URLs to a non existing one), the page returned contains only the message contained in the exceptionally() part, while I need to return the contents of the correct calls plus the error message of the not succeeded call. Any hint on how to do it?
I'm using Play 2.5.1.
Thanks,
Andrea
Basically you just want to handle the .exceptionally(..) individually for each call. Something like this should work:
create a function that returns a CompletionStage for an individual URL, incorporating your error handling (returning JSON of the error)
convert that to a list of completion stages to pass to Futures.sequence
As an aside, you can make the JSON manipulation a bit nicer by building the objects programatically using Jackson's ObjectMapper.createObjectNode() and ObjectMapper.createArrayNode():
private static final ObjectMapper mapper = new ObjectMapper();
private CompletionStage<JsonNode> getDataFromUrl(String url) {
return wsc.url(url)
.get()
.thenApply(WSResponse::asJson)
.exceptionally(ex -> {
ObjectNode error = mapper.createObjectNode();
error.put("error", ex.getMessage());
return error;
});
}
public CompletionStage<Result> getUrlData() throws Exception {
List<String> urls = new ArrayList<>();
urls.add("http://jsonplaceholder.typicode.com/posts/1");
urls.add("http://jsonplaceholder.typicode.com/posts/2");
urls.add("http://jsonplaceholder.typicode.com/posts/3");
// Convert to a list of promises
List<CompletionStage<JsonNode>> stages = urls
.stream()
.map(this::getDataFromUrl)
.collect(Collectors.toList());
return Futures
.sequence(stages)
.thenApply(responses -> {
ArrayNode arrayNode = mapper.createArrayNode();
responses.stream().forEach(arrayNode::add);
return ok(arrayNode);
});
}
We have been transferring our services and MVC4 website to the cloud, overall this process went fine.
Except for caching, since we have moved to Azure it would also be wise to use some kind of caching which azure provides. We choose for co-located / dedicated caching role which has the advantage that the cache is used over all the instances.
Setting up the caching worked fine, I've got a named caching client which I only initialize when its required. It is set up in a inherited layer of the controllers. As soon as one of the functions is called, it checks if the connection to the data-cache is still there or its created. This all seems to work fine, but I'm building a module do retrieve prices. And multiple ajax inserts (views which get inserted into the page with use of javascript) use these functions, some of them are called at the same time, by multiple ajax views. Some of these views then return either a 404 or 500 error, and I cant explain where these are coming from except a non working caching, or something alike.
Can someone help me with a good implementation of the named caching (co-located or dedicated), since all I can find is many examples illustrating the initializing of the DataCacheFactory, but not of the data insertion and retrieval.
Below is the code as I have it now, I've tried more ways with use of locking etc but this one so far worked best.
private static object magicStick = new object();
private static DataCacheFactory dcf = null;
private static DataCache priceCache = null;
protected void CreateCacheFactory()
{
dcf = new DataCacheFactory();
}
protected void CreatePricesCache()
{
if (dcf == null)
{
CreateCacheFactory();
}
priceCache = dcf.GetCache("Prices");
}
protected PriceData GetPrices(int productID)
{
if (priceCache == null)
{
CreatePricesCache();
}
string cacheKey = "something";
lock (magicStick)
{
PriceData datas = priceCache.Get(cacheKey) as PriceData;
if (datas == null)
{
lock (magicStick)
{
Services svc = new Services();
PriceData pData = svc.PriceService.GetPrices(productID);
if (pData != null && pData.Offers != null && pData.Offers.Count() > 0)
{
datas = pData;
datas.Offers = datas.Offers.OrderBy(pr => (pr.BasePrice + pr.ShippingCosts)).ToArray();
priceCache.Add(cacheKey, datas, new TimeSpan(0, cachingTimePricesKK, 0));
}
}
}
return datas;
}
}
As soon as I get to a page where there are pricelists and the function above is called multiple times with the same arguments, there is a 5-10% chance that it returns an error rather then returning the results. Can anybody help me, im totally stuck with this for a week now and its eating me up inside.
First I'd move your cache and cacheFactory instantiation out of your getPrices method. Also, evaluate your need for the lock - this may be causing timeouts. Another VERY important observation - you are using a constant cache key and saving/retrieving data for every productId with the same cache key. You should be using a cache key like: var cacheKey = string.format("priceDatabyProductId-{0}", productId);. You need to set some breakpoints and examine exactly what you are caching and retrieving from the cache. The code as written will save the first productId to the cache and then keep returning that data regardless of the productId.
Here is a full working example we use in production using the "default" named cache in dedicated cache roles:
public static class MyCache
{
private static DataCacheFactory _cacheFactory = null;
private static DataCache ACache
{
get
{
if (_cacheFactory == null)
{
try
{
_retryPolicy.ExecuteAction(() => { _cacheFactory = new DataCacheFactory(); });
return _cacheFactory == null ? null : _cacheFactory.GetDefaultCache();
}
catch (Exception ex)
{
ErrorSignal.FromCurrentContext().Raise(ex);
return null;
}
}
return _cacheFactory.GetDefaultCache();
}
}
public static void FlushCache()
{
ACache.Clear();
}
// Define your retry strategy: retry 3 times, 1 second apart.
private static readonly FixedInterval _retryStrategy = new FixedInterval(3, TimeSpan.FromSeconds(1));
// Define your retry policy using the retry strategy and the Windows Azure storage
// transient fault detection strategy.
private static RetryPolicy _retryPolicy = new RetryPolicy<StorageTransientErrorDetectionStrategy>(_retryStrategy);
// Private constructor to prevent instantiation
// and force consumers to use the Instance property
static MyCache()
{ }
/// <summary>
/// Add an item to the cache with a key and set a absolute expiration on it
/// </summary>
public static void Add(string key, object value, int minutes = 90)
{
try
{
_retryPolicy.ExecuteAction(() => { ACache.Put(key, value, TimeSpan.FromMinutes(minutes)); });
}
catch (Exception ex)
{
ErrorSignal.FromCurrentContext().Raise(ex);
}
}
/// <summary>
/// Add the object with the specified key to the cache if it does not exist, or replace the object if it does exist and set a absolute expiration on it
/// only valid for Azure caching
/// </summary>
public static void Put(string key, object value, int minutes = 90)
{
try
{
_retryPolicy.ExecuteAction(() => { ACache.Put(key, value, TimeSpan.FromMinutes(minutes)); });
}
catch (Exception ex)
{
ErrorSignal.FromCurrentContext().Raise(ex);
}
}
/// <summary>
/// Get a strongly typed item out of cache
/// </summary>
public static T Get<T>(string key) where T : class
{
try
{
object value = null;
_retryPolicy.ExecuteAction(() => { value = ACache.Get(key); });
if (value != null) return (T) value;
return null;
}
catch (DataCacheException ex)
{
ErrorSignal.FromCurrentContext().Raise(ex);
return null;
}
catch (Exception ex)
{
ErrorSignal.FromCurrentContext().Raise(ex);
return null;
}
}
/// <summary>
/// Microsoft's suggested method for cleaning up resources such as this in a static class
/// to ensure connections and other consumed resources are returned to the resource pool
/// as quickly as possible.
/// </summary>
public static void Uninitialize()
{
if (_cacheFactory == null) return;
_cacheFactory.Dispose();
_cacheFactory = null;
}
}
Note: this is also using the Transient Fault Handling block from the Enterprise Library for transient exception fault handling.