In my workflow I call a service which returns me a List. The problem is that in my workflow i use a AddToCollection Activitie to add a new string to the collection, but i get an error right when i get to the activity.
Debugging and checking i got to theworkflow logs and now I see that the error is that "Collection was of a fixed size." Here's the complete log:
System.SZArrayHelper.Add[T](T value)
System.Activities.Statements.AddToCollection`1.Execute(CodeActivityContext context)
System.Activities.CodeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)
System.Activities.ActivityInstance.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)
System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)
What i don't get is why (and how this happened)? Is this a bug? I specifically return a List... why does it says it's fixed sized?!?
EDIT 1: There's something really weird... since my original workflow was quite big I created a new, smaller one, just to reproduce this error... and I can't!
My guess is that WCF is serializing your list to array before sending it over the wire. Don't know if it is possible to avoid.
Anyway, check this and this
You can also create a new Variable on your workflow and assign a List to it when you receive it from the service:
listWFVariable = new List<string>(arrayReceivedFromWebService);
Now you can make Add operations to it.
Related
In the samples for the Cosmos DB SQL API, there are a couple of uses of Database.ReadAsync() which don't seem to be doing anything useful. The remarks second in the method's documentation doesn't really indicate what it might be used for either.
What is the reason for using it in these cases? When would you typically use it?
ChangeFeed/Program.cs#L475 shows getting a database then calling ReadAsync to get another reference to the database
database = await client.GetDatabase(databaseId).ReadAsync();
await database.DeleteAsync();
which seems to be functionally the same as
database = client.GetDatabase(databaseId);
await database.DeleteAsync();
throwing the same exception if the database is not found.
and DatabaseManagement/Program.cs#L80-L83
DatabaseResponse readResponse = await database.ReadAsync();
Console.WriteLine($"\n3. Read a database: {readResponse.Resource.Id}");
await readResponse.Database.CreateContainerAsync("testContainer", "/pk");
which seems to be equivalent to:
Console.WriteLine($"\n3. Read a database: {database.Id}");
await database.CreateContainerAsync("testContainer", "/pk");
producing the same output and creating the container as before.
You are correct that those samples might need polishing, the main difference is:
GetDatabase just gets a proxy object, it does not mean the database actually exists. If you attempt an operation on a database that does not exist, for example, CreateContainer, it can fail with a 404.
ReadAsync will read the DatabaseProperties and allow you obtain any information from there and also would succeed if the database actually exists. Does that guarantee that if I call CreateContainer right away it will succeed? No, because the database could have been deleted right in the middle.
So in summary, ReadAsync is good if you want to get any of the DatabaseProperties or if you want to for some reason verify the database exists.
Most common scenarios would just use GetDatabase because you are probably attempting operations down the chain (like creating a container or executing item level operations in some container in that database).
Short Answer
Database.ReadAsync(...) is useful for reading database properties.
The Database object is useful for performing operations on the database, such as creating a container via Database.CreateContainerIfNotExistsAsync(...).
A bit more detail
The Microsoft Docs page for Database.ReadAsync is kind of confusing and not well written in my opinion:
The definition says:
Reads a DatabaseProperties from the Azure Cosmos service as an
asynchronous operation.
However, the example shows ReadAsync returning a DatabaseResponse object, not a DatabaseProperties object:
// Reads a Database resource where database_id is the ID property of the Database resource you wish to read.
Database database = this.cosmosClient.GetDatabase(database_id);
DatabaseResponse response = await database.ReadAsync();
It's only after a bit more digging that things become clearer. When you look at the documentation page for the DatabaseResponse Class it says the inheritance chain for DatabaseResponse is:
Inheritance: Object -> Response<DatabaseProperties> -> DatabaseResponse
If you then have a look at the Docs page for the Response<T> Class you'll see there is an implicit operator that converts Response<T> to T:
public static implicit operator T (Microsoft.Azure.Cosmos.Response<T> response);
So that means that even though the ReadAsync method returns a DatabaseResponse object, that is implicitly converted to a DatabaseProperties object (since DatabaseResponse inherits Response<DatabaseProperties>).
So Database.ReadAsync is useful for reading database properties.
The Docs page for Database.ReadAsync could have clearer about the implicit link between the DatabaseResponse object returned by the method and the DatabaseProperties object that it wraps.
I read spring-kafka/kafka documentation back and forth, and still cannot find a way, how to do proper transactional behavior with error recovering. I believe this is not trivial question, so please read until end. I believe whole this question revolves around finding way how to reposition over failing record or how to ack in error handler. But mabye there are better ways, I don't know.
So records are flowing in, and some of them are invalid. What I would like to have as a minimal solution is(in which I will then fix sevaral problems you probably see as well):
1) we cannot afford the luxury of stopping the production in case of some trivial mishap, like one or few invalid records. Thus if there is invalid record in kafka topic, I would like to log it, or resend it to different queue, but then proceed with processing following records.
2) there are permanent and temporary failures. Permanent failure is record unable to deserialize, record failing data validation. In this case, I'd like to skip the invalid record, as discussed in 1). Temporary failure might be some specific exception or state, like for example database connection errors, network issues etc. In this case, we do not want to skip failing record, we want to retry, after some delay.
Subject of this question is ONLY implementing skip/don't skip behavior.
Lets say, that this is our starting point:
private Map<String, Object> createKafkaConsumerFactoryProperties(String bootstrapServers, String groupId, Class<?> valueDeserializerClass) {
Map<String, Object> props = new HashMap<>();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, valueDeserializerClass);
props.put(ConsumerConfig.GROUP_ID_CONFIG, groupId);
return props;
}
#Bean(name="SomeFactory")
public ConcurrentKafkaListenerContainerFactory<String, String> kafkaListenerContainerFactory(
#Value("${…}") String bootstrapServers,
#Value("${…}") String groupId) {
ConcurrentKafkaListenerContainerFactory<String, String> factory =
new ConcurrentKafkaListenerContainerFactory<>();
ConsumerFactory<String, String> consumerFactory = new DefaultKafkaConsumerFactory<>(
createKafkaConsumerFactoryProperties(bootstrapServers, groupId, AvroDeserializer.class),
new StringDeserializer(),
new AvroDeserializer(SomeClass.class));
factory.setConsumerFactory(consumerFactory);
// factory.setConcurrency(2);
// factory.setBatchListener(true);
return factory;
}
and we have listener like:
#KafkaListener(topics = "${…}", containerFactory = "SomeFactory")
public void receive(#Valid List<SomeClass> messageList) {/*logic*/}
Now how this behave if I understand correctly:
when listener gets message, ~when we reach inside of receive method, the kafka message will be already acked, and if receive method throw an exception, the next poll will return following record. Because ack happened, and we do not have error handler defined, thus logging error handler will kick in. This is not necessarily what we want. We can use SeekToCurrentErrorHandler to reprocess the message. Or one can specify TransactionManager, and if exception 'leaks' from listener, repositioning will also happen. If someone know performance comparison of these two approaches, please tell me.
when message cannot be deserialized, deserializer will fail, message will not be acked and same record will be polled again. This is some sort of "poison packet" since kafka will spin on this message indefinitelly. We do have retry.backoff.ms to at least slow it down, but I can't see any max number retries or something. So the best thing we can do is to stop/pause container in this situation. Which is way to harsh. Btw. I'm new to kafka/spring-kafka, I did not see anywhere mention, how to manually reposition offset from outside of an application, meaning OK, listener is down, but now what? Another solution would be not to fail deserializer, and return something. But what?? KafkaNull, great, but then our listener will fail because SomeClass ClassCastException. We can send some artificial value of SomeClass, which is again horrible, because this is not a data what we actually get. Also this is architectonically incorrect.
or we can use repositioning error handler, which would be great, well if we know how to do that. I need to seek to next record. But while documentation says, that ErrorHandler should communicate which record caused the failure, it seems that it fails to do so. So even in non-batch listener I have list of records(1 failed + bunch of unprocessed), and have no idea where set offset to.
So what is the solution to this madness?
Well the best I can come up with right now is pretty ugly: do not fail in deserializer (bad), do not accept specific type in listener (bad), filter out KafkaNulls manually (bad) and finally trigger bean validation manually (bad). Is there a better way? Thanks for examplantion, I'd be grateful for every hint or direction given how to achieve this.
See the documentation for the upcoming 2.2 release (due tomorrow).
The DefaultAfterRollbackProcessor (when using transactions) and SeekToCurrentErrorHandler (when not using transactions) can now recover (skip) records that keep failing, and will do so after 10 failures, by default. They can be configured to publish failed records to a dead-letter topic.
Also see the Error Handling Deserializer which catches deserialization problems and passes them to the container so they can be sent to the error handler.
Application background:
I am using async item processor and passing delegate as composite processor. When i get an exception in processors, my write error listener gets called.
onWriteError method signature is (Exception exception, List items)
Issue:
All items on the list to onWriteError method are Future tasks. If i call "get" method on future task, it gives me the same exception which caused write error.
How can i get the original item in writer listener methods during async execution?
I couldn't provide any actual code because my company policy prohibits me from posting code in online forums.
You can't. It's like going back in time. That is one of the disadvantages of the AsyncItemProcessor/AsyncItemWriter combination. As the javadoc points out in the AsyncItemProcessor:
Because the Future is typically unwrapped in the ItemWriter, there are
lifecycle and stats limitations (since the framework doesn't know what
the result of the processor is). While not an exhaustive list, things
like StepExecution.filterCount will not reflect the number of filtered
items and ItemProcessListener.onProcessError(Object, Exception) will
not be called.
I am getting the following error message printed in the console:
Unable to free statement: ERROR: prepared statement "qpsqlpstmt_1" does not exist
It is printed when the the following function is called in the application (or when the object is deleted (if clear() is not called before delete):
sqlQueryModel->clear();
sqlQueryModel object is of type QSqlQueryModel and is used throughout a derived class to communicated with a PostgreSQL database. It also serves as a model for QCompleter. I have never declared or used the name "qpsqlpstmt_1".
Could someone help me interpret the error message please, and explain what might be causing it? Is this indicative of a problem in my code or a Qt bug? (likely the former :))
On reviewing the PostgreSQL log file on the server, the exact same statement appears plus an additional line:
STATEMENT: DEALLOCATE pqsqlpstmt_1
See these Qt issue tracker entries:
https://bugreports.qt.io/browse/QTBUG-8860
https://bugreports.qt.io/browse/QTBUG-16007
https://bugreports.qt.io/browse/QTBUG-15979
... all of which mention your prepared statement name and relate to deletion.
After a considerable amount of time, I realized that I was simply closing the connection to the database before calling clear... not a good strategy.
I'm trying to understand when I should call Query.close(Object) or Query.closeAll();
From the docs I see that it will "close a query result" and "release all resources associated with it," but what does "close" mean? Does it mean I can't use objects that I get out of a Query after I've called close on them?
I ask because I'm trying to compartmentalize my code with functions like getUser(id), which will create a Query, fetch the User, and destroy the query. If I have to keep the Query around just to access the User, then I can't really do that compartmentalization.
What happens if I don't call close on an object? Will it get collected as garbage? Will I be able to access that object later?
Please let me know if I can further specify my question somehow.
You can't use the query results since you closed them (i.e the List object you got back from query.execute). You can access the objects in the results ... if you copied them to your own List, or made references to them in your code. Failure to call close can leak memory
When your query method returns a single object it is easy to simply close the query before returning the single object.
On the other hand, when your query method returns a collection the query method itself can not close the query before returning the result because the query needs to stay open while the caller is iterating through the results.
This puts the responsibility for closing a query that returns a collection on the caller and can introduce leaks if the caller neglects to close the query - I thought there must be a safer way and there is!
Guido, a long time DataNucleus user, created a 'auto closing' collection facade that wraps the collection returned by JDO's Query.execute method. Usage is extremely simple: Wrap the query result inside an instance of the auto closing collection object:
Instead of returning the Query result set like this:
return q.execute();
simply return an 'auto closing' wrapped version of it:
return new JDOQueryResultCollection(q, q.execute());
To the caller it appears like any other Collection but the wrapper keeps a reference to the query that created the collection result and auto closes it when the wrapper is disposed of by the GC.
Guido kindly gave us permission to include his clever auto closing code in our open source exPOJO library. The auto closing classes are completely independent of exPOJO and can be used in isolation. The classes of interest are in the expojo_jdo*.jar file that can be downloaded from:
http://www.expojo.com/
JDOQueryResultCollection is the only class used directly but it does have a few supporting classes.
Simply add the jar to your project and import com.sas.framework.expojo.jdo.JDOQueryResultCollection into any Java file that includes query methods that return results as a collection.
Alternatively you can extract the source files from the jar and include them in your project: They are:
JDOQueryResultCollection.java
Disposable.java
AutoCloseQueryIterator.java
ReadonlyIterator.java