Corda - Queries on QueryableState Child Tables - corda

Background
I have a problem with being able to query child data in a sample Corda application, demonstrating how to persisnt hierarchical data using QueryableState.
For reference:
https://github.com/corda/samples-kotlin/tree/master/Features/queryableState-carinsurance
https://www.corda.net/blog/persisting-corda-states-in-custom-database-tables-using-queryablestate/
The car insurance sample demonstrates how to set up:
a One-to-One relationship between PersistentInsurance and PersistentVehicle
a One-to-Many relationship between PersistentInsurance and PersistentClaim
Problem
What the car insurance sample doesn't demonstrate is how to query that data. Building vault queries from PersistentInsurance is fairly trivial as we can use VaultCustomQueryCriteria to build custom query expressions over the properties of PersistentInsurance, however the same is not true for the child tables in the hierarchy. This is because PersistentInsurance extends PersistentState, whereas PersistentVehicle and PersistentClaim don't.
For Reference, the underlying type hierarchy for PersistentState is as follows:
StatePersistable > DirectStatePersistable > PersistentState
interface StatePersistable
interface DirectStatePersistable : StatePersistable {
val stateRef: PersistentStateRef?
}
class PersistentState(#EmbeddedId override var stateRef: PersistentStateRef? = null) : DirectStatePersistable
data class PersistentStateRef(
#Suppress("MagicNumber") // column width
#Column(name = "transaction_id", length = 144, nullable = false)
var txId: String,
#Column(name = "output_index", nullable = false)
var index: Int
) : Serializable {
constructor(stateRef: StateRef) : this(stateRef.txhash.toString(), stateRef.index)
}
Going back to the car insurance sample, we can't use VaultCustomQueryCriteria for PersistentVehicle or PersistentClaim as VaultCustomQueryCriteria::expression has a generic constraint on StatePersistable:
data class VaultCustomQueryCriteria<L : StatePersistable>(
val expression: CriteriaExpression<L, Boolean>,
...
) : CommonQueryCriteria() { ... }
What I've Tried
Implementing StatePersistable on PersistentVehicle and/or PersistentClaim causes the following error when you create a VaultCustomQueryCriteria using any of the properties from those classes:
net.corda.core.node.services.VaultQueryException: Parsing error: Unable to locate Attribute with the the given name [stateRef] on this ManagedType [...$PersistentClaim]
Implementing DirectStatePersistable or PersistentState on PersistentVehicle and/or PersistentClaim causes an internal error when finalizing the transaction. The flow hangs and eventually times out. I wasn't able to determine exactly where the error occurs as the code is buried inside internal implementations for underlying Corda services.
What Does Work (In a Limited Capacity)
I'm aware that we can execute custom SQL queries; for example:
val sqlQuery = buildString {
appendln("SELECT TRANSACTION_ID, OUTPUT_INDEX")
appendln("FROM CLAIM_DETAIL")
appendln("WHERE claimNumber = '123'")
}
val stateRefs = serviceHub.jdbcSession().executeCaseInsensitiveQuery(sqlQuery).map {
val txId = SecureHash.parse(it.getString("TRANSACTION_ID")
val index = it.getInt("OUTPUT_INDEX")
StateRef(txId, index)
}.toList().toBlocking().first()
VaultQueryCriteria(stateRefs = stateRefs)
Needless to say, that's hideous compared to:
VaultCustomQueryCriteria(PersistentClaim::claimNumber.equal("123"))
As for why it works in a limited capacity, this works fine for queries inside the node, which access to the service hub, but doesn't work over RPC as there's no access to a jdbcSession.
So, any ideas (if it's at all possible), on how we can improve the query capability of child tables?

#Matthew Layton I have put your concerns to the engineering team. The team has looked into this.
To address/fix this issue, requires lot of platform changes.
For time being I would suggest you to manage with using native queries. We are planning to prioritise this issue. I will keep you posted on this.

Related

Binding data to view model when dealing with DAO and Tornadofx

I am making database operations with JetBrains/Exposed. I was able to make use of the DAO approach to display data from the database on the TableView. It was quite challenging with the DSL approach.
After succeeding to display data, data binding was gone. My person class looks like this
class Person(id: EntityID<Int>) : IntEntity(id) {
companion object : IntEntityClass<Person>(Persons)
var name by Persons.name
var title by Persons.title
}
my view model looks like this
class PersonModel(person: Person?) : ViewModel() {
val name = bind {person?.observable(Person::name)}
val title = bind {person?.observable(Person::title)}
}
Each time I want to commit a change to the model I get
java.lang.IllegalStateException: No transaction in context.
I know this is because Person is used in the transaction context to perform Db queries.
I wish to know how I can bind data to view model given that I am using DAO API of JetBrains/Expose.
Thanks.

Use PathElement in objectify query as ancestor

I am migrating from google datastore api to objectify(i have used datastore at compute engine and migrating to 6.0a1 objectify). In previous version i used this code to query users:
KeyFactory userKeyFactory = datastore.newKeyFactory()
.addAncestors(PathElement.of("UserList", "default"))
.setKind(USER_KIND);
//...save entity
Query<Entity> query = Query.newEntityQueryBuilder()
.setKind(USER_KIND)
.setFilter(PropertyFilter.hasAncestor(datastore.newKeyFactory().setKind("UserList").newKey("default")))
.build();
QueryResults<Entity> queryResults = datastore.run(query);
List<User> result = new ArrayList<>();
queryResults.forEachRemaining(entity -> result.add(transformUser(entity)));
return result;
Now i am trying to make same query with ancestor. However objectify don't work with PathElement. So following code:
ofy().load().type(User.class).ancestor(PathElement.of("UserList", "default")).list()
fails with
java.lang.IllegalArgumentException: No class 'com.google.cloud.datastore.PathElement' was registered
at com.googlecode.objectify.impl.Registrar.getMetadataSafe(Registrar.java:115)
at com.googlecode.objectify.impl.Keys.getMetadataSafe(Keys.java:56)
at com.googlecode.objectify.impl.Keys.getMetadataSafe(Keys.java:65)
at com.googlecode.objectify.impl.Keys.rawKeyOf(Keys.java:47)
at com.googlecode.objectify.impl.Keys.anythingToRawKey(Keys.java:117)
at com.googlecode.objectify.impl.QueryImpl.setAncestor(QueryImpl.java:203)
at com.googlecode.objectify.impl.SimpleQueryImpl.ancestor(SimpleQueryImpl.java:69)
at com.googlecode.objectify.impl.LoadTypeImpl.ancestor(LoadTypeImpl.java:23)
What is proper way to use PathElement with objectify? I see i can create com.googlecode.objectify.Key and pass it as ancestor but it require class, but i don't have UserList class (it is small application, all entities related to single group).
I tried to use this code:
ofy().load().type(User.class).ancestor(datastore.newKeyFactory().setKind("UserList").newKey("default")).list()
Now it fails that User don't have field with #Parent annotation. Here is stacktrace:
com.googlecode.objectify.LoadException: Error loading Key{projectId=projectId, namespace=, path=[PathElement{kind=UserList, id=null, name=default}, PathElement{kind=User, id=1, name=null}]}: Loaded Entity has parent but com.package.model.User has no #Parent
at com.googlecode.objectify.impl.EntityMetadata.load(EntityMetadata.java:84)
at com.googlecode.objectify.impl.LoadEngine.load(LoadEngine.java:187)
at com.googlecode.objectify.impl.LoadEngine$1.nowUncached(LoadEngine.java:145)
at com.googlecode.objectify.impl.LoadEngine$1.nowUncached(LoadEngine.java:131)
at com.googlecode.objectify.util.ResultCache.now(ResultCache.java:30)
at com.googlecode.objectify.impl.Round$1.nowUncached(Round.java:66)
at com.googlecode.objectify.util.ResultCache.now(ResultCache.java:30)
at com.googlecode.objectify.impl.HybridQueryResults.lambda$load$1(HybridQueryResults.java:87)
at com.google.common.collect.Iterators$5.transform(Iterators.java:757)
at com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48)
at com.google.common.collect.MultitransformedIterator.next(MultitransformedIterator.java:66)
at com.google.common.collect.Iterators$4.computeNext(Iterators.java:623)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:145)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:140)
at com.googlecode.objectify.impl.HybridQueryResults.hasNext(HybridQueryResults.java:92)
at com.googlecode.objectify.util.IteratorFirstResult.nowUncached(IteratorFirstResult.java:31)
at com.googlecode.objectify.util.ResultCache.now(ResultCache.java:30)
at com.googlecode.objectify.LoadResult.now(LoadResult.java:25)
at com.package.UserObjectifyRepository.getUserByEmail(UserObjectifyRepository.java:43)
...
Caused by: java.lang.IllegalStateException: Loaded Entity has parent but com.package.model.User has no #Parent
at com.googlecode.objectify.impl.KeyMetadata.setKey(KeyMetadata.java:142)
at com.googlecode.objectify.impl.KeyMetadata.setKey(KeyMetadata.java:122)
at com.googlecode.objectify.impl.KeyPopulator.load(KeyPopulator.java:24)
at com.googlecode.objectify.impl.translate.ClassPopulator.load(ClassPopulator.java:118)
at com.googlecode.objectify.impl.translate.ClassTranslator.loadSafe(ClassTranslator.java:109)
at com.googlecode.objectify.impl.translate.NullSafeTranslator.load(NullSafeTranslator.java:21)
at com.googlecode.objectify.impl.EntityMetadata.load(EntityMetadata.java:80)
.. 125 more
I suppose correct way to fix this is to have parent field in my User entity, something like this:
#Parent
private UserList userList;
But i haven't such entity "UserList", i need ancestor just to make query with strong consistency.
UPD: Error is gone if i adding this code:
import com.google.cloud.datastore.Key;
#Parent
private Key userList;
Is it proper way to make consistent query?
The #Parent annotation requires a key.
You just have to create a key, so it can understand that Parent/Child relationship.

LINQ and web service cannot return anonymous types, and you cannot construct an object in a query?

Web services cannot return an anonymous type.
If you are building a LINQ query using classes through a datacontext... you cannot construct instances of those classes in a query.
Why would I want to do this? Say I want to join three "tables" or sets of objects. I have three items with a foreign key to each other. And say the lowest, most detailed of these was represented by a class that had fields from the other two to represent the data from those. In my LINQ query I would want to return a list of the lowest, most detailed class. This is one way I have decided to "join some tables together" and return data from each of them via LINQ to SQL via a WebService. This may be bad practice. I certainly do not like adding the additional properties to the lowest level class.
Consider something like this... (please ignore the naming conventions, they are driven by internal consideration) also for some reason I need to instantiate an anonymous type for the join... I don't know why that is... if I do not do it this way I get an error...
from su in _dataContext.GetTable<StateUpdate>()
join sfs in _dataContext.GetTable<SystemFacetState>()
on new { su.lngSystemFacetState } equals new { lngSystemFacetState = sfs.lngSystemFacetState }
join sf in _dataContext.GetTable<SystemFacet>()
on new { sfs.lngSystemFacet } equals new { lngSystemFacet = sf.lngSystemFacet }
join s in _dataContext.GetTable<System>()
on new { sf.lngSystem } equals new {lngSystem = s.lngSystem}
select new
{
lngStateUpdate = su.lngStateUpdate,
strSystemFacet = sf.strSystemFacet,
strSystemFacetState = sfs.strSystemFacetState,
dtmStateUpdate = su.dtmStateUpdate,
dtmEndTime = su.dtmEndTime,
lngDuration = su.lngDuration,
strSystem = s.strSystem
}
).ToList();
Notice I have to build the anonymous type which is composed of pieces of each type. Then I have to do something like this... (convert it to a known type for transport via the web service)
result = new List<StateUpdate>(from a in qr select(new StateUpdate
{
lngStateUpdate = a.lngStateUpdate,
strSystemFacet = a.strSystemFacet,
strSystemFacetState = a.strSystemFacetState,
dtmStateUpdate = a.dtmStateUpdate,
dtmEndTime = a.dtmEndTime,
lngDuration = a.lngDuration,
strSystem = a.strSystem
}));
It is just awful. And perhaps I have created an awful mess. If I am way way off track here please guide me to the light. I feel I am missing something fundamental here when I am adding all these "unmapped" properties to the StateUpdate class.
I hope someone can see what I am doing here so I can get a better way to do it.
You can create a 'dto' class which just contains the properties you need to return and populate it instead of the anonymous object:
public class Result
{
public string lngStateUpdate
{
get;
set;
}
... // other properties
}
then use it like this:
from su in _dataContext.GetTable<StateUpdate>()
...
select new Result
{
lngStateUpdate = su.lngStateUpdate,
... // other properties
}
Nitpick note - please ditch the Hungarian notation and camel casing for properties :)
I think the answer is to create another object to serve as a DTO. This object would not be mapped to the data context and can contain fields that cross the mapped objects. This solves the problems of repetitive properties in the mapped objects, and allows for instantiation of the DTO class in the query as it is not mapped.
FYI: with respect to the problem with the join- I revisited that and I think I may have had the inner and outer components of the join switched around before.

different hashtable cacheItem with similar data values or separate cacheItems for each data value – which is an efficient approach?

I have broadly two different classes of data caching requirements based on data size:
1) very small data (2-30 characters) – this includes such things as the type code for a given entityId. The system is based upon the concept of parent-child entity hierarchy and actions are authorized against values that are built in combination with entity type code. Caching these type codes for different entities saves time on db fetch.
2) medium/Large data – This is general data like products description and pages.
I'm confused as to which approach is better suited for first class of data.
I can cache it like this:
HttpRuntime.Cache.Insert("typeCode" + entityId, entityTypeCode);
or like this:
Dictionary<int, string> etCodes =
(Dictionary<int, string>)HttpRuntime.Cache["typeCode"];
etCodes[entityId] = entityTypeCode;
Clearly, In the second approach, I'm saving on unnecessary cache items for each entityId.
or, having Cache object populated with several items of such small size is okay.
Which of these approachs is good in terms of performance and overhead?
Personally I would take your second approach of one single object and use a custom object instead of a Dictionary.
This would enable me to later control more aspects like expiration of items within the object or changing the implementation.
I would do it similar to this:
public class MyCacheObject
{
public static MyCacheObject
{
get
{
// ...Omitted locking here for simplification...
var o = HttpRuntime.Cache["MyCacheObject] as MyCacheObject;
if ( o = null )
{
o = new MyCacheObject();
HttpRuntime.Cache["MyCacheObject] = o;
}
return o;
}
}
public object GetEntity( string id, string code )
{
// ...
}
public void SetEntity( object entity, string id, string code )
{
// ...
}
// ...
}
If you have a custome base class for the entities, the GetEntity and SetEntity methods could be optimized further.

ASP.Net Entity Framework Repository & Linq

My scenario:
This is an ASP.NET 4.0 web app programmed via C#
I implement a repository pattern. My repositorys all share the same ObjectContext, which is stored in httpContext.Items. Each repository creates a new ObjectSet of type E. Heres some code from my repository:
public class Repository<E> : IRepository<E>, IDisposable
where E : class
{
private DataModelContainer _context = ContextHelper<DataModelContainer>.GetCurrentContext();
private IObjectSet<E> _objectSet;
private IObjectSet<E> objectSet
{
get
{
if (_objectSet == null)
{
_objectSet = this._context.CreateObjectSet<E>();
}
return _objectSet;
}
}
public IQueryable<E> GetQuery()
{
return objectSet;
}
Lets say I have 2 repositorys, 1 for states and 1 for countrys and want to create a linq query against both. Note that I use POCO classes with the entity framework. State and Country are 2 of these POCO classes.
Repository stateRepo = new Repository<State>();
Repository countryRepo = new Repository<Country>();
IEnumerable<State> states = (from s in _stateRepo.GetQuery()
join c in _countryRepo.GetQuery() on s.countryID equals c.countryID
select s).ToList();
Debug.WriteLine(states.First().Country.country)
essentially, I want to retrieve the state and the related country entity. The query only returns the state data... and I get a null argument exception on the Debug.WriteLine
LazyLoading is disabled in my .edmx... thats the way I want it.
You're doing a join without retrieving anything from it. There are multiple solutions to your problem:
Use Include to load the dependent entities: from s in ((ObjectSet<State>) _stateRepo.GetQuery).Include("Country"). The problem with this approach is that you should expose the ObjectSet directly rather than as a IQueryable if you want to avoid casting.
Use context.LoadProperty(states.First(), s => s.Country) to explicitly load the Country from the database for a given state.
Select both entities in the query: from s in ... join c ... select new { s, c }. You won't be able to access directly the state's Country property but you have it in the anonymous type.
Enable lazy loading.
Your repository implementation is very similar to mine, especially the way you are storing the ObjectContext. It works fine for me, so I don't think it's a conceptual problem.
Try using a static objectcontext (no wrapper) just to see if that fixes the problem. Perhaps there is a bug in your ContextHelper which causes your context to get disposed and recreated.

Resources