I have function from my RPC client that pulls the consumed history of a state. If I pass a custom criteria, it seem to ignore the general criteria with Vault StateStatus.
My general query looks like this:
QueryCriteria generalCriteria = new QueryCriteria.VaultQueryCriteria(Vault.StateStatus.CONSUMED);
My custom criteria like this:
customCriteria = generalCriteria;
if (requestJSON.containsKey("linearId")) {
String linearId = requestJSON.get("linearId").toString();
FieldInfo fieldInfo = getField("linearId", POSchemaV1.POSchemaPersistence.class);
criteriaExpression = Builder.equal(fieldInfo, linearId);
customCriteria = new QueryCriteria.VaultCustomQueryCriteria(criteriaExpression);
customCriteria = generalCriteria.and(customCriteria);
}
The vault query looks as below:
Vault.Page<PurchaseOrder> orderPage = proxy.vaultQueryByCriteria(customCriteria, PurchaseOrder.class);
List<StateAndRef<PurchaseOrder>> orderStateAndRefList = orderPage.getStates();
The result seem to have unconsumed states if I pass linearId, if I ignore linearId then I get all consumed states correctly. pls let me know if I am missing something.
There is a more simple (out of the box) approach to query linear states:
QueryCriteria queryCriteria = new QueryCriteria.LinearStateQueryCriteria()
.withStatus(Vault.StateStatus.CONSUMED)
.withUuid(Collections.singletonList(linearId));
What if you don't have the linearId at hand; but have other custom attributes. It just returns UNCOSUMED states. If it is a linearState you may have to query twice, once to get UNCONSUMED state and then use that linearID to get the CONSUMED states.
How will it work if you have states that are consumed and you want to query for certain customCriteria and don't have the linearID? I don't want to query all the all states ( there can me lots of them ) and look for specific ones that match from all records that were retrieved. I assumed, VaultCustomQueryCriteria was supposed to do that.
Related
How can I get items count for a particular partition key using .net core preferably using Object Persistence Interface or Document Interfaces?
Since I do not see any docs any where, currently I get the number of items count by retrieve all the item and get its count, but it is very expensive to do the reads.
What is the best practices for such item count request? Thank you.
dynamodb is mostly a document oriented key-value db; so its not optimized for functionality of the common relation db functions (like item count).
to minimize the data that is transmitted and to improve speed you may want to do the following:
Create Lambda Function that returns Item Count
To avoid transmitting data outside of AWS; which is slow and expensive.
query options
use only keys in your projection-expression,
reducing the data that is transmitted from db
max page-size, reducing number of calls needed
Stream Option
Streams could also be used for keeping counts; e.g. as described in
https://medium.com/signiant-engineering/real-time-aggregation-with-dynamodb-streams-f93547cfb244
https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/bp-gsi-aggregation.html
Related SO Question
Complexity of finding total records count with partition key in nosql dynamodb table?
I just realized that using low level interface in QueryRequest one can set Select = "COUNT" then when calling QueryAsync() orQuery() will return the count only as a integer only. Please refer to code sample below.
private static QueryRequest getStockRecordCountQueryRequest(string tickerSymbol, string prefix)
{
string partitionName = ":v_PartitionKeyName";
string sortKeyPrefix = ":v_sortKeyPrefix";
var request = new QueryRequest
{
TableName = Constants.TableName,
ReturnConsumedCapacity = ReturnConsumedCapacity.TOTAL,
Select = "COUNT",
KeyConditionExpression = $"{Constants.PartitionKeyName} = {partitionName} and begins_with({Constants.SortKeyName},{sortKeyPrefix})",
ExpressionAttributeValues = new Dictionary<string, AttributeValue>
{
{ $"{partitionName}", new AttributeValue {
S = tickerSymbol
}},
{ $"{sortKeyPrefix}", new AttributeValue {
S = prefix
}}
},
// Optional parameter.
ConsistentRead = false,
ExclusiveStartKey = null,
};
return request;
}
but I would like to point out that this still will consumed the same read units as retrieving all the item and get its count by yourself. but since it is only returning the count as an integer, it is a lot more efficient then transmitting the entire items list cross the wire.
I think using DynamoDB Streams in a more proper way to get the counts for large project. It is just a lot more complicated to implement.
I'm using vaultQueryBy that query all Unconsumed state and sorting with some field of that state but it not working (result is completely random sorting).
Then I use the same vaultQueryBy but add some equal condition, then result is perfectly sorting.
I tried with other field also (type String and long) but it not work also, Unless using with other field condition.
Here some code
QueryCriteria queryCriteria = new QueryCriteria.VaultQueryCriteria(Vault.StateStatus.UNCONSUMED);
Sort.SortColumn sortCol = new Sort.SortColumn(new SortAttribute.Custom(CustomerSchema.CustomerEntity.class, "changeDate"), Sort.Direction.DESC);
List list = new ArrayList();
list.add(sortCol);
List<StateAndRef<CustomerState>> list = rpcOps().vaultQueryBy(queryCriteria, pageSpecification, new Sort(sortList), CustomerState.class).getStates();
And it's not working then I add this condition before vaultQueryBy
QueryCriteria baseCriteria = new QueryCriteria.VaultCustomQueryCriteria(Builder.notNull(getField("linearId", CustomerSchemaV01.CustomerEntity.class)));
queryCriteria = queryCriteria.and(baseCriteria);
And it' work fine get result with correct sorting.
I'm not quite sure that it's Corda's intention or just some bug, but I think it should be able to order without using any field condition.
This is probably expected behaviour since the sorting criteria is only applicable to custom queries, and hence you need to provide a custom query for it to be added to the overall query.
I am fairly new in this realm and any help is appreciated
I have a table in Dynamodb database named Tenant as below:
"TenantId" is the hash primary key and I have no other keys. And I have a field named "IsDeleted" which is boolean
Table Structure
I am trying to run a query to get the record with specified "TenantId" while it is not deleted ("IsDeleted == 0")
I can get a correct result by running the following code: (returns 0 item)
var filter = new QueryFilter("TenantId", QueryOperator.Equal, "2235ed82-41ec-42b2-bd1c-d94fba2cf9cc");
filter.AddCondition("IsDeleted", QueryOperator.Equal, 0);
var dbTenant = await
_genericRepository.FromQueryAsync(new QueryOperationConfig
{
Filter = filter
}).GetRemainingAsync();
But no luck when I try to get it with following code snippet (It returns the item which is also deleted) (returns 1 item)
var queryFilter = new List<ScanCondition>();
var scanCondition = new ScanCondition("IsDeleted", ScanOperator.Equal, new object[]{0});
queryFilter.Add(scanCondition);
var dbTenant2 = await
_genericRepository.LoadAsync("2235ed82-41ec-42b2-bd1c-d94fba2cf9cc", new DynamoDBOperationConfig
{
QueryFilter = queryFilter,
ConditionalOperator = ConditionalOperatorValues.And
});
Any Idea why ScanCondition has no effect?
Later I also tried this: (throw exception)
var dbTenant2 = await
_genericRepository.QueryAsync("2235ed82-41ec-42b2-bd1c-d94fba2cf9cc", new DynamoDBOperationConfig()
{
QueryFilter = new List<ScanCondition>()
{
new ScanCondition("IsDeleted", ScanOperator.Equal, 0)
}
}).GetRemainingAsync();
It throws with: "Message": "Must have one range key or a GSI index defined for the table Tenants"
Why does it complain about Range key or Index? I'm calling
public AsyncSearch<T> QueryAsync<T>(object hashKeyValue, DynamoDBOperationConfig operationConfig = null);
You simply cant query a table only giving a single primary key (only hash key). Because there is one and only one item for that primary key. The result of the Query would be that still that single item, which is actually Load operation not Query. You can only query if you have composite primary key in this case (Hash (TenantID) and Range Key) or GSI (which doesn't impose key uniqueness therefore accepts duplicate keys on index).
The second code attempts to filter the Load. DynamoDBOperationConfig's QueryFilter has a description ...
// Summary:
// Query filter for the Query operation operation. Evaluates the query results and
// returns only the matching values. If you specify more than one condition, then
// by default all of the conditions must evaluate to true. To match only some conditions,
// set ConditionalOperator to Or. Note: Conditions must be against non-key properties.
So works only with Query operations
Edit: So after reading your comments on this...
I dont think there conditional expressions are for read operations. AWS documents indicates they are for put or update operations. However, not being entirely sure on this since I never needed to do a conditional Load. There is no such thing like CheckIfExists functionality as well in general. You have to read the item and see if it exists. Conditional load will still consume read throughput so your only advantage would be only NOT retrieving it in other words saving the bandwith (which is very negligible for single item).
My suggestion is read it and filter it in your application layer. Dont query for it. However what you can also do is if you very need it you can use TenantId as hashkey and isDeleted for range key. If you do so, you always have to query when you wanna get a tenant. With the query you can set rangeKey(isDeleted) to 0 or 1. This isnt how I would do it. As I said, would just read it and filter it at my application.
Another suggestion thing could be setting a GSI on isDeleted field and writing null when it is 0. This way you can only see that attribute in your table when its only 1. GSI on such attribute is called sparse index. Later if you need to get all the tenants that are deleted (isDeleted=1) you can simply scan that entire index without conditions. When you are writing null when its 0 dynamoDB wont put it in the index at the first place.
We have a queryablestate for storing some information when the system is initialized, these states are issued once and never consumed
class DataState implements LinearState, QueryableState {
Party partyA;
Party partyB;
String partyAId;
String partyBId;
String sharedIdentifierNumber;
DataState(Party partyA, Party partyB, String partyAId, String partyBId, String sharedIdentifierNumber) {
this.partyA = partyA;
this.partyB = partyB;
this.partyAId = partyAId;
this.partyBId = partyBId;
this.sharedIdentifierNumber = sharedIdentifierNumber;
}
}
partyA and partyAId must be related to entity A (same for partyB)
some example instances:
new DataState(party1, party2, "00001", "12345", "0001")
new DataState(party3, party1, "05432", "00001", "0022")
new DataState(party2, party1, "12345", "00001", "0123")
we want to have methods that work like a map:
String retrievePartyId(Party party){}
assertTrue(retrievePartyId(party1).equals("00001"))
Party retrieveParty(String partyId){}
assertTrue(retrieveParty("12345").equals(party2))
we have already done this by querying all the states with custom field criteria and comparing through iteration on the
List<StateAndRef<DataState>>
We would like to know if there is some efficient way of doing this, maybe with some custom querycriteria in Corda. We think this is related to sql column projections. The query interface returns list of 'states(.getStates())' OR aggregation results '(.getOtherResults())'. We were wondering if it’s possible (or planned) to get a single column from the db and then filter the duplicates through the vault interface, currently we’re doing that in java.
If your states are QueryableStates, they can be efficiently queried based on their schema's attributes.
For example, suppose in the CorDapp Example (https://github.com/corda/cordapp-example), I wanted to query the vault for IOUStates based on their value and whether I am the lender. I could do this as follows:
try {
Field value = IOUSchemaV1.PersistentIOU.class.getDeclaredField("value");
Field lender = IOUSchemaV1.PersistentIOU.class.getDeclaredField("lender");
QueryCriteria valueOver50Criteria = new QueryCriteria.VaultCustomQueryCriteria(greaterThan(value, 50));
QueryCriteria lenderIsMeCriteria = new QueryCriteria.VaultCustomQueryCriteria(equal(lender, getOurIdentity().getName().toString()));
QueryCriteria criteria = valueOver50Criteria.and(lenderIsMeCriteria);
Vault.Page<IOUState> results = getServiceHub().getVaultService().queryBy(IOUState.class, criteria);
List<StateAndRef<IOUState>> matchingStates = results.getStates();
} catch (NoSuchFieldException e) {
e.printStackTrace();
}
In your case, you would be querying based on the party's ID or the party's name. You would then take the first result and use it to map the party's name to the party's ID, or vice-versa.
I am trying to get the value from other state using the primary key in the current state using a custom query. i couldn't find a way to where to start and what to use. I read about valutqueryby. but no idea about the custom query. please help me
How I am doing it is during creation of your second state you put your first state linear ID into the state, then:-
val generalCriteria = QueryCriteria.VaultQueryCriteria(Vault.StateStatus.UNCONSUMED)
val firstLinearIdExpression = builder { (SecondSchemaV1.SecondEntity::orderfirstLinearId).equal(firstLinearId.id.toString()) }
val results = serviceHub.vaultService.queryBy<Second>(generalCriteria .and(QueryCriteria.VaultCustomQueryCriteria(firstLinearIdExpression))).states
then you will get your results from your vault.