I'm interested in tracking vault updates for states. Whenever a state of a specific contract state type, say Token, is produced or consumed in a transaction, I want to receive an update. I use the following query criteria
val queryCriteria = QueryCriteria.VaultQueryCriteria(Vault.StateStatus.ALL)
.and(
QueryCriteria.VaultQueryCriteria(
timeCondition = QueryCriteria.TimeCondition(
QueryCriteria.TimeInstantType.RECORDED,
ColumnPredicate.BinaryComparison(
BinaryComparisonOperator.GREATER_THAN_OR_EQUAL,
offset
)
)
)
)
together with
ops.vaultTrackByWithSorting(
TokenState::class.java,
queryCriteria,
Sort(
setOf(
Sort.SortColumn(
SortAttribute.Standard(Sort.VaultStateAttribute.RECORDED_TIME),
Sort.Direction.ASC
),
Sort.SortColumn(
SortAttribute.Standard(Sort.CommonStateAttribute.STATE_REF_INDEX),
Sort.Direction.ASC
)
)
)
)
Interestingly, I do see updates which produce Tokens, but I do not see any updates when Tokens get consumed. I'm using Corda 4.6.
I'm aware of the _trackBy's limitations with respect to query criteria, however, according to Corda's vault api documentation, as I understand it, the above code should listen to both produced (UNCONSUMED) and consumed (CONSUMED) state updates. Is this a known bug, or am I doing something wrong?
VaultQueryCriteria() sets Vault.StateStatus.UNCONSUMED by default, if you don't specify it. So, in your case you are doing a query that logically does
(ALL).and(UNCONSUMED) = UNCONSUMED
The result is that you do not see the CONSUMED.
You need to specify the StateStatus.ALL also in your second query in and().
I agree this is not clear in the documentation.
Related
I tried to search for an example but, I presume it's not doable. I am looking to hopefully be proven wrong or to find an official confirmation that it's not doable.
Before using Hasura, I was doing transactional SQL queries that ensured that data was kept consistent.
For example, I would like to create a password reset token if a user requests it, only if the user can be found using an email address. Right now, I have to do 2 queries:
Try to find a user with the specified email address
Insert and assign the token to this user id
In that case, it's not too bad, but now if I want to consume that token, I have to do 3 queries:
Find the valid token
Change the password to the user associated with that token
Delete the token
Obviously, if something goes wrong and the token is not deleted, this could be an issue - so I would be curious to see if there would be ways to merge these queries/mutations into transactions.
Sounds like supporting nested updates would solve this problem for you with the least amount of effort. We are working on a rfc for the feature and hope to start development soon. Please follow this Github issue on our community for future updates.
https://github.com/hasura/graphql-engine/issues/1573
This comment outlines the current scope of the proposed feature. The rfc will provide a more complete explanation.
https://github.com/hasura/graphql-engine/issues/1573#issuecomment-1338057350
You can apply changes to rows that you filter by certain criteria. Here is a sample mutation:
mutation PasswordUpdate($id: uuid!, $token: String!, $new_password: String!) {
update_user(
where: {id: {_eq: $id}, token: {_eq: $token}}
_set: {token: null, password: $new_password}
) {
affected_rows
}
}
That query deletes the token and sets a password for all users (hopefully just one) that have the token assigned.
After some research here is what I found:
For the first example:
Try to find a user with the specified email address
Insert and assign the token to this user id
There are no solutions for this today and as answered by #damel, there is an ongoing RFC to support nested mutations: https://github.com/hasura/graphql-engine/issues/1573#issuecomment-1338057350
Hopefully, this feature will be out soon, but in the meantime, for most cases, it's not such a big deal to have multiple queries as it is possible to catch errors on the first query.
For the second example:
Find the valid token
Change the password to the user associated with that token
Delete the token
When sending multiple mutations in the same query, Hasura treats them as a transaction as announced in 2020.
Of course, it would be nice to do this in the same query (similar to the first example) but since there is a transaction on the mutation, for this case it's still not a problem.
I am sure there are probably cases where this can become a problem but I am not exposed to them right now. Nevertheless, it would be great if the RFC makes it to production, giving more options to Hasura users.
I am using Change Feed processor library to read the Change Feed on a partitioned collection and below is the code for how I have configure it. I ma using most of the default options.
ChangeFeedProcessorOptions feedProcessorOptions = new
{
LeaseRenewInterval = TimeSpan.FromSeconds(15),
};
var docObserverFactory = DocumentFeedObserverFactory.Create(this.destinationCollectionInfo, this.dbRepository);
this.builder
.WithHostName(hostName)
.WithFeedCollection(this.monitoredCollectionInfo)
.WithLeaseCollection(this.leaseCollectionInfo)
.WithProcessorOptions(feedProcessorOptions)
.WithObserverFactory(docObserverFactory);
This runs fine as long as the Change Feed application is running and documents are being inserted/updated in the collection and the Change Feed app picks them up as expected.
The problem happens when I stop the Change Feed app for sometime and insert/update few documents in the Collection. Then when I start the Change Feed app, it doesn't pick changes from where it last left. Those changes that were inserted when the Change Feed app was stopped are lost. But when I set the flag StartFromBeginning to true, it picks everything from the start including changes that were inserted when the Change Feed app was stopped in between for sometime.
My understanding of read from current (StartFromBeginning to false) is that the Change Feed reads documents since it last left. But that doesn't seem to happen. Please help.
There are two ways to continue from exactly where you left it.
The first, and more accurate one, is to store the Continuation token of the last thing you read. That way you can specify it when you start again and it will win over both the StartTime and the StartFromBeginning flags.
The second one is to provide the StartTime property which will try and find the continuation token of a given time automatically. It has an approximate 5 second precision so there is a chance that you might miss some documents though.
This is more of a theoretical how database should be setup, and less about programming.
Lets say I have a news feed full of cards, which each contain a message and a like count. Each user is able to like a mesesage. I want it to be displayed to a user if they have already liked that particular card. (The same way you can see the post you like on facebook, even if you come back days later)
How would you implement that with this Firestore type database? Speed is definitely a concern..
storying it locally isn't an option, my guess would be on each card object, you would have to reference a collection that just kept a list of people who liked it. The only thing is that is a lot more querying.. which feels like it would be slow..
is there a better way to do this?
TL;DR
This approach requires more to setup, ie a cron service, knowledge of Firestore Security Rules and Cloud Functions for Firebase. With that said, the following is the best approach I've come up with. Please note, only pseudo-rules that are required are shown.
Firestore structure with some rules
/*
allow read
allow update if auth.uid == admin_uid and the
admin is updating total_likes ...
*/
messages/{message_key} : {
total_likes: <int>,
other_field:
[,...]
}
/*allow read
allow write if newData == {updated: true} and
docId exists under /messages
*/
messages_updated/{message_key} : {
updated: true
}
/*
allow read
allow create if auth.uid == liker_uid && !counted && !delete and
liker_uid/message_key match those in the docId...
allow update if auth.uid == admin_uid && the admin is
toggling counted from false -> true ...
allow update if auth.uid == liker_uid && the liker is
toggling delete ...
allow delete if auth.uid == admin_uid && delete == true and
counted == true
*/
likes/{liker_uid + '#' + message_key} : {
liker_uid:,
message_key:,
counted: <bool>,
delete: <bool>,
other_field:
[,...]
}
count_likes/{request_id}: {
message_key:,
request_time: <timestamp>
}
Functions
Function A
Triggered every X minutes to count message likes for potentially all messages.
query /messages_updated for BATCH_SIZE docs
for each, set its docId to true in a local object.
go to step 1 if BATCH_SIZE docs were retrieved (there's more to read in)
for each message_key in local object, add to /count_likes a doc w/ fields request_time and message_key.
Function B
Triggered onCreate of count_likes/{request_id}
Delete created docs message_key from /messages_updated.
let delta_likes = 0
query /likes for docs where message_key == created docs message_key and where counted == false.
for each, try to update counted to true (in parallel, not atomically)
if successful, increment delta_likes by 1.
query /likes for docs where message_key == created docs message_key, where delete == true and where counted == true.
for each doc, try to delete it (in parallel, not atomically)
if successful, decrement delta_likes by 1
if delta_likes != 0, transact the total likes for this message under
/messages by delta_likes.
delete this doc from /count_likes.
Function C (optional)
Triggered every Y minutes to delete /count_likes requests that were never met.
query docs under /count_likes that have request_time older than Z.
for each doc, delete it.
On the client
to see if you liked a message, query under /likes for a doc where liker_uid equals your uid, where message_key equals the message's key and where delete == false. if a doc exists, you have liked it.
to like a message, batch.set a like under /likes and batch.set a /messages_updated. if this batch fails, try a batch_two.update on the like by updating its delete field to false and batch_two.set its /messages_updated.
to unlike a message, batch.update on the like by updating its delete field to true and batch.set its /messages_updated.
Pros of this approach
this can be extended to counters for other things, not just messages.
a user can see if they've liked something.
a user can only like something once.
a user can spam toggle a like button and this still works.
any user can see who's liked what message by querying /likes by message_key.
any user can see all the messages any user has liked by querying /likes by liker_uid.
only a cloud function admin updates your like counts.
if a function is fired multiple times for the same event, this function is safe, meaning like counts will not be incremented multiple times for the same like.
if a function is not fired for some event, this approach still works. It just means that the count will not update until the next time someone else likes the same message.
likes are denormalized to only one root level collection, instead of the two that would be required if you had the like under the the message's likes subcollection and under the liker's messages_liked subcollection.
like counts for each message are updated in batches, ie if something has been liked 100 times, only 1 transaction of 100 is required, not 100 transactions of 1. this reduces write rate conflicts due to like counter transactions significantly.
Cons of this approach
Counts are only updated however often your cron job fires.
Relies on a cron service to fire and in general there's just more to set up.
Requires the function to authenticate with limited privileges to perform secure writes under /likes. In the Realtime Database this is possible. In Firestore, it's possible, but a bit hacky. If you can wait and don't want to use the hacky approach, use the regular unrestricted admin in development until Firestore supports authenticating with limited privileges.
May be costly depending on your standpoint. There are function invocations and read/write counts you should think about.
Things to consider
When you transact the count in Function B, you may want to consider trying this multiple times in case the max write rate of 1/sec is exceeded and the transaction fails.
In Function B, you may want to implement batch reading like in Function A if you expect to be counting a lot of likes per message.
If you need to update anything else periodically for the message (in another cron job), you may want to consider merging that function into Function B so the write rate of 1/sec isn't exceeded.
I need help in a scenario when we do multipath updates to a fan-out data. When we calculate the number of paths and then update, in between that, if a new path is added somewhere, the data would be inconsistent in the newly added path.
For example below is the data of blog posts. The posts can be tagged by multiple terms like “tag1”, “tag2”. In order to find how many posts are tagged with a specific tag I can fanout the posts data to the tags path path as well:
/posts/postid1:{“Title”:”Title 1”, “body”: “About Firebase”, “tags”: {“tag1:true, “tag2”: true}}
/tags/tag1/postid1: {“Title”:”Title 1”, “body”: “About Firebase”}
/tags/tag2/postid1: {“Title”:”Title 1”, “body”: “About Firebase”}
Now consider concurrently,
1a) that User1 wants to modify title of postid1 and he builds following multi-path update:
/posts/postid1/Title : “Title 1 modified”
/tags/tag1/postid1/Title : “Title 1 modified”
/tags/tag2/postid1/Title : “Title 1 modified”
1b) At the same time User2 wants to add tag3 to the postid1 and build following multi-path update:
/posts/postid1/tags : {“tag1:true, “tag2”: true, “tag3”: true}
/tags/tag3/postid1: {“Title”:”Title 1”, “body”: “About Firebase”}
So apparently both updates can succeed one after other and we can have tags/tag3/postid1 data out of sync as it has old title.
I can think of security rules to handle this but then not sure if this is correct or will work.
Like we can have updatedAt and lastUpdatedAt fields and we have check if we are updating our own version of post that we read:
posts":{
"$postid":{
".write":true,
".read":true,
".validate": "
newData.hasChildren(['userId', 'updatedAt', 'lastUpdated', 'Title']) && (
!data.exists() ||
data.child('updatedAt').val() === newData.child('lastUpdated').val())"
}
}
Also for tags we do not want to check that again and we can check if /tags/$tag/$postid/updatedAt is same as /posts/$postid/updatedAt.
"tags":{
"$tag":{
"$postid":{
".write":true,
".read":true,
".validate": "
newData.hasChildren(['userId', 'updatedAt', 'lastUpdated', 'Title']) && (
newData.child('updatedAt').val() === root.child('posts').child('$postid').val().child('updatedAt').val())”
}
}
}
By this “/posts/$postid” has concurrency control in it and users can write their own reads
Also /posts/$postid” becomes source of truth and rest other fan-out paths check if updatedAt fields matches with it the primary source of truth path.
Will this bring in consistency or there are still problems? Or can bring performance down when done at scale?
Are multi path updates and rules atomic together by that I mean a rule or both rules are evaluated separately in isolation for multi path updates like 1a and 1b above?
Unfortunately, Firebase does not provide any guarantees, or mechanisms, to provide the level of determinism you're looking for. I have had the best luck front-ending such updates with an API stack (GCF and Lambda are both very easy, server-less methods of doing this). The updates can be made in that layer, and even serialized if absolutely necessary. But there isn't a safe way to do this in Firebase itself.
There are numerous "hack" options you could apply. You could, for example, have a simple lock mechanism using a dedicated collection for tracking write locks. Clients could post to a lock collection, then verify that their key was the only member of that collection, before performing a write. But I hope you'll agree with me that such cooperative systems have too many potential edge cases, potential security issues, and so on. In Firebase, it is best to design such that this component is not a requirement in the first place.
dayPath = ref.path.toString() + '/' + configId + '/screens/' + screenIndex + '/days/',
// the ref for the days object
daysRef = fbutil.ref(dayPath),
// the sync for the days object
daysSync = fbutil.syncObjectReference(daysRef);
// the collection as a $firebase array
var list = daysSync.$asArray(),
items = [],
number;
console.log(list);
list.$add({dummy: 'Test'});
According with the documentation, when I use $add with $asArray, the $add supposed to do a "push". But instead it's creating a hash key instead a numeric index.
So, the dummy: test has a parent containing a hash key. The expected is a numeric index, I mean : array item.
Can someone give me some help? I just have 1 week of experience in this database.
The result is this one...
screens
...0
.......days
..........0
..........1
..........2
.........-JrT5ZATDELIR3gXAvah
................dummy: test
AngulareFire is built on the Firebase JavaScript SDK. So when AngularFire's documentation says it uses push internally, it is not referring to JavaScript's Array.push, but to Firebase's push operation. And Firebase's push generates its own keys, it does not generate regular array indexes.
The reason for that is best explained in Firebase's documentation on arrays, but essentially boils down to: arrays don't work well in distributed environments, because all clients have to agree on the array.length in order to be able to add a new item.
So $firebaseArray.$add will generated a so-called push ID. They are ordered, like array indexes, but can be generated across clients without risk of conflicts.
I noticed that you're on a somewhat older version of AngularFire. I highly recommend that you follow the "official" quickstart and guide for AngularFire.
I would like to comment but i don't have enough reputation yet so i'm doing it here.
The solution in my eyes is very simple:
Instead of:
list.$add({dummy: 'Test'});
Do:
list[index] = {dummy: 'Test'};