I have a scheduled function that runs every three minutes.
It is supposed to look on the database (firestore), query the relevant users, send them emails or perform other db actions.
Once it sends an email to a user, it updates the user with a field 'sent_to_today:true'.
If sent_to_today == true, the function won't touch that user for around 24 hours, which is what's intended.
But, because I have many users, and the function is doing a lot of work, by the time it updates the user with sent_to_today:true, another invocation gets to that user beforehand and processes them for sending emails.
This results in some users getting the same email, twice.
What are my options to make sure this doesn't happen?
Data Model (simplified):
users (Collection)
--- userId (document)
--- sent_to_today [Boolean]
--- NextUpdateTime [String representing a Timestamp in ISO String]
When the function runs, if ("Now" >= NextUpdateTime) && (sent_to_today==false), the user is processed, otherwise, they're skipped.
How do I make sure that the user is only processed by one invocation per day, and not many?
As I said, by the time they're processed by one function invocation (which sets "sent_to_today" to true), the next invocation gets to that user and processes them.
Any help in structuring the data better or using any other logical method would be greatly appreciated.
Here is an idea I'm considering:
Each invocation sets a global document's field, "ex: busy_right_now : true" at the start, and when finished it sets it again to false. If a subsequent invocation runs before the current is finished, it does nothing if busy_right_now is still true.
Option 1.
Do you think you the function can be invoked once in ten minutes, rather every three minutes? If yes - just modify the scheduler, and make sure that 'max instances' attribute is '1'. As the function timeout is only 540 seconds, 10 minutes (600 seconds) is more than enough to avoid overlapping.
Option 2.
When a firestore document is chosen for processing, the cloud function modifies some attribute - i.e. __state - and sets its value to IN_PROGRESS for example. When the processing is finished (email is sent), that attribute value is modified again - to DONE for example. Thus, if the function picks up a document, which has the value IN_PROGRESS in the __state attribute - it simply ignores and continues to the next one.
The drawback - if the function crashes - there might be documents with IN_PROGRESS state, and there should be some mechanism to monitor and resolve such cases.
Option 3.
One cloud function runs through the firestore collection, and for each document, which is to be processed - sends a pubsub message which triggers another cloud function. That one works only with one firestore document. Nevertheless the 'state machine' control is required (like in the Option 2 above). The benefit of the option 3 - higher level of specialisation between functions, and there may be many 'second' cloud functions running in parallel.
Related
I've been trying to make the integration between react-query and react-virtualized's InfiniteLoader possible, however I got stuck at a particular point.
Namely, the InfinteLoader's fetchMore function is invoked with an IndexRange ({ startIndex: number, stopIndex: number }). This is important because the batch size can vary and usually the initially batch size is larger than the subsequent batches. Hence it is imperative for the queryFn to have access to the startIndex, stopIndex or a computed limit. For this integration I've been using the firebase sdk however the principal of a cursor is used more widely. I won't go into the specifics, however, firebase's startAfter collection filter is accepts the reference to the last loaded document. Fortunately, the getFetchMore function is invoked with the last batch and the last document in the batch represents the cursor after which the next batch starts. The point being that there is not way to pass both the limit and the last document reference to the query function in order to load the new batch.
Is there are workaround?
When writing event based cloud functions for firebase firestore it's common to update fields in the affected document, for example:
When a document of users collection is updated a function will trigger, let's say we want to determine the user info state and we have a completeInfo: boolean property, the function will have to perform another update so that the trigger will fire again, if we don't use a flag like needsUpdate: boolean to determine if excecuting the function we will have an infinite loop.
Is there any other way to approach this behavior? Or the situation is a consequence of how the database is designed? How could we avoid ending up in such scenario?
I have a few common approaches to Cloud Functions that transform the data:
Write the transformed data to a different document than the one that triggers the Cloud Function. This is by far the easier approach, since there is no additional code needed - and thus I can't make any mistakes in it. It also means there is no additional trigger, so you're not paying for that extra invocation.
Use granular triggers to ensure my Cloud Function only gets called when it needs to actually do some work. For example, many of my functions only need to run when the document gets created, so by using an onCreate trigger I ensure my code only gets run once, even if it then ends up updating the newly created document.
Write the transformed data into the existing document. In that case I make sure to have the checks for whether the transformation is needed in place before I write the actual code for the transformation. I prefer to not add flag fields, but use the existing data for this check.
A recent example is where I update an amount in a document, which then needs to be fanned out to all users:
exports.fanoutAmount = functions.firestore.document('users/{uid}').onWrite((change, context) => {
let old_amount = change.before && change.before.data() && change.before.data().amount ? change.before.data().amount : 0;
let new_amount = change.after.data().amount;
if (old_amount !== new_amount) {
// TODO: fan out to all documents in the collection
}
});
You need to take care to avoid writing a function that triggers itself infinitely. This is not something that Cloud Functions can do for you. Typically you do this by checking within your function if the work was previously done for the document that was modified in a previous invocation. There are several ways to do this, and you will have to implement something that meets your specific use case.
I would take this approach from an execution time perspective, this means that the function for each document will be run twice. Each time when the document is triggered, a field lastUpdate would be there with a timestamp and the function only updates the document if the time is older than my time - eg 10 seconds.
I have a micro-service which involved in an OAuth 1 interaction. I'm finding myself in a situation where two runs of the Lambda functions with precisely the same starting states have very different outcomes (where state is considered the "event" passed in, environment variables, and "stageParameters" from the API Gateway).
Here's a Cloudwatch log that shows two back-to-back runs:
You can see that while the starting state is identical, the execution path changes pretty quickly. In the second case (failure case), you see the log entry "Auth state changed: null" ... that is very odd indeed because in fact this is logged before even the first line of code of the "handler" is executed. Here's the beginning of the functions handler:
export const handler = (event, context, cb) => {
console.log('EVENT:\n', JSON.stringify(event, null, 2));
So where is this premature logging entry coming from? Well, one must assume that it somehow is left over from prior executions. Let me demonstrate ... it is in fact an event listener that was setup in the prior execution. This function interacts with a Firebase DB and the first time it connects it sets the following up:
auth.signInWithEmailAndPassword(username, password)
.then((result) => {
auth.onAuthStateChanged(this.watchAuthState);
where the watchAuthState function is simply:
watchAuthState(user) {
console.log(`Auth state changed:\n`, JSON.stringify(user, null, 2));
}
This seems to mean that when I run the DB a second time I am already "initialized" with the Firebase DB but apparently the authentication has been invalidated. My number one aim is to just get back to a predictive state model and have it execute precisely the same each time.
If, there are sneaky ways to reuse cached state between Lambda executions in resource useful ways then I guess that too would be interesting but only if we can do that while achieving the predictive state machine.
Regarding the logs order, look at the ID that comes after each timestamp at the beginning of each line. I believe this is the invocation ID. In the two lines you have highlighted in orange, they are from different invocations of the function. The EVENT log is the first line to get logged from the invocation with ID ending in 754ee. The Auth state changed: null line is a log entry coming from the earlier invocation of the function with invocation ID ending in c40d5.
It looks like you are setting auth state to null at the end of an invocation, but the Firebase connection is global, so the second function invocation thinks the Firebase connection is already initialized, but then it throws errors because the authentication was nulled out.
My number one aim is to just get back to a predictive state model and
have it execute precisely the same each time.
Then you need to be aware of Lambda container reuse, and not use any global variables.
I'm dealing with a problem where a user can update a document within a specified time limit, and if he doesn't, the server will.
The update involves incrementing a value and adding an object to an array of a document. I need to ensure that only one of the user/server updates the document. Not both.
To ensure this happens, some checks are run to see if the document has already been updated, but there are times where the user and server run at exactly the same time and both pass the checks and then the document is updated twice.
I've been trying many different ways of fixing this, but I haven't been able to. I tried implement a lock similar to this: http://en.wikipedia.org/wiki/Peterson%27s_algorithm to ensure that only one update will happen and the second update will fail, but I haven't been successful. Any ideas?
To ensure this happens, some checks are run to see if the document has already been updated, but there are times where the user and server run at exactly the same time and both pass the checks and then the document is updated twice.
You can achieve this by using a MongoDB update query that simultaneously checks if the value has been updated and updates it. Like this:
var post = Posts.findOne("ID");
// ... do some stuff with the post ...
Posts.update({counter: post.counter}, {$push: {items: newItem}, $inc: {counter: 1}});
As you can see, in one query we both check the counter and increment it - so if two of these queries run one right after another only one will actually update the document (since the counter won't match anymore).
Say I have a resource defined such that the service https://api.mydomainname.com/v1/stores/ABC/order where "ABC" has a direct relation to a Developer App entry. I also have a Custom Attribute for the maximum number of seconds that can elapse between calls; if the time between calls exceeds that threshold, I'd like to take an action (make an call, raise a Nagios alert, etc).
Example:
ABC (api key: 54c34ce0f7691840093bfba55a10c782) has a 300s elapsed time threshold
XYZ (api key: af9843af2d190f72481183a8645659ac) has a 600s elapsed time threshold
Do we have any ideas on how to do this on for all related message processors in an Organization?
I do not want to have to have a call to check if the threshold has been exceeded.
I'm thinking I have to push this further down to the actual service, but if there is an Apigee based potential solution, I'd be interested.
One option is to use the <LookupCache> and <PopulateCache> policies. Caching is distributed (across MPs) by default.
You could store the threshold value (300s/600s/etc) as a custom attribute on the develop app. After <VerifyAPIKey> policy, this attribute value should be automatically populated and you can use that reference in the <PopulateCache> policy. Example flow steps:
VerifyAPIKey (will get the threshold value for populate.
LookupCache
If cache hit, continue. Else, raise custom fault or execute custom logic.
PopulateCache to create/update entry, reading expiry from variable reference of attribute populated by #1). If the entry already exists, it will re-populate the entry with fresh expiration, essentially reseting the last request time.