I am using Firebase Functions Task Queues to call numerous API endpoints.
Each of those functions is set to retry 5 times if they encounter error.
Now I want to track if the function completes successfully or fails completely (i.e. all retries are exhausted and the function still throws an error). I'll probably update a Firestore document when that happens.
For example, here is a Task Queue function, how do I add the above functionality?
export const someTask = functions.tasks
.taskQueue({
retryConfig: {
maxAttempts: 5,
minBackoffSeconds: 60,
},
rateLimits: {
maxConcurrentDispatches: 1,
},
})
.onDispatch(
async () => {
try {
// Call the API
await apiCall();
return;
} catch (error) {
// Throw error so that the Task Queue will retry
throw new functions.https.HttpsError(
'unknown',
'someTask error'
);
}
}
);
Related
Objective
User on click a purchase button on the web frontend, it will send a POST request to the backend to create a purchase order. First, it will check the number of available stocks. If available is greater than 0, reduce available by 1 and then create the order.
The setup
Backend (NestJS) queries the Firestore for the latest available value, and reduce available by 1. For debugging, I will return the available value.
let available;
try {
await runTransaction(firestore, async (transaction) => {
const sfDocRef = doc(collection(firestore, 'items_available'), documentId);
const sfDoc = await transaction.get(sfDocRef);
if (!sfDoc.exists()) {
throw 'Document does not exist!';
}
const data = sfDoc.data();
available = data.available;
if(available>0){
transaction.update(sfDocRef, {
available: available-1,
});
}
});
} catch (e) {
console.log('Transaction failed: ', e);
}
return { available };
My stress test setup
Our goal is to see all API requests having different available value, this would mean that Firestore Transactions is reducing the value even though there are multiple requests coming in.
I wrote a simple multi-threaded program that queries the backend's create order API, it will query the available value and return the available value. This program will save the available value returned for each API request.
The stress test performed is about 10 transactions per second, as I have 10 concurrent processes querying the backend. Each process will http.get 20 queries:
const http = require('http');
function call(){
http.get('http://localhost:5000/get_item_available', res => {
let data = [];
res.on('data', chunk => {
data.push(chunk);
});
res.on('end', () => {
console.log('Response: ', Buffer.concat(data).toString());
});
}).on('error', err => {
console.log('Error: ', err.message);
});
}
for (var i=0; i<20; i++){
call();
}
The problem
Unfortunately, the available values I got from the requests contains repeated values, that is, having same available values instead of having unique available values.
What is wrong? Isn't Firestore Transactions meant to handle race conditions? Any suggestions on what I could change to handle multiple requests hitting the server and return a new value for each request?
You have a catch clause to handle when the transaction fails, but then still end up returning a value to the caller return { available }. In that situation you should return an error to the caller.
I've made firebase cloud function which adds the claim to a user that he or she has paid (set paid to true for user):
const admin = require("firebase-admin");
exports.addPaidClaim = functions.https.onCall(async (data, context) => {
// add custom claim (paid)
return admin.auth().setCustomUserClaims(data.uid, {
paid: true,
}).then(() => {
return {
message: `Succes! ${data.email} has paid for the course`,
};
}).catch((err) => {
return err;
});
});
However, when I'm running this function: I'm receiving the following error: "Unhandled Rejection (RangeError): Maximum call stack size exceeded". I really don't understand why this is happening. Does somebody see what could cause what's getting recalled which in turn causes the function to never end?
Asynchronous operations need to return a promise as stated in the documentation. Therefore, Cloud Functions is trying to serialize the data contained by promise returned by transaction, then send it in JSON format to the client. I believe your setCustomClaims does not send any object to consider it as an answer to the promise to finish the process so it keeps in a waiting loop that throws the Range Error.
To avoid this error I can think of two different options:
Add a paid parameter to be able to send a JSON response (and remove the setCustomUserClaim if it there isn’t any need to change the user access control because they are not designed to store additional data) .
Insert a promise that resolves and sends any needed information to the client. Something like:
return new Promise(function(resolve, reject) {
request({
url: URL,
method: "POST",
json: true,
body: queryJSON //A json variable I've built previously
}, function (error, response, body) {
if (error) {
reject(error);
}
else {
resolve(body)
}
});
});
When retrieving data from Firestore one has the option of forcing retrieval from the server. The default option is cache and server, as determined by Firestore.
I have a certain usage where a command and control node is issuing real-time commands to remote nodes backed by Firestore. This requires the updates to be done on the server (or fail) so that the C&C node has certainty on the execution (or failure) in real-time. What I would like to do is to disable use of cache with these updates. I have not found a way to do that. Is this possible with current capabilities of Firestore?
Note that it is not desirable to disable Firestore caching at a global level as the cache is beneficial in other situations.
----EDIT-----
Based on the responses I have created this update method that attempts to force updating the server using a transaction.
A couple of notes:
This is dart code.
Utils.xyz is an internal library and in this case it is being used to log.
I have reduced the network speed for the test to simulate a bad network connection.
The timeout is set to 5 seconds.
Here is the output of my log:
I/flutter (22601): [2021-06-06 22:35:30] [LogLevel.DEBUG] [FirestoreModel] [update] [We are here!]
I/flutter (22601): [2021-06-06 22:35:47] [LogLevel.DEBUG] [FirestoreModel] [update] [We are here!]
I/flutter (22601): [2021-06-06 22:36:02] [LogLevel.DEBUG] [FirestoreModel] [update] [We are here!]
I/flutter (22601): [2021-06-06 22:37:18] [LogLevel.DEBUG] [FirestoreModel] [update] [We are here!]
I/flutter (22601): [2021-06-06 22:37:20] [LogLevel.INFO] [FirestoreModel] [update] [Transaction successful in 110929ms.]
Firebase completely ignores the timeout of 5 seconds; tries to update 4 times each time ~15 seconds apart and is finally successful after 110 seconds. I am after a real-time response within seconds (5 sec) or failure.
Future<void> update(
Map<String, dynamic> data, {
WriteBatch batch,
Transaction transaction,
bool forceServer = false,
}) async {
// If updating there must be an id.
assert(this.id != null);
// Only one of batch or transaction can be non-null.
assert(batch == null || transaction == null);
// When forcing to update on server no transaction or batch is allowed.
assert(!forceServer || (batch == null && transaction == null));
try {
if (forceServer) {
DateTime start = DateTime.now();
await FirebaseFirestore.instance.runTransaction(
(transaction) async {
await update(data, transaction: transaction);
Utils.logDebug('We are here!');
},
timeout: Duration(seconds: 5),
);
Utils.logDebug('Transaction successful in ${DateTime.now().difference(start).inMilliseconds}ms.');
} else {
DocumentReference ref =
FirebaseFirestore.instance.collection(collection).doc(this.id);
if (batch != null)
batch.update(ref, data);
else if (transaction != null)
transaction.update(ref, data);
else
await ref.update(data);
}
} catch (e, s) {
Utils.logException('Error updating document $id in $collection.', e, s);
// Propagate the error.
rethrow;
}
}
This requires the updates to be done on the server (or fail)
For that you could use Transactions and batched writes.
Transactions will fail when the client is offline.
Check out doc
To get live data from the server once, you would use:
firebase.firestore()
.doc("somecollection/docId")
.get({ source: "server" })
.then((snapshot) => {
// if here, snapshot.data() is from the server
// TODO: do something with data
})
.catch((err) => {
// if here, get() encountered an error (insufficient permissions, server not available, etc)
// TODO: handle the error
});
To get realtime live data from only the server (ignoring the cache), you would use:
const unsubscribe = firebase.firestore()
.doc("somecollection/docId")
.onSnapshot({ includeMetadataChanges: true }, {
next(snapshot) {
// ignore cache data
if (snapshot.metadata.fromCache) return;
// if here, snapshot.data() is from the server
// TODO: do something with data
},
error(err) {
// if here, onSnapshot() encountered an error (insufficient permissions, etc)
// TODO: handle the error
}
});
To write to the server, you would use the normal write operations - delete(), set(), and update(); as they all return Promises that will not resolve while the client is offline. If they have resolved, the data stored on the server has been updated.
To test if you are online or not, you can try and pull a non existant document down from the server like so:
/**
* Attempts to fetch the non-existant document `/.info/connected` to determine
* if a connection to the server is available.
* #return {Promise<boolean>} promise that resolves to a boolean indicating
* whether a server connection is available
*/
function isCurrentlyOnline() {
// unlike RTDB, this data doesn't exist and has no function
// must be made readable in security rules
return firebase.firestore()
.doc(".info/connected")
.get({ source: "server" })
.then(
() => {
// read data successfully, we must be online
return true;
}, (err) => {
// failed to read data, if code is unavailable, we are offline
// for any other error, rethrow it
if (err.code === "unavailable")
return false;
throw err;
}
);
}
/**
* A function that attaches a listener to when a connection to Firestore has
* been established or when is disconnected.
*
* This function listens to the non-existant `/.info/connected` document and
* uses it's `fromCache` metadata to **estimate** whether a connection to
* Firestore is currently available.
* **Note:** This callback will only be invoked after the first successful
* connection to Firestore
*
* #param {((error: unknown | null, isOnline: boolean) => unknown)} callback the
* callback to invoke when the isOnline state changes
* #return {(() => void)} a function that unsubscribes this listener when
* invoked
*/
function onOnline(callback) {
let hasConnected = false;
// unlike RTDB, this data doesn't exist and has no function
// must be made readable in security rules
return firebase.firestore()
.doc(".info/connected")
.onSnapshot(
{ includeMetadataChanges: "server" },
{
next(snapshot) {
const { fromCache } = snapshot.metadata;
if (!hasConnected) {
if (fromCache) return; // ignore this event
hasConnected = true;
}
callback(null, !fromCache);
},
error(err) {
callback(err);
}
}
);
}
This is my first question ever here:-)
I need to list users in our Cognito use pool. Seems this can be done only using the aws-sdk CognitoIdentityServiceProvider APIs. I got the below code to work perfectly from my local node.js. It lists all users as expected.
However, the same code behaves differently when put into an AWS lambda function. It still runs, but it never waits for the Cognito listUsers() call to return. It just simply completes, like the "await" is not waiting at all. None of the console.log() is invoked from the client.listUsers(params, function(err, data)..._ block.
I tested this inside Lambda directly as well as from AWS API gateway. The same null is return. The call itself is successful, just no data is returned.
See log at the end.
BTW, I did create a role and add a policy giving the role full access to the cognito user pool.
What did I miss? Appreciate your help!
Martin C.
-----------------code------------------------
async function getUserList() {
console.log("enter LAMDA function**********");
var aws = require('aws-sdk');
aws.config.update({accessKeyId: 'xxxxxxxx', secretAccessKey: 'xxxxxxxxxxx'});
var CognitoIdentityServiceProvider = aws.CognitoIdentityServiceProvider;
var client = new CognitoIdentityServiceProvider({ apiVersion: '2016-04-19', region: 'us-east-2' });
var params = {
UserPoolId: 'us-east-xxxxxxxx', /* required */
AttributesToGet: [
'given_name','family_name','phone_number','email','profile'
],
Filter: null,
Limit: 0,
PaginationToken: null
};
console.log("Right before call the listUser method");
let result = await client.listUsers(params, function(err, data) {
console.log("call back reached!");
if (err) {
console.log(err, err.stack); // an error occurred
const response = {
statusCode: 500,
body: JSON.stringify('An error occurred.'),
}
return response;
}
else {
console.log(data);
var count = data.Users.length;
// successful response
const response = {
statusCode: 200,
body: JSON.stringify("sucessful list users! User count="+count)
}
return response;
}
});
console.log("no waiting here. async!!!")
}
getUserList();
***************Lambda log*****************
**************Log when called from node.js****************
getUserList is your lambda function? I don't know why you call it by your self getUserList().
I see, you are using lambda runtime is nodejs version > 8, you use await keyword with a callback function(fail) => you not wait anything.
When a function call by Lambda, the function (async function) will finish when get a return or run to end of function (without return), in your case the function finish when console.log("no waiting here. async!!!") has been executed. In local environment, the funciton finishs when callstack has been clear (do not have any callback function in callstack).
Right way, you have use promise version of aws-sdk then use await syntax to get a result. Relate to How to use Async and Await with AWS SDK Javascript
async function getUserList() {
console.log("enter LAMDA function**********");
var aws = require('aws-sdk');
aws.config.update({ accessKeyId: 'xxxxxxxx', secretAccessKey: 'xxxxxxxxxxx' });
var CognitoIdentityServiceProvider = aws.CognitoIdentityServiceProvider;
var client = new CognitoIdentityServiceProvider({ apiVersion: '2016-04-19', region: 'us-east-2' });
var params = {
UserPoolId: 'us-east-xxxxxxxx', /* required */
AttributesToGet: [
'given_name', 'family_name', 'phone_number', 'email', 'profile'
],
Filter: null,
Limit: 0,
PaginationToken: null
};
console.log("Right before call the listUser method");
try {
let result = await client.listUsers(params).promise(); // use Promise style
console.log(data);
var count = data.Users.length;
// successful response
const response = {
statusCode: 200,
body: JSON.stringify("sucessful list users! User count=" + count)
}
return response; // return to finish function
} catch (err) {
console.log(err, err.stack); // an error occurred
const response = {
statusCode: 500,
body: JSON.stringify('An error occurred.'),
}
return response;
}
}
getUserList(); // remove this line when deploy funtion to Lambda.
When I run my tests on CircleCI, it logs the following message a many times and eventually the tests fail because none of the database methods can retrieve the data due to the deadlocks:
{
"message": "Error running raw sql query in pool.",
"stack": "error: deadlock detected\n at Connection.Object.<anonymous>.Connection.parseE (/home/circleci/backend/node_modules/pg/lib/connection.js:567:11)\n at Connection.Object.<anonymous>.Connection.parseMessage (/home/circleci/-backend/node_modules/pg/lib/connection.js:391:17)\n at Socket.<anonymous> (/home/circleci/backend/node_modules/pg/lib/connection.js:129:22)\n at emitOne (events.js:116:13)\n at Socket.emit (events.js:211:7)\n at addChunk (_stream_readable.js:263:12)\n at readableAddChunk (_stream_readable.js:250:11)\n at Socket.Readable.push (_stream_readable.js:208:10)\n at TCP.onread (net.js:597:20)",
"name": "error",
"length": 316,
"severity": "ERROR",
"code": "40P01",
"detail": "Process 1000 waits for AccessExclusiveLock on relation 17925 of database 16384; blocked by process 986.\nProcess 986 waits for RowShareLock on relation 17870 of database 16384; blocked by process 1000.",
"hint": "See server log for query details.",
"file": "deadlock.c",
"line": "1140",
"routine": "DeadLockReport",
"level": "error",
"timestamp": "2018-10-15T20:54:29.221Z"
}
This is the test command I run: jest --logHeapUsage --forceExit --runInBand
I also tried this: jest --logHeapUsage --forceExit --maxWorkers=2
Pretty much all of the tests run some sort of database function. This issue only started to occur when we added more tests. Has anyone else had this same issue?
Based on the error message we got Deadlock because of RowShareLock;
This means that two transactions (lets call them transactionOne and transactionTwo) have locked resurce which the other transaction requires
Example:
transactionOne locks record in UserTable with userId = 1
transactionTwo locks record in UserTable with userId = 2
transactionOne attempts to update in UserTable for userId = 2, but since it is locked by another transaction - it waits for the lock to be released
transactionTwo attempts to update in UserTable for userId = 1, but since it is locked by another transaction - it waits for the lock to be released
Now the SQL engine detects that there is a deadlock and randomly picks one of the transactions and terminates it.
Lets say the SQL engine picks transactionOne and terminates it. This will result in the exception that is posted in the question.
transactionTwo is now allowed to perform an update in UserTable for user with userId = 1.
transactionTwo completes with success
SQL engines are pretty fast in detecting deadlocks, and the exception will be instant.
This is the reason for the deadlocks.
Deadlocks can have different root causes.
I see you use the pg plugin. Make sure you use it right with the transactions: pg node-postgres transactions
I would suspect a few different root causes and their solutions:
Cause 1: Multiple tests are running against the same database instance
It may be different ci pipelines executing the same test against the same Postgres instance
Solution:
This is the least probable situation, but the CI pipeline should provision its own separate Postgres instance on each run.
Cause 2: Transactions are not handled with appropriate catch("ROLLBACK")
This means that some transactions may stay alive and block others.
Solution: All transactions should have appropriate error handling.
const client = await pool.connect()
try {
await client.query('BEGIN')
//do what you have to do
await client.query('COMMIT')
} catch (e) {
await client.query('ROLLBACK')
throw e
} finally {
client.release()
}
Cause 3: Concurrency. For example: Tests are running in parallel, and they cause deadlocks.
We are writing scalable apps. This means that the deadlocks are inevitable. We have to be prepared for them and handle those appropriately.
Solution: Use the strategy "Let's try again". When we detect in our code that there is a deadlock exception, we just retry finite times. This approach has been proven with all my production apps for more than a decade.
Solution with helper func :
//Sample deadlock wrapper
const handleDeadLocks = async (action, currentAttepmt = 1 , maxAttepmts = 3) {
try {
return await action();
} catch (e) {
//detect it is a deadlock. Not 100% sure whether this is deterministic enough
const isDeadlock = e.stack?.includes("deadlock detected");
const nextAttempt = currentAttepmt + 1;
if (isDeadlock && nextAttempt <= maxAttepmts) {
//try again
return await handleDeadLocks(action, nextAttempt, maxAttepmts);
} else {
throw e;
}
}
}
//our db access functions
const updateUserProfile = async (input) => {
return handleDeadLocks(async () => {
//do our db calls
});
};
If the code becomes to complex/ nested. We can try to do it with another solution using High order function
const handleDeadLocksHOF = (funcRef, maxAttepmts = 3) {
return async (...args) {
const currentAttepmt = 1;
while (currentAttepmt <= maxAttepmts) {
try {
await funcRef(...args);
} catch (e) {
const isDeadlock = e.stack?.includes("deadlock detected");
if (isDeadlock && currentAttepmt + 1 < maxAttepmts) {
//try again
currentAttepmt += 1;
} else {
throw e;
}
}
}
}
}
// instead of exporting the updateUserProfile we should export the decorated func, we can control how many retries we want or keep the default
// old code:
export const updateUserProfile = (input) => {
//out legacy already implemented data access code
}
// new code
const updateUserProfileLegacy = (input) => {
//out legacy already implemented data access code
}
export const updateUserProfile = handleDeadLocksHOF(updateUserProfile)