Firebase Admin SDK Auth error "TOO_MANY_ATTEMPTS_TRY_LATER" - firebase

Question

I'm using firebase admin sdk in my cloud functions and I'm getting error randomly in some executions when trying to get a user by uid .
let userRecord = await admin.auth().getUser(userId);
The error details are:
{"error":{"code":400,"message":"TOO_MANY_ATTEMPTS_TRY_LATER",
"errors":[{ "message":"TOO_MANY_ATTEMPTS_TRY_LATER",
"domain":"global","reason":"invalid"}]
}
}
My cloud function executes on a real time database write and can be triggered for multiple users. In total I have 4 auth function calls in one execution first is the above one, second call is to again get user by uid or email, third call is generateEmailVerificationLink and the last call is generatePasswordResetLink.
I have checked the rate limits in documentation for auth but there is no mention of rate limit for these operation. Also the error TOO_MANY_ATTEMPTS_TRY_LATER was only mentioned in REST API for sign up with email password.
If this error is due to rate limit what should I change to prevent this error given these 4 calls are necessary for the operation needed on database write?.
EDIT:
I have identified the actual call which is throwing too many attempts error. The calls auth().generateEmailVerificationLink() and auth().generatePasswordResetLink() throw this error when called too many times.
I called these two in loop with 100 iterations and waited for the promises. The first executions finishes without any errors i.e. 200 requests. But starting second execution as soon as the first one ends will throw the error of too many attempts. So I think these two calls have limit. Now I'm trying to reduce these calls and reuse the link information. Other calls like getUserByEmail works fine.
let promises = [];
let auth = admin.auth();
let hrstart = process.hrtime()
for (let i = 0; i < 100; i++) {
promises.push(auth.getUserByEmail("user email"));
promises.push(auth.generateEmailVerificationLink("user email", {url: `https://app.firebaseapp.com/path`}));
promises.push(auth.generatePasswordResetLink("user email", {url: `https://app.firebaseapp.com/path`}));
}
Promise.all(promises)
.then(value => {
let hrend = process.hrtime(hrstart);
console.log(hrend);
// console.log(value)
});

Answer 1

The error was specifically in the operation auth.createEmailLink. This function has following limit: 20QPS/I.P address where QPS is (query per second). This limit can be increased by submitting the use case to Firebase.
I got this information from firebase support after submitting my issue.
Link to my github issue: https://github.com/firebase/firebase-admin-node/issues/458

Answer 2

I was way under 20QPS but was receiving this exception. In fact, it would always throw the TOO_MANY_ATTEMPTS_TRY_LATER exception on the 2nd attempt.
It turned out to be usage of FirebaseAuth.DefaultInstance instead of instantiating a static instance thusly:
In class definition:
private readonly FirebaseApp _firebase;
In class constructor:
_firebase = FirebaseAdmin.FirebaseApp.Create();
In function:
var auth = FirebaseAuth.GetAuth(_firebase);
var actionCodeSettings = new ActionCodeSettings()
{
...
};
var link = await auth.GenerateEmailVerificationLinkAsync(email, actionCodeSettings);
return link;

Answer 3

In addition to the answer mentioned in https://stackoverflow.com/a/54782967/5515861, I want to add another solution if you found this issue while trying to create custom email verification.
Inspired by the response in this GitHub isssue https://github.com/firebase/firebase-admin-node/issues/458#issuecomment-933161448 .
I am also seeing this issue. I have not ran admin.auth().generateEmailVerificationLink in over 24hrs (from anywhere else or any user at all) and called it just now only one time (while deployed in the prod functions environment) and got this 400 TOO_MANY_ATTEMPTS_TRY_LATER error ...
But, the client did also call the Firebase.auth.currentUser.sendEmailVerification() method around same time (obviously different IP).
Could that be the issue?
My solution to this issue is by adding a retry. e.g.
exports.sendWelcomeEmail = functions.runWith({failurePolicy: true}).auth.user().onCreate(async (user) => {
functions.logger.log("Running email...");
const email = user.email;
const displayName = user.displayName;
const link = await auth.generateEmailVerificationLink(email, {
url: 'https://mpj.io',
});
await sendWelcomeEmail(email, displayName, link);
});
The .runWith({failurePolicy: true}) is key.
It s giving you an error because your cloud functions/backend call the generateEmailVerificationLink while at the same time the default behaviour of the Firebase is also doing the same and it is counted as 20QPS. It some weird Google Rate Limit accounting rule. So my solution is just to add a retry.
The Downside is, it is calling twice, so if the call is billable, it might be billable twice.