Writing unit tests for orchestration functions in Azure Durable Functions - moq

I'm trying to learn how to write unit tests for durable function. Here is my code:
[FunctionName("OrchestratorFunction")]
public async Task RunOrchestrator([OrchestrationTrigger] IDurableOrchestrationContext context)
{
var jobs = await context.CallActivityAsync<List<Job>>("JobsReaderFunction"), null);
if (jobs != null && jobs .Count > 0)
{
var processingTasks = new List<Task>();
foreach (var job in jobs)
{
Task processTask = context.CallSubOrchestratorAsync("SubOrchestratorFunction"), job);
processingTasks.Add(processTask);
}
await Task.WhenAll(processingTasks);
}
}
[FunctionName("SubOrchestratorFunction")]
public async Task RunSubOrchestrator([OrchestrationTrigger] IDurableOrchestrationContext context)
{
var job = context.GetInput<Job>();
var group = await context.CallActivityAsync<Group>("GroupReaderFunction"), job);
await context.CallActivityAsync("EmailSenderFunction", group);
var canWriteToGroup = await context.CallActivityAsync<bool>("GroupVerifierFunction", job);
await context.CallActivityAsync("JobStatusUpdaterFunction", new JopStatusUpdaterRequest { CanWriteToGroup = canWriteToGroup, Job = job });
await context.CallActivityAsync("TopicMessageSenderFunction", job);
}
How do I write a test that covers Orchestrator, SubOrchestrator and Activity functions? Please let me know.
Here is my test so far:
[TestMethod]
public async Task VerifyOrchestrator()
{
var jobs = <code to get jobs>
var context = new Mock<IDurableOrchestrationContext>();
context.Setup(m => m.CallActivityAsync<List<Job>>("JobsReaderFunction", It.IsAny<object>())).ReturnsAsync(jobs);
await _orchestratorFunction.RunOrchestrator(context.Object);
}
UPDATE:
I updated the test method to:
[TestMethod]
public async Task VerifyJobs()
{
var jobs = <code to get jobs>
var context = new Mock<IDurableOrchestrationContext>();
context.Setup(x => x.CallSubOrchestratorAsync("SubOrchestratorFunction"), It.IsAny<object>())).Returns(() => _orchestratorFunction.RunSubOrchestrator(context.Object));
await _orchestratorFunction.RunOrchestrator(context.Object);
}
which gives me an error:
context.Setup(x => x.CallSubOrchestratorAsync("SubOrchestratorFunction", It.IsAny<object>())).Returns(async () => await It.IsAny<Task>());
await _orchestratorFunction.RunOrchestrator(context.Object);
context.Verify(m => m.CallSubOrchestratorAsync("SubOrchestratorFunction", It.IsAny<string>(), It.IsAny<object>()),Times.Once);
The above gives a null exception.

The context is mocked acording to the line:
var context = new Mock<IDurableOrchestrationContext>();
This means that when you execute Context.CallSubOrchestratorAsync("SubOrchestratorFunction") moq search for setup for the method.
Because the method doesn't have any setup, it runs nothing and returns the default return value.
If you want to execute RunSubOrchestrator, you should setup it:
context.Setup(x => x.CallSubOrchestratorAsync("SubOrchestratorFunction", It.IsAny<object>())).Returns(() => _orchestratorFunction.RunSubOrchestrator(context.Object))

Related

Flutter Firebase async query not retrieving data inside a stream function

I am trying to query a User from firebase within another query but for some reason but I can't get the code to work
The function the wont run is await usersRef.doc(uid).get(); and can be found here:
static getUserData(String uid) async {
return await usersRef.doc(uid).get();
}
static DirectMessageListModel getDocData(QueryDocumentSnapshot qdoc, String uid) {
Userdata postUser = Userdata.fromDoc(getUserData(uid));
return DirectMessageListModel.fromDoc(qdoc, postUser);
}
static DirectMessageListModel fromDoc(QueryDocumentSnapshot doc, Userdata altUser) {
return DirectMessageListModel(
doc['chatId'],
doc['lastMsgContent'],
doc['lastMsgType'],
altUser
);
}
parent function:
Stream<List<DirectMessageListModel>> getMeassageList(){
var snaps = FirebaseFirestore.instance.collection('directMessages').where('users', arrayContains: userdata!.uid).snapshots();
List<String> usersListElement = [];
return snaps.map((event) { return event.docs.map((e) {
usersListElement = [e.get('users')[0], e.get('users')[1]];
usersListElement.remove(userdata!.uid);
return DirectMessageListModel.getDocData(e, usersListElement.first);
}).toList();
});
}
You forgot to wait for the future getUserData(uid) to complete.
Try this:
static Future<DocumentSnapshot<Object>> getUserData(String uid) async {
return await usersRef.doc(uid).get();
}
static DirectMessageListModel getDocData(
QueryDocumentSnapshot qdoc,
String uid,
) async {
Userdata postUser = Userdata.fromDoc(await getUserData(uid)); // await here
return DirectMessageListModel.fromDoc(qdoc, postUser);
}
..
// parent function.
// Also wait for the future in the parent function.
// UPDATE BELOW! Define the parent function like this:
Stream<List<Future<DirectMessageListModel>>> getMeassageList() {
var snaps = FirebaseFirestore.instance
.collection('directMessages')
.where('users', arrayContains: userdata!.uid)
.snapshots();
List<String> usersListElement = [];
return snaps.map((event) {
return event.docs.map((e) async {
usersListElement = [e.get('users')[0], e.get('users')[1]];
usersListElement.remove(userdata!.uid);
return await DirectMessageListModel.getDocData(e, usersListElement.first);
}).toList();
});
}
NB: You are fetching user data (either sender/receiver) for each message in directMessages collection. It might be better to store just sender/receiver name in directMessages collection and simply display that. Then if the user clicks on a message, you can then fetch the full sender/receiver data.

Saving to Firebase storage not possible in an isolate

I have an app feature where the user picks images from his phone and then uploads them to Firebase Storage.
I thought that the upload process should be done in a separate isolate.
I keep getting an exception which I think is related to the Multi Image Picker package.
The exception is:
E/flutter (12961): [ERROR:flutter/lib/ui/ui_dart_state.cc(177)]
Unhandled Exception: Exception: NoSuchMethodError: The getter
'defaultBinaryMessenger' was called on null.
When the user presses on the upload button, this method is called:
Future<void> _initIsolate() async {
ReceivePort receivePort = ReceivePort();
receivePort.listen(
(message) {
print(message.toString());
},
onDone: () => print('Done'),
onError: (error) => print('$error'),
);
await compute(
_function, // This function is called in the separate isolate
{
'sendingPort': receivePort.sendPort,
'images': images,
},
);
}
The _function method is as follows:
static void _function(Map<String, dynamic> parameterMap) async {
SendPort sendingPort = parameterMap['sendingPort'];
List<Asset> images = parameterMap['images'];
List<String> urls = [];
int index = 0;
images.forEach(
(image) async {
String url = await getDownloadUrl(image); // a helper method
urls.add(url);
sendingPort.send('Image number: $index uploaded');
index += 1;
},
);
final CollectionReference collectionRef = FirebaseFirestore.instance.collection('offers');
final user = CurrentUser.getCurrentUser();
await collectionRef.doc(user.uid).set(
{
'time': FieldValue.serverTimestamp(),
'urls': urls,
},
);
}
The helper method _getDownloadUrl is as follows:
Future<String> getDownloadUrl(Asset image) async {
String rannum = Uuid().v1();
final ByteData byteData = await image.getByteData(); // --> This produces a defaultBinaryMessenger
final List<int> imageData = byteData.buffer.asUint8List();
Reference reference = FirebaseStorage.instance.ref().child("offers/$rannum");
UploadTask uploadTask = reference.putData(imageData);
TaskSnapshot downloadUrl = await uploadTask.whenComplete(() => null);
Future<String> futureUrl = downloadUrl.ref.getDownloadURL();
return futureUrl;
}
The getByteData method is part of the multi_image_picker package.
The source code is:
Future<ByteData> getByteData({int quality = 100}) async {
if (quality < 0 || quality > 100) {
throw new ArgumentError.value(
quality, 'quality should be in range 0-100');
}
Completer completer = new Completer<ByteData>();
ServicesBinding.instance.defaultBinaryMessenger // --> Exception here. ServicesBinding.instance is null
.setMessageHandler(_originalChannel, (ByteData message) async {
completer.complete(message);
ServicesBinding.instance.defaultBinaryMessenger
.setMessageHandler(_originalChannel, null);
return message;
});
await MultiImagePicker.requestOriginal(_identifier, quality);
return completer.future;
}
Why is the ServicesBinding.instance null?
Since this method is working fine without using Isolates, does this have something to do with the isolates?

Firebase: Flutter: Optimizing for stability: many Transactions vs. Batch writes vs. Set calls

A few months ago, a glorious soul here taught me about transactions. I may have gone a little overboard thinking they were they best thing since sliced bread. The problem they solved was obvious, guaranteed concurrent writes on a single doc. However, I've noticed already with as little as three closely timed function triggers that I produced the dreaded: ------------------------"10 ABORTED: Too much contention on these documents."...-------------------------
Optimizing for stability, my question is: Would it be best practice to use a mixed bag of these write calls for different situations? For example: if a cloud function is writing to a location where I do not expect contention, should it just be a set call? Instead of 4 transactions to various locations, should I use a batch?
Reading the Firebase limitations I assumed I was in the clear with max 60w/doc/sec. However, I've learned now that Transactions can timeout AND only try to write 5 times.
Some background on the app and the contention error:
- It's a basic social media app.
- The contention error came from making three posts in close succession from a single user.
- Each post triggers a cloud function that does several transactions to link the post to appropriate places. i.e. followers, feed, groups, sends notifications, and sets activity feed docs for each follower.
Side question: Am I wrongly understanding that firebase can handle an app with this level of activity?
EDIT: I was aware of these firebase limitations early on and did my best work to keep documents and collections spread apart appropriately.
CODE EDIT: index.js: adminPostReview is the specific function to throw the error (did the best I could to simplify).
The specific transaction to throw the error, I believe, is the call to transactionDayIndexAdd().
function transactionDelete(docRef) {
return db.runTransaction(async t => {
var doc = await t.get(docRef);
if (doc.exists)
t.delete(docRef);
})
}
// THIS FUNCTION. Is it bad to read and set two documents?
function transactionDayIndexAdd(docRef, dayPosted, postId, userId) {
return db.runTransaction(async (t) => {
var postMap = {};
const doc = await t.get(docRef.doc(dayPosted));
if (doc.exists) {
postMap = doc.data().pids;
} else {
const indexDoc = await t.get(docRef.doc('index'));
var newIndex = indexDoc.exists ? indexDoc.data().index : {};
newIndex[dayPosted] = true;
t.set(docRef.doc('index'), { 'index': newIndex });
}
postMap[postId] = userId;
t.set(doc.ref, { 'pids': postMap });
})
}
exports.adminPostReview = functions.firestore
.document('/adminPostReview/{postId}')
.onUpdate(async (change, context) => {
const postId = context.params.postId;
const userId = change.before.data().ownerId;
const approvedMaks = change.after.data().approvedMaks;
const approvedRita = change.after.data().approvedRita;
var promises = [];
if (approvedMaks == false || approvedRita == false) {
promises.push(transactionDelete(db.collection('posts').doc(userId).collection('userPosts').doc(postId)));
}
else if (approvedMaks == true || approvedRita == true) {
var newPost = change.after.data();
promises.push(postLive(newPost));
}
if (approvedMaks != null || approvedRita != null) {
promises.push(transactionDelete(db.collection('activityFeed').doc(MAKS_ID).collection('feedItems').doc(`${postId}_review`)));
promises.push(transactionDelete(db.collection('activityFeed').doc(RITA_ID).collection('feedItems').doc(`${postId}_review`)));
}
});
async function postLive(newPost) {
const userId = newPost.ownerId;
const postId = newPost.postId;
const dayPosted = newPost.dayPosted;
var postToFeed = newPost.postToFeed;
var postToGroups = newPost.postToGroups;
newPost.approved = true;
delete newPost.postToFeed;
delete newPost.postToGroups;
var batch = db.batch();
var promises = [];
if (postToFeed == true) {
batch.set(
db.collection('posts').doc(userId).collection('userPosts').doc(postId),
newPost
);
batch.update(
db.collection('userActivity').doc(userId),
'numPosts',
admin.firestore.FieldValue.increment(1),
)
promises.push(batch.commit());
promises.push(transactionDayIndexAdd(db.collection("feedRandom"), dayPosted, postId, userId));
var querySnap = await db.collection('followers')
.doc(userId)
.collection('userFollowers')
.get();
querySnap.docs.forEach(async follower => {
promises.push(transactionDayIndexAdd(
db.collection('feedFollowing').doc(follower.id).collection('feedItems'),
dayPosted, postId, userId));
promises.push(transactionSet(db.collection('activityFeed').doc(follower.id)
.collection('feedItems').doc(postId),
{
media1Url: newPost.media1Url,
media2Url: newPost.media2Url,
postId: newPost.postId,
timestamp: newPost.timestamp,
type: 'newFollowingPost',
userId: userId,
userProfileImg: newPost.ownerProfileImg,
username: newPost.username,
displayName: newPost.displayName,
}
));
if (follower.data().notificationToken != null) {
const payload = {
notification: {
title: 'Someone you follow made a new post!',
body: `${newPost.username} has a new post.`
},
data: {
click_action: "FLUTTER_NOTIFICATION_CLICK",
vestiq_type: 'newFollowingPost',
vestiq_uid: follower.id,
vestiq_fid: userId,
vestiq_pid: postId,
vestiq_displayName: newPost.displayName,
vestiq_photoUrl: newPost.ownerProfileImg,
vestiq_username: newPost.username,
}
};
var user = await db.collection('users').doc(follower.id).get();
if (user.data().notificationOp3 == true)
promises.push(pushNotification(follower.data().notificationToken, payload));
}
});
if (postToGroups != null && postToGroups.length > 0) {
promises.push(pushGroupPosts(postToGroups, userId, postId, newPost));
return Promise.all(promises);
} else return Promise.all(promises);
}
else if (postToGroups != null && postToGroups.length > 0) {
promises.push(pushGroupPosts(postToGroups, userId, postId, newPost));
return Promise.all(promises);
}
}
async function pushGroupPosts(postToGroups, userId, postId, newPost) {
var groupBatch = db.batch();
postToGroups.forEach((gid) => {
groupBatch.set(
db.collection('groups').doc(gid).collection('posts').doc(postId),
newPost,
);
groupBatch.set(
db.collection('usersGroupPosts').doc(userId).collection(gid).doc(postId),
{ 'gid': gid, 'postId': postId },
);
});
return push(groupBatch.commit());
}
I was able to fix the contention problem by splitting transactionDayIndexAdd() into two separate transactions. I flip a bool to determine if the second should run instead.
This leads me to believe that the nested t.get/t.set transaction significantly increases chances for contention issues. Since the split, I have not been able to reproduce the error. Here is the new transactionDayIndexAdd() for those who are curious.
HOWEVER, my original question still stands regarding optimising for stability.
async function transactionDayIndexAdd(docRef, dayPosted, postId, userId) {
var dayAdd = 0;
var promises = [];
await db.runTransaction(async (t) => {
var postMap = {};
const doc = await t.get(docRef.doc(dayPosted));
if (doc.exists)
postMap = doc.data().pids;
else {
dayAdd = 1;
}
postMap[postId] = userId;
t.set(doc.ref, { 'pids': postMap });
});
if (dayAdd == 1) {
promises.push(db.runTransaction(async (t) => {
const indexDoc = await t.get(docRef.doc('index'));
var newIndex = indexDoc.exists ? indexDoc.data().index : {};
newIndex[dayPosted] = true;
t.set(indexDoc.ref, { 'index': newIndex });
}));
}
return await Promise.all(promises);
}

AWS Lambda: Async Calls outside handler (initialization section, invoke lambda)

I would like to call an asynchronous function outside the lambda handler with by the following code:
var client;
(async () => {
var result = await initSecrets("MyWebApi");
var secret = JSON.parse(result.Payload);
client= new MyWebApiClient(secret.API_KEY, secret.API_SECRET);
});
async function initSecrets(secretName) {
var input = {
"secretName" : secretName
};
var result = await lambda.invoke({
FunctionName: 'getSecrets',
InvocationType: "RequestResponse",
Payload: JSON.stringify(input)
}).promise();
return result;
}
exports.handler = async function (event, context) {
var myReq = await client('Request');
console.log(myReq);
};
The 'client' does not get initialized. The same code works perfectly if executed within the handler.
initSecrets contains a lambda invocation of getSecrets() which calls the AWS SecretsManager
Has anyone an idea how asynchronous functions can be properly called for initialization purpose outside the handler?
Thank you very much for your support.
I ran into a similar issue trying to get next-js to work with aws-serverless-express.
I fixed it by doing the below (using typescript so just ignore the :any type bits)
const appModule = require('./App');
let server: any = undefined;
appModule.then((expressApp: any) => {
server = createServer(expressApp, null, binaryMimeTypes);
});
function waitForServer(event: any, context: any){
setImmediate(() => {
if(!server){
waitForServer(event, context);
}else{
proxy(server, event, context);
}
});
}
exports.handler = (event: any, context: any) => {
if(server){
proxy(server, event, context);
}else{
waitForServer(event, context);
}
}
So for your code maybe something like
var client = undefined;
initSecrets("MyWebApi").then(result => {
var secret = JSON.parse(result.Payload);
client= new MyWebApiClient(secret.API_KEY, secret.API_SECRET)
})
function waitForClient(){
setImmediate(() => {
if(!client ){
waitForClient();
}else{
client('Request')
}
});
}
exports.handler = async function (event, context) {
if(client){
client('Request')
}else{
waitForClient(event, context);
}
};
client is being called before it has initialised; the client var is being "exported" (and called) before the async function would have completed. When you are calling await client() the client would still be undefined.
edit, try something like this
var client = async which => {
var result = await initSecrets("MyWebApi");
var secret = JSON.parse(result.Payload);
let api = new MyWebApiClient(secret.API_KEY, secret.API_SECRET);
return api(which) // assuming api class is returning a promise
}
async function initSecrets(secretName) {
var input = {
"secretName" : secretName
};
var result = await lambda.invoke({
FunctionName: 'getSecrets',
InvocationType: "RequestResponse",
Payload: JSON.stringify(input)
}).promise();
return result;
}
exports.handler = async function (event, context) {
var myReq = await client('Request');
console.log(myReq);
};
This can be also be solved with async/await give Node v8+
You can load your configuration in a module like so...
const fetch = require('node-fetch');
module.exports = async () => {
const config = await fetch('https://cdn.jsdelivr.net/gh/GEOLYTIX/public/z2.json');
return await config.json();
}
Then declare a _config outside the handler by require / executing the config module. Your handler must be an async function. _config will be a promise at first which you must await to resolve into the configuration object.
const _config = require('./config')();
module.exports = async (req, res) => {
const config = await _config;
res.send(config);
}
Ideally you want your initialization code to run during the initialization phase and not the invocation phase of the lambda to minimize cold start times. Synchronous code at module level runs at initialization time and AWS recently added top level await support in node14 and newer lambdas: https://aws.amazon.com/blogs/compute/using-node-js-es-modules-and-top-level-await-in-aws-lambda/ . Using this you can make the init phase wait for your async initialization code by using top level await like so:
const sleep = ms => new Promise(resolve => setTimeout(resolve, ms))
console.log("start init");
await sleep(1000);
console.log("end init");
export const handler = async (event) => {
return {
statusCode: 200,
body: JSON.stringify('Hello from Lambda!'),
};
};
This works great if you are using ES modules. If for some reason you are stuck using commonjs (e.g. because your tooling like jest or ts-node doesn't yet fully support ES modules) then you can make your commonjs module look like an es module by making it export a Promise that waits on your initialization rather than exporting an object. Like so:
const sleep = ms => new Promise(resolve => setTimeout(resolve, ms))
const main = async () => {
console.log("start init");
await sleep(1000);
console.log("end init");
const handler = async (event) => {
return {
statusCode: 200,
body: JSON.stringify('Hello from Lambda!'),
};
};
return { handler };
};
# note we aren't exporting main here, but rather the result
# of calling main() which is a promise resolving to {handler}:
module.exports = main();

awaiting on an async function does not execute correctly in dart

I have a few async functions in my dart program which interact with SQLite database. I use await expression to invoke those functions and mostly the functions are executed when they're awaited on but one function does not execute and the calling function continues execution without awaiting on the called function. Here's the code:
Future<int> addShoppingList(String listName) async {
var dbClient = await db;
String now = new DateTime.now().toString();
await dbClient.transaction((txn) async {
int res = await txn.rawInsert("insert into lists(list_name,list_created_at) values(\'$listName\',\'$now\')");
print('result of adding a new shopping list: $res');
return res;
});
List<Map> resList = await dbClient.rawQuery("select list_id from lists where list_name=\'$listName\'");
if (resList.length > 0) {
return resList[0]['list_id'];
}
return 0;
//await dbClient.rawInsert("insert into lists(list_name,list_created_at) values(\'$listName\',\'$now\')");
}
Future<int> addShoppingListItems(int listId, Map<String,String> listItems) async {
var dbClient = await db;
int res = 0;
listItems.forEach((itemName, quantity) async{
int itemId = await getItemId(itemName);
print('adding item $itemName with id $itemId');
await dbClient.transaction((txn) async {
res = await txn.rawInsert("insert into list_items values($listId,$itemId,\'$quantity\')");
print('result of adding item in list_items: $res');
});
return res;
});
return 0;
}
Future<int> addItemsToShoppingList(String listName, Map<String,String> listItems) async {
int listId = await getListId(listName);
if (listId == 0) {
listId = await addShoppingList(listName);
print('got list id of $listId after adding new list');
}
print('in additemstoshoppinglist list id: $listId');
print('in additemstoshoppinglist ${listItems.toString()}');
int res = await addShoppingListItems(listId, listItems);
print('result after adding item in addItemsToShoppingList: $res');
return res;
}
In my external class I await on addItemsToShoppingList function to start the whole chain of functions. When I run the code, I see that the functions getListId(), addShoppingList() are awaited correctly(the execution of addItemsToShoppingList does not proceed until the awaited functions are executed) but when I await on addShoppingListItems, the calling function's execution does not wait until addShoppingListItems returns. What am I missing here and how can I make sure addShoppingListItems returns before the execution of calling function can proceed?
I figured out that the problem was with the way async functions are awaited upon in the for each loop of the map being used. I solved this problem with the solution given here: Map forEach function exits with an await inside

Resources