I use DynamoDB with nodeJS on a Lambda function using serverless.
When I scan item from my local computer it works but when I deploy my function scan does not respond. No errors
const docClient = new AWS.DynamoDB.DocumentClient({
apiVersion: "2012-08-10",
});
const checkApiKey = async (apiKey, ) => {
try {
log.debug("before scan");
let result = await docClient
.scan({
"MY_TABLE",
FilterExpression: "#apiKey = :apiKey",
ExpressionAttributeNames: {
"#apiKey": "apiKey",
},
ExpressionAttributeValues: { ":apiKey": apiKey },
})
.promise();
log.debug("after scan");
} catch (error) {
log.error("Can not get dynamo object", { message: error.message });
throwError(error);
}
};
When I call this function on AWS, I can see in my log before scan but I don't see after scan nor error message from catch.
DynamoDB operations like "create" works fine.
I have been looking for a solution for several days ... Without success
I'm not sure if it's causing your problem, but the first thing that stuck out to me is how you are defining the table name in the call to scan:
.scan({
"MY_TABLE",
...
According to the docs, that should be a key/value pair
.scan({
TableName: "MY_TABLE",
...
If you are using the Serverless Framework, do you get different results if you run the function local vs remote?
For example, running the function locally from the command line:
sls invoke local --function <FUNCATION NAME from serverless.yml>
vs running the function remotely (in AWS) from the command line
sls invoke --function <FUNCATION NAME from serverless.yml>
Please try to add the callbacks for the promise of await docClient.scan().promise()
I mean:
let result = await docClient
.scan({
TableName: "MY_TABLE",
FilterExpression: "#apiKey = :apiKey",
ExpressionAttributeNames: {
"#apiKey": "apiKey",
},
ExpressionAttributeValues: { ":apiKey": apiKey },
})
.promise()
.then(data => console.log)
.catch(error => console.error)
and check the result, maybe then the picture became clear.
Related
I'm fighting with this issue for over 3 days. I have no idea what is happening.
Firebase Functions throws error only when I try to use Emulator. I try to execute this function in useEffect hook. Again, when I call deployed Cloud Functions everything seems fine, unfortunately in case of using Emulator things don't want to go so well.
const resolvePromise = async () => {
functions.useEmulator("https://0.0.0.0:5001");
const query = functions.httpsCallable("helloWorld");
query()
.then((result) => console.log(result))
.catch((err) => console.log(err));
};
I receive this useless (for me) error.
Error: ENOENT: no such file or directory, open 'HttpsErrorImpl#http://192.168.0.104:19000/index.bundle?platform=android&dev=true&hot=false&minify=false'
at Object.openSync (node:fs:585:3)
at Object.readFileSync (node:fs:453:35)
at getCodeFrame (Z:\repo\PTCG_Marketplace\node_modules\metro\src\Server.js:1296:18)
at Z:\repo\PTCG_Marketplace\node_modules\metro\src\Server.js:1367:24
at Generator.next (<anonymous>)
at asyncGeneratorStep (Z:\repo\PTCG_Marketplace\node_modules\metro\src\Server.js:146:24)
at _next (Z:\repo\PTCG_Marketplace\node_modules\metro\src\Server.js:168:9)
at runMicrotasks (<anonymous>)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
internal
at HttpsErrorImpl#http://192.168.0.104:19000/index.bundle?platform=android&dev=true&hot=false&minify=false:197178:29 in <unknown>
at http://192.168.0.104:19000/index.bundle?platform=android&dev=true&hot=false&minify=false:197273:29 in _errorForResponse
at http://192.168.0.104:19000/index.bundle?platform=android&dev=true&hot=false&minify=false:197751:39 in <unknown>
at http://192.168.0.104:19000/index.bundle?platform=android&dev=true&hot=false&minify=false:170357:26 in step
at http://192.168.0.104:19000/index.bundle?platform=android&dev=true&hot=false&minify=false:170287:21 in <unknown>
at http://192.168.0.104:19000/index.bundle?platform=android&dev=true&hot=false&minify=false:170241:31 in fulfilled
at http://192.168.0.104:19000/index.bundle?platform=android&dev=true&hot=false&minify=false:31526:15 in tryCallOne
at http://192.168.0.104:19000/index.bundle?platform=android&dev=true&hot=false&minify=false:31627:26 in <unknown>
at http://192.168.0.104:19000/index.bundle?platform=android&dev=true&hot=false&minify=false:31955:16 in _callTimer
at http://192.168.0.104:19000/index.bundle?platform=android&dev=true&hot=false&minify=false:31994:16 in _callImmediatesPass
at http://192.168.0.104:19000/index.bundle?platform=android&dev=true&hot=false&minify=false:32211:32 in callImmediates
at http://192.168.0.104:19000/index.bundle?platform=android&dev=true&hot=false&minify=false:3457:34 in __callImmediates
at http://192.168.0.104:19000/index.bundle?platform=android&dev=true&hot=false&minify=false:3236:33 in <unknown>
at http://192.168.0.104:19000/index.bundle?platform=android&dev=true&hot=false&minify=false:3440:14 in __guard
at http://192.168.0.104:19000/index.bundle?platform=android&dev=true&hot=false&minify=false:3235:20 in flushedQueue
This is all the text which I can see after visiting
http://192.168.0.104:19000/index.bundle -- https://pastebin.com/ggsCMN0W
http://192.168.0.104:19000/index.bundle?platform=android&dev=true&hot=false&minify=false -- https://pastebin.com/LSeufs8H
It don't make any sense for me. At second address it seems like it's correlated to metro dependency so I updated it, it didn't work.
Any ideas? Thanks in advance :D
Edit 1: All errors are logged at client site, it seems like client can't even call emulator.
Edit 2:
I tried to update entire firebase to v.9 aswell as Expo to 44 SKD with react-native to 0.64.3
This is how my Request function looks now:
const requestApi = () => {
const functions = getFunctions(app);
connectFunctionsEmulator(functions, "127.0.0.1", 5001);
const helloWorld = httpsCallable(functions, "helloWorld");
helloWorld()
.then((result) => {
console.log(result);
})
.catch((error) => {
console.log(error.message, error.code, error.details);
});
};
I receive only this from catch block :(
internal functions/internal undefined
I also receive warning about Timer after executing that function
Setting a timer for a long period of time, i.e. multiple minutes, is a
performance and correctness issue on Android as it keeps the timer
module awake, and timers can only be called when the app is in the
foreground. See https://github.com/facebook/react-native/issues/12981
for more info. (Saw setTimeout with duration 70000ms)
It don't work only when I try to use Emulator
Couple of potential issues here:
Assuming you're using the latest version of Firebase, functions is actually a method: firebase.functions().useEmulator("localhost", 5001); - Note the () after functions. See docs for more info.
Maybe you've already done this, but have you made sure that the functions emulator is actually running and connectable on port 5001)? Could be useful to test it via Postman or similar.
Make sure you're using the correct IP address for the functions emulator given your setup. 0.0.0.0 probably doesn't map where you want it to... assuming the app is running locally and the functions emulator is too, try 127.0.0.1 or "localhost" ... this answer has more options to troubleshoot.
I am not sure if it is your case, but I had a function:
exports.findUserInAuth = functions.https.onCall((data, context) => {
let field = data.field;
let value = data.value;
if (!field || !value) {
return false;
}
if (field === "email") {
return admin.auth().getUserByEmail(value);
}
});
This one returns a promise, I had to change it to wait for the result before doing a return and problem solved...
exports.findUserInAuth = functions.https.onCall((data, context) => {
let field = data.field;
let value = data.value;
if (!field || !value) {
return false;
}
if (field === "email") {
admin.auth().getUserByEmail(value).then((result) => {
return result;
})
.catch((error) => {
if (error.code === "auth/user-not-found") {
return "Email or Password is incorrect";
}
return `${error.code} ${error.message}`;
});
}
return false;
});
Ok, so after almost a week of fighting with this sh!t.
When you use Expo Go like me. You should copy the host address on which you are emulating your app, and use the same address you emulate your functions (or other tools).
app.json
{
"firestore": {
"rules": "firestore.rules",
"indexes": "firestore.indexes.json"
},
"emulators": {
"functions": {
"host": "192.168.0.104",
"port": 5001
}
}
}
and final code of requestApi function
const requestApi = async () => {
const functions = firebase.functions()
functions.useEmulator("192.168.0.104", 5001); <--- ADDRESS!!!
const helloWorld = functions.httpsCallable("helloWorld");
helloWorld()
.then((result) => {
console.log(result);
})
.catch((error) => {
console.log(error.message, error.code, error.details);
});
};
I'm adding a sort field to one of my AppSync tables using GraphQL. The new schema looks like:
type MyTable
#model
#auth(rules: [{allow: owner}])
#key(name: "BySortOrder", fields: ["sortOrder"], queryField: "tableBySortOrder")
{
id: ID!
name: String!
sortOrder: Int
}
However, when retrieving a list using tableBySortOrder I get an empty list because the new field sortOrder is null.
My question is, how do I backfill this data in the DynamoDB table so that my existing users will not be disrupted by this new change? With a traditional database, I would run a SQL update: UPDATE MyTable SET sortOrder = #.
However, I'm new to NoSQL/AWS and couldn't find a way to do this except build a backfill script whenever a user logs into my app. That feels very hacky. What is the best practice for handling this type of scenario?
Have you already created the new field in DDB?
If yes, I think you should backfill it before making the client side change.
Write a script to iterate through and update the table. Options for this:
Java - Call updateItem to update the table if you have any integ tests running.
Bash - Use AWS CLI: aws dynamodb scan --table-name item_attributes --projection-expression "whatever" > /tmp/item_attributes_table.txt and then aws dynamodb update-item --table-name item_attributes --key. This is a dirty way.
Python - Same logic as above.
Ended up using something similar to what Sunny suggested with a nodejs script:
const AWS = require('aws-sdk')
AWS.config.update({
region: 'us-east-1'
})
// To confirm credentials are set
AWS.config.getCredentials(function (err) {
if (err) console.log(err.stack)
// credentials not loaded
else {
console.log('Access key:', AWS.config.credentials.accessKeyId)
console.log('Secret access key:', AWS.config.credentials.secretAccessKey)
}
})
const docClient = new AWS.DynamoDB.DocumentClient()
const table = 'your-table-dev'
const params = {
TableName: table
}
const itemMap = new Map()
// Using scan to retrieve all rows
docClient.scan(params, function (err, data) {
if (err) {
console.error('Unable to query. Error:', JSON.stringify(err, null, 2))
} else {
console.log('Query succeeded.')
data.Items.forEach(item => {
if (itemMap.has(item.owner)) {
itemMap.set(item.owner, [...itemMap.get(item.owner), item])
} else {
itemMap.set(item.owner, [item])
}
})
itemMap.forEach(ownerConnections => {
ownerConnections.forEach((connection, index) => {
connection.sortOrder = index
update(connection)
})
})
}
})
function update(connection) {
const params = {
TableName: table,
Key: {
'id': connection.id
},
UpdateExpression: 'set sortOrder = :s',
ExpressionAttributeValues: {
':s': connection.sortOrder,
},
ReturnValues: 'UPDATED_NEW'
};
console.log('Updating the item...');
docClient.update(params, function (err, data) {
if (err) {
console.error('Unable to update item. Error JSON:', JSON.stringify(err, null, 2));
} else {
console.log('UpdateItem succeeded:', JSON.stringify(data, null, 2));
}
});
}
My DynamoDB table alexas has this item with key "abc" as seen in the DynamoDB console below:
However, the following query returns no result:
const params = { TableName: "alexas",
KeyConditionExpression: "deviceId = :deviceId",
ExpressionAttributeValues: { ":deviceId": "abc"}
}
const docClient = new AWS.DynamoDB.DocumentClient();
docClient.query(params, (err, data) => {
if (err) {
console.error(err);
} else {
console.log(data);
}
});
The above code returns null for err and in data:
{ Items: [], Count: 0, ScannedCount: 0 }
I am new to the DynamoDB style of expressions. Is there anything wrong with my code which I took from here.
If instead of query, I used the scan method and just have TableName in params, I get the items in my table. This confirms that I am performing the operations on the correct table that has data.
The query returned no data because the key value does not match.
The item's deviceId is the string "abc" and not abc. Note the extra quotation marks.
The item was inserted using the DynamoDB console's Create editor and there is no need to include "" if the value is already expected to be of type string.
DynamoDB's Scan operation doesn't take a KeyConditionExpression - only the Query operation takes this parameter. Scan always scans the entire table, and has a FilterExpression to post-filter these results (however please note that you still pay for scanning the entire table).
For example, here is the official documentation of Scan: https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Scan.html
Check QueryAPI
const params = { TableName: "alexas",
KeyConditionExpression: "deviceId = :deviceId",
ExpressionAttributeValues: {
":devideId":{
S: "abc", // here
}
}
}
const docClient = new AWS.DynamoDB.DocumentClient();
docClient.query(params, (err, data) => {
if (err) {
console.error(err);
} else {
console.log(data);
}
});
ExpressionAttributeValues needs to be passed in a different manner.
Update:
Try using Exp attribute names, (I'm not sure if this will make a difference)
var params = {
TableName: "alexas",
KeyConditionExpression: "#d = :dId",
ExpressionAttributeNames:{
"#d": "email"
},
ExpressionAttributeValues: {
":dId": "abc"
}
};
This is my first question ever here:-)
I need to list users in our Cognito use pool. Seems this can be done only using the aws-sdk CognitoIdentityServiceProvider APIs. I got the below code to work perfectly from my local node.js. It lists all users as expected.
However, the same code behaves differently when put into an AWS lambda function. It still runs, but it never waits for the Cognito listUsers() call to return. It just simply completes, like the "await" is not waiting at all. None of the console.log() is invoked from the client.listUsers(params, function(err, data)..._ block.
I tested this inside Lambda directly as well as from AWS API gateway. The same null is return. The call itself is successful, just no data is returned.
See log at the end.
BTW, I did create a role and add a policy giving the role full access to the cognito user pool.
What did I miss? Appreciate your help!
Martin C.
-----------------code------------------------
async function getUserList() {
console.log("enter LAMDA function**********");
var aws = require('aws-sdk');
aws.config.update({accessKeyId: 'xxxxxxxx', secretAccessKey: 'xxxxxxxxxxx'});
var CognitoIdentityServiceProvider = aws.CognitoIdentityServiceProvider;
var client = new CognitoIdentityServiceProvider({ apiVersion: '2016-04-19', region: 'us-east-2' });
var params = {
UserPoolId: 'us-east-xxxxxxxx', /* required */
AttributesToGet: [
'given_name','family_name','phone_number','email','profile'
],
Filter: null,
Limit: 0,
PaginationToken: null
};
console.log("Right before call the listUser method");
let result = await client.listUsers(params, function(err, data) {
console.log("call back reached!");
if (err) {
console.log(err, err.stack); // an error occurred
const response = {
statusCode: 500,
body: JSON.stringify('An error occurred.'),
}
return response;
}
else {
console.log(data);
var count = data.Users.length;
// successful response
const response = {
statusCode: 200,
body: JSON.stringify("sucessful list users! User count="+count)
}
return response;
}
});
console.log("no waiting here. async!!!")
}
getUserList();
***************Lambda log*****************
**************Log when called from node.js****************
getUserList is your lambda function? I don't know why you call it by your self getUserList().
I see, you are using lambda runtime is nodejs version > 8, you use await keyword with a callback function(fail) => you not wait anything.
When a function call by Lambda, the function (async function) will finish when get a return or run to end of function (without return), in your case the function finish when console.log("no waiting here. async!!!") has been executed. In local environment, the funciton finishs when callstack has been clear (do not have any callback function in callstack).
Right way, you have use promise version of aws-sdk then use await syntax to get a result. Relate to How to use Async and Await with AWS SDK Javascript
async function getUserList() {
console.log("enter LAMDA function**********");
var aws = require('aws-sdk');
aws.config.update({ accessKeyId: 'xxxxxxxx', secretAccessKey: 'xxxxxxxxxxx' });
var CognitoIdentityServiceProvider = aws.CognitoIdentityServiceProvider;
var client = new CognitoIdentityServiceProvider({ apiVersion: '2016-04-19', region: 'us-east-2' });
var params = {
UserPoolId: 'us-east-xxxxxxxx', /* required */
AttributesToGet: [
'given_name', 'family_name', 'phone_number', 'email', 'profile'
],
Filter: null,
Limit: 0,
PaginationToken: null
};
console.log("Right before call the listUser method");
try {
let result = await client.listUsers(params).promise(); // use Promise style
console.log(data);
var count = data.Users.length;
// successful response
const response = {
statusCode: 200,
body: JSON.stringify("sucessful list users! User count=" + count)
}
return response; // return to finish function
} catch (err) {
console.log(err, err.stack); // an error occurred
const response = {
statusCode: 500,
body: JSON.stringify('An error occurred.'),
}
return response;
}
}
getUserList(); // remove this line when deploy funtion to Lambda.
My objects in Dynamodb look roughly like this:
{
userId: "GEFOeE8EsaWmq4NQ3oh7tbeVkLx1",
url: 'objectURL',
object: {}
}
I have this simple piece of code for deleting an object, when the user that owns the object requests a delete. The user argument here is a parsed JWT, by the way.
export async function deleteObject(user, url) {
let params = {
TableName: OBJECTS_TABLE,
Key: {
url: url,
},
ConditionExpression: `userId = :uid`,
ExpressionAttributeValues: {
":uid": {
S: user.sub
}
}
};
let deleteResult = await dynamoDb.delete(params).promise();
return deleteResult;
}
The problem is that it doesn't work, and I've made sure that the problem stems from the ConditionExpression by changing = to <>. I simply get this:
ConditionalCheckFailedException: The conditional request failed
I'm sure solving the problem wouldn't be difficult, but I barely have any information
Questions:
Why is the condition expression failing? Everything looks alright, and it should work. Right?
How could I debug this issue better?
The await/async is not supported by AWS SDK at the moment. Please refer this similar issue.
The SDK currently relies on CLS to trace the call context. It doesn't
work with async/await functionality right now. You can see the
discussion here.
It should work if you remove the await. Example below:-
let deleteResult = dynamodb.deleteItem(params).promise();
deleteResult.then(function (data) {
console.error("Delete item result :", JSON.stringify(data,
null, 2));
}).catch(function (err) {
console.error("Delete item result error :", JSON.stringify(err,
null, 2));
});
I figured it out. ExpressionAttributeValues can be directed used, without mentioning the datatype. The Javascript SDK does that automatically.
export async function deleteObject(user, url) {
let params = {
TableName: OBJECTS_TABLE,
Key: {
url: url,
},
ConditionExpression: `userId = :uid`,
ExpressionAttributeValues: {
":uid": user.sub
}
};
let deleteResult = await dynamoDb.delete(params).promise();
return deleteResult;
}