Problems structuring and querying a table properly - DynamoDB - amazon-dynamodb

I am new to dynamoDB. I am having difficulty developing a table structure. I have data that can best be thought of as a folder structure. There are folders which are nested in parent folders. Most of the time, I will be querying for all folders with a given parent folder, however, there are times when I will be querying individual folders.
If I use the parent_id (parent folder) as the partition key and the id of the individual folder as the sort key, I believe that this creates a table where all related files are stored together and I can query them efficiently. However, I have questions.
First, the query "works" in that it returns the data, but is it written so that it queries the data correctly and is not merely scrolling through the whole table?
router.get("/api/children_folders/:parent_id", (req, res, next) => {
let parent_id = req.params.parent_id;
let params = {
TableName: tableName,
KeyConditionExpression: "parent_id = :pid",
ExpressionAttributeValues: {
":pid": parent_id,
},
ScanIndexForward: false,
};
docClient.query(params, (err, data) => {
if (err) {
console.log(err);
return res.status(err.statusCode).send({
message: err.message,
status: err.statusCode,
});
} else {
return res.status(200).send(data);
}
});
});
Second, if I want to query for individual tags, do I need to pass in a combination of the parent folder ID and the actual ID, or is this OK?
router.get("/api/folder/:folder_id", (req, res, next) => {
let tag_id = req.params.folder_id;
let params = {
TableName: tableName,
KeyConditionExpression: "folder_id = :fid",
ExpressionAttributeValues: {
":fid": folder_id,
},
Limit: 1,
};
docClient.query(params, (err, data) => {
if (err) {
console.log(err);
return res.status(err.statusCode).send({
message: err.message,
status: err.statusCode,
});
} else {
if (!_.isEmpty(data.Items)) {
return res.status(200).send(data.Items[0]);
} else {
return res.status(404).send();
}
}
});
});
I just feel like I am missing some thing here and I want to make sure that I am grabbing the data correctly.

The PK, should be something that would divide the load equally (ideally). I don't the fully picture of your problem but assuming you can chose a good parent folder as a partition key, then you can insert every file/dir with a sort key representing its full path
For example:
PK SK
/home /username/pictures/cat.jpg
This way if you want to get a specific item you can use the get item request
var params = {
Key: {
"PK": { "S": "/home" },
"SK": { "S": "/username/pictures/cat.jpg" }
},
TableName: tableName
};
var result = await dynamodb.getItem(params).promise()
Now if you want to list all the files in "/home/username/pictures" you can use begins with query
const params = {
TableName: 'tablenName',
KeyConditionExpression: '#PK = :root_path and begins_with(#SK, :sub_path)',
ExpressionAttributeNames:{
"#user_id": "root_path",
"#user_relation": 'sub_path'
},
ExpressionAttributeValues: {
":root_path": "/home",
":sub_path": "/username/pictures"
}
}

Related

How do I access data in GraphQL if not by the id?

I'm using GraphQL with Meteor and Pup v2, and I have a problem accessing the users data via a special ID provided to every user on signup, this ID will be used in a link (mysite.com/user/specialId) so other users can view the searched users account. Problem is, I can't get the data with the special ID, I can't get any data back if I don't pass in the users _id provided by MongoDB. Below I have a bunch of the code used:
Attempt 1
I tried to use a custom on-the-go way just to be able to at least access the data to see if it works (and then implement it correctly later)
const GET_USER_DETAILS = gql`
query user($userId: String) {
user(userId: $userId) {
userId
username
_id
}
}
`;
Here I export so I can get the data:
export default compose(
graphql(GET_USER_DETAILS, {
options: ({ match }) => ({
fetchPolicy: 'no-cache',
variables: {
// existing specialId for testing purposes, to be replaced with match.params.userId
userId: "J4xZzvvhBDSEufnBn",
},
}),
}),
)(PublicProfileView);
This returns a 400 error Network error: Response not successful: Received status code 400 error and after multiple attempts, I could not fix it, so I tried a different approach...
Attempt 2
I tried to go deep into the files and change the GraphQL. Created a new query:
query userById($userId: String) {
userById(userId: $userId) {
...UserAttributes
}
}
(Mentioned fragment)
fragment UserAttributes on User {
_id
name {
...
}
username
emailAddress
oAuthProvider
roles {
...
}
settings {
...
}
userId
}
Tried to add new item in API:
type Query {
...
userById(userId: String): User
...
}
Resolver:
resolvers: {
Query: {
...
userById: (parent, args) => {
// Assuming args equals an object like { _id: '123' };
return UserQueries.userById(args);
},
},
},
query.js, attempt 1:
userById: (parent) => queryUsers.find({ userId: parent.userId }, { sort: { createdAt: 1 } }).fetch()
Attempt 2:
userById: (parent, args, context) => {
return queryUsers({
userId: parent.userId,
});
},
And finally
Attempt 3
I tried to modify the get query
const getQueryModified = (options) => {
// console.log(options.userId)
try {
return options.userId
? { 'userId': options.userId }
: { userId: options.userId };
} catch (exception) {
throw new Error(`[queryUsers.getQuery] ${exception.message}`);
}
};
Here is the original query I tried to modify:
const getQuery = (options) => {
try {
return options.search
? {
_id: { $ne: options.currentUser._id },
$or: [
{ 'profile.name.first': options.search },
{ 'profile.name.last': options.search },
{ 'emails.address': options.search },
// { 'userId': options.search },
{ 'services.facebook.first_name': options.search },
{ 'services.facebook.last_name': options.search },
{ 'services.facebook.email': options.search },
],
}
: { _id: options.currentUser._id };
} catch (exception) {
throw new Error(`[queryUsers.getQuery] ${exception.message}`);
}
};
Unfortunately this was also unsuccessful, the best I get from these when executing the below query is null...
userById(userId: "J4xZzvvhBDSEufnBn"){
username
}
All I want is to get the user data from their userId and not their _id, but I can't seem to figure out how to do it

Nodejs Sequelize recursive async/await

I'm struggling with a recursive loop and nested create/select statements. I'm receiving an object from a post request with the following structure:
11.6042
---11.6042_01
---11.6042_02
---11.6042_02
---14x10-100
------14x10-100_01
---14x10-100
------14x10-100_01
---14x10-100
------14x10-100_01
---M10-DIN929_14020
---M10-DIN929_14020
---11.6042_05
Wanted behaviour: travel through the structure recursive, add record to Part table, self join with parent part, join with PartLib table, if no match present create PartLib record and match created record. Process next part.
The problem: part 14x10-100 occurs three times in the structure. I want to create a record for part 14x10-100 in the part_lib table and refer to that record three times. What actually happens is that for each 14x10-100 part a corresponding record in the part_lib table is created in stead of one create and two matches. If I run it again it will match like excpected. I suspect I'm lost in the promise/async await parts of the code.
Below the relevant code. I've removed some attribute mappings for readability. My thoughts behind it: I'm not returning new promises like normal in a async function since Sequelize already returns a promise. When creating a part I'm awaiting (or at least I think so) the partLibController calls to ensure that all matching/creating/joining is done before proceeding to the next part in the structure.
Thanks a bunch!!
Recursive loop
function parseChild(child, modelId, parentId, userId, level) {
return new Promise((resolve, reject) => {
partController.create({
parent_id: parentId
, name: child.name
}, { id: userId }).then((part) => {
resolve({ child: child, level: level });
if (child.children) {
child.children.forEach(grandChild => {
parseChild(grandChild, modelId, part.part_id, userId, level + '---');
});
}
}).catch(error => { console.log(error); });
}).then((obj) => { console.log(`${obj.level} ${obj.child.name}`); });
}
PartController Create
async function create(partBody, currentUser) {
let { parent_id, name } = partBody;
const match = await partLibController.match(name);
let partLibId = null;
if (match.length == 0) {
const partLib = await partLibController.createFromPart(partBody, currentUser);
partLibId = partLib.part_lib_id;
} else {
partLibId = match[0].dataValues.part_lib_id
}
return ModelAssembly.create({
parent_id: parent_id
, name: name
, part_lib_id: partLibId
});
}
PartLibController Match
function match(name) {
return PartLib.findAll({
where: {
name: name
},
});
}
PartLibController CreateFromPart
function createFromPart(partBody, currentUser) {
let { name } = partBody;
return PartLib.create({
name,
});
}
Thanks to AKX I've solved the problem: hero
The problem was in the recursive call itself I suppose but here's the working code:
async function parseChild(child, modelId, parentId, userId, level) {
const body = {
parent_id: parentId
, name: child.name
};
const ma = await partController.create(body, { id: userId });
if (child.children) {
for (const grandChild of child.children) {
await parseChild(grandChild, modelId, ma.part_id, userId, level + '---');
}
}
return;
}

DynamoDb Nested Map Update

After working a bit with DynamoDb I’ve run into an issue that from what I’ve read so far is not really ideal for DynamoDb. So before I make the switch to RDS, I’d like to see if there’s anyway I can achieve what I need with DynamoDb. I’ve also thought about breaking this out into multiple tables for DynamoDb
Below of my Data schema. There is a list nested inside the item. I need to be able to append strings to the list.
{
“server-id”: “123345678”,
“server-name”: “my-server”
“topics”: [
{
“name”: “my-topic”,
“subscribers”: [] //This is what I need to append
}
]
}
Yes, this is possible.
var AWS = require('aws-sdk');
AWS.config.update({region: 'us-east-1'});
var ddb = new AWS.DynamoDB({apiVersion: '2012-08-10'});
var params = {
ExpressionAttributeNames: {
"#T": "topics",
"#S": "subscribers"
},
ExpressionAttributeValues: {
":vals": {
L: [
{ N: "123" },
{ N: "456" }
]
}
},
Key: {
'server-id': { S: '123345678' }
},
ReturnValues: "ALL_NEW",
TableName: 'dummy-table',
UpdateExpression: "SET #T[0].#S = list_append(#T[0].#S, :vals)"
};
ddb.updateItem(params, function(err, data) {
if (err) {
console.log("Error", err);
} else {
console.log("Success", data);
}
});

How to Publish joined Data from Array of IDs in Meteor

I just want to Publish the relational Data for a Publication to client, but the issue is my Relational Data field is array of ID's of a Different Collection, I tried Different Packages but all works with single Relational ID but not working with Array of relational ID's, let assume I have two Collection Companies and Meteor.users below is my Company Document Looks like
{
_id : "dYo4tqpZms9j8aG4C"
owner : "yjzakAgYWejmJcuHz"
name : "Labbaik Waters"
peoples : ["yjzakAgYWejmJcuHz", "yjzakAgYWejmJcuHz"],
createdAt: "2019-09-18T15:33:29.952+00:00"
}
here you can see peoples field contains the user ID's as Array, so How I publish this userId's as user Documents, as for example I tried the most popular meteor package named publishComposit, when I tried Loop in Children's find, I got undefined in children i.e below
publishComposite('compoundCompanies', {
find() {
// Find top ten highest scoring posts
return Companies.find({
owner: this.userId
}, {sort: {}});
},
children: [
{
find(company) {
let cursors = company.peoples.forEach(peopleId => {
console.log(peopleId)
return Meteor.users.find(
{ _id: peopleId },
{ fields: { profile: 1 } });
})
//here cursor undefined
console.log(cursors)
return cursors
}
}
]
});
and if I implement async loop in children's find I got error like below code
publishComposite('compoundCompanies', {
find() {
// Find top ten highest scoring posts
return Companies.find({
owner: this.userId
}, {sort: {}});
},
children: [
{
async find(company) {
let cursors = await company.peoples.forEach(peopleId => {
console.log(peopleId)
return Meteor.users.find(
{ _id: peopleId },
{ fields: { profile: 1 } });
})
//here cursor undefined
console.log(cursors)
return cursors
}
}
]
});
the error occured in above code is Exception in callback of async function: TypeError: this.cursor._getCollectionName is not a function
I don't know what I am exactly doing wrong here, or implementing package function not as intended any help will be greatly appropriated
EDIT: my desired result should be full user documents instead of ID no matter it mapped in same peoples array or as another fields I just want as below
{
_id: "dYo4tqpZms9j8aG4C",
owner: "yjzakAgYWejmJcuHz",
name: "Labbaik Waters",
peoples: [
{
profile: {firstName: "Abdul", lastName: "Hameed"},
_id: "yjzakAgYWejmJcuHz"
}
],
createdAt: "2019-09-18T15:33:29.952+00:00"
}
I ran into a similar problem couple of days ago. There are two problems with the provided code. First, using async; it's not needed and rather complicates things. Second, publishComposite relies on receiving one cursor not multiple within its children to work properly.
Below is a snippet of the code used to solve the problem I had, hopefully you can replicate it.
Meteor.publishComposite("table.conversations", function(table, ids, fields) {
if (!this.userId) {
return this.ready();
}
check(table, String);
check(ids, Array);
check(fields, Match.Optional(Object));
return {
find() {
return Conversation.find(
{
_id: {
$in: ids
}
},
{ fields }
);
},
children: [
{
find(conversation) {
// constructing one big cursor that entails all of the documents in one single go
// as publish composite cannot work with multiple cursors at once
return User.find(
{ _id: { $in: conversation.participants } },
{ fields: { profile: 1, roles: 1, emails: 1 } }
);
}
}
]
};
});

How to dynamically update an attribute in a dynamodb item?

I created an item in dynamodb using Node js, the item has multiple attributes such as brand, category, discount, validity, etc. I am using uuid to generate ids for each item. Now let's say I want to update the validity attribute of the item, in which case I am currently sending the entire json object with the value of validity modified to the new value.
This is definitely not optimal, please help me find an optimal solution.
const params = {
TableName: process.env.PRODUCT_TABLE,
Key: {
id: event.pathParameters.id,
},
ExpressionAttributeNames: {
'#discount': 'discount',
},
ExpressionAttributeValues: {
':brand': data.brand,
':category': data.category,
':discount': data.discount,
':denominations': data.denominations,
":validity": data.validity,
":redemption": data.redemption
},
UpdateExpression: 'SET #discount = :discount, denominations = :denominations, brand = :brand, category = :category, validity = :validity, redemption = :redemption',
ReturnValues: 'ALL_NEW',
};
I want to send just the attribute I want to update with the new value, if I want to change the validity from 6 months to 8 months, I should just send something like:
{
"validity": "8 months"
}
And it should update the validity attribute of the item.
Same should apply to any other attribute of the item.
'use strict';
const AWS = require('aws-sdk');
const dynamoDb = new AWS.DynamoDB.DocumentClient();
module.exports.update = (event, context, callback) => {
const data = JSON.parse(event.body);
let attr = {};
let nameobj = {};
let exp = 'SET #';
let arr = Object.keys(data);
let attrname = {};
arr.map((key) => {attr[`:${key}`]=data[key]});
arr.map((key) => {
exp += `${key} = :${key}, `
});
arr.map((key) => {nameobj[`#${key}`]=data[key]});
attrname = {
[Object.keys(nameobj)[0]] : nameobj[Object.keys(nameobj)[0]]
}
const params = {
TableName: process.env.PRODUCT_TABLE,
Key: {
id: event.pathParameters.id,
},
ExpressionAttributeNames: attrname,
ExpressionAttributeValues: attr,
UpdateExpression: exp,
ReturnValues: 'ALL_NEW',
};
// update the todo in the database
dynamoDb.update(params, (error, result) => {
// handle potential errors
if (error) {
console.error(error);
callback(null, {
statusCode: error.statusCode || 501,
headers: { 'Content-Type': 'text/plain' },
body: 'Couldn\'t update the card',
});
return;
}
// create a response
const response = {
statusCode: 200,
body: JSON.stringify(result.Attributes),
};
callback(null, response);
});
};
Contrary to others comments, this is very possible, use the UpdateItem action.
Language agnostic API docs
JavaScript specific API docs
If you want to dynamically create the query, try something like this:
const generateUpdateQuery = (fields) => {
let exp = {
UpdateExpression: 'set',
ExpressionAttributeNames: {},
ExpressionAttributeValues: {}
}
Object.entries(fields).forEach(([key, item]) => {
exp.UpdateExpression += ` #${key} = :${key},`;
exp.ExpressionAttributeNames[`#${key}`] = key;
exp.ExpressionAttributeValues[`:${key}`] = item
})
exp.UpdateExpression = exp.UpdateExpression.slice(0, -1);
return exp
}
let data = {
'field' : { 'subfield': 123 },
'other': '456'
}
let expression = generateUpdateQuery(data)
let params = {
// Key, Table, etc..
...expression
}
console.log(params)
Output:
{
UpdateExpression: 'set #field = :field, #other = :other',
ExpressionAttributeNames: {
'#field': 'field',
'#other': 'other'
},
ExpressionAttributeValues: {
':field': {
'subfield': 123
},
':other': '456'
}
}
Using Javascript SDK V3:
Import from the right package:
import { DynamoDBClient PutItemCommandInput, UpdateItemCommandInput, UpdateItemCommand } from '#aws-sdk/client-dynamodb';
Function to dynamically do partial updates to the item:
(the code below is typescript can be easily converted to Javascript, just remove the types!)
function updateItem(id: string, item: any) {
const dbClient = new DynamoDBClient({region: 'your-region-here });
let exp = 'set ';
let attNames: any = { };
let attVal: any = { };
for(const attribute in item) {
const valKey = `:${attribute}`;
attNames[`#${attribute}`] = attribute;
exp += `#${attribute} = ${valKey}, `;
const val = item[attribute];
attVal[valKey] = { [getDynamoType(val)]: val };
}
exp = exp.substring(0, exp.length - 2);
const params: UpdateItemCommandInput = {
TableName: 'your-table-name-here',
Key: { id: { S: id } },
UpdateExpression: exp,
ExpressionAttributeValues: attVal,
ExpressionAttributeNames: attNames,
ReturnValues: 'ALL_NEW',
};
try {
console.debug('writing to db: ', params);
const command = new UpdateItemCommand(params);
const res = await dbClient.send(command);
console.debug('db res: ', res);
return true;
} catch (err) {
console.error('error writing to dynamoDB: ', err);
return false;
}
}
And to use it (we can do partial updates as well):
updateItem('some-unique-id', { name: 'some-attributes' });
What i did is create a helper class.
Here is a simple function : Add all the attribute and values that goes into, if the value is null or undefined it won't be in the expression.
I recommande to create a helper class with typescript and add more functions and other stuff like generator of expressionAttributeValues , expressionAttributeNames ... , Hope this help.
function updateExpression(attributes, values) {
const expression = attributes.reduce((res, attribute, index) => {
if (values[index]) {
res += ` #${attribute}=:${attribute},`;
}
return res;
}, "SET ");
return expression.slice(0, expression.length - 1)
}
console.log(
updateExpression(["id", "age", "power"], ["e8a8da9a-fab0-55ba-bae3-6392e1ebf624", 28, undefined])
);
You can use code and generate the params object based on the object you provide. It's just a JavaScript object, you walk through the items so that the update expression only contains the fields you have provided.
This is not really a DynamoDB question in that this is more a general JS coding question.
You can use UpdateItem; to familiarize yourself with DynamoDb queries I would suggest you DynamoDb NoSQL workbench:
https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/workbench.settingup.html
It can generate snippets for you based on your queries.
DynamoDb NoSQL workbench screenshot query

Resources