I'm struggling with a recursive loop and nested create/select statements. I'm receiving an object from a post request with the following structure:
11.6042
---11.6042_01
---11.6042_02
---11.6042_02
---14x10-100
------14x10-100_01
---14x10-100
------14x10-100_01
---14x10-100
------14x10-100_01
---M10-DIN929_14020
---M10-DIN929_14020
---11.6042_05
Wanted behaviour: travel through the structure recursive, add record to Part table, self join with parent part, join with PartLib table, if no match present create PartLib record and match created record. Process next part.
The problem: part 14x10-100 occurs three times in the structure. I want to create a record for part 14x10-100 in the part_lib table and refer to that record three times. What actually happens is that for each 14x10-100 part a corresponding record in the part_lib table is created in stead of one create and two matches. If I run it again it will match like excpected. I suspect I'm lost in the promise/async await parts of the code.
Below the relevant code. I've removed some attribute mappings for readability. My thoughts behind it: I'm not returning new promises like normal in a async function since Sequelize already returns a promise. When creating a part I'm awaiting (or at least I think so) the partLibController calls to ensure that all matching/creating/joining is done before proceeding to the next part in the structure.
Thanks a bunch!!
Recursive loop
function parseChild(child, modelId, parentId, userId, level) {
return new Promise((resolve, reject) => {
partController.create({
parent_id: parentId
, name: child.name
}, { id: userId }).then((part) => {
resolve({ child: child, level: level });
if (child.children) {
child.children.forEach(grandChild => {
parseChild(grandChild, modelId, part.part_id, userId, level + '---');
});
}
}).catch(error => { console.log(error); });
}).then((obj) => { console.log(`${obj.level} ${obj.child.name}`); });
}
PartController Create
async function create(partBody, currentUser) {
let { parent_id, name } = partBody;
const match = await partLibController.match(name);
let partLibId = null;
if (match.length == 0) {
const partLib = await partLibController.createFromPart(partBody, currentUser);
partLibId = partLib.part_lib_id;
} else {
partLibId = match[0].dataValues.part_lib_id
}
return ModelAssembly.create({
parent_id: parent_id
, name: name
, part_lib_id: partLibId
});
}
PartLibController Match
function match(name) {
return PartLib.findAll({
where: {
name: name
},
});
}
PartLibController CreateFromPart
function createFromPart(partBody, currentUser) {
let { name } = partBody;
return PartLib.create({
name,
});
}
Thanks to AKX I've solved the problem: hero
The problem was in the recursive call itself I suppose but here's the working code:
async function parseChild(child, modelId, parentId, userId, level) {
const body = {
parent_id: parentId
, name: child.name
};
const ma = await partController.create(body, { id: userId });
if (child.children) {
for (const grandChild of child.children) {
await parseChild(grandChild, modelId, ma.part_id, userId, level + '---');
}
}
return;
}
Related
I am new to dynamoDB. I am having difficulty developing a table structure. I have data that can best be thought of as a folder structure. There are folders which are nested in parent folders. Most of the time, I will be querying for all folders with a given parent folder, however, there are times when I will be querying individual folders.
If I use the parent_id (parent folder) as the partition key and the id of the individual folder as the sort key, I believe that this creates a table where all related files are stored together and I can query them efficiently. However, I have questions.
First, the query "works" in that it returns the data, but is it written so that it queries the data correctly and is not merely scrolling through the whole table?
router.get("/api/children_folders/:parent_id", (req, res, next) => {
let parent_id = req.params.parent_id;
let params = {
TableName: tableName,
KeyConditionExpression: "parent_id = :pid",
ExpressionAttributeValues: {
":pid": parent_id,
},
ScanIndexForward: false,
};
docClient.query(params, (err, data) => {
if (err) {
console.log(err);
return res.status(err.statusCode).send({
message: err.message,
status: err.statusCode,
});
} else {
return res.status(200).send(data);
}
});
});
Second, if I want to query for individual tags, do I need to pass in a combination of the parent folder ID and the actual ID, or is this OK?
router.get("/api/folder/:folder_id", (req, res, next) => {
let tag_id = req.params.folder_id;
let params = {
TableName: tableName,
KeyConditionExpression: "folder_id = :fid",
ExpressionAttributeValues: {
":fid": folder_id,
},
Limit: 1,
};
docClient.query(params, (err, data) => {
if (err) {
console.log(err);
return res.status(err.statusCode).send({
message: err.message,
status: err.statusCode,
});
} else {
if (!_.isEmpty(data.Items)) {
return res.status(200).send(data.Items[0]);
} else {
return res.status(404).send();
}
}
});
});
I just feel like I am missing some thing here and I want to make sure that I am grabbing the data correctly.
The PK, should be something that would divide the load equally (ideally). I don't the fully picture of your problem but assuming you can chose a good parent folder as a partition key, then you can insert every file/dir with a sort key representing its full path
For example:
PK SK
/home /username/pictures/cat.jpg
This way if you want to get a specific item you can use the get item request
var params = {
Key: {
"PK": { "S": "/home" },
"SK": { "S": "/username/pictures/cat.jpg" }
},
TableName: tableName
};
var result = await dynamodb.getItem(params).promise()
Now if you want to list all the files in "/home/username/pictures" you can use begins with query
const params = {
TableName: 'tablenName',
KeyConditionExpression: '#PK = :root_path and begins_with(#SK, :sub_path)',
ExpressionAttributeNames:{
"#user_id": "root_path",
"#user_relation": 'sub_path'
},
ExpressionAttributeValues: {
":root_path": "/home",
":sub_path": "/username/pictures"
}
}
I have this helper function in my reducer, which has the given state:
type CustomerCollection = { [number]: Customer }
type CustomerState = {
+customers: ?CustomerCollection,
+newItem: ?(Customer | Review),
+searchResults: ?(Customer[]),
+error: ?string,
+isLoading: boolean
};
function customerWithReview(review: Review): Customer {
const id: number = review.customerId;
const oldCustomer: Customer = state.customers[id];
const newReviews: Review[] = [review, ...oldCustomer.reviews];
return Object.assign(oldCustomer, { reviews: newReviews });
}
I get a Flow error on the id of const oldCustomer: Customer = state.customers[id]; saying Cannot get state.customers[id] because an index signature declaring the expected key/value type is missing in null or undefined.
This is happening because of the nullable/optional ?CustomerCollection type of state.customers.
I can silence the error by making sure customers isn't null:
if (state.customers) {
const oldCustomer: Customer = state.customers[id];
const newReviews: Review[] = [review, ...oldCustomer.reviews];
return Object.assign(oldCustomer, { reviews: newReviews });
}
But then the problem just goes up the chain because I don't have anything to return from the function.
I can certainly expand it to:
function customerWithReview(review: Review): Customer {
if (!state.customers) {
return new Customer();
} else {
const id: number = review.customerId;
const oldCustomer: Customer = state.customers[id];
const newReviews: Review[] = [review, ...oldCustomer.reviews];
return Object.assign(oldCustomer, { reviews: newReviews });
}
}
But in actual practice, the action that gets us to this branch of the reducer will never be called if state.customers is null, and we'd never return new Customer() and would have no use for it if we did. state.customers is nullable in order to tell the difference between "we haven't fetched the customers yet (state.customers == null)" and "we've fetched the customers but there are none (state.customers == {}).
It would be a lot easier if I could just assert that state.customers would always exist in these cases, which in Swift I would do with force-unwrapping:
const oldCustomer: Customer = state.customers![id];
Can I do anything like this with Flow?
Or, given that only my GET_CUSTOMERS_FAILURE action would ever deal with state.customers == null, is there some other way to restructure my reducer so that this is a little easier? An entirely separate fetchReducer that is has a nullable customer collection while the rest of the actions fall under a different reducer?
You can use invariant function (Check that it works here):
type Customer = { id: number, reviews: Array<Review> };
type Review = { customerId: number };
type CustomerCollection = { [number]: Customer }
type CustomerState = {
+customers: ?CustomerCollection,
+newItem: ?(Customer | Review),
+searchResults: ?(Customer[]),
+error: ?string,
+isLoading: boolean
};
declare var state: CustomerState;
declare function invariant(): void;
function customerWithReview(review: Review): Customer {
const id: number = review.customerId;
invariant(state.customers, 'No customers and I don\'t know why');
const oldCustomer: Customer = state.customers[id];
const newReviews: Review[] = [review, ...oldCustomer.reviews];
return Object.assign(oldCustomer, { reviews: newReviews });
}
You can implement it somewhere in your project and import when necessary.
You can implement it like this:
export function invariant<T>(value: ?T, falsyErrorMessage: string, errorParams?: Object): void {
if (!value) {
log.error(falsyErrorMessage, errorParams || {});
throw new Error(INVARIANT_ERROR_MESSAGE);
}
}
Unfortunately, the name of the function is hard-coded in flow.
Alternative variant is just to add an if and to throw an error in your customerWithReview function directly.
I have a collection named campgrounds in which every document contains an array of document reference to the documents in the comments collections.
It looks like this Campground
I'm trying to figure out a way to populate this comments array before sending it to my ejs template.
My code looks like this
app.get("/campgrounds/:docId", function(req, res) {
var docRef = firestore.collection("campgrounds").doc(req.params.docId);
try {
docRef.get().then(doc => {
if (!doc.exists) {
res.send("no such document");
} else {
// res.send(doc.data());
res.render("campground", {
doc: doc.data(),
title: doc.data().title,
id: req.params.docId
});
}
});
} catch (error) {
res.send(error);
}
});
In your array you store DocumentReferences. If you want to get the data of the corresponding documents in order to include this data in your object you should use Promise.all() to execute the variable number (1 or more) of get() asynchronous operations.
The following should work (not tested at all however):
app.get("/campgrounds/:docId", function(req, res) {
var docRef = firestore.collection("campgrounds").doc(req.params.docId);
try {
var campground = {};
docRef.get()
.then(doc => {
if (!doc.exists) {
res.send("no such document");
} else {
campground = {
doc: doc.data(),
title: doc.data().title,
id: req.params.docId
};
var promises = [];
doc.data().comments.forEach((element, index) => {
promises.push(firestore.doc(element).get());
});
return Promise.all(promises);
}
})
.then(results => {
var comments = {};
results.forEach((element, index) => {
comments[index] = element.data().title //Let's imagine a comment has a title property
});
campground.comments = comments;
res.render("campground", campground);
})
} catch (error) {
res.send(error);
}
});
Note that with this code you are doing 1 + N queries (N being the length of the comments array). You could denormalize your data and directly store in the campground doc the data of the comments: you would then need only one query.
I am using normalizr to organize my redux-store state.
Let's say that I have normalized todo-list:
{
result: [1, 2],
entities: {
todo: {
1: {
id: 1,
title: 'Do something'
},
2: {
id: 2,
title: 'Second todo'
}
}
}
}
Then I would like to implement addTodo action. I need to have an id in todo object, so I generate a random one:
function todoReducer(state, action) {
if(action.type == ADD_TODO) {
const todoId = generateUUID();
return {
result: [...state.result, todoId],
enitities: {
todos: {
...state.entities.todos,
[todoId]: action.todo
}
}
}
}
//...other handlers...
return state;
}
But the problem is that eventually all data will be saved to server and generated id should be replaced with real server-assigned id. Now I merge them like this:
//somewhere in reducer...
if(action.type === REPLACE_TODO) {
// copy todos map, add new entity, remove old
const todos = {
...state.entities.todos
[action.todo.id]: action.todo
};
delete todos[action.oldId];
// update results array as well
const result = state.result.filter(id => id !== oldId).concat(action.todo.id);
// return new state
return {entities: {todos}, result};
}
It seems to be a working solution, but there also a lot of overhead. Do you know any way to simplify this and don't make REPLACE_TODO operation?
I’m having issues getting two dependant types of data from a PouchDB database.
I have a list of cars that I get like so:
localDB.query(function(doc) {
if (doc.type === ‘list’) {
emit(doc);
}
}, {include_docs : true}).then(function(response) {
console.log(“cars”, response);
// Save Cars List to app
for(var i = 0; i < response.rows.length; i++) {
addToCarsList(response.rows[i].id, response.rows[i].carNumber);
}
console.log(“Cars List: " + carsListToString());
return response;
}).then(function(listRecord) {
listRecord.rows.forEach(function(element, index){
console.log(index + ' -> ', element);
localDB.query(function(doc) {
console.log("filtering with carNb = " + element.carNb);
if (doc.type === 'defect' && doc.listId == getCurrentListId() && doc.carNb == element.carNb ) {
emit(doc);
}
}, {include_docs : false}).then(function(result){
console.log("defects", result);
}).catch(function(err){
console.log("an error has occurred", err);
});
});
}).catch(function(err) {
console.log('error', err);
});
Here's what happens. After getting the list of cars, then for each cars I would like to query the defects and store then in some arrays. Then when all that querying is done, I want to build the UI with the data saved.
But what's happening is that the forEach gets processed quickly and does not wait for the inner async'd localDb.query.
How can I query some documents based on an attribute from a parent query? I looked into promises in the PouchDB doc but I can't understand how to do it.
(please forget about curly quotes and possible lint errors, this code was anonymized by hand and ultra simplified)
The method you are looking for is Promise.all() (execute all promises and return when done).
However, your query is already pretty inefficient. It would be better to create a persistent index, otherwise it has to do a full database scan for every query() (!). You can read up on the PouchDB query guide for details.
I would recommend installing the pouchdb-upsert plugin and then doing:
// helper method
function createDesignDoc(name, mapFunction) {
var ddoc = {
_id: '_design/' + name,
views: {}
};
ddoc.views[name] = { map: mapFunction.toString() };
return ddoc;
}
localDB.putIfNotExists(createDesignDoc('my_index', function (doc) {
emit([doc.type, doc.listId, doc.carNb]);
})).then(function () {
// find all docs with type 'list'
return localDB.query('my_index', {
startkey: ['list'],
endkey: ['list', {}],
include_docs: true
});
}).then(function (response) {
console.log("cars", response);
// Save Cars List to app
for(var i = 0; i < response.rows.length; i++) {
addToCarsList(response.rows[i].id, response.rows[i].carNumber);
}
console.log("Cars List: " + carsListToString());
return response;
}).then(function (listRecord) {
return PouchDB.utils.Promise.all(listRecord.rows.map(function (row) {
// find all docs with the given type, listId, carNb
return localDB.query('my_index', {
key: ['defect', getCurrentListId(), row.doc.carNb],
include_docs: true
});
}));
}).then(function (finalResults) {
console.log(finalResults);
}).catch(function(err){
console.log("an error has occurred", err);
});
I'm using a few tricks here:
emit [doc.type, doc.listId, doc.carNb], which allows us to query by type or by type+listId+carNb.
when querying for just the type, we can do {startkey: ['list'], endkey: ['list', {}]}, which matches just those with the type "list" because {} is the "higher" than strings in CouchDB object collation order.
PouchDB.utils.Promise is a "hidden" API, but it's pretty safe to use if you ask me. It's unlikely we'll change it.
Edit Another option is to use the new pouchdb-find plugin, which offers a simplified query API designed to replace the existing map/reduce query() API.
Another approach would be to pull both the list docs and the defect docs down at the same time then merge them together using a reduce like method that will convert them into an array of objects:
{
_id: 1,
type: 'list',
...
defects: [{
type: 'defect'
listId: 1
...
}]
}
By pulling the list and the defects down in one call you save a several calls to the pouchdb query engine, but you do have to iterate through every result to build your collection of lists objects with and embedded array of defects.
// This is untested code so it may not work, but you should get the idea
var _ = require('underscore');
// order documents results by list then defect
var view = function (doc) {
if (doc.type === 'list') {
emit([doc._id, doc.carNumber, 1);
} else if (doc.type === 'defect') {
emit([doc.listId, doc.carNb, 2])
}
}
localDB.query(view, { include_docs: true })
.then(function(response) {
return _(response.rows)
.reduce(function(m, r) {
if (r.key[2] === 1) {
// initialize
r.doc.defects = [];
m.push(r.doc)
return m;
}
if (r.key[2] === 2) {
var list = _(m).last()
if (list._id === r.key[0] && list.carNumber === r.key[1]) {
list.defects.push(r.doc);
}
return m;
}
}, []);
})
.then(function(lists) {
// bind to UI
});
With couch, we found reducing calls to the couch engine to be more performant, but I don't know if this approach is better for PouchDB, but this should work as a solution, especially if you are wanting to embed several collections into one list document.