I have a Queue that looks like this
new Queue(queueRef, options, ({post, user, postId}, progress, resolve, reject) => {
rootRef.child(`users/${user.user_id}/followers`).once('value', (snapshot) => {
const followers = toArray(snapshot.val())
for (var i = 0; i < followers.length; i++) {
rootRef.child(`users/${followers[i].user_id}/feed/${postId}`).set(post, (err) => {
if (err) {
reject(err)
} else if (i >= followers.length - 1) {
resolve({post, user, postId})
}
})
}
}, reject)
})
My issue is that I'm really only resolving once all the sets have finished and rejecting if any of those fail. What I'd like to do is somehow pass each iteration of a loop to another Queue which can then reject/resolve for that specific request rather than the whole collection.
This looks like it's probably an XY problem and probably has a better solution. But you're looking for something like Q.all().
In essence, call a method that does each op and returns a promise, and resolve/reject when the entire set is done.
new Queue(queueRef, options, ({post, user, postId}, progress, resolve, reject) => {
rootRef.child(`users/${user.user_id}/followers`).once('value', (snapshot) => {
var promiseList = [], p;
const followers = toArray(snapshot.val())
for (var i = 0; i < followers.length; i++) {
p = processNextFollower(followers[i]);
// p.then(progress);
promiseList.push(p);
}
Q.all(promiseList).then(resolve, reject);
}, reject)
})
function processNextFollower(follower, postId) {
var def = Q.defer();
rootRef.child(`users/${follower.user_id}/feed/${postId}`).set(post, (err) => {
if (err) {
def.reject(err)
} else if (i >= followers.length - 1) {
def.resolve({post, user, postId})
}
})
}
return def.promise;
}
Related
i'm trying to add persistent attributes to my lambda function.
i created a dynamoDB table and added it to the triggers of my lambda function.
i copied a sample code from github, but when i try to launch the skill i get an error. The console log shows:
{
"errorMessage": "Could not read item (amzn1.ask.account.AGIIYNRXWDLBD6XEPW72QS2BHGXNP7NWYBEWSH2XLSXZP64X3NCYEMVK233VFDWH77ZB6DAK6YJ53SZLNUFVQ56CYOVCILS7QFZI4CIRDWC3PAHS4QG27YUY5PTT6QEIK46YFNTJT54YAKNGOWV2UO66XZACFDQ5SEXKJYOBNFNIZNUXKNTIAAYZG4R5ZU4FMLPDZZN64KLINNA) from table (Spiele): The provided key element does not match the schema",
"errorType": "AskSdk.DynamoDbPersistenceAdapter Error",
"stackTrace": [
"Object.createAskSdkError (/var/task/node_modules/ask-sdk-dynamodb-persistence-adapter/lib/utils/AskSdkUtils.js:22:17)",
"DynamoDbPersistenceAdapter.<anonymous> (/var/task/node_modules/ask-sdk-dynamodb-persistence-adapter/lib/attributes/persistence/DynamoDbPersistenceAdapter.js:123:49)",
"step (/var/task/node_modules/ask-sdk-dynamodb-persistence-adapter/lib/attributes/persistence/DynamoDbPersistenceAdapter.js:44:23)",
"Object.throw (/var/task/node_modules/ask-sdk-dynamodb-persistence-adapter/lib/attributes/persistence/DynamoDbPersistenceAdapter.js:25:53)",
"rejected (/var/task/node_modules/ask-sdk-dynamodb-persistence-adapter/lib/attributes/persistence/DynamoDbPersistenceAdapter.js:17:65)",
"<anonymous>",
"process._tickDomainCallback (internal/process/next_tick.js:228:7)"
]
}
the table contains a primary key "name" and sort key "UserId". is that wrong?
here is my index.js:
const Alexa = require('ask-sdk');
// Define the skill features
let skill;
/**
* If this is the first start of the skill, grab the user's data from Dynamo and
* set the session attributes to the persistent data.
*/
const GetUserDataInterceptor = {
process(handlerInput) {
let attributes = handlerInput.attributesManager.getSessionAttributes();
if (handlerInput.requestEnvelope.request.type === 'LaunchRequest' && !attributes['isInitialized']) {
return new Promise((resolve, reject) => {
handlerInput.attributesManager.getPersistentAttributes()
.then((attributes) => {
attributes['isInitialized'] = true;
saveUser(handlerInput, attributes, 'session');
resolve();
})
.catch((error) => {
reject(error);
})
});
}
}
};
function saveUser(handlerInput, attributes, mode) {
if(mode === 'session'){
handlerInput.attributesManager.setSessionAttributes(attributes);
} else if(mode === 'persistent') {
console.info("Saving to Dynamo: ",attributes);
return new Promise((resolve, reject) => {
handlerInput.attributesManager.getPersistentAttributes()
.then((persistent) => {
delete attributes['isInitialized'];
handlerInput.attributesManager.setPersistentAttributes(attributes);
resolve(handlerInput.attributesManager.savePersistentAttributes());
})
.catch((error) => {
reject(error);
});
});
}
}
const LaunchHandler = {
canHandle(handlerInput) {
return handlerInput.requestEnvelope.request.type === 'LaunchRequest';
},
handle(handlerInput) {
console.info("LaunchRequest");
let attributes = handlerInput.attributesManager.getSessionAttributes();
console.info("Test the load: " + attributes['isInitialized']);
attributes['FOO'] = "BAR";
saveUser(handlerInput, attributes, 'persistent');
return handlerInput.responseBuilder
.speak('Hello')
.reprompt('Hello')
.getResponse();
}
}
exports.handler = Alexa.SkillBuilders.standard()
.addRequestHandlers(
LaunchHandler
)
.addRequestInterceptors(GetUserDataInterceptor)
.withTableName('Spiele')
.withAutoCreateTable(true)
.withDynamoDbClient()
.lambda();
can anyone tell me what i'm doing wrong?
please confirm the partition key is 'userId' not 'UserId' (notice the uppercase U).
Also I would suggest using 'this' object.
Let me know if that helps.
Cheers
Below code is for python lambda function
from ask_sdk_core.skill_builder import CustomSkillBuilder
from ask_sdk_dynamodb.adapter import DynamoDbAdapter
sb = SkillBuilder()
sb = CustomSkillBuilder(persistence_adapter = dynamodb_adapter)
I have a collection named campgrounds in which every document contains an array of document reference to the documents in the comments collections.
It looks like this Campground
I'm trying to figure out a way to populate this comments array before sending it to my ejs template.
My code looks like this
app.get("/campgrounds/:docId", function(req, res) {
var docRef = firestore.collection("campgrounds").doc(req.params.docId);
try {
docRef.get().then(doc => {
if (!doc.exists) {
res.send("no such document");
} else {
// res.send(doc.data());
res.render("campground", {
doc: doc.data(),
title: doc.data().title,
id: req.params.docId
});
}
});
} catch (error) {
res.send(error);
}
});
In your array you store DocumentReferences. If you want to get the data of the corresponding documents in order to include this data in your object you should use Promise.all() to execute the variable number (1 or more) of get() asynchronous operations.
The following should work (not tested at all however):
app.get("/campgrounds/:docId", function(req, res) {
var docRef = firestore.collection("campgrounds").doc(req.params.docId);
try {
var campground = {};
docRef.get()
.then(doc => {
if (!doc.exists) {
res.send("no such document");
} else {
campground = {
doc: doc.data(),
title: doc.data().title,
id: req.params.docId
};
var promises = [];
doc.data().comments.forEach((element, index) => {
promises.push(firestore.doc(element).get());
});
return Promise.all(promises);
}
})
.then(results => {
var comments = {};
results.forEach((element, index) => {
comments[index] = element.data().title //Let's imagine a comment has a title property
});
campground.comments = comments;
res.render("campground", campground);
})
} catch (error) {
res.send(error);
}
});
Note that with this code you are doing 1 + N queries (N being the length of the comments array). You could denormalize your data and directly store in the campground doc the data of the comments: you would then need only one query.
I'm using Node/Puppeteer in the code below, passing in a large list of URL's for traversal and scraping. It has been difficult to do it asynchronously, though I find that I am getting closer and closer to the answer. I am currently stuck on an issue related to the following error.
UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 17): Error: Protocol error (Target.createTarget): Target closed.
This error occurs once upon every iteration of the while loop. Though I'm not sure what I may be doing incorrectly.
Could someone help me do the following:
1) Diagnose the source of the error.
2) Potentially find a more effective way to traverse a large list of URLs asynchronously.
async function subProc(list, batchSize) {
let subList = null;
let i = 0;
while (list.length > 0) {
let browser = await puppeteer.launch();
subList = list.splice(0, batchSize);
console.log("Master List Size :: " + list.length);
console.log("SubList Size :: " + subList.length);
for (let j = 0; j < subList.length; j++) {
promiseArray.push(new Promise((resolve, reject) => {
resolve(pageScrape(subList[j], browser));
}));
}
Promise.all(promiseArray)
.then(response => {
procArray.concat(response);
});
promiseArray = new Array();
try {
await browser.close();
} catch(ex){
console.log(ex);
}
};
}
async function pageScrape(url, browser) {
let page = await browser.newPage();
await page.goto(url, {
timeout: 0
});
await page.waitFor(1000);
return await page.evaluate(() => {
let appTitle = document.querySelector('').innerText;
let companyName = document.querySelector('').innerText;
let dateListed = document.evaluate("", document, null, XPathResult.FIRST_ORDERED_NODE_TYPE, null).singleNodeValue.innerText;
let category = document.evaluate("']//a//strong", document, null, XPathResult.FIRST_ORDERED_NODE_TYPE, null).singleNodeValue.innerText;
/* */
return {
appTitle,
companyName,
dateListed,
category
}
}).then(response => {
let urlData = {
id: subList[j],
appName: response.appTitle,
companyName: response.companyName,
dateListed: response.dateListed,
category: response.category
}
return urlData;
});
};
I figured out the solution to the problem I was having.
Every computer is limited in its processing ability, so instead of iterating through 1000 urls simultaneously you have to break it down into smaller pieces.
By using a PromiseAll, and iterating and scraping 10 urls at a time and storing these values in an array, I was able to throttle the processing required to iterate through all 1000 urls.
processBatch(subData, 10, procArray).then((processed)=>{
for(let i = 0; i < procArray.length; i++){
for(let j = 0; j < procArray[i].length; j++){
results.push(procArray[i][j]);
}
}
function processBatch(masterList, batchSize, procArray){
return Promise.all(masterList.splice(0, batchSize).map(async url =>
{
return singleScrape(url)
})).then((results) => {
if (masterList.length < batchSize) {
console.log('done');
procArray.push(results);
return procArray;
} else {
console.log('MasterList Size :: ' + masterList.length);
procArray.push(results);
return processBatch(masterList, batchSize, procArray);
}
})
}
i have a ionic project.
I want to unsubscribe when i get the cities from subscription of firebase return.
But something goes wrong and it throws following error when i try to unsubscribe.
How can is solve that?
My code.
getCities() {
let i: any;
let a = this.firebaseProvider.getOtoparks()
.subscribe(data => {
for (i = 0; i < data.length; i++) {
this.cities.push({
name: data[i]['details']['sehir'],
value: i,
}
);
}
// try to unsubscribe
a.unsubscribe();
});
}
The best practice is to use take(1) when you want to unsubscribe without a condition.
getCities() {
let i: any;
let a = this.firebaseProvider.getOtoparks()
.take(1)
.subscribe(data => {
for (i = 0; i < data.length; i++) {
this.cities.push({
name: data[i]['details']['sehir'],
value: i,
});
}
});
}
for loop not waiting for reponse of getRoleName function.
how to do this?Issue with for loop and promise.
for(var i = 0; i < data.rows.length; i++) {
alert("inside" + i);
this.getRoleName(data.rows.item(i).role_id).then((data1)=>{
this.users.push({
id: data.rows.item(i).id,
text: data.rows.item(i).username,
role: data1.rows.item(i).role_text
});
},(error)=>{
});
}
getRoleName(id):Promise<any> {
let text : String;
let sql = "select role_text from roles where id = "+id;
return this.db.executeSql(sql, {}).then((data) => {
return data;
},(error) => { alert("i im in eerror");
console.error("Unable to execute sql", JSON.stringify(error));
});
}