I'm trying to crawl movie data from TMDB website. I finished my code with pure javascript, but I want to change the code into functional programming style by using ramda.js.
I attached my code below. I want to get rid of for-loop (if it is possible) and use R.pipe function.
(async () => {
for (let i = 0; i < 1000; i++) {
(() => {
setTimeout(async () => {
let year = startYr + Math.floor(i / 5);
await request.get(path(year, i % 5 + 1), async (err, res, data) => {
const $ = cheerio.load(data);
let list = $('.results_poster_card .poster.card .info .flex a');
_.forEach(list, (element, index) => {
listJSON.push({
MovieID: $(element).attr('id').replace('movie_', ''),
Rank: (i % 5) * 20 + index + 1,
Year: year
});
});
if(i === 1000 - 1) {
await pWriteFile(`${outputPath}/movieList.json`, JSON.stringify(listJSON, null, 2));
}
});
}, 1000 * i);
})(i);
}
})().catch(error => console.log(error));
Steps:
1- Break your code in small functions
2- Stop using async await and use promise.then(otherFunction)
3- When using promise, you could create a sleep function like these: const sleep = (time) => new Promise(resolve => setTimeout(resolve, time));
Ex.:
const process = index => sleep(1000)
.then(() => makeRequest(index))
.then(processData);
R.range(0, 1000)
.reduce(
(prev, actual) => prev.then(() => process(actual),
Promise.resolve()
) // Sequential
.then(printResult);
R.range(0, 1000)
.map(process) // Parallel
.then(printResult);
You can use the Ramda range() function to replace your loop.
https://ramdajs.com/docs/#range
R.range(0, 1000);
That will provide you with a collection of integers (your i) that you can work with (map() or whatever you need).
Related
I am trying to get to grips with the composition API. Struggling with watch:
const totalValuation = ref(0);
const values = ref([1, 2, 3]);
totalValuation.value = watch(finalTasksFiltered.value, () => {
return values.value.reduce((prev, curr) => {
console.log(prev + curr);
return prev + curr;
}, 0);
});
return {
finalTasksFiltered,
totalValuation,
};
The console.log works exactly like it should (1,3,6) but I cannot seem to render it to the DOM.
When I check to console it is fine but in the DOM like so {{ totalValuation }} it returns:
() => { (0,_vue_reactivity__WEBPACK_IMPORTED_MODULE_0__.stop)(runner); if (instance) { (0,_vue_shared__WEBPACK_IMPORTED_MODULE_1__.remove)(instance.effects, runner); } }
Oh, I am using Quasar - not sure if that makes a difference.
I am sure its something small.
I have imported ref and watch from vue.
I have a computed function working fine too.
watch is meant to execute a function when some value changes. It is not meant to be assigned as a ref. What you're looking to do seems like a better fit for computed
watch: (no assignment)
watch(finalTasksFiltered.value, () => {
totalValuation.value = values.value.reduce((prev, curr) => {
console.log(prev + curr);
return prev + curr;
}, 0);
});
computed: (uses assignment)
const totalValuation = computed(() => {
return values.value.reduce((prev, curr) => {
console.log(prev + curr);
return prev + curr;
}, 0);
});
I have been trying to use the riot games api to compute all the previous custom games and then find the win loss streaks for individual players, I have built the following code to grab matches for a particular user.
See https://github.com/FriendlyUser/deno-riot-games-custom-games
But I feel like the riot games api is only returning data with its v4 api up to season 11, if anyone could clarify how the api works or explain how I could possibly get more data, that would be fantastic.
import { writeJson } from "https://deno.land/std/fs/mod.ts"
import "https://deno.land/x/dotenv/load.ts"
const player_id = Deno.env.get('ACCOUNT_ID')
const region_url = 'https://na1.api.riotgames.com'
let riot_URL = new URL(`${region_url}/lol/match/v4/matchlists/by-account/${player_id}`)
enum HTTP {
GET = 'GET',
POST = 'POST',
PUT = 'PUT',
DELETE = 'DELETE'
}
interface MatchlistDto {
startIndex: number
totalGames: number
endIndex: number
matches: Array<any>
}
function makeFetchOptions(
riotKey = Deno.env.get('RIOT_API_KEY'),
method: HTTP = HTTP.GET
): object {
return {
method: method,
headers: {
"Accept-Charset": "application/x-www-form-urlencoded; charset=UTF-8",
"Accept-Language": "en-US,en;q=0.9",
'X-Riot-Token': riotKey
}
}
}
function appendMatchHistory(riot_endpoint: string): Promise<MatchlistDto> {
const riotKey = Deno.env.get('RIOT_API_KEY')
console.log(riotKey)
const options = makeFetchOptions(riotKey)
return fetch(riot_endpoint, options)
.then( (resp: any) => {
console.log(resp)
return resp.json()
})
.then( (matchData: MatchlistDto) => {
return matchData
})
}
const max_iterations = 1000
let bIndex = 0
let eIndex = 100
let current_url = riot_URL
let riot_endpoint = null
let allMatches = []
let customGames = []
const sleep = (milliseconds: number) => {
return new Promise(resolve => setTimeout(resolve, milliseconds))
}
for (let i = 0; i < max_iterations; i++) {
console.log(`beginIndex: ${bIndex} endIndex: ${eIndex}`)
riot_endpoint = current_url.toString()
const newMatches = await appendMatchHistory(riot_endpoint)
await sleep(1500)
current_url.searchParams.delete('beginIndex')
current_url.searchParams.delete('endIndex')
const {matches} = newMatches
if (matches.length == 0) {
console.log(`ENDING SCRIPT AT ${eIndex} with ${matches.length}`)
break
}
// startIndex becomes endIndex
bIndex = eIndex
eIndex = eIndex + 100
allMatches.push(newMatches.matches)
// get new url
current_url.searchParams.append('beginIndex', String(bIndex))
current_url.searchParams.append('endIndex', String(eIndex))
}
await writeJson(
"./allData.json",
allMatches
);
Sorry if this answer is late. But yes the Riot API is only for "current" data, and that is why sites like U.GG, OP.GG, etc actually run scripts to store data continuously. So to get statistics you would have to write scripts to store it into your own DB over time.
Sadly, there is no way to get previous season data
I try to insert multiple rows in React Native using SQLite.
this is the code:
rows = responseJson.rows;
for (i = 0; i < rows.length; i++) {
row=rows[i];
query = `insert into ComuniUserAccountSync values (
${row.IDComuniUserAccountSync},
${row.IdAzienda},
${row.IdComune},
${row.IdUserAccount},
'${row.DescrizioneComune}',
'${row.DateLastUpdateMaster}'
)`;
db.transaction(
tx => {
tx.executeSql(query, [], (a,b) =>
console.log("!OK!!", JSON.stringify(b)), (a, b) =>
console.log("!ERROR!!", a, b)
)
}
);
}
but the result is that I insert only the last row, many times! This is the output with
db.transaction(
tx => {
tx.executeSql("select IDComuniUserAccountSync from ComuniUserAccountSync", [], (a,b) =>
console.log("!OK!", JSON.stringify(b)), (a,b) =>
console.log("!ERROR!!", JSON.stringify(b))
);
}
);
!OK! {"rowsAffected":0,"rows":{"_array":[{"IDComuniUserAccountSync":72},{"IDComuniUserAccountSync":72},{"IDComuniUserAccountSync":72},{"IDComuniUserAccountSync":72},{"IDComuniUserAccountSync":72}, .......
ANY HELP??
Max
insertCategories(arrCateData){
let keys = Object.keys(arrCateData[0])
let arrValues = []
var len = arrCateData.length;
for (let i = 0; i < len; i++) {
arrCateData[i].image = `"${arrCateData[i].image}"`;
arrCateData[i].thumbnail = `"${arrCateData[i].thumbnail}"`;
arrCateData[i].name = `"${arrCateData[i].name}"`;
arrCateData[i].path = `"${arrCateData[i].path}"`;
arrValues.push("(" + Object.values(arrCateData[i]) +")");
}
// console.log(arrValues)
return new Promise((resolve) => {
this.initDB().then((db) => {
db.transaction((tx) => {
tx.executeSql("INSERT INTO category ("+ keys + ") VALUES " + String(arrValues)).then(([tx, results]) => {
resolve(results);
});
}).then((result) => {
this.closeDatabase(db);
}).catch((err) => {
console.log(err);
});
}).catch((err) => {
console.log(err);
});
});
}
this code may help you i just pass my array into function and it will insert data jusr change data according to ur requirement
I am trying to write a function that will:
Create documents in a sub collection
Allow for a then/catch call back after all sub documents have been created
export const doCreateSubs = (Id, count) => {
if (count > 0 && count <= 50){
const times = n => f => {
let iter = i => {
if (i === n) return;
f(i);
iter(i + 1);
};
return iter(0);
};
times(count)(i => {
db
.collection("parent")
.doc(`${Id}`)
.collection("sub")
.add({
subName: `name ${i + 1}`,
dateCreated: new Date()
});
});
}
}
I've played around with batch but it doesn't work with .collection. I know my function is really poor - is there a generally bettery way of doing this?
So i've just realised you can .doc() with no value and it will create a uid for the key. I can also return .commit and recieve a call back when it's complete!
export const doCreateSubs = (Id, count) => {
if (count > 0 && count <= 50){
const times = n => f => {
let iter = i => {
if (i === n) return;
f(i);
iter(i + 1);
};
return iter(0);
};
const batch = db.batch();
times(count)(i => {
const ref = db
.collection("parent")
.doc(`${Id}`)
.collection("subs")
.doc();
batch.set(ref, {
boxName: `Sub ${i + 1}`,
dateCreated: new Date()
});
});
return batch.commit();
}
}
I'm using Node/Puppeteer in the code below, passing in a large list of URL's for traversal and scraping. It has been difficult to do it asynchronously, though I find that I am getting closer and closer to the answer. I am currently stuck on an issue related to the following error.
UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 17): Error: Protocol error (Target.createTarget): Target closed.
This error occurs once upon every iteration of the while loop. Though I'm not sure what I may be doing incorrectly.
Could someone help me do the following:
1) Diagnose the source of the error.
2) Potentially find a more effective way to traverse a large list of URLs asynchronously.
async function subProc(list, batchSize) {
let subList = null;
let i = 0;
while (list.length > 0) {
let browser = await puppeteer.launch();
subList = list.splice(0, batchSize);
console.log("Master List Size :: " + list.length);
console.log("SubList Size :: " + subList.length);
for (let j = 0; j < subList.length; j++) {
promiseArray.push(new Promise((resolve, reject) => {
resolve(pageScrape(subList[j], browser));
}));
}
Promise.all(promiseArray)
.then(response => {
procArray.concat(response);
});
promiseArray = new Array();
try {
await browser.close();
} catch(ex){
console.log(ex);
}
};
}
async function pageScrape(url, browser) {
let page = await browser.newPage();
await page.goto(url, {
timeout: 0
});
await page.waitFor(1000);
return await page.evaluate(() => {
let appTitle = document.querySelector('').innerText;
let companyName = document.querySelector('').innerText;
let dateListed = document.evaluate("", document, null, XPathResult.FIRST_ORDERED_NODE_TYPE, null).singleNodeValue.innerText;
let category = document.evaluate("']//a//strong", document, null, XPathResult.FIRST_ORDERED_NODE_TYPE, null).singleNodeValue.innerText;
/* */
return {
appTitle,
companyName,
dateListed,
category
}
}).then(response => {
let urlData = {
id: subList[j],
appName: response.appTitle,
companyName: response.companyName,
dateListed: response.dateListed,
category: response.category
}
return urlData;
});
};
I figured out the solution to the problem I was having.
Every computer is limited in its processing ability, so instead of iterating through 1000 urls simultaneously you have to break it down into smaller pieces.
By using a PromiseAll, and iterating and scraping 10 urls at a time and storing these values in an array, I was able to throttle the processing required to iterate through all 1000 urls.
processBatch(subData, 10, procArray).then((processed)=>{
for(let i = 0; i < procArray.length; i++){
for(let j = 0; j < procArray[i].length; j++){
results.push(procArray[i][j]);
}
}
function processBatch(masterList, batchSize, procArray){
return Promise.all(masterList.splice(0, batchSize).map(async url =>
{
return singleScrape(url)
})).then((results) => {
if (masterList.length < batchSize) {
console.log('done');
procArray.push(results);
return procArray;
} else {
console.log('MasterList Size :: ' + masterList.length);
procArray.push(results);
return processBatch(masterList, batchSize, procArray);
}
})
}