I have script tied to Google sheet that collects data from 3 columns (date, userID, value) on the sheet and pass them to object that is then fetched to Firebase DB.
The script correctly push the object into Firebase but I have troubles to achieve structure of data object that I need (so the nodes are correctly structured in Firebase). In the “date” column in the sheet I have date under which I want to group userID:value pair but when I iterate over rows in the sheet it only stores last userID:value pair with given date and not all.
My code:
function writeValues() {
// Database reference
const databaseURL = "https://sampledatabaseURL.firebaseio.com/"
const token = ScriptApp.getOAuthToken();
// Get spreadsheet, range of data and all values within
var spreadSheet = SpreadsheetApp.getActiveSpreadsheet();
var sheet = spreadSheet.getSheetByName("test");
const range = sheet.getDataRange();
const allValues = range.getValues();
// Get indexes for headers in columns
const headers = {};
allValues[0].forEach(function (value, index) {
headers[value] = index
});
const dbData = {};
allValues.forEach(function (row, index) {
if (index === 0) { return } // skip header row
dbData[row[headers.date]] = {
[row[headers.userID]] : row[headers.value]
}
})
const url = databaseURL + "/test.json?access_token=" + encodeURIComponent(token)
const response = UrlFetchApp.fetch(url, {
method: 'put',
payload: JSON.stringify(dbData)
})
Logger.log(response.getResponseCode());
}
Output JSON file it produces and that’s fetched to Firebase:
{
"20180118" : {"userID2" : "value2"},
"20200705" : {"userID4" : "value4"},
"20210324" : {"userID6" : "value6"}
}
Desired output structure:
{
"20180118" : {"userID1" : "value1", "userID2" : "value2"},
"20200705" : {"userID3" : "value3", "userID4" : "value4"},
"20210324" : {"userID5" : "value5", "userID6" : "value6"}
}
I believe your goal as follows.
You want to achieve the following situation.
From: This image is from your question.
To: You want to retrieve the following output as the value of dbData.
{
"20180118" : {"userID1" : "value1", "userID2" : "value2"},
"20200705" : {"userID3" : "value3", "userID4" : "value4"},
"20210324" : {"userID5" : "value5", "userID6" : "value6"}
}
In this case, how about the following modidfication? When your script is modified, it becomes as follows.
From:
const allValues = range.getValues();
// Get indexes for headers in columns
const headers = {};
allValues[0].forEach(function (value, index) {
headers[value] = index
});
const dbData = {};
allValues.forEach(function (row, index) {
if (index === 0) { return } // skip header row
dbData[row[headers.date]] = {
[row[headers.userID]] : row[headers.value]
}
})
To:
const [, ...values] = range.getValues();
const dbData = values.reduce((o, [a,b,c]) => {
if (!o[a]) o[a] = {};
o[a][b] = c;
return o;
}, {});
console.log(dbData)
When above modified script is run for your sample Spreadsheet, the value of dbData is the output you expect.
Reference:
reduce()
Related
Is there a function in the API of IFCJS to get the guid of the elements grouped in an ifcgroup?
for example, if I group a column with a wall
getElementsFromIfcGroup(guidGroup) ---> return [guidWall, guidColumn]
According to the IFC schema, IfcGroup instances group elements together using an indirect relationship object called IfcRelAssignsToGroup. This means that you can retrieve the elements contained within that group like this:
import { IFCRELASSIGNSTOGROUP as REL } from 'web-ifc';
async function getItemsOfGroup(modelID, groupID) {
const manager = ifcLoader.ifcManager;
const relIDs = await manager.getAllItemsOfType(modelID, REL);
for(relID of groupsIDs) {
const groupRel = await manager.getItemProperties(modelID, relID);
if(groupRel.RelatingGroup.value === groupID) {
return groupRel.RelatedObjects;
}
}
return [];
}
based on Antonio's answer, it looks like this:
async function getItemsOfGroup(modelID, groupID) {
const manager = viewer.IFC.loader.ifcManager
// Get all ifcgroups
const relIDs = await manager.getAllItemsOfType(modelID, IFCRELASSIGNSTOGROUP);
let relID, relObj, props;
var guIDs = [];
for(relID of relIDs) {
const groupRel = await manager.getItemProperties(modelID, relID);
// Find the groupID
if(groupRel.GlobalId.value === groupID) {
// Search all related objects
for(relObj of groupRel.RelatedObjects) {
//get object properties
props = await manager.getItemProperties(modelID, relObj.value);
//Add guid to array
guIDs[guIDs.length] = props.GlobalId.value;
}
return guIDs;
}
}
return guIDs;
}
I have been trying to use the riot games api to compute all the previous custom games and then find the win loss streaks for individual players, I have built the following code to grab matches for a particular user.
See https://github.com/FriendlyUser/deno-riot-games-custom-games
But I feel like the riot games api is only returning data with its v4 api up to season 11, if anyone could clarify how the api works or explain how I could possibly get more data, that would be fantastic.
import { writeJson } from "https://deno.land/std/fs/mod.ts"
import "https://deno.land/x/dotenv/load.ts"
const player_id = Deno.env.get('ACCOUNT_ID')
const region_url = 'https://na1.api.riotgames.com'
let riot_URL = new URL(`${region_url}/lol/match/v4/matchlists/by-account/${player_id}`)
enum HTTP {
GET = 'GET',
POST = 'POST',
PUT = 'PUT',
DELETE = 'DELETE'
}
interface MatchlistDto {
startIndex: number
totalGames: number
endIndex: number
matches: Array<any>
}
function makeFetchOptions(
riotKey = Deno.env.get('RIOT_API_KEY'),
method: HTTP = HTTP.GET
): object {
return {
method: method,
headers: {
"Accept-Charset": "application/x-www-form-urlencoded; charset=UTF-8",
"Accept-Language": "en-US,en;q=0.9",
'X-Riot-Token': riotKey
}
}
}
function appendMatchHistory(riot_endpoint: string): Promise<MatchlistDto> {
const riotKey = Deno.env.get('RIOT_API_KEY')
console.log(riotKey)
const options = makeFetchOptions(riotKey)
return fetch(riot_endpoint, options)
.then( (resp: any) => {
console.log(resp)
return resp.json()
})
.then( (matchData: MatchlistDto) => {
return matchData
})
}
const max_iterations = 1000
let bIndex = 0
let eIndex = 100
let current_url = riot_URL
let riot_endpoint = null
let allMatches = []
let customGames = []
const sleep = (milliseconds: number) => {
return new Promise(resolve => setTimeout(resolve, milliseconds))
}
for (let i = 0; i < max_iterations; i++) {
console.log(`beginIndex: ${bIndex} endIndex: ${eIndex}`)
riot_endpoint = current_url.toString()
const newMatches = await appendMatchHistory(riot_endpoint)
await sleep(1500)
current_url.searchParams.delete('beginIndex')
current_url.searchParams.delete('endIndex')
const {matches} = newMatches
if (matches.length == 0) {
console.log(`ENDING SCRIPT AT ${eIndex} with ${matches.length}`)
break
}
// startIndex becomes endIndex
bIndex = eIndex
eIndex = eIndex + 100
allMatches.push(newMatches.matches)
// get new url
current_url.searchParams.append('beginIndex', String(bIndex))
current_url.searchParams.append('endIndex', String(eIndex))
}
await writeJson(
"./allData.json",
allMatches
);
Sorry if this answer is late. But yes the Riot API is only for "current" data, and that is why sites like U.GG, OP.GG, etc actually run scripts to store data continuously. So to get statistics you would have to write scripts to store it into your own DB over time.
Sadly, there is no way to get previous season data
I created an item in dynamodb using Node js, the item has multiple attributes such as brand, category, discount, validity, etc. I am using uuid to generate ids for each item. Now let's say I want to update the validity attribute of the item, in which case I am currently sending the entire json object with the value of validity modified to the new value.
This is definitely not optimal, please help me find an optimal solution.
const params = {
TableName: process.env.PRODUCT_TABLE,
Key: {
id: event.pathParameters.id,
},
ExpressionAttributeNames: {
'#discount': 'discount',
},
ExpressionAttributeValues: {
':brand': data.brand,
':category': data.category,
':discount': data.discount,
':denominations': data.denominations,
":validity": data.validity,
":redemption": data.redemption
},
UpdateExpression: 'SET #discount = :discount, denominations = :denominations, brand = :brand, category = :category, validity = :validity, redemption = :redemption',
ReturnValues: 'ALL_NEW',
};
I want to send just the attribute I want to update with the new value, if I want to change the validity from 6 months to 8 months, I should just send something like:
{
"validity": "8 months"
}
And it should update the validity attribute of the item.
Same should apply to any other attribute of the item.
'use strict';
const AWS = require('aws-sdk');
const dynamoDb = new AWS.DynamoDB.DocumentClient();
module.exports.update = (event, context, callback) => {
const data = JSON.parse(event.body);
let attr = {};
let nameobj = {};
let exp = 'SET #';
let arr = Object.keys(data);
let attrname = {};
arr.map((key) => {attr[`:${key}`]=data[key]});
arr.map((key) => {
exp += `${key} = :${key}, `
});
arr.map((key) => {nameobj[`#${key}`]=data[key]});
attrname = {
[Object.keys(nameobj)[0]] : nameobj[Object.keys(nameobj)[0]]
}
const params = {
TableName: process.env.PRODUCT_TABLE,
Key: {
id: event.pathParameters.id,
},
ExpressionAttributeNames: attrname,
ExpressionAttributeValues: attr,
UpdateExpression: exp,
ReturnValues: 'ALL_NEW',
};
// update the todo in the database
dynamoDb.update(params, (error, result) => {
// handle potential errors
if (error) {
console.error(error);
callback(null, {
statusCode: error.statusCode || 501,
headers: { 'Content-Type': 'text/plain' },
body: 'Couldn\'t update the card',
});
return;
}
// create a response
const response = {
statusCode: 200,
body: JSON.stringify(result.Attributes),
};
callback(null, response);
});
};
Contrary to others comments, this is very possible, use the UpdateItem action.
Language agnostic API docs
JavaScript specific API docs
If you want to dynamically create the query, try something like this:
const generateUpdateQuery = (fields) => {
let exp = {
UpdateExpression: 'set',
ExpressionAttributeNames: {},
ExpressionAttributeValues: {}
}
Object.entries(fields).forEach(([key, item]) => {
exp.UpdateExpression += ` #${key} = :${key},`;
exp.ExpressionAttributeNames[`#${key}`] = key;
exp.ExpressionAttributeValues[`:${key}`] = item
})
exp.UpdateExpression = exp.UpdateExpression.slice(0, -1);
return exp
}
let data = {
'field' : { 'subfield': 123 },
'other': '456'
}
let expression = generateUpdateQuery(data)
let params = {
// Key, Table, etc..
...expression
}
console.log(params)
Output:
{
UpdateExpression: 'set #field = :field, #other = :other',
ExpressionAttributeNames: {
'#field': 'field',
'#other': 'other'
},
ExpressionAttributeValues: {
':field': {
'subfield': 123
},
':other': '456'
}
}
Using Javascript SDK V3:
Import from the right package:
import { DynamoDBClient PutItemCommandInput, UpdateItemCommandInput, UpdateItemCommand } from '#aws-sdk/client-dynamodb';
Function to dynamically do partial updates to the item:
(the code below is typescript can be easily converted to Javascript, just remove the types!)
function updateItem(id: string, item: any) {
const dbClient = new DynamoDBClient({region: 'your-region-here });
let exp = 'set ';
let attNames: any = { };
let attVal: any = { };
for(const attribute in item) {
const valKey = `:${attribute}`;
attNames[`#${attribute}`] = attribute;
exp += `#${attribute} = ${valKey}, `;
const val = item[attribute];
attVal[valKey] = { [getDynamoType(val)]: val };
}
exp = exp.substring(0, exp.length - 2);
const params: UpdateItemCommandInput = {
TableName: 'your-table-name-here',
Key: { id: { S: id } },
UpdateExpression: exp,
ExpressionAttributeValues: attVal,
ExpressionAttributeNames: attNames,
ReturnValues: 'ALL_NEW',
};
try {
console.debug('writing to db: ', params);
const command = new UpdateItemCommand(params);
const res = await dbClient.send(command);
console.debug('db res: ', res);
return true;
} catch (err) {
console.error('error writing to dynamoDB: ', err);
return false;
}
}
And to use it (we can do partial updates as well):
updateItem('some-unique-id', { name: 'some-attributes' });
What i did is create a helper class.
Here is a simple function : Add all the attribute and values that goes into, if the value is null or undefined it won't be in the expression.
I recommande to create a helper class with typescript and add more functions and other stuff like generator of expressionAttributeValues , expressionAttributeNames ... , Hope this help.
function updateExpression(attributes, values) {
const expression = attributes.reduce((res, attribute, index) => {
if (values[index]) {
res += ` #${attribute}=:${attribute},`;
}
return res;
}, "SET ");
return expression.slice(0, expression.length - 1)
}
console.log(
updateExpression(["id", "age", "power"], ["e8a8da9a-fab0-55ba-bae3-6392e1ebf624", 28, undefined])
);
You can use code and generate the params object based on the object you provide. It's just a JavaScript object, you walk through the items so that the update expression only contains the fields you have provided.
This is not really a DynamoDB question in that this is more a general JS coding question.
You can use UpdateItem; to familiarize yourself with DynamoDb queries I would suggest you DynamoDb NoSQL workbench:
https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/workbench.settingup.html
It can generate snippets for you based on your queries.
DynamoDb NoSQL workbench screenshot query
This question already has answers here:
Firebase search by child value
(2 answers)
Closed 3 years ago.
{
"users" : {
"1320104182" : {
"datas" : {
"bio" : "some_data",
"picture" : "some_data",
"username" : "some_data",
"website" : "some_data",
"followers" : 14,
}
},
"3271376571" : {
"datas" : {
"bio" : "some_data",
"picture" : "some_data",
"username" : "some_data",
"website" : "some_data",
"followers" : 10,
}
}
}
}
I'm new to Firebase and i'm trying to do multiple think here without any success so far.
How can i retrieve a user by his "username" without knowing the key ?
Or how can i order the users by followers ?
I tried everything i could find in the documentation for a few hours i'm desperate.
This seems fairly easy:
var ref = firebase.database().ref("users");
var query = ref.orderByChild("database/username").equalTo("some_data");
query.once("value", function(snapshot) {
snapshot.forEach(function(child) {
console.log(child.key, child.val().bio);
});
});
Firebase is not so good when you need fancy queries. You must handle everything in your client (JavaScript) which is not the best approach when dealing with large data. In this case, I'd suggest you something like this:
const nameToSearch = 'John';
firebase.ref('users').once('value') //get all content from your node ref, it will return a promise
.then(snapshot => { // then get the snapshot which contains an array of objects
snapshot.val().filter(user => user.name === nameToSearch) // use ES6 filter method to return your array containing the values that match with the condition
})
To order by followers, you can either also apply sort() (see example 1) or any of firebase default methods orderByChild() (see example 2), orderByKey (see example 3), or orderByValue (see example 4)
Example 1:
firebase.database().ref("users").once('value')
.then(snapshot => {
const sortedUsers = snapshot.sort((userA, userB) => {
if (userA.name < userB.name) {
return -1;
}
if (userA.name > userB.name) {
return 1;
}
return 0;
})
})
Example 2:
var ref = firebase.database().ref("dinosaurs");
ref.orderByChild("height").on("child_added", function(snapshot) {
console.log(snapshot.key + " was " + snapshot.val().height + " m tall");
});
Example 3:
var ref = firebase.database().ref("dinosaurs");
ref.orderByKey().on("child_added", function(snapshot) {
console.log(snapshot.key);
});
Example 4:
var scoresRef = firebase.database().ref("scores");
scoresRef.orderByValue().limitToLast(3).on("value", function(snapshot) {
snapshot.forEach(function(data) {
console.log("The " + data.key + " score is " + data.val());
});
});
Note: there might be typos in the examples, I wrote just to show you the idea of the concepts.
Check the following docs for more info:
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/filter
https://firebase.google.com/docs/reference/js/firebase.database.Query#orderByChild
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/sort
Hope it helps
I'm a newbie to the field of javascript/angularJS, so please bear with me.I need a way to convert .xls/.xlsx files into .zip files by using jsZip library. I'm making use of alasql for generating the .xls file. I've looked all over for any possible solutions to create zip file of all xls files, but haven't come across any demo. (.txt and .doc files generate just fine, but .xls files does not open if jsZip is used). Any help would be appreciated!!
What I need is an xls file to be generated dynamically, and the same file to be compressed as zip
EDIT :-
Here's some of the code which I tried (but with no success)
var newExcelData = {'Name':'abc'};
//var res = alasql("SELECT * INTO XLSX('Summary.xlsx',{headers:true}) FROM ? ", [newExcelData]);
var zip = new JSZip();
zip.file(alasql("SELECT * INTO XLSX('Summary.xlsx',{headers:true}) FROM ? ", [newExcelData]));
zip.generateAsync({ type: "blob" })
.then(function (content) {
saveAs(content, "example.zip");
});
PS:- I'm able to make it work in case of generating .xls file.
Please refer below code:-
var newExcelData = {'Name':'abc', 'Age':'12'};
var zip = new JSZip();
zip.file("test.xls", [newExcelData]);
zip.generateAsync({ type: "blob" })
.then(function (content) {
saveAs(content, "example.zip");
});
But although excel sheet is generated, on opening excel sheet is blank.
Please help!!
Hi, here's an update :-
I've tried to make use of js-xlsx library - https://github.com/SheetJS/js-xlsx - to generate xls file and then zip it. Please refer the below code..
function Create_Zip() {
function datenum(v, date1904) {
if (date1904) v += 1462;
var epoch = Date.parse(v);
return (epoch - new Date(Date.UTC(1899, 11, 30))) / (24 * 60 * 60 * 1000);
}
function sheet_from_array_of_arrays(data, opts) {
var ws = {};
var range = { s: { c: 10000000, r: 10000000 }, e: { c: 0, r: 0 } };
for (var R = 0; R != data.length; ++R) {
for (var C = 0; C != data[R].length; ++C) {
if (range.s.r > R) range.s.r = R;
if (range.s.c > C) range.s.c = C;
if (range.e.r < R) range.e.r = R;
if (range.e.c < C) range.e.c = C;
var cell = { v: data[R][C] };
if (cell.v === null) continue;
var cell_ref = XLSX.utils.encode_cell({ c: C, r: R });
if (typeof cell.v === 'number') cell.t = 'n';
else if (typeof cell.v === 'boolean') cell.t = 'b';
else if (cell.v instanceof Date) {
cell.t = 'n'; cell.z = XLSX.SSF._table[14];
cell.v = datenum(cell.v);
}
else cell.t = 's';
ws[cell_ref] = cell;
}
}
if (range.s.c < 10000000) ws['!ref'] = XLSX.utils.encode_range(range);
return ws;
}
var data = [[1, 2, 3], [true, false, null, "sheetjs"], ["foo", "bar", new Date("2014-02-19T14:30Z"), "0.3"], ["baz", null, "qux"]];
var ws_name = "SheetJS";
function Workbook() {
if (!(this instanceof Workbook)) return new Workbook();
this.SheetNames = [];
this.Sheets = {};
}
var wb = new Workbook(), ws = sheet_from_array_of_arrays(data);
/* add worksheet to workbook */
wb.SheetNames.push(ws_name);
wb.Sheets[ws_name] = ws;
var wbout = XLSX.write(wb, { bookType: 'xlsx', bookSST: true, type: 'binary' });
function s2ab(s) {
var buf = new ArrayBuffer(s.length);
var view = new Uint8Array(buf);
for (var i = 0; i != s.length; ++i) view[i] = s.charCodeAt(i) & 0xFF;
return buf;
}
var jsonse = JSON.stringify([s2ab(wbout)]);
var testblob = new Blob([jsonse], { type: "application/json" });
console.log(testblob);
var zip = new JSZip();
zip.file("trial.xls", testblob);
var downloadFile = zip.generateAsync({ type: "blob" });
saveAs(downloadFile, 'test.zip');
}
But, the problem here is that I keep getting this error: 'The data of 'trial.xls' is in an unsupported format !' in the console :(. Is there any way I can make this work?
I'm at my wits end now :(
Not an answer (see below) but an explanation of what's going on:
To add a file, JSZip needs its binary content (as Blob, Uint8Array, etc). The line zip.file("test.xls", [newExcelData]); can't work for example: [newExcelData] is not a binary content but an array of js object.
What you need to figure out is how to get the content of the xlsx file. SELECT * INTO XLSX('Summary.xlsx') will trigger a download and return 1, it's not what you want. I searched on my side but can't find a way to do it with alasql.
Once/if you find the solution, the JSZip part looks correct.
Edit, following your switch to js-xlsx:
You use JSZip v2 (needed by js-xlsx) which doesn't support Blob inputs. However, wbout is a binary string which is supported:
zip.file("trial.xls", wbout, {binary: true});
Then, replace zip.generateAsync (added in JSZip v3):
var downloadFile = zip.generate({type: "blob" });
saveAs(downloadFile, 'test.zip');
Here is the solution I found using JSZip, XLSX and File Saver libraries.
Import:
import * as XLSX from "xlsx";
import JSZip from 'jszip';
import { saveAs } from 'file-saver';
Here is an example of compressing a .xlsx inside a zip:
let zip = new JSZip();
const jsonData = [
{
"Product": "Red Velvet Cupcake",
"Price": "6",
"GluttenFree": "Yes",
},
{
"Product": "Cheesecake",
"Price": "15",
"GluttenFree": "No",
}
];
const workBook: XLSX.WorkBook = XLSX.utils.book_new();
const workSheet: XLSX.WorkSheet = XLSX.utils.json_to_sheet(jsonData);
XLSX.utils.book_append_sheet(workBook, workSheet, 'Bakery');
const workBookBuffer = XLSX.write(workBook, { bookType: 'xlsx', type: 'array' });
const fileData = new Blob([workBookBuffer], {type: 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet;charset=UTF-8'});
zip.file('Products.xlsx', fileData);
zip.generateAsync({type:"blob"}).then(function (blob) {
saveAs(blob, "WorkBooks.zip");
});
This code generates a zip file named 'WorkBooks.zip' that contains the file 'Products.xlsx'. This is how the excel looks like:
Some file-saver examples: https://www.tabnine.com/code/javascript/modules/file-saver.
Here is the JSZip method used:
https://stuk.github.io/jszip/documentation/api_jszip/file_data.html