AppSpreadsheet (GAS): avoid some problems with sistematic tested data - asynchronous

In my current job with spreadsheet, all inserted data passes through a test, checking if the same value is found on the same index in other sheets. Failing, a caution message is put in the current cell.
//mimimalist algorithm
function safeInsertion(data, row_, col_)
{
let rrow = row_ - 1; //range row
let rcol = col_ - 1; // range col
const active_sheet_name = getActiveSheetName(); // do as the its name suggest
const all_sheets = SpreadsheetApp.getActiveSpreadsheet().getSheets();
//test to evaluate the value to be inserted in the sheet
for (let sh of all_sheets)
{
if (sh.getName() === active_sheet_name)
continue;
//getSheetValues do as its name suggest.
if( getSheetValues(sh)[rrow][rcol] === data )
return "prohibited insertion"
}
return data;
}
// usage (in cell): =safeInsertion("A scarce data", ROW(), COLUMN())
The problems are:
cached values confuse me sometimes. The script or data is changed but not perceived by the sheet itself until renewing manually the cell's content or refreshing all table. Is there any relevant configuration available to this issue?
Sometimes, at loading, a messing result appears. Almost all data are prohibited, for example (originally, all was fine!).
What can I do to obtain a stable sheet using this approach?
PS: The original function does more testing on each data insertion. Those tests consist on counting the frequency in the actual sheet and in all sheets.
EDIT:
In fact, I can't create a stable sheet. For test, a let you a copy of my code with minimal adaptations.
function safelyPut(data, max_onesheet, max_allsheet, row, col)
{
// general initialization
const data_regex = "\^\s*"+data+"\s*$"
const spreadsheet = SpreadsheetApp.getActiveSpreadsheet();
const activesheet = spreadsheet.getActiveSheet();
const active_text_finder = activesheet.createTextFinder(data_regex)
.useRegularExpression(true)
.matchEntireCell(true);
const all_text_finder = spreadsheet.createTextFinder(data_regex)
.useRegularExpression(true)
.matchEntireCell(true);
const all_occurrences = all_text_finder.findAll();
//test general data's environment
const active_freq = active_text_finder.findAll().length;
if (max_onesheet <= active_freq)
return "Too much in a sheet";
const all_freq = all_occurrences.length;
if (max_allsheet <= all_freq)
return "Too much in the work";
//test unicity in a position
const active_sname = activesheet.getName();
for (occurrence of all_occurrences)
{
const sname = occurrence.getSheet().getName();
//if (SYSTEM_SHEETS.includes(sname))
//continue;
if (sname != active_sname)
if (occurrence.getRow() == row && occurrence.getColumn() == col)
if (occurrence.getValue() == data)
{
return `${sname} contains same data with the same indexes.`;
};
}
return data;
}
Create two or three cells and put randomly in a short range short range a value following the usage
=safeInsertion("Scarce Data", 3; 5; ROW(), COLUMN())
Do it, probably you will get a unstable sheet.

About cached values confuse me sometimes. The script is changed but not perceived by the sheet until renewing manually the cell's content or refreshing all table. No relevant configuration available to this issue?, when you want to refresh your custom function of safeInsertion, I thought that this thread might be useful.
About Sometimes, at loading, a messing result appears. Almost all data are prohibited, for example (originally, all was fine!). and What can I do to obtain a stable sheet using this approach?, in this case, for example, how about reducing the process cost of your script? I thought that by reducing the process cost of the script, your situation might be a bit stable.
When your script is modified by reducing the process cost, how about the following modification?
Modified script:
function safeInsertion(data, row_, col_) {
const ss = SpreadsheetApp.getActiveSpreadsheet();
const range = ss.createTextFinder(data).matchEntireCell(true).findNext();
return range && range.getRow() == row_ && range.getColumn() == col_ && range.getSheet().getSheetName() != ss.getActiveSheet().getSheetName() ? "prohibited insertion" : data;
}
The usage of this is the same with your current script like =safeInsertion("A scarce data", ROW(), COLUMN()).
In this modification, TextFinder is used. Because I thought that when the value is searched from all sheets in a Google Spreadsheet, TextFinder is suitable for reducing the process cost.
References:
createTextFinder(findText) of Class Spreadsheet
findNext()

Related

Scrape Highchart, missing data

I've been trying to scrape a specific highchart, using console commands, something in the line off:
data = $('div#graphCont2').highcharts().series[0].data; { console.log(data)}
This code works on the following site, I retrieve all data.
test-hichart1
However, when I rework the code for the graph I intend to scrape (chart, Its the uppermost chart, APX-PSE for all X and Y entries), I miss data. It varies somehow (based on the timestamps, it seems to vary by the selected period), but I only get data from around timestamp 1562284800000 and onwards when the period is set to "all" (thus missing 2/3 of all entries).
I use this code:
data = $('div#stockchart_apx').highcharts().series[0].data; { console.log(data) }
My idea was to use a console.table to get the info I need, though I'm unsure if the table is usable past 999 entries anyway.
Does anyone have an idea of why the readout fluctuates and how I can retrieve all the information?
Thanks!
EDIT~ so, after a couple more hours, I managed to get all data by opening the graph in full-window mode. I'm unsure to where the differences originate from, but it worked. I scraped the data with:
data = $('div#stockchart_apx').highcharts().series[0].data;
const getCircularReplacer1 = () => {
const seen = new WeakSet();
return (key, value) => {
if (typeof value === "object" && value !== null) {
if (seen.has(value)) {
return;
}
seen.add(value);
}
return value;
};
};
JSON.stringify(data, getCircularReplacer1());

IMPORTDATA not grabbing live data from XML

I'm using Google Sheet's IMPORTDATA function to grab information from an XML file that is pulling from an API but the information I pull into the sheet isn't up to date.
How can I modify my sheet to pull in up-to-date data?
Compare the sheet: https://docs.google.com/spreadsheets/d/1W0Bt5z-Tky-tNhG_JtfE4FfjTRgQNRu_eQu2qVhQ-_E/edit?usp=sharing (LiveScores sheet)
To the XML: https://www67.myfantasyleague.com/2019/export?TYPE=liveScoring&L=64741&APIKEY=&W=14&DETAILS=1&JSON=0
Observe franchise id="0015" in both sets of data.
The sheet states <franchise id="0005" score="0.00" gameSecondsRemaining="21600" playersYetToPlay="6" playersCurrentlyPlaying="0" isHome="0">
The XML states <franchise id="0015" score="11.14" gameSecondsRemaining="20004" playersYetToPlay="4" playersCurrentlyPlaying="2"> (This data is for a football game that is currently being played as I'm writing this so the above example may not be exact, but it WON'T be score of 0.00, for example.
Any help would be amazing, thanks!
Have you tried using IMPORTXML? Google Sheets IMPORTXML Page
In IMPORTXML, you can just use the Inspect Element feature to pull the xpath.
Hope this helps. Let me know if I can help further.
Edit: Instructions To Change When Data Is Imported
In the toolbar go to the script editor
Now in the scripts, paste the code listed bellow
/**
* Go through all sheets in a spreadsheet, identify and remove all spreadsheet
* import functions, then replace them a while later. This causes a "refresh"
* of the "import" functions. For periodic refresh of these formulas, set this
* function up as a time-based trigger.
*
* Caution: Formula changes made to the spreadsheet by other scripts or users
* during the refresh period COULD BE OVERWRITTEN.
*
* From: https://stackoverflow.com/a/33875957/1677912
*/
function RefreshImports() {
var lock = LockService.getScriptLock();
if (!lock.tryLock(5000)) return; // Wait up to 5s for previous refresh to end.
// At this point, we are holding the lock.
var id = "YOUR-SHEET-ID";
var ss = SpreadsheetApp.openById(id);
var sheets = ss.getSheets();
for (var sheetNum=0; sheetNum<sheets.length; sheetNum++) {
var sheet = sheets[sheetNum];
var dataRange = sheet.getDataRange();
var formulas = dataRange.getFormulas();
var tempFormulas = [];
for (var row=0; row<formulas.length; row++) {
for (col=0; col<formulas[0].length; col++) {
// Blank all formulas containing any "import" function
// See https://regex101.com/r/bE7fJ6/2
var re = /.*[^a-z0-9]import(?:xml|data|feed|html|range)\(.*/gi;
if (formulas[row][col].search(re) !== -1 ) {
tempFormulas.push({row:row+1,
col:col+1,
formula:formulas[row][col]});
sheet.getRange(row+1, col+1).setFormula("");
}
}
}
// After a pause, replace the import functions
Utilities.sleep(5000);
for (var i=0; i<tempFormulas.length; i++) {
var cell = tempFormulas[i];
sheet.getRange( cell.row, cell.col ).setFormula(cell.formula)
}
// Done refresh; release the lock.
lock.releaseLock();
}
}
This snippet of code came from Periodically refresh IMPORTXML() spreadsheet function
Last and definitely the least, replace the "YOUR-SHEET-ID"
NOTE: I have not personally tested this code and I cannot vouch for it. I recommend making a copy and testing it there first.
Hopefully, this solves the issue of your data not being imported as often as you want. If you want to manually get "fresh" data, you can just delete/cut the import function and paste it back in.
try in A2:
=ARRAYFORMULA(IFNA(VLOOKUP(C2:C, PlayerList!A:F, {2, 6}, 0)))
and in C2:
=ARRAYFORMULA(QUERY(REGEXEXTRACT(QUERY(IMPORTDATA(
"https://www67.myfantasyleague.com/2019/export?TYPE=liveScoring&L=64741&APIKEY=&W=14&DETAILS=1&JSON=0?273"),
"where Col1 contains 'player id'", 0),
"(player id=""(\d+)).+?(score=""(\d+.\d+))"),
"select Col2,Col4"))
spreadsheet demo

Google Sheets: delete rows containing specified data

I'm new to Java scripting and Google Apps Scripts so i am sorry if this has already been answered. I was not able to find what i was looking for over the last few months of working on this project.
I am working on a variant of the scripts here:
Delete row in Google Sheets if certain "word" is found in cell
AND
Google Sheet Script - Find Value in Col and Delete Row
I want to create a button, or menu, that will allow someone to enter specific data, and have each row in the spreadsheet containing that data deleted.
I have a test sheet here that illustrates the data i'm working with, formulas i'm using, and has the beginning of the script attached to it:
https://docs.google.com/spreadsheets/d/1e2ILQYf8MJD3mrmUeFQyET6lOLYEb-4coDTd52QBWtU/edit?usp=sharing
The first 4 sheets are pulling data from the "Form Responses 1" sheet via a formula in cell A:3 in each sheet so the data would only need to be deleted from the "Form Responses 1" sheet to clear it from the rest of the sheets.
I tried working this in but i do not think i am on the right track.
https://developers.google.com/apps-script/guides/dialogs
I also posted this on Google Docs Help Forum 60 days ago, but have not received any responses.
Any help would be greatly appreciated.
There's a few steps. For usability of UI this takes a little longer code. In concise form:
The user activates a dialog and enters a string.
Rows w/ the string are deleted (with error handling and confirmation)
(Hopefully this gets you started and you can tailor it to your needs)
Function that initiates the menu:
function onOpen(){
SpreadsheetApp.getUi()
.createMenu('My Menu')
.addItem('Delete Data', 'deleteFunction')
.addToUi();
}
The main workhorse:
function deleteFunction(){
//declarations
var sheetName = "Form Responses 1";
var ss = SpreadsheetApp.getActive();
var sheet = ss.getSheetByName(sheetName);
var dataRange = sheet.getDataRange();
var numRows = dataRange.getNumRows();
var values = dataRange.getValues();
var delete_string = getUIstring();//open initial UI, save value
if (delete_string.length < 3) return shortStringError()//UI to protect your document from an accidental entry of a very short string.
//removing the rows (start with i=2, so don't delete header row.)
var rowsDeleted = 0;
for (var i = 2; i <= numRows; i++){
var rowValues = values[i-1].toString();//your sheet has various data types, script can be improved here to allow deleting dates, ect.
if (rowValues.indexOf(delete_string) > -1){
sheet.deleteRow(i - rowsDeleted);//keeps loop and sheet in sync
rowsDeleted++;
}
}
postUIconfirm(rowsDeleted);//Open confirmation UI
}
Isolated UI functions to help make above function more concise:
function getUIstring(){
var ui = SpreadsheetApp.getUi();
var response = ui.prompt("Enter the target data element for deletion")
return response.getResponseText()
}
function postUIconfirm(rowsDeleted){
var ui = SpreadsheetApp.getUi();
ui.alert("Operation complete. There were "+rowsDeleted+" rows deleted.")
}
function shortStringError(){
var ui = SpreadsheetApp.getUi();
ui.alert("The string is too short. Enter a longer string to prevent unexpected deletion")
}
I'll just show a way to delete the cell value if it matches your search criteria. It's up to you to connect it to buttons ,etc.
You'll loop through a Sheet Range. When you find the word match, delete it using clearContent()
function deleteSpecificData() {
var ss = SpreadsheetApp.getActiveSpreadsheet();
var sheet = ss.getSheets()[0];
var range = sheet.getRange("Sheet1!A1:C4");
var values = range.getValues();
var numArray = [1,2,3,4,5,6,7,8,9];
var deleteItem = "Garen";
Logger.log(range);
for(var i=0; i< values.length; i++){
for(var j=0; j<values[i].length; j++){
if(values[i][j] == deleteItem){
var row = numArray[i];
var col = numArray[j];
var range = sheet.getRange(row,col).clearContent();
}
}
}
}
Before:
After:

DocumentDB Change Feed and saving Checkpoint

After reading the documentation, I'm having a hard time conceptualizing the change feed. Let's take the code from the documentation below. The second change feed is picking up the changes from the last time it was run via the checkpoints. Let's say it is being used to create summary data and there was an issue and it needed to be re-run from a prior time. I don't understand the following:
How to specify a particular time the checkpoint should start. I understand I can save the checkpoint dictionary and use that for each run, but how do you get the changes from X time to maybe rerun some summary data
Secondly, let's say we are rerunning some summary data and we save the last checkpoint used for each summarized data so we know where that one left off. How does one know that a record is in or before that checkpoint?
Code that runs from collection beginning and then from last checkpoint:
Dictionary < string, string > checkpoints = await GetChanges(client, collection, new Dictionary < string, string > ());
await client.CreateDocumentAsync(collection, new DeviceReading {
DeviceId = "xsensr-201", MetricType = "Temperature", Unit = "Celsius", MetricValue = 1000
});
await client.CreateDocumentAsync(collection, new DeviceReading {
DeviceId = "xsensr-212", MetricType = "Pressure", Unit = "psi", MetricValue = 1000
});
// Returns only the two documents created above.
checkpoints = await GetChanges(client, collection, checkpoints);
//
private async Task < Dictionary < string, string >> GetChanges(
DocumentClient client,
string collection,
Dictionary < string, string > checkpoints) {
List < PartitionKeyRange > partitionKeyRanges = new List < PartitionKeyRange > ();
FeedResponse < PartitionKeyRange > pkRangesResponse;
do {
pkRangesResponse = await client.ReadPartitionKeyRangeFeedAsync(collection);
partitionKeyRanges.AddRange(pkRangesResponse);
}
while (pkRangesResponse.ResponseContinuation != null);
foreach(PartitionKeyRange pkRange in partitionKeyRanges) {
string continuation = null;
checkpoints.TryGetValue(pkRange.Id, out continuation);
IDocumentQuery < Document > query = client.CreateDocumentChangeFeedQuery(
collection,
new ChangeFeedOptions {
PartitionKeyRangeId = pkRange.Id,
StartFromBeginning = true,
RequestContinuation = continuation,
MaxItemCount = 1
});
while (query.HasMoreResults) {
FeedResponse < DeviceReading > readChangesResponse = query.ExecuteNextAsync < DeviceReading > ().Result;
foreach(DeviceReading changedDocument in readChangesResponse) {
Console.WriteLine(changedDocument.Id);
}
checkpoints[pkRange.Id] = readChangesResponse.ResponseContinuation;
}
}
return checkpoints;
}
DocumentDB supports check-pointing only by the logical timestamp returned by the server. If you would like to retrieve all changes from X minutes ago, you would have to "remember" the logical timestamp corresponding to the clock time (ETag returned for the collection in the REST API, ResponseContinuation in the SDK), then use that to retrieve changes.
Change feed uses logical time in place of clock time because it can be different across various servers/partitions. If you would like to see change feed support based on clock time (with some caveats on skew), please propose/upvote at https://feedback.azure.com/forums/263030-documentdb/.
To save the last checkpoint per partition key/document, you can just save the corresponding version of the batch in which it was last seen (ETag returned for the collection in the REST API, ResponseContinuation in the SDK), like Fred suggested in his answer.
How to specify a particular time the checkpoint should start.
You could try to provide a logical version/ETag (such as 95488) instead of providing a null value as RequestContinuation property of ChangeFeedOptions.

SQLite storage API Insert statement freezes entire firefox in bootstrapped(Restartless) AddOn

Data to be inserted has just two TEXT columns whose individual length don't even exceed 256.
I initially used executeSimpleSQL since I didn't need to get any results.
It worked for simulataneous inserts of upto 20K smoothly i.e. in the bakground no lag or freezing observed.
However, with 0.1 million I could see horrible freezing during insertion.
So, I tried these two,
Insert in chunks of 500 records - This didn't work well since even for 20K records it showed visible freezing. I didn't even try with 0.1million.
So, I decided to go async and used executeAsync alongwith Bind etc. This also shows visible freezing for just 20K records. This was the whole array being inserted and not in chunks.
var dirs = Cc["#mozilla.org/file/directory_service;1"].
getService(Ci.nsIProperties);
var dbFile = dirs.get("ProfD", Ci.nsIFile);
var dbService = Cc["#mozilla.org/storage/service;1"].
getService(Ci.mozIStorageService);
dbFile.append('mydatabase.sqlite');
var connectDB = dbService.openDatabase(dbFile);
let insertStatement = connectDB.createStatement('INSERT INTO my_table
(my_col_a,my_col_b) VALUES
(:myColumnA,:myColumnB)');
var arraybind = insertStatement.newBindingParamsArray();
for (let i = 0; i < my_data_array.length; i++) {
let params = arraybind.newBindingParams();
// Individual elements of array have csv
my_data_arrayTC = my_data_array[i].split(',');
params.bindByName("myColumnA", my_data_arrayTC[0]);
params.bindByName("myColumnA", my_data_arrayTC[1]);
arraybind.addParams(params);
}
insertStatement.bindParameters(arraybind);
insertStatement.executeAsync({
handleResult: function(aResult) {
console.log('Results are out');
},
handleError: function(aError) {
console.log("Error: " + aError.message);
},
handleCompletion: function(aReason) {
if (aReason != Components.interfaces.mozIStorageStatementCallback.REASON_FINISHED)
console.log("Query canceled or aborted!");
console.log('We are done inserting');
}
});
connectDB.asyncClose(function() {
console.log('[INFO][Write Database] Async - plus domain data');
});
Also, I seem to get the async callbacks after a long time. Usually, executeSimpleSQL is way faster than this.If I use SQLite Manager Tool extension to open the DB immediately this is what I get ( as expected )
SQLiteManager: Error in opening file mydatabase.sqlite - either the file is encrypted or corrupt
Exception Name: NS_ERROR_STORAGE_BUSY
Exception Message: Component returned failure code: 0x80630001 (NS_ERROR_STORAGE_BUSY) [mozIStorageService.openUnsharedDatabase]
My primary objective was to dump data as big as 0.1 million + and then later on perform reads when needed.

Resources