Scraping from an interactive chart - web-scraping

I would like to retrieve information from charts that shows you the information of a datapoint when you are with the arrow of your mouse on the Y coordinate The link to an image that shows my situation. The red X is where my mouse was pointing.
In my mind I wanted to create a scraper that simulates my mouse moving a pixel after another and each time save the data showed.
Can you help my please?
Thank you very much

They are loading data from https://bitcoinvisuals.com/static/data/data_daily.csv URL as CSV and then passing to the graph
They are using this Javascript code to manipulate data
// Iterate through chunk data
for (i = 0; i < data_csv.length; i++) {
var day = ( data_csv[i].day == "" ? null : new Date(data_csv[i].day).getTime() );
var btc = ( data_csv[i].capacity_total == "" ? null : +data_csv[i].capacity_total );
// If btc exists, try to convert
if (btc) {
var usd = ( data_csv[i].price_btc == "" ? null : (btc * +data_csv[i].price_btc) );
}
// Push to series
if (day) {
// If either exists
if (btc || usd) {
series_btc.addPoint([day, btc], redraw = false)
series_usd.addPoint([day, usd], redraw = false)
}
}
}
Note that capacity_total is the column that exists in that CSV

Related

Google Sheets: How to input timestamp to every changes made in one cell value field?

I’m not an expert in Google Sheets but can anyone can help me on how to put a timestamp on every updates made on one certain cell, please?
Please check link:
Let’s say that B2 is the total amount of B4-B7 and I want to update the Acct1 from 100.00 to 600.00 that auto updates the B2 to 1500.00. My question is, how do I keep track the updates that has been made on B2 i.e., putting timestamps to another sheet or somewhere in the active sheet. Thank you in advance!
I've some people keeping them in notes
function onEdit(e) {
e.range.setNote(e.range.getNote() + '\n' + new Date());
}
Create another sheet for logging. And in onEdit, you can log the changed information to the sheet.
Following is an example code.
function onEdit(e) {
try{
var dataSheetName = "Sheet1";
var logSheetName = "Sheet2";
if ( SpreadsheetApp.getActiveSheet().getName() !== dataSheetName ){ // check it is the target sheet name
return;
}
var range = e.range;
var row = range.getRow();
var col = range.getColumn();
if ( row < 3 || col !== 2 ){ // check it is Amount cell.
return;
}
var spreadSheet = SpreadsheetApp.getActiveSpreadsheet();
var dataSheet = spreadSheet.getSheetByName(dataSheetName);
var logSheet = spreadSheet.getSheetByName(logSheetName);
var nextRow = logSheet.getLastRow() + 1;
// set timestamp
logSheet.getRange(nextRow, 1).setValue(new Date());
// copy data
var srcRange = dataSheet.getRange(row, 1, 1, 2);
var dstRange = logSheet.getRange(nextRow, 2, 1, 2);
dstRange.setValues(srcRange.getValues());
}
catch(e){
Browser.msgBox(e);
}
}
I made an example spreadsheet, feel free to make a copy and check the behavior.
https://docs.google.com/spreadsheets/d/1g0pL7hwuzaS5Yr7Dh7hm_SyRG5TYxwwrlND2Zx3Bo7w/edit?usp=sharing

Scrape Highchart, missing data

I've been trying to scrape a specific highchart, using console commands, something in the line off:
data = $('div#graphCont2').highcharts().series[0].data; { console.log(data)}
This code works on the following site, I retrieve all data.
test-hichart1
However, when I rework the code for the graph I intend to scrape (chart, Its the uppermost chart, APX-PSE for all X and Y entries), I miss data. It varies somehow (based on the timestamps, it seems to vary by the selected period), but I only get data from around timestamp 1562284800000 and onwards when the period is set to "all" (thus missing 2/3 of all entries).
I use this code:
data = $('div#stockchart_apx').highcharts().series[0].data; { console.log(data) }
My idea was to use a console.table to get the info I need, though I'm unsure if the table is usable past 999 entries anyway.
Does anyone have an idea of why the readout fluctuates and how I can retrieve all the information?
Thanks!
EDIT~ so, after a couple more hours, I managed to get all data by opening the graph in full-window mode. I'm unsure to where the differences originate from, but it worked. I scraped the data with:
data = $('div#stockchart_apx').highcharts().series[0].data;
const getCircularReplacer1 = () => {
const seen = new WeakSet();
return (key, value) => {
if (typeof value === "object" && value !== null) {
if (seen.has(value)) {
return;
}
seen.add(value);
}
return value;
};
};
JSON.stringify(data, getCircularReplacer1());

how do I place an input that filters my markers

I have two dropdown menu that allows me to filter markers. I would like to add an input that allows users to type in values for another parameter. the parameter I have contains U.S. zip code, but some of them containers more than one zip code.
For example
A. 48333
B. 49272, 39847
C. 47483, 30494,48333
what I would like is that when user type in the zip code value, only markers that contains this zip code will show. so if I type 48333, point A and C will be shown and B is hidden.
I am not sure what method I should add for this functionality. I don't think I can use
marker.<zipcode value> == <zipcode value> ||
<zipcode value>.length === 0
because the values user enter may not absolutely equal to the value I have.
filterMarkers = function (gender, people) {
for (i = 0; i < markers1.length; i++) {
marker = gmarkers1[i];
var gender = document.getElementById("gender_filter").value;
var people = document.getElementById("people_filter").value;
var service = document.getElementById("filterzipcode").value;
infowindow.close();
// If is same category or category not picked
if( (marker.gender == gender || gender.length === 0
)&& (
marker.people == people || people.length === 0
)&&(marker.service contains any value that users include) {
marker.setVisible(true);
console.log('vis');
}
// Categories don't match
else {
marker.setVisible(false);
console.log('hide');
}
map.panTo(marker.getPosition());
}
Thank you inadvance

PHPexcel - getOldCalculatedValue and rangeToArray

Searched for quite a while now, but I'm stuck at the following problem.
I am using PHPexcel 1.8.0
The spreadsheet is read using the following code:
$rowData = $sheet->rangeToArray('A' . $row . ':' . $highestColumn . $row, NULL, TRUE, TRUE);
So far ok and it works well.
But some spreadsheets contain external referenced data.
And for that I want to use "getOldCalculatedValue".
How do I combine "getOldCalculatedValue" with "rangeToArray" ?
Or is "rangeToArray" inappropriate for this ?
Thanks for any help or hints !
Simple answer, you can't combine the two
rangeToArray() is a simple method for a simple purpose, it doesn't try to do anything clever, simply to return the data from the worksheet as efficiently and quickly as possible
getOldCalculatedValue() is used for a very specific circumstance, and isn't guaranteed to be correct even then, because it retrieves the last value calculated for the cell in MS EXcel itself, which ,ay not be correct if the external workbook wasn't available to MS Excel in that circumstance, or MS Excel formula evaluation was disable.
When calculating cells values from a formula, the PHPExcel calculation engine should use the getOldCalculatedValue() as a fallback if it finds an external reference, and rangeToArray() will try to use this method, but it isn't perfect, especially when that reference in nested deep inside other formulae referenced in other cells.
If you know that a formula in a cell contains an external reference, you should use getOldCalculatedValue() directly for that cell
I came up with the following solution.
Maybe not perfect, but it currently does the job. Thanks for any improvements!
With PHPExcel included and the excel file uploaded and ready, I continue with:
$sheet = $objPHPExcel->getSheet(0);
$highestRow = $sheet->getHighestRow();
Create a new array to store the cell values of a row
$arr_row = array();
Loop through the rows
for ($rownumber = 2; $rownumber <= $highestRow; $rownumber++){
$row = $sheet->getRowIterator($rownumber)->current();
$cellIterator = $row->getCellIterator();
$cellIterator->setIterateOnlyExistingCells(false);
Then loop through the cells of the current row
foreach ($cellIterator as $cell) {
Find cells with a formula
$cellcheck = substr($cell->getValue(),0,1);
if($cellcheck == '='){
$cell_content = $cell->getOldCalculatedValue();
}
else{
$cell_content = $cell->getValue();
}
Add the cell values to the array
array_push($arr_row,$cell_content);
Close cell loop
}
At this point I use the $arr_row to do further calculations and string formatting, before finally inserting it into a mysql table.
Close row loop
}
I made some changes in the function rangeToArray() inside Worksheet.php.
Worked fine!
public function rangeToArray($pRange = 'A1', $nullValue = null, $calculateFormulas = true, $formatData = true, $returnCellRef = false) {
// Returnvalue
$returnValue = array();
// Identify the range that we need to extract from the worksheet
list($rangeStart, $rangeEnd) = PHPExcel_Cell::rangeBoundaries($pRange);
$minCol = PHPExcel_Cell::stringFromColumnIndex($rangeStart[0] -1);
$minRow = $rangeStart[1];
$maxCol = PHPExcel_Cell::stringFromColumnIndex($rangeEnd[0] -1);
$maxRow = $rangeEnd[1];
$maxCol++;
// Loop through rows
$r = -1;
for ($row = $minRow; $row <= $maxRow; ++$row) {
$rRef = ($returnCellRef) ? $row : ++$r;
$c = -1;
// Loop through columns in the current row
for ($col = $minCol; $col != $maxCol; ++$col) {
$cRef = ($returnCellRef) ? $col : ++$c;
// Using getCell() will create a new cell if it doesn't already exist. We don't want that to happen
// so we test and retrieve directly against _cellCollection
if ($this->_cellCollection->isDataSet($col.$row)) {
// Cell exists
$cell = $this->_cellCollection->getCacheData($col.$row);
if ($cell->getValue() !== null) {
if ($cell->getValue() instanceof PHPExcel_RichText) {
$returnValue[$rRef][$cRef] = $cell->getValue()->getPlainText();
} else {
if ($calculateFormulas)
{ ##################################### CHANGED LINES
if(!preg_match('/^[=].*/', $cell->getValue()))
{
$returnValue[$rRef][$cRef] = $cell->getCalculatedValue(); # THE ORIGINAL CODE ONLY HAD THIS LINE
}
else
{
$returnValue[$rRef][$cRef] = $cell->getOldCalculatedValue();
}
} ##################################### CHANGED LINES
else
{
$returnValue[$rRef][$cRef] = $cell->getValue();
}
}
if ($formatData) {
$style = $this->_parent->getCellXfByIndex($cell->getXfIndex());
$returnValue[$rRef][$cRef] = PHPExcel_Style_NumberFormat::toFormattedString(
$returnValue[$rRef][$cRef],
($style && $style->getNumberFormat()) ?
$style->getNumberFormat()->getFormatCode() :
PHPExcel_Style_NumberFormat::FORMAT_GENERAL
);
}
} else {
// Cell holds a NULL
$returnValue[$rRef][$cRef] = $nullValue;
}
} else {
// Cell doesn't exist
$returnValue[$rRef][$cRef] = $nullValue;
}
}
}
// Return
return $returnValue;
}

How do I get a bounding box for the current selection?

I'd like to do fancy things the selection indicator. How do I get the bounding box for the currently selected characters?
This was non-trivial. First, a selection may require more than one rectangle. Next, there's no convenient way to do it.
Here's what I had to do:
var start:int = op.activePosition < op.anchorPosition ? op.activePosition : op.anchorPosition;
var end:int = op.activePosition > op.anchorPosition ? op.activePosition : op.anchorPosition;
var textFlow:TextFlow = this.textFlow;
var rectangles:Dictionary = new Dictionary();
// For each selected character, make a box
for( var i:int=start; i < end; i++) {
var flowLine:TextFlowLine = textFlow.flowComposer.findLineAtPosition( i, true );
if( rectangles[ flowLine.absoluteStart ] == null ) {
rectangles[ flowLine.absoluteStart ] = new Rectangle();
(rectangles[ flowLine.absoluteStart ] as Rectangle).x = 0xffffff;
(rectangles[ flowLine.absoluteStart ] as Rectangle).right = 0;
}
var currentRect:Rectangle = rectangles[ flowLine.absoluteStart ];
var textLine:TextLine = flowLine.getTextLine(true);
var atomIndex:int = textLine.getAtomIndexAtCharIndex( i );
if( atomIndex >= 0) {
var atomBounds:Rectangle = textLine.getAtomBounds( atomIndex );
var pt:Point = this.globalToLocal( textLine.localToGlobal( new Point( atomBounds.left, atomBounds.top ) ) );
if( pt.x <= currentRect.left ) {
currentRect.left = pt.x;
currentRect.top = pt.y;
}
pt = this.globalToLocal( textLine.localToGlobal( new Point( atomBounds.right, atomBounds.bottom) ) );
if( pt.x >= currentRect.right ) {
currentRect.right = pt.x;
currentRect.bottom = pt.y;
}
}
}
return rectangles;
I don't believe there's a trivial way to get total control over this, from looking around at the docs a bit I saw this:
http://opensource.adobe.com/wiki/display/flexsdk/Spark+Text+Primitives#SparkTextPrimitives-FTE
[Style(name="focusedTextSelectionColor", type="uint", format="Color", inherit="yes")]
[Style(name="inactiveTextSelectionColor", type="uint", format="Color", inherit="yes")]
[Style(name="unfocusedTextSelectionColor", type="uint", format="Color", inherit="yes")]
also to note:
anchor position - A character index specifying the end of the selection that stays fixed when you extend the selection with the arrow keys.
active position - A character index specifying the end of the selection that moves when you extend the selection with the arrow keys.
Since these are all only colors (or indices) I don't know if they'll get at all the fanciness you'd like to do. I'd worked on something in Flex 3 to deal with custom text selection controls and ended up using an "off screen buffer" of sorts where I put a TextField offscreen with the same properties as the one on screen then dump the characters 1 by 1 until I got to the desired width then I could figure out where in the characters the control was dropped(kind of like android selection).
I would suggest searching for the above styles in the SDK you have (particularly in RichEditableText and its super classes, I would do this but there's quite a few versions out there now and don't know which one your using, TLF and FTE are both a bit in flux it seems). Once you find where these styles are used you'll probably be in the vicinity of the selection indicator drawing code and will likely need to extend from whatever class that is in order to override the appropriate methods.
Sorry I can't give you a straight answer but hopefully this helps or someone else is able to chime in if there's an easier way.
Shaun

Resources