I'm trying to read a big excel file of about 20mb to import into mysql.
I've searched across internet and found the "Chunks reading" solution, however is not working... or is SO slowly for me, and I'm not sure why.
This is what im doing:
// .....
// into MyReadFilter class.. this is the most important function:
public function readCell($column, $row, $worksheetName = '') {
// Only read the rows and columns that were configured
if (($row == 1) || ($row >= $this->_startRow && $row < $this->_endRow)) {
if (in_array($column,$this->_columns)) {
return true;
}
}
return false;
}
// .....
$filter = new MyReadFilter(1, 22000);
$chunkSize = 10;
$objReader = PHPExcel_IOFactory::createReader($inputFileType);
$objReader->setReadFilter($filter);
$objReader->setReadDataOnly(false); //not sure if this should be true
for ($startRow = 2; $startRow <= 65536; $startRow += $chunkSize) {
echo "Reading";
$filterSubset->setRows($startRow, $chunkSize);
$objPHPExcel = $objReader->load($inputFileName); // this line takes like 40 seconds... for 10 rows?
echo "chunk done! ";
}
However, inside the for, the $objReader->load() is taking like 40 seconds, and in fact, after 2 loops I got a memory error.
If I unset the $objReader inside the for I can make it run about 20 times inside the for... (although it take like 10 minutes) and.. memory error.
I'm wondering why the load function seems to read all the file if im using a filter, also the filter strategy seems to parse all rows and return false for all rows that are not being required... is not posible to abort reading or really read just the required ones?
I've tried a couple of FilterClass and code snippets but got same results...
If You're using a filter, then the Reader is still reading the whole file but only populating the PHPExcel object cells that are defined by the filter; and the Reader still needs to read the whole file each pass of the filtering process, which is what makes it slower.
The Reader needs to read the whole file because of the structure of the raw spreadsheet files. Cell data is not stored with cell formatting, and cell content may also be stored separately. The Reader needs to pull all this together. You can't simply abort the reader when the filter condition is met, because the reader has no way of knowing that it has been completed... if you have a filter that is limiting the load to cells A1:C3, then you can't abort after B3 has been read because you don't know if cell B2 comes after that in the file, or there may be comments associated with cell A1 further on in the file. Until the whole file has been loaded and parsed, you can't start to filter.
The main memory usage in PHPExcel is the PHPExcel object, and specifically the cells (typically about 1k/cell on 32-bit PHP).... the main solution provided to reduce memory here is cell caching. This can (using SQLite caching) reduce cell memory usage to 0k/cell, though at a cost in speed.
The Reader doesn't use much more memory than the size of the Excel file (decompressed) itself, so is normally far less of a memory problem; but this is being addressed (for XML-based spreadsheet formats) by switching from SimpleXML to XMLReader. But it is dependent on the format of the file being loaded; xls format files are very different to xlsx files (xlsx will benefit from this, xls won't) and also dependent on the developers being able to find the time to do this - but it is on the roadmap for the coming year, and work has already started.
I'm trying to create a simple setup to load a template, insert some data and save as a new file. However I need some conditional formatting on some of the cells and when I get the newly created files the conditional formatting is missing. It's not being overridden by some other formatting, the rules are missing from the conditional formatting menu. I'm using PHP 5.2, PHPExcel 1.7.8 and Excel 2010.
<?php
class template {
static $objPHPExcel;
function __construct() {
define('EOL',(PHP_SAPI == 'cli') ? PHP_EOL : '<br />');
require_once '../Classes/PHPExcel/IOFactory.php';
if (!file_exists("template.xlsx")) {
exit("template missing." . EOL);
}
echo date('H:i:s') , " Load from Excel2007 file" , EOL;
$objReader = PHPExcel_IOFactory::createReader('Excel2007');
$objReader->setReadDataOnly(false);
self::$objPHPExcel = $objReader->load("template.xlsx");
}
function insertdata($dataArray){ /* unused */ }
function save($name){
define('EOL',(PHP_SAPI == 'cli') ? PHP_EOL : '<br />');
require_once '../Classes/PHPExcel/IOFactory.php';
echo date('H:i:s') , " Write to Excel2007 format" , EOL;
$objWriter = PHPExcel_IOFactory::createWriter(self::$objPHPExcel, 'Excel2007');
$objWriter->save($name);
echo date('H:i:s') , " File written to: ".$name , EOL;
}
$temp=new template();
$temp->save("savethis.xlsx");
I'm trying to preserve a Graded 2 color scale (Formatting based on cell values, Minimum is type Number=1, Maximum is type Number=10). The cell in question has a formula attached that references another sheet (all data has been saved correctly).
I have found that is is hopeless to read, modify and save an XLS/XLSX file getting the "auto filter", "Data Validation" and "Conditional Format" from the original file, the final solution I found is to make the template using the PHPExcel library.
Sadly, as it was stated on the other answer (https://stackoverflow.com/a/13172890/218418):
PHPExcel is not a library for "editing" workbook files: you're not using PHPExcel to change a file, you're changing a PHPExcel object that can be loaded from a file, and can subsequently be written to a file.
I'm amused that something like "editing" an Excel file using a template with just data validation, conditional formating and auto filter is not possible, but I understand.
This is an old thread, but I recently ran into the same issue. It seems like PHPSpreadsheet (the new incarnation of PHPExcel) supports creating spreadsheets with conditional formatting, but when I tried to read an existing spreadsheet and then write it out to a new file, the conditional formatting was broken.
I'm planning to use xlsx-populate instead - it's nodeJS rather than PHP, but it seems to be better at writing Excel files while maintaining features of the spreadsheet (like conditional formatting) that it doesn't understand.
In my case, I have a need to populate one sheet of a large, complex spreadsheet. Values from that sheet are used in calculations and conditional formatting on other sheets, which I don't need to modify directly.
With xlsx-populate, I was able to open the spreadsheet, populate it, and save it without breaking the formatting.
After loading a PHPExcel object with my data, I want to output the contents directly into a php variable, instead of writing to a file. Have I missed the method to do that, for it seems that the only way to save is to save to disk.
You can use a writer to save to disk, or save to php://output. If the latter, you could use output buffering to capture that output and then store in a variable.... not sure quite why you'd want to do this though.
$objWriter = PHPExcel_IOFactory::createWriter($objPHPExcel, 'Excel2007');
ob_start();
$objWriter->save('php://output');
$excelOutput = ob_get_clean();
Since PHPExcel writes the file to the disk in any case, if you capture stdout, what happens internally is that the file is written to the disk, then read from the disk (and deleted), printed to stdout and then captured in the output buffer.
You might as well do:
$tmpfile = tempnam($tmp_dir, 'phpxltmp');
$objWriter = PHPExcel_IOFactory::createWriter($objPHPExcel, 'Excel2007');
$objWriter->save($tmpfile);
$excelOutput = file_get_contents($tmpfile);
unlink($tmpfile);
I'm having this problem when I tried to extract information from excel files. Here's my situation, I have 34 Excel files which I received from my various users.
I'm using PHP version 5 to extract from the Excel files. My script will loop for every files, and looping again according to sheet name, and lastly looping again according to cell addresses.
The problem arised when the users had entered into a cell for e.g. =+A1 which means the users referencing the cell value to another cell due to it has the same value with cell A1.
When I checked in mysql (as I saved those for future use) I found from the record for a particular cell is identical with another record obtained from the same cell but in different excel file. What I meant is that, as my php script will loop from one file to another file, the first time PHPExcel read for e.g cell C3 which has some value USD3,000.00 the next files the PHPExcel may go to the same cell C3 but this time the C3 cell contain a formula that referencing to cell A1 ("=+A1" formula)which has value USD5,000.00.
PHP script suppose to record in mysql for USD5,000.00 but it didn't. I suspect that the PHPExcel script did not clear the variable at first round. I've tried unset($objPHPExcel) and destroy the variable but it still happening.
My coding is simple as follows:
if(file_exists($inputFileName))
{
$inputFileType = PHPExcel_IOFactory::identify($inputFileName);
$objReader = PHPExcel_IOFactory::createReader($inputFileType);
$objReader->setReadDataOnly(true);
$objPHPExcel = $objReader->load($inputFileName);
//to obtain date from FILE and store in DB for future comparison
$validating_date_reporting = $objPHPExcel->getSheet(0)->getCell('C10')->getValue();
$validating_date_reporting = PHPExcel_Style_NumberFormat::toFormattedString($validating_date_reporting,"YYYY-MMM-DD");
$validating_date_reporting = date('Y-m-d',strtotime($validating_date_reporting));
//first entry
$entry = mysql_query('INSERT INTO `'.$table.'`(`broker_code`, `date`, `date_from_submission`) VALUES("'.$broker_code.'","'.$reporting_date.'","'.$reporting_date.'")') or die(mysql_error());
foreach($cells_array as $caRef=>$sName)
{
foreach($sName as $sNameRef=>$cells)
{
$wksht_page = array_search($caRef, $sheetNameArray);
$cell_column = $wksht_page.'_'.$cells;
echo $inputFileName.' '.$caRef.' '.$cell_column.'<br>';
$value = $objPHPExcel->setActiveSheetIndexByName($caRef)->getCell($cells)->getCalculatedValue();
echo $value.'<br>';
if($value)
{
$isdPortal->LoginDB($db_periodic_submission);
$record = mysql_query('UPDATE `'.$table.'` SET `'.$cell_column.'` = "'.$value.'" WHERE broker_code = "'.$broker_code.'" AND date_from_submission = "'.$validating_date_reporting.'"') or die(mysql_error());
}
}
}
}
I really hope that you can help me out here..
thank you in advance.
PHPExcel holds a calculation cache as well, and this is not cleared when you unset a workbook: it has to be cleared manually using:
PHPExcel_Calculation::flushInstance();
or
PHPExcel_Calculation::getInstance()->clearCalculationCache();
You can also disable calculation caching completely (although this may slow things down if you have a lot of formulae that reference cells containing other formulae) using:
PHPExcel_Calculation::getInstance()->setCalculationCacheEnabled(FALSE);
before you start processing your files
This is because currently PHPExcel uses a singleton for the calculation engine. It is in the roadmap to switch to using a multiton pattern later this year, which will effectively maintain a separate cache for each workbook, alleviating this problem.
EDIT
Note that simply unsetting $objPHPExcel does not work. You need to detach the worksheets before unsetting $objPHPExcel.
$objPHPExcel->disconnectWorksheets();
unset($objPHPExcel);
as described in section 4.3 of the Developer Documentation. And this is the point where you should also add the PHPExcel_Calculation::flushInstance();
I have made a function that finds all the URLs within an html file and repeats the same process for each html content linked to the discovered URLs. The function is recursive and can go on endlessly. However, I have put a limit on the recursion by setting a global variable which causes the recursion to stop after 100 recursions.
However, php returns this error:
Fatal error: Maximum function nesting level of '100' reached,
aborting! in
D:\wamp\www\crawler1\simplehtmldom_1_5\simple_html_dom.php on line
1355
I found a solution here: Increasing nesting function calls limit but this is not working in my case.
I am quoting one of the answers from the link mentioned above. Please do consider it.
"Do you have Zend, IonCube, or xDebug installed? If so, that is probably where you are getting this error from.
I ran into this a few years ago, and it ended up being Zend putting that limit there, not PHP. Of course removing it will let >you go past the 100 iterations, but you will eventually hit the memory limits."
Is there a way to increase the maximum function nesting level in PHP
Increase the value of xdebug.max_nesting_level in your php.ini
A simple solution solved my problem. I just commented this line:
zend_extension = "d:/wamp/bin/php/php5.3.8/zend_ext/php_xdebug-2.1.2-5.3-vc9.dll
in my php.ini file. This extension was limiting the stack to 100 so I disabled it. The recursive function is now working as anticipated.
Another solution is to add xdebug.max_nesting_level = 200 in your php.ini
Rather than going for a recursive function calls, work with a queue model to flatten the structure.
$queue = array('http://example.com/first/url');
while (count($queue)) {
$url = array_shift($queue);
$queue = array_merge($queue, find_urls($url));
}
function find_urls($url)
{
$urls = array();
// Some logic filling the variable
return $urls;
}
There are different ways to handle it. You can keep track of more information if you need some insight about the origin or paths traversed. There are also distributed queues that can work off a similar model.
Rather than disabling the xdebug, you can set the higher limit like
xdebug.max_nesting_level=500
It's also possible to fix this directly in php, for example in the config file of your project.
ini_set('xdebug.max_nesting_level', 200);
Go into your php.ini configuration file and change the following line:
xdebug.max_nesting_level=100
to something like:
xdebug.max_nesting_level=200
on Ubuntu using PHP 5.59 :
got to `:
/etc/php5/cli/conf.d
and find your xdebug.ini in that dir, in my case is 20-xdebug.ini
and add this line `
xdebug.max_nesting_level = 200
or this
xdebug.max_nesting_level = -1
set it to -1 and you dont have to worry change the value of the nesting level.
`
probably happened because of xdebug.
Try commenting the following line in your "php.ini" and restart your server to reload PHP.
";xdebug.max_nesting_level"
Try looking in /etc/php5/conf.d/ to see if there is a file called xdebug.ini
max_nesting_level is 100 by default
If it is not set in that file add:
xdebug.max_nesting_level=300
to the end of the list so it looks like this
xdebug.remote_enable=on
xdebug.remote_handler=dbgp
xdebug.remote_host=localhost
xdebug.remote_port=9000
xdebug.profiler_enable=0
xdebug.profiler_enable_trigger=1
xdebug.profiler_output_dir=/home/drupalpro/websites/logs/profiler
xdebug.max_nesting_level=300
you can then use #Andrey's test before and after making this change to see if worked.
php -r 'function foo() { static $x = 1; echo "foo ", $x++, "\n"; foo(); } foo();'
php.ini:
xdebug.max_nesting_level = -1
I'm not entirely sure if the value will ever overflow and reach -1, but it'll either never reach -1, or it'll set the max_nesting_level pretty high.
You could convert your recursive code into an iterative code, which simulates the recursion. This means that you have to push the current status (url, document, position in document etc.) into an array, when you reach a link, and pop it out of the array, when this link has finished.
Check recursion from command line:
php -r 'function foo() { static $x = 1; echo "foo ", $x++, "\n"; foo(); } foo();'
if result > 100 THEN check memory limit;
You could try to wiggle down the nesting by implementing parallel workers (like in cluster computing) instead of increasing the number of nesting function calls.
For example: you define a limited number of slots (eg. 100) and monitor the number of "workers" assigned to each/some of them. If any slots become free, you put the waiting workers "in them".
<?php
ini_set('xdebug.max_nesting_level', 9999);
... your code ...
P.S. Change 9999 to any number you want.
Stumbled upon this bug as well during development.
However, in my case it was caused by an underlying loop of functions calling eachother - as a result of continuous iterations during development.
For future reference by search engines - the exact error my logs provided me with was:
Exception: Maximum function nesting level of '256' reached, aborting!
If, like in my case, the given answers do not solve your problem, make sure you're not accidentally doing something along the lines of the following simplified situation:
function foo(){
// Do something
bar();
}
function bar(){
// Do something else
foo();
}
In this case, even if you set ini_set('xdebug.max_nesting_level', 9999); it will still print out the same error message in your logs.
If you're using Laravel, do
composer update
This should be work.
In your case it's definitely the crawler instance is having more Xdebug limit to trace error and debug info.
But, in other cases also errors like on PHP or core files like CodeIgniter libraries will create such a case and if you even increase the x-debug level setting it would not vanish.
So, look into your code carefully :) .
Here was the issue in my case.
I had a service class which is library in CodeIgniter. Having a function inside like this.
class PaymentService {
private $CI;
public function __construct() {
$this->CI =& get_instance();
}
public function process(){
//lots of Ci referencing here...
}
My controller as follow:
$this->load->library('PaymentService');
$this->process_(); // see I got this wrong instead it shoud be like
Function call on last line was wrong because of the typo, instead it should have been like below:
$this->Payment_service->process(); //the library class name
Then I was keeping getting the exceed error message. But I disabled XDebug but non helped. Any way please check you class name or your code for proper function calling.
I had a error when i was installing many plugins So the error 100 showed including the location of the last plugin that i installed C:\wamp\www\mysite\wp-content\plugins\"..." so i deleted this plugin folder on the C: drive then everything was back to normal.I think i have to limit the amount of plug-in i install or have activated .good luck i hope it helps
I had this issue with WordPress on cloud9. It turns out it was the W3 Caching plugin. I disabled the plugin and it worked fine.
Another solution if you are running php script in CLI(cmd)
The php.ini file that needs edit is different in this case. In my WAMP installation the php.ini file that is loaded in command line is:
\wamp\bin\php\php5.5.12\php.ini
instead of \wamp\bin\apache\apache2.4.9\bin\php.ini which loads when php is run from browser
You can also modify the {debug} function in modifier.debug_print_var.php, in order to limit its recursion into objects.
Around line 45, before :
$results .= '<br>' . str_repeat(' ', $depth * 2)
. '<b> ->' . strtr($curr_key, $_replace) . '</b> = '
. smarty_modifier_debug_print_var($curr_val, ++$depth, $length);
After :
$max_depth = 10;
$results .= '<br>' . str_repeat(' ', $depth * 2)
. '<b> ->' . strtr($curr_key, $_replace) . '</b> = '
. ($depth > $max_depth ? 'Max recursion depth:'.(++$depth) : smarty_modifier_debug_print_var($curr_val, ++$depth, $length));
This way, Xdebug will still behave normally: limit recursion depth in var_dump and so on.
As this is a smarty problem, not a Xdebug one!
I had the same problem and I resolved it like this:
Open MySQL my.ini file
In [mysqld] section, add the following line: innodb_force_recovery =
1
Save the file and try starting MySQL
Remove that line which you just added and Save