PHPUnit_Framework_TestCase memory leak with large DataProvider - symfony

When I run PHPUnit, it appears to me as if it had a memory-leak when running many tests inside a single test class. But I don't know if this is a bug or it was the expected behaviour.
To reproduce:
I create a simple testHello() with a silly assertTrue(true).
I feed it from providerHello(). Just feeding 3 dummy params.
With $numberOfTests = 1;, consumed memory is 5.75MB.
PHPUnit output = Time: 0 seconds, Memory: 5.75Mb
With $numberOfTests = 10000;, I don't expect the memory to grow so much, just the size of the new array. But the used memory is 99.75MB which I feel it is too much.
PHPUnit output = Time: 4 seconds, Memory: 99.75Mb
I added a dirty echo() in the provider, just to know how much memory the array made the script to consume.
With 1 test: Memory = 5294552 (5.2MB)
With 10.000 tests: Memory = 15735352 (15.7MB)
The questions:
Why do I loose 84MB in the way? (99.75 really consumed - 15.75 really used by the array)
Is it hormal that it allocates memory at each iteration, probably its internal setUp(), but does not free the same amount at the internal tearDown()?
Am I doing anything wrong?
My version:
phpunit --version gives PHPUnit 3.6.10 by Sebastian Bergmann..
This is the code:
<?php
class DemoTest extends \PHPUnit_Framework_TestCase
{
/** #dataProvider providerHello */
public function testHello( $a, $b, $c )
{
$this->assertTrue( true );
}
public function providerHello()
{
$numberOfTests = 10000;
$data = array();
for( $i = 0; $i < $numberOfTests; $i++ )
{
$data[] = array( 1, 2, 3 );
}
echo( "Memory = " . memory_get_peak_usage() . PHP_EOL );
return $data;
}
}
?>

you need to set backupGlobals and backupStaticAttributes to false in your phpunit.xml file. If you don't use an config file you can also do so on the command line.
--no-globals-backup
--static-backup

Related

Segfault in factory constructed class

I have an issue where a Segfault occurs in random locations but those locations seem to always be the same. The issue also is made more confusing by the fact that the location of the Segfault can change depending on whether I'm using gdb or valgrind to try and solve the issue. This would seem to be a race condition problem but I don't always see any of my destructors being called so I'm not sure why that would be happening. I have not defined my copy constructors which I suppose could be the problem but I would like to understand why that may be.
The tests that I have on the midLevel class to exercise its functionality don't have this problem. Is there something flawed with my basic construction?
My use case is:
In highest level class:
returnObject highLevelClass::performTask( ){
std::shared_ptr< midLevelClass > midClass;
std::vector< someType > dataForClass;
for ( auto it = _someIterate.begin( ); it != _someIterate.end( ); it++ ){
...
buildMidClass( midClass, &dataForClass );
}
...
return returnObject;
}
returnObject highLevelClass::buildMidClass( std::shared_ptr< midLevelClass > &midClass,
std::vector< someType > *dataForClass ){
...
midClass = midLevelClass( _configurationInfo ).create( )
midClass.loadData( dataForClass );
midClass->processData( ); //SOMETIMES IT SEGFAULTS HERE DURING std::vector ALLOCATIONS
...
return returnObject;
}
highLevelClass::~highLevelClass( ){
//SOMETIMES IT SEGFAULTS HERE
return;
}
In the mid-level class:
midLevelClass::loadData( std::vector< someType > *data ){
_data = data; //_data is a std::vector< someType >*
}
std::shared_ptr< midLevelClass > midLevelClass::create( configurationType &_configInfo ){
if ( _configInfo[ "type" ] == childMidLevelClass ){
return std::make_shared< childMidLevelClass >( _configInfo );
}
_error = new errorNode( "create", "The type is not defined" );
return std::make_shared< volumeReconstructionBase >( _config, _error );
}
The answer turned out to be that another std::vector ( that wasn't being accessed by any other part of the code ) was overflowing through a bad access using []. Nothing in gdb or valgrind showed that this was the problem. The answer is probably be careful about using [] to access a std::vector and consider using std::vector::at( ).

Get content of an ArrayCollection

I would like to upgrade my symfony 2 project from 2.3 to 2.7 LTS version. I have a problem in a repository to get result of a query. In 2.3, this query give me something :
public function findProtectedPublications( $steps, $start, $end)
{
$query= $this->getEntityManager()
->createQueryBuilder()
->select('d.pubRefs')
->from('ImpressionDemandBundle:Event', 'h')
->innerJoin('h.demand','d')
->where('d.protectedPublications = :pub')
->setParameter('pub', 1 )
->andWhere('h.date >= :start')
->setParameter('start', $start )
->andWhere('h.date <= :end')
->setParameter('end', $end )
->andWhere('h.stepId in (:steps)')
->setParameter('steps', $steps )
->orderBy('d.id','ASC')
->getQuery();
$results = $query->getResult();
$publications = array();
if ($results && ! empty ($results)){
foreach($results as $result){
$pubs = $result['pubRefs'];
if ($pubs && ! empty($pubs)){
foreach($pubs as $pub){
$publications[] = $pub;
}
}
}
}
return $publications;
}
But this code doesn't work in earlier version because $pubs variable in an ArrayCollection. So I changed the end of my code with this :
$results = $query->getResult();
$publications = array();
if ($results && ! empty ($results)){
foreach($results as $result){
$pubs = $result['pubRefs'];
var_dump($pubs);
if (! $pubs->isEmpty()){
$arrayPubs = $pubs->toArray();
foreach($arrayPubs as $pub){
$publications[] = $pub;
}
}
}
}
return $publications;
In this part, when I dump the $pubs variable, I have :
object(Doctrine\Common\Collections\ArrayCollection)#131 (2) {
["elements":"Doctrine\Common\Collections\ArrayCollection":private]=>
NULL
["_elements":"Doctrine\Common\Collections\ArrayCollection":private]=>
array(1) {
[0]=>
object(Impression\DemandBundle\Entity\Publication)#125 (5) {
["editor":"Impression\DemandBundle\Entity\Publication":private]=>
string(24) "Journal Le Monde 4-10-13"
["coauthors":"Impression\DemandBundle\Entity\Publication":private]=>
string(12) "Machin Machin"
["title":"Impression\DemandBundle\Entity\Publication":private]=>
string(57) "La tragédie de Lampedusa: s"émouvoir, comprendre, agir."
["nbPages":"Impression\DemandBundle\Entity\Publication":private]=>
float(1)
["nbCopies":"Impression\DemandBundle\Entity\Publication":private]=>
float(40)
}
}
}
So it seems that there are elements in this ArrayCollection, but the test $pubs->isEmpty() gives a true result, so I have nothing in $publications array.
Edit: In fact, the problem seems to be due to my data in the database : for an object previous from my upgrade, I have something like this in the database :
O:43:"Doctrine\Common\Collections\ArrayCollection":1:{s:54:"Doctrine\Common\Collections\ArrayCollection_elements";a:1:{i:0;O:42:"Impression\DemandBundle\Entity\Publication":5:{s:50:"Impression\DemandBundle\Entity\Publicationeditor";s:5:"BREAL";s:53:"Impression\DemandBundle\Entity\Publicationcoauthors";s:5:"MONOT";s:49:"Impression\DemandBundle\Entity\Publicationtitle";s:18:"USA Canada mexique";s:51:"Impression\DemandBundle\Entity\PublicationnbPages";d:150;s:52:"Impression\DemandBundle\Entity\PublicationnbCopies";d:150;}}}
and this gives the error.
For a object add after my upgrade, I have something like this in the database :
O:43:"Doctrine\Common\Collections\ArrayCollection":1:{s:53:"Doctrine\Common\Collections\ArrayCollectionelements";a:1:{i:0;O:42:"Impression\DemandBundle\Entity\Publication":5:{s:50:"Impression\DemandBundle\Entity\Publicationeditor";s:8:"dfg dfgd";s:53:"Impression\DemandBundle\Entity\Publicationcoauthors";s:7:"dfg dfg";s:49:"Impression\DemandBundle\Entity\Publicationtitle";s:5:"fdg d";s:51:"Impression\DemandBundle\Entity\PublicationnbPages";d:5;s:52:"Impression\DemandBundle\Entity\PublicationnbCopies";d:3;}}}
and the function findProtectedPublications() works without errors.
The difference between the two versions is ArrayCollection_elements for the first and ArrayCollectionelements for the second.
To correct this data, I tried with
UPDATE demand SET pub_refs = REPLACE (pub_refs, "ArrayCollection_elements', 'ArrayCollectionelements')
but this doesn't work because of special chars. Trying with
UPDATE demand SET pub_refs = REPLACE (pub_refs, "ArrayCollection�_elements', 'ArrayCollection�elements')
doesn't work better. How can I correct this data ?
Doctrine can populate results as an Array instead of an ArrayCollection, simply change the getResult() call to:
$results = $query->getResult(\Doctrine\ORM\AbstractQuery::HYDRATE_ARRAY);
This would be the most efficient way to complete your task however you could also use ArrayCollection's built-in toArray() method to convert its own data to array format:
$publications = $results->toArray();
As the problem seems to be due to a change in the storage of ArrayCollection in database between 2.3 and 2.7 version of symfony, I created an line command to update these in database.

Doctrine Batch Processing doesn't clear memory?

I am trying to use a technique suggested by Doctrine 2 to process a large number of objects. The technique suggests that by using an iterator and by detaching after processing each iteration, memory usage should be kept to a minimum (they speak about an increase of a few KB for processing 10000 records).
However, when I try to do this, I do not see any objects freed. In fact, I am retrieving a little more than 2000 assets and this increases my memory usage by 90 MB. Clearly, these objects are not freed. Can anyone tell me what I am doing wrong? My code looks as follows:
$profiles = //array of Profile entities
$qb = $this->createQueryBuilder('a')
->addSelect('file')
->leftJoin($profileNodeEntityName, 'pn', JOIN::WITH, 'pn.icon = a OR pn.asset = a')
->leftJoin(
$profileEntityName,
'p',
JOIN::WITH,
'pn.profile = p OR p.logo = a OR p.background = a OR p.pricelistAsset = a OR p.pdfTplBlancAsset = a OR p.pdfTplFrontAsset = a OR p.pdfTplBackAsset = a'
)
->innerJoin('a.currentFile', 'file')
->where('p IN (:profiles)')
->setParameter('profiles', $profiles)
->distinct(true);
$iterableResult = $qb->getQuery()->iterate();
$start = memory_get_usage() / 1024;
while (($row = $iterableResult->next()) !== false) {
// process $row[0]
$this->getEntityManager()->detach($row[0]);
}
$end = memory_get_usage() / 1024 - $start;
// $end is more of less equal to 90000 or 90 MB
Thanks!
You should also detach the related entities, or set cascade={"detach"} on the associations.

AMFPHP working for ArrayCollection in Flex

I have a function in my php class which must receive an array of Objects. In flex, I send the data (as ArrayCollection) calling the service. If I work locally, the PHP receive the data and store all the records in the database, but i I place such service in the server, the function doesnt work.
public function putPrecioBaseProductos($data) {
$priveID = $data[0]->priveID;
$date = $data[0]->date;
$res = mysql_query("DELETE FROM db.prices WHERE priveID=".$priveID." AND date='".$date."'");
if (!$res) return '0';
$cadena = "";
for ($i=0; $i < count($data); $i++) {
if ($cadena != '') $cadena .= ', ';
$cadena .= "(".$priveID.", ".$data[$i]->productID.", '".$data[$i]->precio1."', '".$data[$i]->precio2."', '".$data[$i]->precio3."', '".$data[$i]->precio4."', '".$data[$i]->precio5."', '".$date."')";
}
$res = mysql_query( "INSERT INTO tabo4.precios_base (proveedorID, productoID, precio1, precio2, precio3, precio4, precio5, fecha) VALUES ".$cadena );
if ($res) return '1'; else return '0';
}
I have been googling and found that amfphp do not support ArrayColletion as parameter but as i have just said, locally (using MAMP), data is received as desired but in server not.
Anyone know why?
Thanks.
Try sending data as Array instead of ArrayCollection.
ArrayCollection does not work well with AMFPhp ...
To get the array just use:
myArrayCollection.source;

Log4perl category as log file name

I'm sure I'm being dim and missing the obvious but is there a simple way of using the current category as the filename in a config file without resorting to a subroutine call?
So that in the following one could use ${category}.log instead of repeating bin.nh.tpp in the filename line
log4perl.logger.**bin.nh.tpp**=INFO, BIN_NH_TPP_LOGFILE
log4perl.appender.BIN_NH_TPP_LOGFILE=Log::Log4perl::Appender::File
log4perl.appender.BIN_NH_TPP_LOGFILE.filename=${LOGS}/nh/**bin.nh.tpp**.log
log4perl.appender.BIN_NH_TPP_LOGFILE.mode=append
log4perl.appender.BIN_NH_TPP_LOGFILE.layout=PatternLayout
log4perl.appender.BIN_NH_TPP_LOGFILE.layout.ConversionPattern=[%d] %F{1} %L %c - %m%n
It's somewhat more involved than a subroutine, I'm afraid. Subroutines in l4p conf files allow for including variables known at conf file parsing time, e.g. the time/date or a user id. You can't modify log time behavior that way.
The easiest way I can think of right now to accomplish what you want is a custom appender like
package FileByCategoryAppender;
use warnings;
use strict;
use base qw( Log::Log4perl::Appender::File );
sub new {
my( $class, %options ) = #_;
$options{filename } = "no-category.log";
my $self = $class->SUPER::new( %options );
bless $self, $class;
}
sub log {
my( $self, %params ) = #_;
my $category = $params{ log4p_category };
$self->SUPER::file_switch( $category . ".log" );
$self->SUPER::log( %params );
}
1;
and then use it in your script like
use strict;
use warnings;
use Log::Log4perl qw( get_logger );
my $conf = q(
log4perl.category = WARN, Logfile
log4perl.appender.Logfile = FileByCategoryAppender
log4perl.appender.Logfile.create_at_logtime = 1
log4perl.appender.Logfile.layout = \
Log::Log4perl::Layout::PatternLayout
log4perl.appender.Logfile.layout.ConversionPattern = %d %F{1} %L> %m %n
);
Log::Log4perl::init(\$conf);
my $logger = get_logger("Bar::Twix");
$logger->error("twix error");
$logger = get_logger("Bar::Mars");
$logger->error("mars error");
which will result in two log files being created at log time:
# Bar.Mars.log
2012/11/18 11:12:12 t 21> mars error
and
# Bar.Twix.log
2012/11/18 11:12:12 t 21> twix error

Resources