symfony2 + doctrine2 - how lazy loading affect our applications? - symfony

This will be a sort of Q/A for everyone that wonder about symfony2 + doctrine2 and performance.
I'm writing this Q/A because I've started to playing around SF2+doctrine for a new project and I wanted to do some performance tests. I've surfed the net but didn't find a complete answer for my doubts so I've decided to make my own one. Hope this will be useful for someone
Let's say we have two entities like the following
//all ORM declaration here
class Lodging
{
//some declaration here
/**
* #ORM\ManyToOne(targetEntity="Currency")
* #ORM\JoinColumn(name="currency_iso_code", referencedColumnName="iso_code")
*/
protected $currency;
//some methods here
}
//all ORM declaration here
class Currency
{
/**
* #ORM\Id
* #ORM\Column(type="string", length=3)
*/
protected $iso_code;
/**
* #ORM\Column(type="string")
*/
protected $description;
//some methods here
}
Now you will made an entity type for Lodging where you will be able to create or - if the entity passed to the type has some prefetched values - to edit a Lodging object (let's concentrate on that case). In that form you also need the description of Currency, not only the $iso_code.
Question: is better to use ->findOneById() doctrine2 method or to write a DQL (or use query builder facility)?
Why?

PREMABLE
This answer has sense only if you're experiencing a slow page that's using lazy loads. For more common actions (load just one entity, load a "static page" or a cached one and so on) you could continue to use built in function provided by doctrine
Well, let's analyze some of the different methodologies you can follow
1 - findOneById()
Into controller you have chosen to follow more "common" and "fast" way to procede: use pre-existent findOneById() method. Wonderful.
Let's check performance
Number of queries done: three query were done as we need one to retrieve Lodging object, the second one to fetch currently associated Currency (lazy loaded) and one to fetch all Currency entities (because you can change from one currency to other)
Page loading time: about 500ms
Memory usage: about 32MB
2 - Write a custom repository method
2.1 "Basic" DQL repository function
public function findLodgingById($id)
{
$lodging = null;
$q = $this->createQueryBuilder('lodging')
->select('lodging')
->where('lodging.id = :id')
->setParameter('id', $id)
->getQuery();
$lodging_array = $q->getResult(); //will not fetch a single object but array
if ($lodging_array) {
$lodging = reset($lodging_array);
}
return $lodging;
}
Let's check performace
Number of queries done: it shouldn't be a surprise for you but ... number of query done is always three! Of course you're not doing anything but same of findOneById() (an, maybe, even in a worst way!). You're taking advantage of lazy loading again.
Page loading time: about 500ms. Loading time didn't change
Memory usage: about 34MB. Memory usage is increased of 6,25 % (due to array?)
2.2 DQL with JOIN
public function findLodgingById($id)
{
$lodging = null;
$q = $this->createQueryBuilder('lodging')
->select('lodging')
->leftJoin('lodging.currency', 'currency')
->where('lodging.id = :id')
->setParameter('id', $id)
->getQuery();
$lodging_array = $q->getResult(); //will not fetch a single object but array
if ($lodging_array) {
$lodging = reset($lodging_array);
}
return $lodging;
}
Let's check performace
Number of queries done: The number of queries dind't change! But ... why? We are telling explicitly to doctrine2 to join currency entity but it seems to ignore that instruction. The answer is that we're not selecting currency entity also so doctrine2 will use, again, lazy loading facility.
Page loading time: about 500ms. Loading time didn't change 2.1
Memory usage: about 34MB. Memory usage didn't changed from 2.1
2.3 Let's try something better: Join with Currency selection
public function findLodgingById($id)
{
$lodging = null;
$q = $this->createQueryBuilder('lodging')
->select('lodging', 'currency')
->leftJoin('lodging.currency', 'currency')
->where('lodging.id = :id')
->setParameter('id', $id)
->getQuery();
$lodging_array = $q->getResult(); //will not fetch a single object but array
if ($lodging_array) {
$lodging = reset($lodging_array);
}
return $lodging;
}
Let's check performace
Number of queries done: Finally number of queries decreased! We reach two query instead of three. What query gone? Lazy loading of associated (current) currency is gone but, of course, you have to fetch all possible currency.
Page loading time: about 350ms.
Memory usage: about 34MB. Memory usage didn't changed
Definitive Solution (?)
public function findLodgingById($id)
{
$lodging = null;
$q = $this->createQueryBuilder('lodging')
->select('lodging')
->where('lodging.id = :id')
->setParameter('id', $id)
->getQuery();
$q->setHint(Query::HINT_FORCE_PARTIAL_LOAD, true);
$lodging_array = $q->getResult(); //will not fetch a single object but array
if ($lodging_array) {
$lodging = reset($lodging_array);
}
return $lodging;
}
The $q->setHint(Query::HINT_FORCE_PARTIAL_LOAD, true); line of code seems to save time and memory compared to other solutions.
Number of queries done: Two, of course
Page loading time: about 340ms.
Memory usage: about 32MB.
PLEASE PAY ATTENTION
This solution doesn't let you change associated (Currency) entity as it will be betched with Query::HINT_FORCE_PARTIAL_LOAD, true
Comments
Results seems to be good for page loading time (memory usage of course will not change) and though performances seems to be only "a little" better, you shouldn't not focus ONLY onto results: here we're taking as an example only a simple snippet of code with just one lazy loading operation: think about an entity (or, worst, a lot of entity like blog posts with related comments) that will do wrong(*) lazy loading for every entity fetched and managed: you could reach even 50-70 query for a single page (and of course, in this case, you could notice easily performace benefits due to "single" query)
(*) Why I say wrong? Because if you can migrate the fetching-logic of your objects elsewhere or if you already know what entity/attribute you need, so your contents aren't dynamic and you can know them before use, lazy-loading is not only useless but also harmful.
Contrariwise if you can't know at "coding writing time" what properties or entities you'll need, of course lazy loading could save you memory (and time) wasting for useless objects/associations.
Final thoughts
It's better to "lose" some minutes for write DQL query (that seems to be even silly) that use "built-in" ones. Moreover you should use array (and not objects) for read-only operation (list elements that couldn't be modificated) changing getResult() method call as follows: getResult(Doctrine\ORM\Query::HYDRATE_ARRAY);. That will change "default" value (HYDRATE::OBJECT)

It seems like premature optimisation to be worrying about it before you're seeing a problem.
Write whatever's quickest, and in this case, that's often going to be $em->findOneById($id) style. Then use valgrind to look for bottlenecks.
Looking at the amount of time you spend writing all that custom DQL, the overall peformance could be improved by fixing a bigger problem somewhere else in your application.

Related

Akeneo 2.2.8 : How can I get the original attribute data in the akeneo.storage.pre_save event?

I'm using Akeneo 2.2.8 and I'm trying to use the akeneo.storage.pre_save-event to compare the original product data with the new data provided. I do this by subscribing to the akeneo.storage.pre_save-event:
In event_subscribers.yml:
parameters:
vendor.bundle.event_subscriber.product_save.class: Vendor\Bundle\CustomBundle\EventSubscriber\ProductSaveSubscriber
services:
vendor.bundle.event_subscriber.product_save:
class: '%vendor.bundle.event_subscriber.product_save.class%'
arguments:
- '#pim_catalog.repository.product'
tags:
- { name: kernel.event_listener, event: akeneo.storage.pre_save, method: onPreSave, priority: 255 }
In ProductSaveSubscriber.php:
/**
* #var ProductRepositoryInterface
*/
protected $productRepository;
public function __construct(ProductRepositoryInterface $productRepository)
{
$this->productRepository = $productRepository;
}
public function onPreSave(GenericEvent $event)
{
/** #var Product $subject */
$subject = $event->getSubject();
if ($subject instanceof Product) {
$originalProduct = $this->productRepository->findOneByIdentifier($subject->getIdentifier());
foreach ($subject->getAttributes() as $attribute) {
if ($attribute->getReadOnly()) {
echo "{$attribute->getCode()} : {$subject->getValue($attribute->getCode())}\n";
echo "{$attribute->getCode()} : {$originalProduct->getValue($attribute->getCode())}\n";
}
}
}
}
Now when I run this code, I expect the second echo-statement to give the original data (since I've loaded that anew). However, the original product I load from the repository also has the new data.
Another thing to note here is that if I add a die()-statement, the data is not stored in the database. So it seems that the repository returns the in-memory model or something like that.
Can anyone point me in the right direction? Or am I using the wrong approach to compare newly-entered data with already existing data?
Now when I run this code, I expect the second echo-statement to give
the original data (since I've loaded that anew). However, the original
product I load from the repository also has the new data.
This might be because the object is referenced in doctrine's unit of work. So when you use the repository to fetch what you think is the original object, might actually be the same object you've updated.
Another thing to note here is that if I add a die()-statement, the
data is not stored in the database. So it seems that the repository
returns the in-memory model or something like that.
That's because since your subscriber is listening on the PRE_SAVE event, the updated product has not been flushed in the database yet. Saving a product goes this way:
PRE_SAVE event thrown
COMMIT / FLUSH
POST_SAVE event thrown
So if you call die during the PRE_SAVE event, the COMMIT / FLUSH won't be called.
Can anyone point me in the right direction? Or am I using the wrong
approach to compare newly-entered data with already existing data?
I don't know your particular use case but you might want to use a Query function (see https://github.com/akeneo/pim-community-dev/blob/2.2/src/Pim/Bundle/CatalogBundle/Doctrine/ORM/Query/FindAttributesForFamily.php). It's purpose is to directly fetch in the database the data you need (it will be the original value since the product hasn't been flushed in DB on PRE_SAVE)
I hope this helps.

Dynamically eager loading deep relationships with Doctrine

I'm currently working on an API using the following stack;
Symfony (3)
FOSRestBundle
Fractal
I'm wanting to integrate the ability to specify, via query parameter, which relationships to include when retrieving an entity/collection, e.g;
[GET] /users?include=friends.addresses
Fractal comes with the ability to handle includes however, as this happens around the serialization point of the response building, each related entity is retrieved via lazy loading, thus triggering additional queries.
Is there a way to tell Doctrine, when retrieving a collection, to dynamically also retrieve relationships specified? Ive seen the following from the Doctrine docs which shows how to dynamically change the fetch mode however this only seems to work with associations on the target entity (friends in the example above) and not deeper relations (addresses of friends in the example).
Thanks!
If I remember correctly you can "preload" relations by joining them in rather than letting the lazy loading mechanism handle it. An idea could be to create a service that creates a query builder based on your criteria. This is a crude snippet of what I mean:
class EagerService
{
protected $em;
public function __construct(EntityManager $em)
{
$this->em = $em;
}
public function resolveIncludes($class, $alias, $includes)
{
// Parse includes into an array
if (strpos($includes, '.') !== false) {
$relations = explode('.', $includes);
} else {
$relations = [$includes];
}
// The next relation is owned by the previous one, so we keep track of the previous relation
$previousRelation = $alias;
$qb = $em->getRepository($class)->getQueryBuilder($previousRelation);
foreach ($relations as $relation) {
// Add inner joins to the query builder referencing the new relation
$qb->innerJoin("{$previousRelation}.{$relation}", $relation);
$previousRelation = $relation;
}
// Return query builder or the result of the query
return $qb;
}
}

PHP: Symfony2 Doctrine "The discriminator column "discr" is missing" HydrationException with filter

fellow programmers :)
I'm having a bad case of no idea what's going on.
This concerns two entities: Ware and File.
In my repository, I have a function that returns File objects along with wares to avoid lazy loading:
The relevant part of this function ( because the exception happens even if i trigger this bit only ) is this:
public function findByWithFilesTotal($params, $page = false, $per_page = false){
$res = $this->_em->createQuery('SELECT w, f FROM ShopBundle:Ware w JOIN w.files f');
// same result occurs with LEFT JOIN
return count($res->getResult());
}
Important stuff:
1) Ware and File classes are direct descendants of class Data. Discriminators are all right.
2) Ware has OneToMany relation with File - that means File table has ware_id column.
3) This is the most important part ( IMHO ). I use this filter to separate deleted items in all Data descendants.
class UndeletedFilter extends SQLFilter
{
public function addFilterConstraint(ClassMetadata $targetEntity, $targetTableAlias)
{
if(!$targetEntity->hasField('deleted'))
return "";
return "( $targetTableAlias.deleted = 0 OR $targetTableAlias.deleted IS NULL)";
}
}
It works for all the other entities, but causes a HydrationException with message 'The discriminator column "discr" is missing for "Acme\CoreBundle\Entity\File" using the DQL alias "f".' for this one query.
This, however, DOES NOT happen if I either remove JOIN w.files from DQL query, and leave it to lazy loading, or return an empty string from addFilterConstraint() method even if it does have the 'deleted' field.
So, if anyone knows: What exactly causes this, and how do solve it?
Thank you in advance :)
One reason for trigger this exception in when you are working inherent classes, then for example, if your discriminator column has null values the QueryBuilder will do not know how to convert this unknown type based in type.

Symfony2 Doctrine2 exceeding memory limit, Allowed memory size of 1073741824 bytes exhausted

I have a relational database for posts.
Post Table <-- OneToMany
Post cat Table <-- ManyToOne
Category Table <-- OneToMany
If I use the Doctrine #ORM to join the tables, in Entities using annotation. I get a white screen and in the error logs show the error:
emergency.EMERGENCY: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 1052508160 bytes) {"type":1,"file":"/[PATH TO SYMFONY]/vendor/twig/twig/lib/Twig/Extension/Debug.php","line":66}
I have upped the memory limit several times, from 64M to 1024M.
Has anyone else found this problem? I think that a file executing > a Gig of memory is no good.
If I write the query using the query builder I get the results I expect. I think if I could get the Doctrine relational mapping to work this would be better. does any one have an opinion on this?
I would love a bit of advice on this matter.
Thanks In advance.
Joe
------------ Edit in response to comment ---------------------------------------
Thanks for your comment #Cerad. There are only about 10 rows in the database. Also i'm in app_dev.php.
Here are some excerpts form my files.
post table
class Post
{
//... ^ table collumns
/**
* #ORM\OneToMany(targetEntity="PostCats", mappedBy="Post")
*/
protected $PostCats;
public function __construct()
{
$this->PostCats = new ArrayCollection();
}
}
post cat joining table.
class PostCats
{
//... ^ table collumns
/**
* #ORM\ManyToOne(targetEntity="Post", inversedBy="PostCats")
* #ORM\JoinColumn(name="postid", referencedColumnName="id")
*/
protected $Post;
}
the controller
$posts = $this->getDoctrine()
->getRepository('comPostBundle:Post')
->find(7);
if (!$posts) {
throw $this->createNotFoundException(
'No product found for id '.$posts
);
}
return new Response(print_r($posts))
Result.... White screen...
I have also tried returning the results dumped into a twig template.
Do you think its ok to skip the Doctrine relational mapping and just write joins in the entity repositories?
So The problem has been solved thanks to #Cerad.
The issue was that I was ether doing a print_r() in the PHP, or a {{ dump() }} in the twig template. these functions do not like entities or displaying large arrays/objects.
now I am just calling what parts of the returned values I want and not dumping the whole data. and it works fine!.
EDIT:
This works for dumping data
\Doctrine\Common\Util\Debug::dump($object);

$em->remove() symfony2 erasing all rows

I have an issue when erasing something from a BD.
The problem is that it not only erase the object i looked for (using findOneBy), but all the objects related to the principal id.
//---Controller
$new = $this->getDoctrine()->getManager();
$OBJcar = $new->getRepository('SomeOtherBundle:CarEntityClass')
->findOneBy(array('idOwner' => $idowner, 'idCar' => $idcar));
if($OBJcar){
$new->remove($OBJcar);
$new->flush();
$msj="The car for an specific owner has been erased.";
}
//---Profiler (Query)
"START TRANSACTION"
Parameters: { }
Time: 0.22 ms
DELETE FROM schema.CarTable WHERE id_owner = ?
Parameters: ['123456']
Time: 0.63 ms
"COMMIT"
Parameters: { }
Time: 0.63 ms
How to erase the one row i am getting from the db?
I voted down the answer above, because I'm tired of people using string DQLs.
It's not standartized, non-object oriented(even though in background dql operates with objects), it doesn't use caching mechanisms query builder provides, it's non-flexible and simply looks unclean.
Here is the "right way"(IMHO):
You add the repository class for entity
You add the method you need with a query builder in it
You call the method while passing parameters needed for specific REPOSITORY OBJECT ORIENTED ACTION
You get an easy-to-handle result
Here's the code:
namespace ProjectName\BundleName\Repository;
use Doctrine\ORM\EntityRepository;
class CarRepository extends EntityRepository
{
public function deleteCarWithOwner($ownerId,$carId)
{
$isDeleted = $this->createQueryBuilder("car")
->delete()
->where('car.id = :carId')->setParameter("carId", $carId)
->andWhere('car.idOwner = :ownerId')->setParameter("ownerId", $ownerId)
->getQuery()->execute();
return $isDeleted;
}
}
Also, refer to http://doctrine-orm.readthedocs.org/en/latest/reference/query-builder.html for query builder details. There are a lot of "pros" and I see no "cons" for using builder.
LATE UPDATE
Also, a lot of Doctrine's entity events are not dispatched when using DQL.
Use DQL
$query = $em->createQuery('DELETE SomeOtherBundle:CarEntityClass c WHERE c.idOwner = 4 AND c.id = 10');
$query->execute();
This will remove only single car with ID 10 and owner with ID 4.

Resources