Symfony2 $em->createQuery() also selects removed objects - symfony

Foreword: My script is a bit complicated so I try to reduce the complexity down to a simple example.
Imagine we have an entity "company". We have also a repository-function that looks like this:
public function delete(){
// get company-objects to delete
[...]
// delete some companies
$this->getEntityManager()->remove($company1);
$this->getEntityManager()->remove($company2);
// do other things
[...]
// get companies via createQuery (I really need to do it via
// createQuery because of different reasons that would be too
// complicated to explain for this problem)
$query = $this->getEntityManager()
->createQuery('
SELECT company
FROM MyOwnBundle:Company company
WHERE [...irrelevant...]
');
$companies = $query->getResult();
dump($companies);
exit;
}
The problem: The createQuery also selects the removed companies.
My thoughts:
I thought that the entityManager knows that the companies are removed and therefore doesn't select them in a query. But this thought was wrong... (But why is this? Isn't the entityManager a global singleton-object?)
I know that when I would persist the data with flush() before using createQuery the results would be fine. But I can't persist the data here because then the whole process wouldn't be in a transaction.
I also can't manipulate the where-part of createQuery to exclude the already deleted companies because the whole process is separated into many functions and it would be hard to transport an array with the removed companies through the whole process.
The Question: How can I get the companies via createQuery without the removed ones?

You could wrap your code like below to make your whole process be in a transaction. This way you could flush right after remove the data.
$this->getEntityManager()->transactional(function($em) {
// put your code here
// $em is instanceof EntityManager
});

Related

Entity Framework Core many-to-many relationship, EFC updates junction table ONLY when there is at least one existing relationship

everyone, sorry to bother you, hopefully someone can answer. I've been working on this project for a week now and I've come across a very weird bug to me and I can't wrap my mind around why it won't work. The thing is that it actually works just fine, but doesn't in one case scenario.
So it's an app like Spotify, I have Records (music) and Genres and it's a many-to-many relationship (created automatically by entity framework core convention of putting ICollection on both classes). This is the code in controller:
// this just gets the record with specified id and includes genres, tested and working
var record = await _unitOfWork.Records.Get(x => x.Id == id, new List<string> { "Genres" });
// checking if record is null
// mapping view model (dto) to the record from db to capture any changes on record itself
_mapper.Map(recordDto, record);
record.Genres ??= new List<Genre>();
foreach (var currentGenre in record.Genres)
{
// Attach() really only attaches
_unitOfWork.Genres.Attach(currentGenre);
if (!recordDto.GenresIds.Contains(currentGenre.Id))
{
record.Genres.Remove(currentGenre);
}
}
foreach (var genreId in recordDto.GenresIds)
{
var genre = await _unitOfWork.Genres.GetById(genreId);
// check if null
_unitOfWork.Genres.Attach(genre);
record.Genres.Add(genre);
}
//update function attaches the record and uses EntityState.Modified
_unitOfWork.Records.Update(record);
// save is just calling SaveChangesAsync() on the dbcontext
await _unitOfWork.Save();
return NoContent();
The bug is that when there are no relationship between Records and Genres in the database, it does nothing, no matter what IDs i put in a json array. but when there is at least one reliotionship in the junction table (the record must have at least one genre already), it works perfectly, tested on all cases. Insert works just fine, though. adds the record and relations with no problems. if you need more code, let me know! literally first time i had to ask for help, always tryna fix things myself. I could solve it by not allowing genres for records to be empty, but I really wanna
know the issue, cause one time that wont be the option. btw no errors, it jus tdoesn't work. I tested in debug mode with a breakpoint, it adds the genres nicely to the record object, but it just doesn't create relationships in the db. If anyone tries helping, thank you!!
EDIT: still wanna know what the issue is, but I'll probably end up just creating the junction table(s) myself to have more control. Should also be better on performance as I won't need to load any genres from db. It's just that I have to create a lot of them and wanted to save time.

TYPO3 does not persist consistently

I'm creating Events and want to bundle them into consolidated objects matched by title so I created an EventBundle repository which holds these objects and I register single events against it matching them by title into the Bundles.
Since I have a lot of troubles saving them I already went so far as to cache them locally which does help somewhat but still it's pretty bad.
public function registerEvent($event) {
//We are matching with the title of the event so we get that first
$title = $event->getEvTitle();
if(!isset($this->aBundles[$title]))
//Then we look up the event bundle for this title, if it does not exist this will return null
$this->aBundles[$title] = $this->findEventBundleByTitle($title);
if($this->aBundles[$title] != NULL) {
$this->aBundles[$title]->copyDetails($event);
$this->aBundles[$title]->setEvTitle($title);
$this->update($this->aBundles[$title]);
print_r("Update: $title\n");
}
else {
$objectManager = \TYPO3\CMS\Core\Utility\GeneralUtility::makeInstance('TYPO3\CMS\Extbase\Object\ObjectManager');
$this->aBundles[$title] = $objectManager->get('Ext\MyEvents\Domain\Model\EventBundle');
$this->aBundles[$title]->copyDetails($event);
$this->aBundles[$title]->setEvTitle($title);
$this->add($this->aBundles[$title]);
print_r("Add: $title\n");
}
}
public function findEventBundleByTitle($title){
$query = $this->createQuery();
$query->getQuerySettings()->setRespectStoragePage(FALSE);
$query->matching(
$query->equals('ev_title', $title)
);
$res = $query->execute();
$bundle = ($res->count()==0?NULL:$res->getFirst());
return $bundle;
}
Now running this I would expect to see one add for each title and then updates - which is true for the first run.
But on subsequent runs there are again some adds, it does not match some of the events to the title. With each subsequent run there are less and less adds until there are only updates. But when looking into the Database it shows multiple records with the same title now. A unique index will cause errors on the second run too as the lookup of the Object fails, sometimes without any pattern.
Any idea why this might happen? I can check to see the entries in the database between the runs so it's most likely that the lookup fails for some reason. But I'm totally out of ideas why that might be the case as it does work eventually, but there are a lot more than just 1-2 entries in the database for some of the events then...
Also confusing is the fact that after 5 runs all events do match consistently with some events being in the database 5 times at this point of time. But all matches are to the FIRST of those entries so it's not like it is not matched by the query, it's just being ignored until there are enough of them?!, all entries created due to the database lookup not returning anything are ignored after this point. Deleting them from the database by hand restarts the adding of spurious content again.
To answer it myself... I just found that within the copy function I copied over some properties of the Model that I probably should not copy which confuses TYPO3 and does break the saving to the DB.
So if someone stumbles across this, make sure you only copy valid data and not all properties of the Model as some of the properties might break functionality.

Trying to add virtual property to query result in Symfony2 with JMS

I have a simple task ahead of me, yet I find myself pretty much incapable of completing it.
My model is pretty complex, so I'll try to simplify for the sake of being specific.
I have two entities, Call and Caller, with entity repositories that I access via custom services. I am using JMS Serializer Bundle. All entities are mapped correctly, the the whole thing is working pretty fine. But I have this idea that I just can't make happen. To the point...
These are my entities described:
--Call--
#call_id
location
date_created
Caller
--Caller--
#caller_id
phone_num
The idea is to have a list of all calls with their fields for example:
New York, 2015.12.12. 20:07:06, Novak Djokovic, 3816976548 [YY]
London, 2015.12.13. 20:07:06, Jelena Jankovic, 3811116333 [XX]
Fields YY and XX represent the number of calls already in a database with that specific number.
I have a query that returns the list without YY and XX values, and I also have a separate query that returns number of calls from a specific number. The thing gets complicated when I try to join them. Not sure how to do that.
I read about VirtualProperty annotation for JMS, but failed to actually see how to use it this time (since it's not a good practice to access your repository or service from an Entity).
These are my methods:
1 - get list of all calls with callers
public function findAllCalls()
{
$callerAlias = "c";
//getAlias method returns the alias of the current entity (call)
$qb = $this->createQueryBuilder($this->getAlias());
$qb->leftJoin($this->getAlias() . '.caller', $callerAlias);
return $qb->getQuery()->getResult();
}
2 - get number of calls from a specific number based on a call as a parameter
public function getNumberOfCalls(Call $call) {
$callerAlias = "c";
$qb = $this->createQueryBuilder($this->getAlias());
$qb->leftJoin($this->getAlias() . '.caller', $callerAlias);
$qb->select("COUNT(" . $this->getAlias() . ".call_id)");
$qb->where($callerAlias.".phonenbr = ".$call->getPhoneNumber());
return $qb->getQuery()->getScalarResult();
}
Hoping to hear your opinions on this, 'cause I really struggled to find the sollution.

Doctrine: Why can't I free memory when accessing entities through an association?

I have an Application that has a relationship to ApplicationFile:
/**
* #ORM\OneToMany(
* targetEntity="AppBundle\Entity\ApplicationFile",
* mappedBy="application",
* cascade={"remove"},
* orphanRemoval=true
* )
*/
private $files;
A file entity has a field that stores binary data, and can be up to 2MB in size. When iterating over a large list of applications and their files, PHP memory usage grows. I want to keep it down.
I've tried this:
$applications = $this->em->getRepository('AppBundle:Application')->findAll();
foreach ($applications as $app) {
...
foreach ($app->getFiles() as $file) {
...
$this->em->detach($file);
}
$this->em->detach($app);
}
Detaching the object should tell the entity manager to stop caring about this object and de-referencing it, but it surprisingly has no effect on the amount of memory usage - it keeps increasing.
Instead, I have to manually load the application files (instead of retrieving them through the association method), and the memory usage does not increase. This works:
$applications = $this->em->getRepository('AppBundle:Application')->findAll();
foreach ($applications as $app) {
...
$appFiles = $this
->em
->getRepository('AppBundle:ApplicationFile')
->findBy(array('application' => $application));
foreach ($appFiles as $file) {
...
$this->em->detach($file);
}
$this->em->detach($app);
}
I used xdebug_debug_zval to track references to the $file object. In the first example, there's an extra reference somewhere, which explains why memory is ballooning - PHP is not able to garbage collect it!
Does anyone know why this is? Where is this extra reference and how do I remove it?
EDIT: Explicitly calling unset($file) at the end of its loop has no effect. There are still TWO references to the object at this point (proven with xdebug_debug_zval). One contained in $file (which I can unset), but there's another somewhere else that I cannot unset. Calling $this->em->clear() at the end of the main loop has no effect either.
EDIT 2: SOLUTION: The answer by #origaminal led me to the solution, so I accepted his answer instead of providing my own.
In the first method, where I access the files through the association on $application, this has a side effect of initializing the previously uninitialized $files collection on the $application object I'm iterating over in the outer loop.
Calling $em->detach($application) and $em->detach($file) only tells Doctrine's UOW to stop tracking the objects, but it doesn't affect the array of $applications I'm iterating on, which now have populated collection of $files which eat up memory.
I have to unset each $application object after I'm done with it to remove all references to the loaded $files. To do this, I modified the loops as such:
$applications = $em->getRepository('AppBundle:Application')->findAll();
$count = count($applications);
for ($i = 0; $i < $count; $i++) {
foreach ($applications[$i]->getFiles() as $file) {
$file->getData();
$em->detach($file);
unset($file);
}
$em->detach($applications[$i]);
unset($applications[$i]);
// Don't NEED to force GC, but doing so helps for testing.
gc_collect_cycles();
}
Cascade
EntityManager::detach should indeed remove all references Doctrine has to the enities. But it does not do the same for associated entities automatically.
You need to cascade this action by adding detach the cascade option of the association:
/**
* #ORM\OneToMany(
* targetEntity="AppBundle\Entity\ApplicationFile",
* mappedBy="application",
* cascade={"remove", "detach"},
* orphanRemoval=true
* )
*/
private $files;
Now $em->detach($app) should be enough to remove references to the Application entity as well as its associated ApplicationFile entities.
Find vs Collection
I highly doubt that loading the ApplicationFile entities through the association, in stead of using the repository to findBy() them, is the source of your issue.
Sure that when loaded through the association, the Collection will have a reference to those child-entities. But when the parent entity is dereferenced, the entire tree will be garbage collected, unless there are other references to those child entities.
I suspect the code you show is pseudo/example code, not the actual code in production. Please examine that code thoroughly to find those other references.
Clear
Sometimes it worth clearing the entire EntityManager and merging a few entities back in. You could try $em->clear() or $em->clear('AppBundle\Entity\ApplicationFile').
Clear has no effect
You're saying that clearing the EntityManager has no effect. This means the references you're searching for are not within the EntityManager (of UnitOfWork), because you've just cleared that.
Doctrine but not Doctrine
Are you using any event-listeners or -subscribers? Any filters? Any custom mapping types? Multiple EntityManagers? Anything else that could be integrated into Doctrine or its life-cycle, but is not necessarily part of Doctrine itself?
Especially event-listeners/subscribers are often overlooked when searching for the source of issues. So I'd suggest you start to look there.
If we are speaking about your first implementation you have extra links to the collection in the PersistentCollection::coll of the Application::files property - this object is created by Doctrine on Application instantiation.
With detach you are just deleting UoW links to the object.
There are different ways to fix this but a lot of hacks should be applied. Most nice way probably to detach also Application object and unset it.
But it is still preferable to use more advanced ways for a batch processing: some were listed in the other answer. The current way forces doctrine to make use proxies and throws extra queries to DB to get the files of the current object.
Edit
The difference between the first and the second implementation is that there are no circular references in the second case: Application::files stays with uninitialized PersistenceCollection (with no elements in coll).
To check this - can you try to drop the files association explicitly?
The trick is in PHP's garbage collector, that works a bit odd. First off all each time the scripts need memory it will allocate memory from RAM, even if you use unset(), $object = null, or other tricks to free the memory, the allocated memory will not be returned to Operating System till the script is not finished and the process associated with it killed.
How to fix that ?
Usually is done on Linux Systems
Create commands that runs the needed script with limit, offset parameters and re-run the needed scripts in small batches more times. In this way, the script will use less memory, and each time the memory will be freed when the script will be finished.
Get rid of Doctrine it balloons by himself the memory, PDO is much faster and less costly.
For that kind of task where objects in memory could lead to that leaks, you should use Doctrine2 iterators
$appFiles = $this
->em
->getRepository('AppBundle:ApplicationFile')
->findBy(array('application' => $application));
should be refactored in order to return a query object and not an ArrayCollection, then from that query object, you can easily call iterate() method and clean the memory after every object inspection
Edit
You have "hidden references" because detach operation will not delete the object in memory, it only tells to EntityManager not to handle it anymore. This is why you should use my solution or unset() the object with php function.

How to prevent Doctrine from lazy loading one to one relationsip?

EDIT: If you're having similar issues This Topic will be of interest to you
I have User and UserSettings with one to one bidirectional relationship. It appears that even if i do not use any UserSettings values in my page, doctrine lazy loads it anyways.
Is this expected behavior? Why is Doctrine fetching this data even though I'm not using it in my page? If I'm unable to stop it, I would have to join this UserSettings to User every time I retrieve user object, but this is so unnecessary.
What can I do to prevent this from happening?
Code that loads data:
->createQuery('SELECT p, u, s FROM TestPostBundle:Post p LEFT JOIN p.user u LEFT JOIN p.sub s WHERE p.id IN (:ids)')
->setParameter('ids', $ids)
->getResult();
The in twig I loop through posts and display Post data and associated user, but I never request any UserSettings variables, I'm not accessing them at all.
I've seen this question asked in a few places and am adding my answer here:
I came across this same problem and remember that the symblog tutorial gave an example of how to reduce the lazy loading by explicitly add left joins on the tables that you do not need. It seems strange to include tables on a join when you do not even want that data at all, but this way you will reduce all of those extra queries down to 1 and it does run faster.
Search for lazy loading - about 1/5 of the way down http://tutorial.symblog.co.uk/docs/customising-the-view-more-with-twig.html
To fix this for the user/userdata issue try adding this to the user repository and use to whenever you need to get all users even if you do not want userdata. It can be further enhanced by selecting partial: ->select('partial p.{user_id,name,}')
public function getAll($limit = 500) {
$qb = $this->createQueryBuilder('u')
->select('u', 'd')
->leftJoin('p.userdata', 'd')
if (false === is_null($limit))
$qb->setMaxResults($limit);
return $qb->getQuery()->getResult();
}
UPDATE
The symblog tutorial seems to be down and I'm leaving the link here for the time being in case its temporary. The relevant code is here in the answer.
I also faced same problem. It seems that when querying from inverse side doctrine also queries the owning side. See this discussion.

Resources