EDIT: If you're having similar issues This Topic will be of interest to you
I have User and UserSettings with one to one bidirectional relationship. It appears that even if i do not use any UserSettings values in my page, doctrine lazy loads it anyways.
Is this expected behavior? Why is Doctrine fetching this data even though I'm not using it in my page? If I'm unable to stop it, I would have to join this UserSettings to User every time I retrieve user object, but this is so unnecessary.
What can I do to prevent this from happening?
Code that loads data:
->createQuery('SELECT p, u, s FROM TestPostBundle:Post p LEFT JOIN p.user u LEFT JOIN p.sub s WHERE p.id IN (:ids)')
->setParameter('ids', $ids)
->getResult();
The in twig I loop through posts and display Post data and associated user, but I never request any UserSettings variables, I'm not accessing them at all.
I've seen this question asked in a few places and am adding my answer here:
I came across this same problem and remember that the symblog tutorial gave an example of how to reduce the lazy loading by explicitly add left joins on the tables that you do not need. It seems strange to include tables on a join when you do not even want that data at all, but this way you will reduce all of those extra queries down to 1 and it does run faster.
Search for lazy loading - about 1/5 of the way down http://tutorial.symblog.co.uk/docs/customising-the-view-more-with-twig.html
To fix this for the user/userdata issue try adding this to the user repository and use to whenever you need to get all users even if you do not want userdata. It can be further enhanced by selecting partial: ->select('partial p.{user_id,name,}')
public function getAll($limit = 500) {
$qb = $this->createQueryBuilder('u')
->select('u', 'd')
->leftJoin('p.userdata', 'd')
if (false === is_null($limit))
$qb->setMaxResults($limit);
return $qb->getQuery()->getResult();
}
UPDATE
The symblog tutorial seems to be down and I'm leaving the link here for the time being in case its temporary. The relevant code is here in the answer.
I also faced same problem. It seems that when querying from inverse side doctrine also queries the owning side. See this discussion.
Related
With Doctrine and Symfony in my PHPUnit test method :
// Change username for user #1 (Sheriff Woody to Chuck Norris)
$form = $crawler->selectButton('Update')->form([
'user[username]' => 'Chuck Norris',
]);
$client->submit($form);
// Find user #1
$user = $em->getRepository(User::class)->find(1);
dump($user); // Username = "Sheriff Woody"
$user = $em->createQueryBuilder()
->from(User::class, 'user')
->andWhere('user.id = :userId')
->setParameter('userId', 1)
->select('
user
')
->getQuery()
->getOneOrNullResult()
;
dump($user); // Username = "Chuck Norris"
Why my two methods to fetch the user #1 return different results ?
diagnosis / explanation
I assume* you already created the User object you're editing via crawler before in that function and checked that it is there. This leads to it being a managed entity.
It is in the nature of data, to not sync itself magically with the database, but some automatism must be in place or some method executed to sync it.
The find() method will always try to use the cache (unless explicitly turned off, also see side note). The query builder won't, if you explicitly call getResult() (or one of its varieties), since you explicitly want a query to be executed. Executing a different query might lead to the cache not being hit, producing the current result. (it should update the first user object though ...) [updated, due to comment from Arno Hilke]
((( side note: Keeping objects in sync is hard. It's mainly about having consistency in the database, but all of ACID is wanted. Any process talking to the database should assume, that it only is working with the state at the moment of its first query, and is the only user of the database. Unless additional constraints must be met and inconsistent reads can occur, in which case isolation levels should be raised (See also: transactions or more precisely: isolation). So, automatically syncing is usually not wanted. Doctrine uses certain assumptions for performance gains (mainly: isolation / locking is optimistic). However, in your particular case, all of those things are of no actual concern... since you actually want a non-repeatable read. )))
(* otherwise, the behavior you're seeing would be really unexpected)
solution
One easy solution would be, to actively and explicitly sync the data from the database by either calling $em->refresh($user), or - before fetching the user again - to call $em->clear(), which will detach all entities (clearing the cache, which might have a noticable performance impact) and allowing you to call find again with the proper results being returned.
Please note, that detaching entities means, that any object previously returned from the entity manager should be discarded and fetched again (not via refresh).
alternate solution 1 - everything is requests
instead of checking the database, you could instead do a different request to a page that displays the user's name and checks that it has changed.
alternate solution 2 - using only one entity manager
using only one entity manager (that is: sharing the entity manager / database in the unit test with the server on the request) may be a reasonable solution, but it comes with its own set of problems. mainly omitted commits and flushes may avoid detection.
alternate solution 3 - using multiple entity managers
using one entity manager to set up the test, since the server is using a new entity manager to perform its work, you should theoretically - to do this actually properly - create yet another entity manager to check the server's behavior.
comment: the alternate solutions 1,2 and 3 would work with the highest isolation level, the initial solution probably wouldn't.
I'm creating Events and want to bundle them into consolidated objects matched by title so I created an EventBundle repository which holds these objects and I register single events against it matching them by title into the Bundles.
Since I have a lot of troubles saving them I already went so far as to cache them locally which does help somewhat but still it's pretty bad.
public function registerEvent($event) {
//We are matching with the title of the event so we get that first
$title = $event->getEvTitle();
if(!isset($this->aBundles[$title]))
//Then we look up the event bundle for this title, if it does not exist this will return null
$this->aBundles[$title] = $this->findEventBundleByTitle($title);
if($this->aBundles[$title] != NULL) {
$this->aBundles[$title]->copyDetails($event);
$this->aBundles[$title]->setEvTitle($title);
$this->update($this->aBundles[$title]);
print_r("Update: $title\n");
}
else {
$objectManager = \TYPO3\CMS\Core\Utility\GeneralUtility::makeInstance('TYPO3\CMS\Extbase\Object\ObjectManager');
$this->aBundles[$title] = $objectManager->get('Ext\MyEvents\Domain\Model\EventBundle');
$this->aBundles[$title]->copyDetails($event);
$this->aBundles[$title]->setEvTitle($title);
$this->add($this->aBundles[$title]);
print_r("Add: $title\n");
}
}
public function findEventBundleByTitle($title){
$query = $this->createQuery();
$query->getQuerySettings()->setRespectStoragePage(FALSE);
$query->matching(
$query->equals('ev_title', $title)
);
$res = $query->execute();
$bundle = ($res->count()==0?NULL:$res->getFirst());
return $bundle;
}
Now running this I would expect to see one add for each title and then updates - which is true for the first run.
But on subsequent runs there are again some adds, it does not match some of the events to the title. With each subsequent run there are less and less adds until there are only updates. But when looking into the Database it shows multiple records with the same title now. A unique index will cause errors on the second run too as the lookup of the Object fails, sometimes without any pattern.
Any idea why this might happen? I can check to see the entries in the database between the runs so it's most likely that the lookup fails for some reason. But I'm totally out of ideas why that might be the case as it does work eventually, but there are a lot more than just 1-2 entries in the database for some of the events then...
Also confusing is the fact that after 5 runs all events do match consistently with some events being in the database 5 times at this point of time. But all matches are to the FIRST of those entries so it's not like it is not matched by the query, it's just being ignored until there are enough of them?!, all entries created due to the database lookup not returning anything are ignored after this point. Deleting them from the database by hand restarts the adding of spurious content again.
To answer it myself... I just found that within the copy function I copied over some properties of the Model that I probably should not copy which confuses TYPO3 and does break the saving to the DB.
So if someone stumbles across this, make sure you only copy valid data and not all properties of the Model as some of the properties might break functionality.
Foreword: My script is a bit complicated so I try to reduce the complexity down to a simple example.
Imagine we have an entity "company". We have also a repository-function that looks like this:
public function delete(){
// get company-objects to delete
[...]
// delete some companies
$this->getEntityManager()->remove($company1);
$this->getEntityManager()->remove($company2);
// do other things
[...]
// get companies via createQuery (I really need to do it via
// createQuery because of different reasons that would be too
// complicated to explain for this problem)
$query = $this->getEntityManager()
->createQuery('
SELECT company
FROM MyOwnBundle:Company company
WHERE [...irrelevant...]
');
$companies = $query->getResult();
dump($companies);
exit;
}
The problem: The createQuery also selects the removed companies.
My thoughts:
I thought that the entityManager knows that the companies are removed and therefore doesn't select them in a query. But this thought was wrong... (But why is this? Isn't the entityManager a global singleton-object?)
I know that when I would persist the data with flush() before using createQuery the results would be fine. But I can't persist the data here because then the whole process wouldn't be in a transaction.
I also can't manipulate the where-part of createQuery to exclude the already deleted companies because the whole process is separated into many functions and it would be hard to transport an array with the removed companies through the whole process.
The Question: How can I get the companies via createQuery without the removed ones?
You could wrap your code like below to make your whole process be in a transaction. This way you could flush right after remove the data.
$this->getEntityManager()->transactional(function($em) {
// put your code here
// $em is instanceof EntityManager
});
I am trying to do some Bulk inserts into my database. I have read the article about it on the doctrine side and wanted to use sporadic flushs and clears in order to prevent high memory consumptions. Unfortunately all entities get detached in the process, not only the ones I am inserting, but also the relations to it.
I tried to remerge them or use references instead. In my current case I am using a reference and still I get the following error message:
[Doctrine\ORM\ORMInvalidArgumentException]
A new entity was found through the relationship
'Strego\TippBundle\Entity\Bet#betRound' that was not configured to
cascade persist operations for entity: LoadTest GameGroup. To
solve this issue: Either explicitly call EntityManager#persist() on
this unknown entity or configure cascade persist this association in
the mapping for example #ManyToOne(..,cascade={"persist"}).
The relevant coding is this:
// BetRound
print(PHP_EOL."Search for BetROund");
$betRounds = $em->getRepository('StregoTippBundle:BetRound')->findAll();
print(PHP_EOL.'found Betrrounds:'.count($betRounds));
$betRound = current($betRounds);
...
// References
$betRoundRef = $em->getReference('Strego\\TippBundle\\Entity\\BetRound', $betRound->getId());
and here the insert:
foreach($gameRefs as $game){
$bet = new GameBet();
$bet->setBetround($betRoundRef);
$bet->setUser($genuser);
$bet->setGame($game);
$bet->setScoreT1($this->getScore());
$bet->setScoreT2($this->getScore());
$bet->recalculatePoints();
$em->persist($bet);
}
if(($i % self::$batchSize) == 0){
$em->persist($userGroup);
$em->persist($mySelf);
$em->flush();
$em->clear();
$em->merge($betRound);
$em->merge($userGroup);
$em->merge($mySelf);
}
My whole Fixture for loading this data can be found here: https://gist.github.com/KeKs0r/a3006768db267311bb35
When calling the clear method everything is detached (Detaching entities).
You'll need to reload each previously loaded entity (in your case $betRoundRef, $genuser and probably $game too).
Have a look at this Stack Overflow answer
On my meteor project users can post events and they have to choose (via an autocomplete) in which city it will take place. I have a full list of french cities and it will never be updated.
I want to use a collection and publish-subscribes based on the input of the autocomplete because I don't want the client to download the full database (5MB). Is there a way, for performance, to tell meteor that this collection is "static"? Or does it make no difference?
Could anyone suggest a different approach?
When you "want to tell the server that a collection is static", I am aware of two potential optimizations:
Don't observe the database using a live query because the data will never change
Don't store the results of this query in the merge box because it doesn't need to be tracked and compared with other data (saving memory and CPU)
(1) is something you can do rather easily by constructing your own publish cursor. However, if any client is observing the same query, I believe Meteor will (at least in the future) optimize for that so it's still just one live query for any number of clients. As for (2), I am not aware of any straightforward way to do this because it could potentially mess up the data merging over multiple publications and subscriptions.
To avoid using a live query, you can manually add data to the publish function instead of returning a cursor, which causes the .observe() function to be called to hook up data to the subscription. Here's a simple example:
Meteor.publish(function() {
var sub = this;
var args = {}; // what you're find()ing
Foo.find(args).forEach(function(document) {
sub.added("client_collection_name", document._id, document);
});
sub.ready();
});
This will cause the data to be added to client_collection_name on the client side, which could have the same name as the collection referenced by Foo, or something different. Be aware that you can do many other things with publications (also, see the link above.)
UPDATE: To resolve issues from (2), which can be potentially very problematic depending on the size of the collection, it's necessary to bypass Meteor altogether. See https://stackoverflow.com/a/21835534/586086 for one way to do it. Another way is to just return the collection fetch()ed as a method call, although this doesn't have the benefits of compression.
From Meteor doc :
"Any change to the collection that changes the documents in a cursor will trigger a recomputation. To disable this behavior, pass {reactive: false} as an option to find."
I think this simple option is the best answer
You don't need to publish your whole collection.
1.Show autocomplete options only after user has inputted first 3 letters - this will narrow your search significantly.
2.Provide no more than 5-10 cities as options - this will keep your recordset really small - thus no need to push 5mb of data to each user.
Your publication should look like this:
Meteor.publish('pub-name', function(userInput){
var firstLetters = new RegExp('^' + userInput);
return Cities.find({name:firstLetters},{limit:10,sort:{name:1}});
});