Trigger UniqueEntity after persisting and before flushing data - symfony

Question

I'm importing a kind of csv data via doctrine entity manger, however, I have a loop that performs a batch processing as it is mentioned at the level of the doc.
https://www.doctrine-project.org/projects/doctrine-orm/en/2.7/reference/batch-processing.html#bulk-inserts
$validationErrors = [];
foreach($data as $itrationNumber => $item) {
/** #var User|null $user **/
$user = $this->em->getRepository(User::class)->findOnBy(['email' => $item['email']]);
$user = ($user) ? $user : new User();
$user->setName('test');
$errors = $this->validator->validate($user);
if ($errors->count() === 0) {
$this->persist($user);
} else {
$validationErrors[] = $errors;
}
if ($itratioNumber% 100 === 0) {
$this->em->flush();
}
}
return $validationErrors;
Here is my User class, which has a unique constraint on the email field:
/**
* #ORM\Entity
* #UniqueEntity("email")
*/
class User
{
/**
* #ORM\Column(name="email", type="string", length=255, unique=true)
* #Assert\Email
*/
protected $email;
}
Unfortunately, if there is more than one row in my data that has an identical email address, UniqueEntity validation will not be triggered, simply, because users are persisted but are not flushed into the database.
Solution 1: Avoid the batch and do a flush at each iteration which is very violating and may provoke connection closed doctrine or kind of memory leak.
Solution 2: It was to create a custom constraint, which is inspired by UniqueEntity("email") and then check for each item's email address if there's a user already persisted with the same mail.
The problem that if the user already exists in the database, and we call em-> persist(), I cannot find any persisted object neither in $entityManager->getUnitOfWork()->getScheduledEntityInsertions() nor in $entityManager->getUnitOfWork()->getScheduledEntityUpdates().
Only during a new insertion that I hole the object persisted in the response of the function ->getScheduledEntityInsertions()
I would be very grateful if anyone has any idea how I could recover the entities persisted after the $entityManager->persist() step.
Or simply a 3rd solution which allows me to trigger a validation on the uniqueness of the email even in a batch context.

Answer 1

You should keep a second bit of information to see if an e-mail address was already processed:
$emails = [];
foreach($data as $itrationNumber => $item) {
if (isset($emails[$item['email']])) {
continue;
}
/** #var User|null $user **/
$user = $this->em->getRepository(User::class)->findOnBy(['email' => $item['email']]);
$user = ($user) ? $user : new User();
$user->setName('test');
if ($this->validator->validate($user)->count() === 0) {
$this->persist($user);
$emails[$item['email']] = true;
}
if ($itratioNumber% 100 === 0) {
$this->em->flush();
}
}
This code snippet does not normalize the e-mail address, this should be done first (strtolower and maybe removal of +foo gmail local parts for example).

Answer 2

Except if yo are WILLING to load the whole emails in a memory cache and play around that with REDIS or direct arrays, I can't see a second way but using the DB constraint of uniqueness (i.e doctrine flush)
Regarding your fear of memory overload, you can simply detach the persisted entity from the managed pool by calling the clear method. So your code would look something like this:
foreach($data as $itrationNumber => $item) {
/** #var User|null $user **/
$user = $this->em->getRepository(User::class)->findOnBy(['email' => $item['email']]);
$user = ($user) ? $user : new User();
$user->setName('test');
if ($this->validator->validate($user)->count() === 0) {
$this->persist($user);
$this->em->flush();
}
$this->em->clear();
}
I personnaly used this solution for a batch processing of huge CSV flow, and got inspired from this doctrine documentation section.
P.S I would think about DB transactions but might need more tricks.