It seems that the default doctrine listener used by FOSElasticaBundle does not support bulk index by default. I have an application where I want to add support for more complex search queries through ElasticSearch. The search engine only will perform queries through one unique entity Post. When I create, edit or delete there is not any problem, the index in elasticsearch is updated automatically through the listener. My problem comes when I want to do bulk updates to hide or show more than one post at once, the listener is not receiving the signal to make the bulk index update in elasticsearch.
I am new to FOSElasticSearch so I do not know if I am missing something. I am using FOSElasticaBundle 6, Symfony 5.2 and ElasticSearch 7
Here you can find my fos_elastica.yaml
fos_elastica:
messenger: ~
clients:
default: { host: 127.0.0.1, port: 9200 }
indexes:
product:
settings:
index:
analysis:
analyzer:
my_analyzer:
type: snowball
language: English
persistence:
# the driver can be orm, mongodb or phpcr
driver: orm
model: App\Entity\Post
listener: { enabled: true }
provider: ~
finder: ~
elastica_to_model_transformer:
ignore_missing: true
properties:
title: { boost: 10, analyzer: my_analyzer }
tags: { boost: 8, analyzer: my_analyzer }
description: { boost: 6, analyzer: my_analyzer }
ispublished: { type: integer}
And here you can find the way I am updating more than once entity element at once in PostRepository (the function is to update all post from one unique author, it is just an example):
public function bulkUpdate($ispublished, $authorid){
return $this->createQueryBuilder('p')
->update()
->set('p.ispublished', $ispublished)
->andWhere('p.authorid = :id')
->setParameter('id', $authorid)
->getQuery()
->execute();
}
Also I found that I could disable default listener, dispatch messages for each create, update or delete action through symfony/messenger and consume them async in the background. I guess that I should create my own handler and dispatch specific messages (although I could not find an example about this in the doc) in each modifying action, although at the end I also have the same problem, as I do not know how to send a bulk index update/delete action
In the other hand I was thinking in executing all time a background script in python to check what rows were modified in mysql database and update those index with the script directly through ElasticSearch Api
I do not think that I will need to update more than 1k posts at once, so I would like to keep using the default listener to update posts automatically and avoid gaps between that an entity is modified and the index is updated in ElasticSearch. I just need to find the best way to update indexes in bulk as I have everything else already implemented and working
Sorry for all the text but I wanted to give all details about what I need to do
Related
I use the symfony bundle Foselasticabundle and i am facing a problem.
I have an entity (B) which can be deleted via gedmo softdeleteable.
I created the mapping, via the YAML file, and when I execute the following command fos:elastica:populate i get an error.
Entity of type 'App\Entity\B' for IDs id(XX) was not found
In fact, value was previously deleted in my database...
I would have liked him to insert an empty value in the field
Do you have a solution?
Thank you for your answers
fos_elastica.yaml
clients:
default: { url: '%env(ELASTICSEARCH_URL)%/' }
indexes:
app:
types:
A:
properties:
id: ~
name: ~
B:
type: object
id: ~
persistence:
driver: orm
model: App\Entity\A
I am having troubles while writing a controller-action inside a Symfony project, that should return data (in this particular case orders of a web-shop). Yeah ... It's a kind of a REST-API. That route just get's called from some JavaScript. And the data has to be visualized on the client-side.
The Problem:
I cannot find out, why the serialization of related entities results in empty objects. In this example it is the user of an order, which is empty.
This is a sample output:
orders: [
{
id: 2,
created: '2016-05-04T11:40:27+00:00',
user: {},
}
]
When I do something like
$orders = $this->getDoctrine()->getRepository('AppBundle:Order')
->findAllCompleted();
$serializationContext->setSerializeNull(true);
$serializationContext->setGroups(['statistics']);
$json = $serializer->serialize($orders, 'json', $serializationContext);
$response = new Response($json, $statusCode, [
'Content-Type' => 'application/json',
]);
return $response;
... i get a nice JSON response from the server, but every related entity of each order, like let's say user is {} (empty).
Even if I dump the related entity before it gets serialized like so:
[...]
$myOrder = array_filter($orders, function($order) {
if ($order->getId() == 2) {
return true;
}
return false;
});
dump($myOrder[0]->getUser());
die();
... it results in an empty (unhydrated) entity.
But if I change this debugging code to:
$myOrder = array_filter($orders, function($order) {
if ($order->getId() == 2) {
return true;
}
return false;
});
dump($myOrder[0]->getUser()->getUsername());
die();
... I get a clear output (string) with the value of the username of that entity.
So I think the issue is about a non hydrated entity, and not the serializer or its wrong configuration.
How can I get the JMSSerializer to take care of the hydration of those related entities?
I didn't find any hint in the docs ...
BTW, this are the JMS entity configs of order and user.
AppBundle\Entity\User:
exclusion_policy: ALL
properties:
userMeta:
expose: true
address:
expose: true
username:
expose: true
email:
expose: true
isReseller:
expose: true
acceptPublicOrders:
expose: true
vatNumber:
expose: true
taxRate:
expose: true
AppBundle\Entity\Order:
exclusion_policy: NONE
properties:
id:
groups: ['statistics']
user:
groups: ['statistics']
configuration:
groups: ['statistics']
created:
groups: ['statistics']
invoiceItems:
groups: ['statistics']
exclude: true
I think your problem is caused by doctrine lazy loading maybe you can try to change the fetch mode of the User association to EAGER in your Order entity
#ManyToOne(targetEntity="Cart", cascade={"all"}, fetch="EAGER")
By default i think it doesn't fetch the associations unless you call it directly like you did here
dump($myOrder[0]->getUser()->getUsername());
https://www.doctrine-project.org/projects/doctrine-orm/en/latest/reference/annotations-reference.html#annref-onetoone
Or this if you use DQL
14.7.6.6. Temporarily change fetch mode in DQL
http://doctrine-orm.readthedocs.io/en/latest/reference/dql-doctrine-query-language.html#temporarily-change-fetch-mode-in-dql
Edit : i was wrong
I made some tests everything worked fine with lazy loading or eager until i tried with groups, even if the fields are exposed you don't use the Default group so it only take the things with the 'statistics' group on it
Try to add the default group here
$serializationContext->setGroups(['Default','statistics']);
Or add the statistics group on your user fields both worked for me
I have generated a Grid with CRUD actions for my Labellisation entity on Sylius.
The grid is displayed well, but I would like to get associated elements too (the defaultAdresse of the -> customer of the -> current labellisation), so I need to use a custom repository method.
I tried to do this with this conf :
labellisation_grid:
resource: |
alias: grid.label
criteria:
valide: true
except: ['create', 'delete', 'update']
grid: public_labels
templates: LabelBundle:public/Crud
type: sylius.resource
defaults:
_sylius:
repository:
method: findAllValides
(adding all the defaults block), but I have an error because the method findAllValides is not defined. I do have a findAllValides method in my LabellisationRepository.
Debugging the ResourcesResolver, I saw in the getResource that the $repository passed to this function has a customRepositoryClassName = LabelBundle\Repository\LabellisationRepository (this path is the good one to my LabellisationRepository).
Is there something wrong with my code ?
I'm trying to start sending my logs into elastic search using monolog. (I'm using Symfony2).
I've set up monolog like this:
monolog:
handlers:
elasticsearch:
elasticsearch:
host: %logger_elastic_host%
port: %logger_elastic_port%
type: elasticsearch
level: info
It worked only few minutes until it broke with this error messages(a fatal error, I removed useless stuff):
create: /monolog/logs/AVQKYsGRPmEhlo7mDfrN caused
MapperParsingException[failed to parse [context.stack.args]]; nested:
ElasticsearchIllegalArgumentException[unknown property [class]];
I've been looking with my collegue how to fix that. What we found out is:
Elastic search receive the first logs and automatically build a mapping
We send new logs with another mapping or slightly different to what was sent before and it breaks.
In this case it's breaking here: context.stack.args.
The problem is that the context will always be very different.
What we would like is:
is anyone out there using Monolog to log to Elasticsearch
How do you guys manage to avoid this issue. (How can we manage to avoid it)?
thanks guys.
This is happening because ES creates a mapping from the first document. If any document that is inserted after has the same property but with other type/format then ES will throw an error.
A solution is to create a custom monolog formatter and register it:
config.yml:
elasticsearch:
type: elasticsearch
elasticsearch:
host: elasticsearch
ignore_error: true
formatter: my_elasticsearch_formatter
This line will make Monolog\Handler\ElasticSearchHandler ignore any other errors from Ruflin's Elastica package:
ignore_error: true
Then register a service with this name: my_elasticsearch_formatter:
<service id="my_elasticsearch_formatter" class="AppBundle\Services\MyFormatter">
<argument type="string">monolog</argument>
<argument type="string">logs</argument>
</service>
first argument is the index name, second arg is the type.
And the formatter class:
<?php
namespace AppBundle\Services;
use function json_encode;
use Monolog\Formatter\ElasticaFormatter;
use function var_dump;
class MyFormatter extends ElasticaFormatter
{
/**
* #param string $index
* #param string $type
*/
public function __construct($index, $type)
{
parent::__construct($index, $type);
}
/**
* #param array $record
* #return array|\Elastica\Document|mixed|string
*/
public function format(array $record)
{
$record['context'] = json_encode($record['context']);
return parent::format($record);
}
}
The downside of this solution is that it will json_encode the context. You will not be able to filter by inner properties of the context in ES but at least you will not lose important information about your logs.
I use FOSElasticaBundle in my Symfony 2 project. Since today reindexing is resulting in the below error:
index: /app/hotel/1 caused MapperParsingException[failed to parse
[priceFrom]]; nested: NumberFormatException[For input string:
"410.00"];
In my doctrine orm yml the priceFrom field is defined as followed:
priceFrom:
type: decimal
nullable: true
precision: 7
scale: 2
comment: ''
column: price_from
My fos_elastica config looks like this (config.yml):
fos_elastica:
clients:
default: { host: localhost, port: 9200 }
indexes:
app:
types:
hotel:
mappings:
id: ~
active: ~
priceFrom: { type: integer }
persistence:
driver: orm
model: XXX\XXXBundle\Entity\Hotel
provider: ~
listener:
immediate: ~
finder: ~
The command I use to reindex: php app/console fos:elastica:populate
The above setup has worked until now. I hope someone can point my to the good direction to solve this problem.
Versions:
ruflin/elastica (2.1.0)
friendsofsymfony/elastica-bundle (v3.1.5)
symfony/symfony (v2.6.11)
PS: No other entities in my project are using a priceFrom field.
In mappings, you define PriceFrom as integer but then you pass a decimal.
I haven't tested it but it definitely seems the major candidate as the culprit.
Francesco Abeni is right with answer. If you are already pushed something in ES as integer (or ES defined it as integer) it will generate exception when you will try to save decimal data here.
I always explicitly specify type in mapping like:
id: {"type" : "integer"}
shop_id: {"type" : "integer"}
source: {"type" : "string", "index" : "not_analyzed"}
There I see two ways to solve problem.
index alias and index merge
specify type in mapping; kill index; populate in again
I used second variant on a dev :)