elasticsearch doesn't returns all hits - symfony

I'm using Symfony 2.3 and ElasticSearchBundle 3.0. I implemented two fields for the search. The search works correctly but it doesn't display all results. For example: when I search for the a keyword, the number of hits are 33 hits but it returns only 10 results.
config.php
fos_elastica:
clients:
default: { host: localhost, port: 9200 }
serializer:
callback_class: FOS\ElasticaBundle\Serializer\Callback
serializer: serializer
indexes:
hortis:
finder: ~
client: default
settings:
index:
analysis:
analyzer:
custom_search_analyzer:
type: custom
tokenizer: standard
filter : [standard, lowercase, asciifolding]
custom_index_analyzer:
type: custom
tokenizer: standard
filter : [standard, lowercase, asciifolding, custom_filter]
filter:
custom_filter:
type: edgeNGram
side: front
min_gram: 3
max_gram: 100
types:
business:
mappings:
name: { search_analyzer: custom_search_analyzer, index_analyzer: custom_index_analyzer, type:string }
enabled: ~
gouvernaurat: ~
delegation: ~
postal_code: ~
# activities.principal: { search_analyzer: custom_search_analyzer, index_analyzer: custom_index_analyzer, type:string }
activities :
type : object
properties :
principal : ~
persistence:
driver: orm
model: Toto\AdminBundle\Entity\EntityName
provider: ~
listener: ~
finder: ~
controller.php
public function searchEngineAction(Request $request) {
$finder = $this->container->get('fos_elastica.index.hortis.business');
// get data from both fields
$querystring = strip_tags($request->get('name'));
$querystring2 = strip_tags($request->get('location'));
$boolQuery = new \Elastica\Query\Bool();
// if both fields are empty then display all businesses
if (empty($querystring) and empty($querystring2)) {
$query = new \Elastica\Query\MatchAll();
$boolQuery->addMust($query);
} else {
// create a boolean query
if (!empty($querystring)) {
$fieldQuery = new \Elastica\Query\QueryString();
$fieldQuery->setFields(array('name', 'activities.principal'));
$fieldQuery->setQuery($querystring);
$boolQuery->addMust($fieldQuery);
}
if (!empty($querystring2)) {
$fieldQuery2 = new \Elastica\Query\QueryString();
$fieldQuery2->setFields(array(
'gouvernaurat', 'delegation', 'postal_code'));
$fieldQuery2->setQuery($querystring2);
$boolQuery->addMust($fieldQuery2);
}
}
// select only enbaled business
$enabled = new \Elastica\Query\Term();
$enabled->setTerm('enabled', true);
$boolQuery->addMust($enabled);
$findAll = \Elastica\Query::create($boolQuery);
$findAll->setSize(27);
// trigger search function
$elasticaResultSet = $finder->search($findAll);
dump($elasticaResultSet);
// get results from
$findbusinesses = $elasticaResultSet->getResults();
$noresult = '';
if (!$findbusinesses) {
$noresult = 'no result';
}
$em = $this->getDoctrine()->getManager();
$FrontSettings = $em->getRepository('TotoAdminBundle:FrontSettings')->getFrontSettings();
if (!$FrontSettings) {
throw $this->createNotFoundException('Unable to find frontSettings entity');
}
// get all categories and activities
$categories = $em->getRepository('TotoAdminBundle:Category')
->findBy(array(), array('order' => 'ASC'));
if (!$categories) {
throw $this->createNotFoundException('unable to find categories and activities');
}
$paginator = $this->get('knp_paginator');
$businesses = $paginator->paginate(
$findbusinesses, $this->get('request')->query->get('page', 1)/* page number */, 9/* limit per page */
);
return $this->render('TotoFrontBundle:Front:search_result.html.twig', array(
'querystring' => $querystring, 'businesses' => $businesses,
'FrontSettings' => $FrontSettings, 'noresult' => $noresult,
'categories' => $categories,
));
}
How can I display the all the hits?

Elasticsearch by default only returns the first 10 results. This setting can be modified by specifing the from and size parameters. Note that it rarely makes sense to display all results on one page, instead use a pagination with a controllable amount of viewed items.
If you want all hits on one page also consider using the scroll api as deep pagination can get very inefficiently when having a high amount of results.

Related

api_platform produces Error "no handler found for uri [/index/_doc/_search] and method [POST]"

When trying to implement elasticsearch (v7.9.3) via the fos_elastica-bundle (v6.0.0) into my Symfony (v5.3.10) - App with api_platform (v2.6.6), I keep on getting this error:
"{"error":"no handler found for uri [//posts/_doc/_search] and method [POST]"}",
My api_platform.yaml reads:
api_platform:
[...]
elasticsearch:
hosts: [ '%env(ELASTICSEARCH_URL)%' ]
mapping:
App\Document\Post:
index: posts
and my fos_elastica.yaml:
fos_elastica:
clients:
default: { url: '%env(ELASTICSEARCH_URL)%' }
indexes:
posts:
properties:
id:
"type": "keyword"
source: ~
title: ~
description: ~
body: ~
children: ~
tags: ~
originalContent: ~
persistence:
driver: mongodb
model: App\Document\Post
By debugging the fos-elastica Bundle, I found out that the Elastica-Connector correctly triggers a [POST]-Request to "/posts/_doc/_search" with this request body:
{"sort":[{"id":{"order":"asc"}}],"query":{"match_all":{}},"size":30,"from":0}
If I use the Kibana Dev Tools Console and trigger an identical request
POST /posts/_doc/_search
{"sort":[{"id":{"order":"asc"}}],"query":{"match_all":{}},"size":30,"from":60}
I do get results from elasticsearch as expected:
#! Deprecation: [types removal] Specifying types in search requests is deprecated.
{
"took" : 12,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 3082,
"relation" : "eq"
},
"max_score" : null,
"hits" : [
{
"_index" : "posts",
"_type" : "_doc",
[...]
Apart from the deprecation notice, everything seems fine.
Does anyone have an idea why the api_platform integration of the fos_elastica-bundle does not work as expected and keeps on returning the "no handler found"-error message?
I have now helped myself by creating a custom ApiResource - filter
#[ApiFilter(FulltextFilter::class, arguments: ['index' => 'post'], properties: ['body','description','tag'])]
My custom filter implements ApiPlatform\Core\Bridge\Doctrine\MongoDbOdm\Filter\FilterInterface, directly communicates with the ElasticSearch server, sends a query to search the specified index (posts) and adds another match()-directive to the aggregationBuilder with a set of IDs matching the original search:
<?php
declare(strict_types=1);
namespace App\Filter;
use ApiPlatform\Core\Bridge\Doctrine\MongoDbOdm\Filter\FilterInterface;
use Doctrine\ODM\MongoDB\Aggregation\Builder;
use Elastica\Result;
use Elastica\Client;
use Elastica\Query;
use Symfony\Component\PropertyInfo\Type;
/**
* Filter the collection by given properties.
*
*/
final class FulltextFilter implements FilterInterface
{
protected $index = '';
protected $properties = [];
protected $client;
protected $searchParameterName;
protected $maxResultsParameterName;
const DEFAULT_MAX_RESULTS = 200;
public function __construct(Client $client, string $index = '', string $maxResultsParameterName = 'amount', string $searchParameterName = 'query', array $properties = []) {
$this->index = $index;
$this->properties = $properties;
$this->client = $client;
$this->searchParameterName = $searchParameterName;
$this->maxResultsParameterName = $maxResultsParameterName;
}
public function getFilteredIds($searchterm, $index = null, $properties = null, $maxResults = null) {
$matches = [];
if (is_null($properties)) {
$properties = array_keys($this->properties);
}
foreach ($properties as $propertyName) {
array_push($matches, ['match'=>[$propertyName => $searchterm]]);
}
$queryObject = ['query' => ['bool' => ['should' => $matches]]];
$queryObject['size'] = (int) $maxResults >0 ? (int) $maxResults : self::DEFAULT_MAX_RESULTS;
$query = new Query();
$response = $this->client->getIndex($index ?? $this->index)
->search($query->setRawQuery($queryObject))
->getResults();
return array_map(function(Result $result) {return $result->getHit()['_source']['id'];}, $response);
}
public function apply(Builder $aggregationBuilder, string $resourceClass, string $operationName = null, array &$context = [])
{
$maxResults = $context['filters'][$this->maxResultsParameterName] ?? null;
$searchterm = $context['filters'][$this->searchParameterName] ?? false;
if ($searchterm !== false) {
$aggregationBuilder->match()->field('id')->in($this->getFilteredIds($searchterm, null, null, $maxResults));
}
}
public function getDescription(string $resourceClass): array
{
return [];
}
}
This solution might not be as elegant as using the ElasticSearch-Connector natively provided by api_platform, but it is fairly performant and it works.
However, if someone comes up with a solution to fix the depicted ES-Connector issue with api_platform, please feel free to share it.
The problem is that, FOS Elastica requires an ES URL with an ending slash. But Api Platform requires a URL without ending slash.
We usually define the URL in .env file and then recall it in config files.
To solve the problem, we could define the URL in .env without endling slash and add the slash to the FOS Elastica config.
# .env
###> friendsofsymfony/elastica-bundle ###
ELASTICSEARCH_URL=http://localhost:9200
###< friendsofsymfony/elastica-bundle ###
# config/packages/api_platform.yaml
api_platform:
elasticsearch:
enabled: true
hosts: [ '%env(ELASTICSEARCH_URL)%' ]
# config/packages/fos_elastica.yaml
fos_elastica:
clients:
default: { url: '%env(ELASTICSEARCH_URL)%/' }

FOSElasticaBundle / Custom filter

I have an entity with
title, url, description, text, enabled,...
I want to search only through enabled = true articles.
How can I do it in elastica?
My config:
fos_elastica:
clients:
default: { host: localhost, port: 9200 }
indexes:
appletrh:
index_name: 'domain.com'
client: default
types:
products:
properties:
title: { type: string, analyzer: czech }
persistence:
driver: orm
model: Web\MagazineBundle\Entity\Article
elastica_to_model_transformer:
**query_builder_method: search**
provider: ~
listener: ~
finder: ~
EntityRepository search function
public function search()
{
$qb = $this->createQueryBuilder('p');
$qb->where('p.enabled = true');
return $qb;
}
Action:
public function searchAction(Request $request)
{
$keyword = $request->query->get('keyword');
$finder = $this->get('fos_elastica.finder.domain.articles');
$paginator = $this->get('knp_paginator');
$articles= $finder->createPaginatorAdapter($keyword);
$pagination = $paginator->paginate($articles, $request->query->get('page', 1), 12);
return $this->render('WebMagazineBundle:Search:search.html.twig', ['articles' => $pagination]);
}
I thought that this is the right solution, but it's still returning all of the data from DB.
I feel like you should use "provider", not "elastica_to_model_transformer" option in the bundle config for that purpose, like:
provider:
query_builder_method: search

FOSElasticaBundle multiple nested query

I use FOSElasticaBundle to handle searching. All works great when I have one level of nesting. However, when I have two levels of nesting results which should match the innermost nest are not returned (e.g. searching for 'xx' category does produce results, but searching for 'yy' brand does not - and should).
Here's my fos_elastica configuration:
fos_elastica:
clients:
default: { host: localhost, port: 9200 }
indexes:
my_index:
client: default
types:
product:
mappings:
title: { boost: 1 }
articleNumber: ~
introductionDateSearch: { type: integer }
delistingDateSearch: { type: integer }
deleted: { type: boolean }
category:
type: "nested"
properties:
name: { boost: 1 }
brand:
type: "nested"
properties:
name: { boost: 1 }
persistence:
driver: orm
model: MyBundle\Entity\Product
provider: ~
finder: ~
listener: ~
And my query handler:
public function searchForKeyword($keyword, AbstractUser $user)
{
$this->setFilters($user);
$keyword = trim($keyword);
if ($keyword !== '') {
$mainQuery = new \Elastica\Query\Bool();
$mainProductQuery = new \Elastica\Query\Bool();
//searching in Product title
$productQuery = new \Elastica\Query\Text();
$productQuery->setFieldQuery('title', $keyword);
$productQuery->setFieldParam('title', 'boost', 5);
$productQuery->setFieldParam('title', 'type', 'phrase_prefix');
//searching in Product articleNumber
$articleNumberQuery = new \Elastica\Query\Text();
$articleNumberQuery->setFieldQuery('articleNumber', $keyword);
$articleNumberQuery->setFieldParam('articleNumber', 'boost', 5);
$articleNumberQuery->setFieldParam('articleNumber', 'type', 'phrase_prefix');
//searching in Category name
$categoryQuery = new \Elastica\Query\Text();
$categoryQuery->setFieldQuery('name', $keyword);
$categoryQuery->setFieldParam('name', 'boost', 3);
$categoryQuery->setFieldParam('name', 'type', 'phrase_prefix');
$nestedCategoryProductQuery = new \Elastica\Query\Nested();
$nestedCategoryProductQuery->setPath('category');
$nestedCategoryProductQuery->setQuery($categoryQuery);
//searching in Brand name
$brandQuery = new \Elastica\Query\Text();
$brandQuery->setFieldQuery('name', $keyword);
$brandQuery->setFieldParam('name', 'boost', 3);
$brandQuery->setFieldParam('name', 'type', 'phrase_prefix');
$nestedBrandCategoryQuery = new \Elastica\Query\Nested();
$nestedBrandCategoryQuery->setPath('category.brand');
$nestedBrandCategoryQuery->setQuery($brandQuery);
$mainProductQuery->addShould($productQuery);
$mainProductQuery->addShould($articleNumberQuery);
$mainProductQuery->addShould($nestedCategoryProductQuery);
$mainProductQuery->addShould($nestedBrandCategoryQuery);
$mainQuery->addMust($mainProductQuery);
$esFilteredQuery = new \Elastica\Query\Filtered($mainQuery, $this->filters);
} else {
$esFilteredQuery = new \Elastica\Query\Filtered(new \Elastica\Query\MatchAll(), $this->filters);
}
$this->query = new \Elastica\Query();
$this->query->setQuery($esFilteredQuery);
}
How is the $nestedBrandCategoryQuery added to the $mainProductQuery?
Thanks for your help!
gtb
FOSElasticaBundle uses the Elastica Library. So this should not be an issue of FOSElasticaBundle.
Have a Look at http://elastica.io/ for more Details about the Lib. In my experience,there is nothing you can not do with Elastica if it is supported by Elasticsearch. Even when there is no Mapper in Elastica, just use the Raw Array Query (http://elastica.io/example/raw-array-query.html) to build the desired Query.

Symfony add Avatar field to sfGuardUser model

I have a project in symfony that I would like to let my users upload an image for their "avatar" field. I have found many posts about how to "extend" the table which I have with the schema below:
Member:
inheritance:
type: column_aggregation
extends: sfGuardUser
columns:
idmember: { type: integer }
birthday: { type: date }
avatar: { type: string(255) }
bio: { type: string(255) }
The columns get added to the table just fine, but when I go to change the widget to a sfWidgetFormInputFileEditable it breaks. Here is the Form.class file:
$file_src = $this->getObject()->getAvatar();
if ($file_src == '')
{
$file_src = 'default_image.png';
}
$this->widgetSchema['avatar'] = new sfWidgetFormInputFileEditable(array(
'label' => ' ',
'file_src' => '/uploads/avatar/'.$file_src,
'is_image' => true,
'edit_mode' => true,
'template' => '<div>%file%<br />%input%</div>',
));
and "save" function of the form:
if($this->isModified())
{
$uploadDir = sfConfig::get('sf_upload_dir');
$thumbnail = new sfThumbnail(150, 150);
$thumbnail2 = new sfThumbnail(800, 800);
if($this->getAvatar())
{
$thumbnail->loadFile($uploadDir.'/avatar/'.$this->getAvatar());
$thumbnail->save($uploadDir.'/avatar/thumbnail/'. $this->getAvatar());
$thumbnail2->loadFile($uploadDir.'/avatar/'.$this->getAvatar());
$thumbnail2->save($uploadDir.'/avatar/big/'. $this->getAvatar());
}
}
When I submit the form, I get this error message:
This form is multipart, which means you need to supply a files array as the bind() method second argument.
In the action where you bind the form you should use something like this:
$form->bind($request->getParamater($form->getName()), $request->getFiles($form->getName()));
So you need to pass the uploaded files as the second parameter to the bind method.

How to use nested queries with FOQElasticaBundle for Symfony2

I have a problem with query building with FOQElasticaBundle
I have 3 entities
User
Hotel
Ambiance
Users can have 1 or more Hotels, and each Hotel has only 1 Ambiance.
In my config file, I have:
foq_elastica:
clients:
default: { host: %elasticsearch.host%, port: %elasticsearch.port% }
indexes:
MyBundle:
client: default
finder:
types:
user:
mappings:
id:
boost: 10
analyzer: fr_case_analyzer
name:
boost: 5
analyzer: fr_case_analyzer
hotels:
type: "nested"
properties:
name:
boost: 10
analyzer: fr_case_analyzer
ambiance:
boost: 1
I want to be able to search for User by typing his name or the name of his hotels, and possibly add a filter on the Ambiance type.
So the query should look like something like this :
$mainQuery = new \Elastica_Query_Bool();
$nameQuery = new \Elastica_Query_Bool();
$filtersQuery = new \Elastica_Query_Bool();
//searching in Users' names
$nameQuery = new \Elastica_Query_Text();
$nameQuery->setFieldQuery('name', $searchName);
$nameQuery->setFieldParam('name', 'boost', 5);
$nameQuery->setFieldParam('name', 'type', 'phrase_prefix');
//searching in Hotels' names
$hotelNameQuery = new \Elastica_Query_Text();
$hotelNameQuery->setFieldQuery('name', $searchName);
$hotelNameQuery->setFieldParam('name', 'boost', 3);
$hotelNameQuery->setFieldParam('name', 'type', 'phrase_prefix');
$nestedHotelNameQuery = new \Elastica_Query_Nested();
$nestedHotelNameQuery->setPath('hotels');
$nestedHotelNameQuery->setQuery($hotelNameQuery);
$nameQuery->addShould($nameQuery);
$nameQuery->addShould($nestedHotelNameQuery);
//if filter on ambiance
$ambianceQuery = new \Elastica_Query_Term();
$ambianceQuery->setTerm('ambiance', $arrFilters['ambiance']);
$nestedAmbianceQuery = new \Elastica_Query_Nested();
$nestedAmbianceQuery->setPath('hotels');
$nestedAmbianceQuery->setQuery($ambianceQuery);
$filtersQuery->addMust($nestedAmbianceQuery);
//adding the parameters to the main query
$mainQuery->addMust($nameQuery);
$mainQuery->addMust($filtersQuery);
Unfortunately this doesn't work and returns no result if the Ambiance filter is activated, but works perfectly if I only search with the name.
What do I do wrong ?
I found why it wouldn't work.
The bundle actually uses __toString() on the object.
So, instead of querying on the "id" of the ambiance, I modified my html inputs so the value is the ambiance's name.
Here's my own version of the solution :
According to the elasticsearch documentation we should implement a structure that's smilar to the json bellow :
{
"query": {
"bool": {
"must": [
{ "match": { "title": "eggs" }},
{
"nested": {
"path": "comments",
"query": {
"bool": {
"must": [
{ "match": { "comments.name": "john" }},
{ "match": { "comments.age": 28 }}
]
}}}}
]
}}}
So to do this with symfony 2 and according to the FOSElasticaBundle bundle we will make the following code lines :
//if filter on ambiance
$ambianceQuery = new \Elastica_Query_Term();
$ambianceQuery->setTerm('ambiance', $arrFilters['ambiance']);
// We will add the the term to the query bool
$filtersQuery->addMust($ambianceQuery)
$nestedAmbianceQuery = new \Elastica_Query_Nested();
$nestedAmbianceQuery->setPath('hotels');
// we will set the result as argument in the setQuery method.
$nestedAmbianceQuery->setQuery($filtersQuery);
// And finally we add the nested query to the main query bool through addMust.
$mainQuery->addMust($nestedAmbianceQuery);
Hope that will help others.
A+
Just had the same Problem. This works for me:
..
$ambianceQuery->setTerm('hotels.ambiance', $arrFilters['ambiance']);
..
Coudn't find any examples on this in FOQElasticaBundle (now FOSElasticaBundle) for Symfony2 but it should end in an elasticsearch query like this one here:
http://www.elasticsearch.org/guide/reference/query-dsl/nested-query/
One can do/test raw elasticsearch queries like this:
$queryString = '{..JSON..}';
$boolQuery = new \Elastica_Query_Builder($queryString);

Resources