How to configure correctly FOS Elastica analyzers and filters? - symfony

In my symfony2 application, I am using FOS Elastica bundle to perform searches.
I have tried to set up analyzers and filters but it seems they just have no effect. For instance, if I search for the word 'cake', the objects containing the sentence case 'Cake' won't be returned.
How can I configure those analyzers and filters correctly ?
My config :
#Elastic Search
fos_elastica:
default_manager: orm
clients:
default: { host: localhost, port: 9200 }
indexes:
website:
client: default
settings:
index:
analysis:
analyzer:
custom_index_analyzer :
type : custom
tokenizer: nGram
filter : [stopwords, asciifolding ,lowercase, snowball, elision, worddelimiter]
custom_search_analyzer :
type : custom
tokenizer: nGram
filter : [stopwords, asciifolding ,lowercase, snowball, elision, worddelimiter]
tokenizer:
nGram:
type: nGram
min_gram: 1
max_gram: 2
filter:
snowball:
type: snowball
language: French
elision:
type: elision
articles: [l, m, t, qu, n, s, j, d]
stopwords:
type: stop
stopwords: [_french_]
ignore_case : true
worddelimiter :
type: word_delimiter
index_name: foodmeup
types:
recipe:
mappings:
name:
boost: 5
index_analyzer : custom_index_analyzer
search_analyzer : custom_search_analyzer
nickName:
index_analyzer : custom_index_analyzer
search_analyzer : custom_search_analyzer
content:
index_analyzer : custom_index_analyzer
search_analyzer : custom_search_analyzer
recipeIngredients:
type: "nested"
properties:
name:
index_analyzer : custom_index_analyzer
search_analyzer : custom_search_analyzer
product:
type: "nested"
properties:
name: { type: string, boost: 10}
nickName: { type: string }
content: { type: string }
tags:
type: "nested"
boost: 5
properties:
name: { type: string }
userRecipes:
type: "nested"
properties:
name:
index_analyzer : custom_index_analyzer
search_analyzer : custom_search_analyzer
content:
index_analyzer : custom_index_analyzer
search_analyzer : custom_search_analyzer
tags:
type: "nested"
boost: 5
properties:
name:
index_analyzer : custom_index_analyzer
search_analyzer : custom_search_analyzer
persistence:
driver: orm
model: AppBundle\Entity\FoodAnalytics\Recipe
repository: AppBundle\Repository\FoodAnalytics\RecipeRepository
provider: ~
finder: ~
listener: ~ # by default, listens to "insert", "update" and "delete"
product:
mappings:
name: { type: string, boost: 10}
nickName: { type: string }
content: { type: string }
userIngredients:
type: "nested"
properties:
name: { type: string }
content: { type: string }
tags:
type: "nested"
boost: 5
properties:
name: { type: string }
persistence:
driver: orm
model: AppBundle\Entity\MarketPlace\Product
repository: AppBundle\Repository\MarketPlace\ProductRepository
provider: ~
finder: ~
listener: ~ # by default, listens to "insert", "update" and "delete"
user:
mappings:
firstName: { type: string, boost: 3}
lastName: { type: string, boost: 10 }
content: { type: string }
username: { type: string }
email: { type: string }
jobSeeker:
type: "nested"
properties:
skills:
type: "nested"
properties:
name: { type: string }
experiences:
type: "nested"
properties:
position:
type: "nested"
properties:
name: { type: string }
content: { type: string }
trainings:
type: "nested"
properties:
name: { type: string }
content: { type: string }
diploma:
type: "nested"
properties:
name: { type: string }
persistence:
driver: orm
model: AppBundle\Entity\User\User
repository: AppBundle\Repository\User\UserRepository
provider: ~
finder: ~
listener: ~ # by default, listens to "insert", "update" and "delete"
organization:
mappings:
name: { type: string, boost: 10}
legalName: { type: string, boost: 10}
shortDescription: { type: string, boost: 3}
route: { type: string}
content: { type: string }
persistence:
driver: orm
model: AppBundle\Entity\User\Organization
repository: AppBundle\Repository\User\OrganizationRepository
provider: ~
finder: ~
listener: ~ # by default, listens to "insert", "update" and "delete"
offer:
mappings:
name: { type: string, boost: 10}
content: { type: string }
responsibilities: { type: string }
skills:
type: "nested"
properties:
name: { type: string }
contractType:
type: "nested"
properties:
name: { type: string }
position:
type: "nested"
properties:
name: { type: string, boost: 10 }
persistence:
driver: orm
model: AppBundle\Entity\Job\Offer
repository: AppBundle\Repository\Job\OfferRepository
provider: ~
finder: ~
listener: ~
post:
mappings:
name: { type: string, boost: 10}
content: { type: string }
summary: { type: string }
tags:
type: "nested"
boost: 5
properties:
name: { type: string }
comments:
type: "nested"
properties:
content: { type: string }
persistence:
driver: orm
model: AppBundle\Entity\Social\Post
repository: AppBundle\Repository\Social\PostRepository
provider: ~
finder: ~
listener: ~
The query is a basic one :
$finder = $this->container->get('website.recipe')
$elasticaResults = $finder->find($search);

Each time you update your fos_elastica config file, you need in a first time to update the index(es) with the command:
php app/console fos:elastica:populate --index website --type= <<you type>>
This command will reindex your data with erasing previous one. In a second time, you should have to check your mapping information to check everything is fine and synchronous.
You could check your analyser with this command:
curl 'http://<< yourdomain >>:9200/website/_analyze?pretty=true&analyzer=custom_french_analyzer' -d "Cake"

Related

Unrecognized option "types" | FOS Elastica & Symfony

I am able to create indexes 1 by 1 but my purpose is to have 1 big index with subindexes to search. (I am reproducing an existing app but upgrading dependencies)
Symfony : 5.4*
FOS-Elastica : 6.1
Elastic : 7*
This is my error message and if I change types to properties I have similar errors. I have been trying to switch and indent differently all day:
Unrecognized option "types" under "fos_elastica.indexes.app". Available options are "_id", "_routing", "_source", "analyzer", "client", "date_detection", "dynamic", "dynamic_date_formats", "dynamic_templates", "finder", "index_
name", "index_prototype", "indexable_callback", "numeric_detection", "persistence", "properties", "serializer", "settings", "use_alias".
What am I doing wrong please ?
#app/config/config.yml
fos_elastica:
clients:
default: { host: localhost, port: 9200 }
indexes:
app:
settings:
analysis:
analyzer:
fr:
tokenizer: standard
filter: [ lowercase, stop_fr, snowball_fr ]
autocomplete:
type: custom
tokenizer: whitespace
filter: [ lowercase, engram, elision ]
csv:
type: pattern
pattern: '\s*,\s*'
lowercase: false
filter:
snowball_fr:
type: "snowball"
language: "French"
stop_fr:
type: "stop"
stopwords: "_french_"
engram:
type: edge_ngram
min_gram: 2
max_gram: 15
types:
# ---------
# USER
# ---------
user:
properties:
username: ~
email: ~
organization:
type: object
properties:
id: { index: true }
code: { index: true }
name:
index: true
type: text
fields:
source: { type: text, index: true }
persistence:
driver: orm # orm, mongodb, propel are available
model: App\Entity\User
provider: ~
listener: ~
finder: ~
So I was also figuring out how to deal with the same issue as yours, and I do confirm lot of the docs and blogposts out there use the config struture you posted. However when I went back to the v6.1 documentation of the bundle I found this:
Dynamic templates
Dynamic templates allow to define mapping templates
that will be applied when dynamic introduction of fields / objects
happens.
Documentation
fos_elastica:
indexes:
user:
dynamic_templates:
my_template_1:
match: apples_*
mapping:
type: float
my_template_2:
match: *
match_mapping_type: text
mapping:
type: keyword
properties:
username: { type: text }
So in your case a working config would look like:
#app/config/config.yml
fos_elastica:
clients:
default: { host: localhost, port: 9200 }
indexes:
user:
settings:
analysis:
analyzer:
fr:
tokenizer: standard
filter: [ lowercase, stop_fr, snowball_fr ]
autocomplete:
type: custom
tokenizer: whitespace
filter: [ lowercase, engram, elision ]
csv:
type: pattern
pattern: '\s*,\s*'
lowercase: false
filter:
snowball_fr:
type: "snowball"
language: "French"
stop_fr:
type: "stop"
stopwords: "_french_"
engram:
type: edge_ngram
min_gram: 2
max_gram: 15
properties:
username: ~
email: ~
organization:
type: object
properties:
id: { index: true }
code: { index: true }
name:
index: true
type: text
fields:
source: { type: text, index: true }
persistence:
driver: orm # orm, mongodb, propel are available
model: App\Entity\User
provider: ~
listener: ~
finder: ~

Include schema components inside paths in swagger OpenApi 3.0 yaml file

I`m working on API for drupal and I created some reusable components, when I try to "include" components inside paths one of them not showing up
here are my files:
header component
#HEADER (JSONAPI)
header:
required:
- jsonapi
- data
- links
properties:
jsonapi:
required:
- version
- meta
properties:
version:
type: "string"
example: "1.0"
meta:
required:
- links
properties:
self:
required:
- href
properties:
href:
type: "string"
example: "http://jsonapi.org/format/1.0/"
type: object
type: object
type: object
type: object
#END OF HEADER
links component
#Links
links:
required:
- self
properties:
self:
required:
- href
properties:
href:
type: "string"
example: "http://localhost/jsonapi/link/be336992-26b9-448e-af00-2fb55642d933"
type: object
type: object
data component
#Data Link
data:
type: "array"
items:
type: "object"
properties:
type:
type: "string"
example: "node--link"
id:
type: "string"
example: "12c800a8-ee1d-4fde-9bf6-9479a57f9588"
attributes:
required:
- "drupal_internal__nid"
- "drupal_internal__vid"
- "status"
- "title"
- "created"
- "path"
- "body"
- "url"
properties:
drupal_internal__nid:
type: "number"
example: 8
drupal_internal__vid:
type: "number"
example: 8
status:
type: "boolean"
example: true
title:
type: "string"
example: "Test Link"
created:
type: "string"
example: "2019-12-04T10:11:58+00:00"
path:
required:
- "alias"
- "pid"
- "langcode"
properties:
alias:
type: "string"
example: "/link/test-link"
pid:
type: "number"
example: 8
langcode:
type: "string"
example: "en"
type: "object"
body:
required:
- "value"
- "format"
- "processed"
- "summary"
properties:
value:
type: "string"
example: "<p>This is a test link</p>\r\n"
format:
type: "string"
example: "basic_html"
processed:
type: "string"
example: "<p>This is a test link</p>"
summary:
type: "string"
example: ""
type: "object"
url:
type: "string"
example: "http://www.example.com"
type: "object"
relationships:
required:
- "node_type"
- "uid"
- "link"
properties:
node_type:
required:
- "data"
- "links"
properties:
data:
required:
- "type"
- "id"
properties:
type:
type: "string"
example: "node_type"
id:
type: "string"
example: "a95e770e-9035-4b8c-8c80-679ad9703174"
type: "object"
links:
required:
- "self"
- "related"
properties:
self:
required:
- "href"
properties:
href:
type: "string"
example: "http://localhost/jsonapi/link/12c800a8-ee1d-4fde-9bf6-9479a57f9588/relationships/node_type?resourceVersion=id%3A8"
type: "object"
related:
required:
- "href"
properties:
href:
type: "string"
example: "http://localhost/jsonapi/link/12c800a8-ee1d-4fde-9bf6-9479a57f9588/node_type?resourceVersion=id%3A8"
type: "object"
type: "object"
type: "object"
uid:
required:
- "data"
- "links"
properties:
links:
required:
- "self"
properties:
self:
required:
- "href"
properties:
href:
type: "string"
example: "http://localhost/jsonapi/link/12c800a8-ee1d-4fde-9bf6-9479a57f9588/relationships/uid?resourceVersion=id%3A8"
type: "object"
type: "object"
type: "object"
link:
required:
- "data"
- "links"
properties:
links:
required:
- "self"
properties:
self:
required:
- "href"
properties:
href:
type: "string"
example: "http://localhost/jsonapi/link/12c800a8-ee1d-4fde-9bf6-9479a57f9588/relationships/link?resourceVersion=id%3A8"
type: "object"
type: "object"
type: "object"
type: "object"
#end of data for link
here is my path:
paths:
/link:
get:
responses:
'200':
description: OK
content:
application/json:
schema:
allOf:
- $ref: '#/components/schemas/header'
- $ref: '#/components/schemas/data'
- $ref: '#/components/schemas/links'
In swagger app only header and links are showing up, data component is missing
Sorry for posting such a long file
The problem is incompatible subschema types in allOf: the header and links schemas are objects but data is an array. This part:
allOf:
- $ref: '#/components/schemas/header'
- $ref: '#/components/schemas/data'
- $ref: '#/components/schemas/links'
is equivalent to:
allOf:
- # header
type: object
properties:
jsonapi: ...
- # data
type: array # <-----
items: ...
- # links
type: object
properties:
self: ...
which is an impossible schema because an instance cannot be an object and an array at the same time.
Assuming the data array is supposed to be wrapped into the data property, you need to change the data schema as follows:
components:
schemas:
data:
type: object
properties:
data:
type: array
items:
...

Index a foreign key label with FosElasticaBundle

When I index my relationnal database (Postgresql), I want to get the string value intead of the foreign key id (recreate the relation between the two tables personn and country..)
I wish to do that in the fos_elastica.yml mapping file
Example:
fos_elastica:
clients:
default: { host: %elastic_host%, port: %elastic_port% }
indexes:
ex_app:
client: default
types:
personn:
mappings:
id:
type: integer
name:
type: text
boost: 5
firstname:
type: text
boost: 3
dateofbirth:
type: date
country:
type: integer
persistence:
driver: orm
model: AppBundle\Entity\Personn
finder: ~
provider: ~
listener: ~
I want to find the person by typing the name of the country and not the country id.
I'm pretty sure that you should use "nested" :
fos_elastica:
clients:
default: { host: %elastic_host%, port: %elastic_port% }
indexes:
ex_app:
client: default
types:
personn:
mappings:
id:
type: integer
name:
type: text
boost: 5
firstname:
type: text
boost: 3
dateofbirth:
type: date
country:
type: "nested"
properties:
id:
type : integer
label:
type: text
persistence:
driver: orm
model: AppBundle\Entity\Personn
finder: ~
provider: ~
listener: ~

symfony elastic search match exact term not working

I use symfony with fos elastica bundle but i have some problems
I have an entity content with link with other entity categorie (with name and slugname) x content have only 1 category.
I have install the bundle and configure the yml files like this
fos_elastica:
clients:
default: { host: 127.0.0.1, port: 9200 }
serializer:
callback_class: FOS\ElasticaBundle\Serializer\Callback
serializer: serializer
indexes:
portail:
client: default
settings:
index:
analysis:
analyzer:
custom_analyzer :
type: custom
tokenizer: nGram
char_filter: [html_strip]
filter: [stopwords, lowercase, snowball, elision]
custom_analyzer_simple :
type: custom
tokenizer: whitespace
filter: [lowercase]
custom_search_analyzer :
type: custom
char_filter: [html_strip]
tokenizer: whitespace
filter: [stopwords, lowercase, snowball, elision]
custom_search_analyzer_simple :
type: custom
tokenizer: whitespace
filter: [lowercase]
tokenizer:
nGram:
type: nGram
min_gram: 4
max_gram: 20
filter:
snowball:
type: snowball
language: French
elision:
type: elision
articles: [l, m, t, qu, n, s, j, d]
stopwords:
type: stop
stopwords: [_french_]
ignore_case : true
worddelimiter :
type: word_delimiter
types:
Content:
mappings:
titre:
type: string
index_analyzer: custom_analyzer
search_analyzer : custom_search_analyzer
preview:
type: string
index_analyzer: custom_analyzer
search_analyzer : custom_search_analyzer
contenu:
type: string
index_analyzer: custom_analyzer
search_analyzer : custom_search_analyzer
categorie:
type: object
properties:
slugnom:
type: string
index: not_analyzed
search_analyzer : custom_search_analyzer_simple
persistence:
driver: orm
model: PO\EtablissementBundle\Entity\Content
provider: ~
listener:
logger: true
finder: ~
Page:
mappings:
titre:
type: string
index_analyzer: custom_analyzer
search_analyzer : custom_search_analyzer
persistence:
driver: orm
model: PO\EtablissementBundle\Entity\PageEtablissement
provider: ~
listener:
logger: true
finder: ~
Etablissement:
mappings:
nom:
type: string
index_analyzer: custom_analyzer
search_analyzer : custom_search_analyzer
contents:
type: object
categories:
type: nested
properties:
slugnom:
type: string
index: not_analyzed
search_analyzer : custom_search_analyzer_simple
sous_categories:
type: nested
properties:
slugnom:
type: string
index: not_analyzed
persistence:
driver: orm
model: PO\EtablissementBundle\Entity\Etablissement
provider: ~
listener:
logger: true
finder: ~
I want to have some flexibility but on the content categorie slugname i want to search exactly the word for example arts-de-la-rue but when i make a search i found result with categorie = arts or categorie = lecture, i don't understand the problem :(
If somebody can help me :)
You have to setup your mapping for that field
your field: { type: "string", index: "not_analyzed" }
And just to be sure destroy the type and populate it again.
You need to setup an keyword based analyzer for that.
The ngram on your analyzer will destroy any slug.
It is possible to setup multiple different analyzers per field, but I dont know if it is supported by the FOS\ElasticaBundle: https://www.elastic.co/guide/en/elasticsearch/reference/current/multi-fields.html
Later you can do a "filtered" query (term filter) against this new raw field.
Thanks for your help i have done some mistake in my configuration. I have put a type nested and in fact it's objet and the name of the properties are not good.
Now it's working perfectly :)

Auto index child objects in fos elastica

I have created an elastic search mapping in config.yml which is using doctrine orm, the issue is that when I create a new document the index is auto populated for the top most object which is "documents" but not for any child objects such as "Folder, Authors, Comments ...", can someone suggest if I am missing any flag or any settings. Below is my Elastica configuration:
# Elastica Configuration
fos_elastica:
clients:
default: { host: localhost, port: 9200 }
indexes:
website:
client: default
index_name: docova
types:
documents:
mappings:
Doc_Title: ~
Description: ~
Date_Created: ~
Doc_Status: ~
Keywords: ~
folder:
type: "object"
properties:
id: ~
Library:
type: "object"
properties:
id: ~
Author:
type: "object"
properties:
username: ~
Comments:
type: "object"
properties:
comment: ~
Form_Values:
type: "object"
properties:
Field_Value: ~
Bookmarks:
type: "object"
properties:
Target_Folder:
type: "object"
properties:
id: ~
DocType:
type: "object"
properties:
id: ~
Doc_Name: ~
Attachments:
type: "object"
properties:
File_Name:
content:
type: attachment
persistence:
driver: orm
model: Docova\DocovaBundle\Entity\Documents
provider: ~
listener: ~
finder: ~
Thnx.
i've found a solution here.
Symfony2 FOSElasticaBundle update index for all entities related to the entity updated
Note : In my version of FOSElastica, Listener is not in the ORM folder but in FOS\ElasticaBundle\Doctrine and the argument has changed from LifecycleEventArgs $eventArgs to \Doctrine\Common\EventArgs $eventArgs.
EDIT : For this case, you have to override postPersist too.
public function postPersist(\Doctrine\Common\EventArgs $eventArgs)
{
$entity = $eventArgs->getEntity();
if ($entity instanceof $this->objectClass && $this->isObjectIndexable($entity)) {
$this->scheduledForInsertion[] = $entity;
$this->inicializaPost();
$this->objectPersisterPost->replaceOne($entity->getPost());
}
}

Resources