Is it feasible to run queries created by Kibana using Java API?
I mean get a ready queries from Kibana dashboards that are created by users dynamically, and pass it like a parameter in Java ?
this is an example of a query coming from Kibana dashboards :
{
"size": 0,
"query": {
"filtered": {
"query": {
"query_string": {
"analyze_wildcard": true,
"query": "*"
}
},
"filter": {
"bool": {
"must": [
{
"range": {
"#timestamp": {
"gte": 1274879129857,
"lte": 1432645529858
}
}
}
],
"must_not": []
}
}
}
},
"aggs": {
"3": {
"terms": {
"field": "ruleid",
"size": 20,
"order": {
"_count": "desc"
}
}
}
}
}
Related
I have problem with my queries when I'm using " or ' - then I expect match_phrase, but I don't know how I can retrieve posts when I'm using match_phrase with &
For example I'm using Something & Something as phrase, and when I'm didn't using ' and " I can see posts with Something & Something but there I'm using multi_match.
Something what I've tried:
{
"from": 0,
"size": 10,
"sort": {
"post_date": {
"order": "desc"
}
},
"query": {
"function_score": {
"query": {
"bool": {
"must": [
{
"match_phrase": {
"query": "Something & Something"
}
}
]
}
},
"exp": {
"post_date_gmt": {
"scale": "270d",
"decay": 0.5,
"offset": "90d"
}
},
"score_mode": "avg",
"boost_mode": "sum"
}
},
"post_filter": {
"bool": {
"must": [
{
"terms": {
"post_type.raw": [
"post"
]
}
},
{
"terms": {
"post_status": [
"publish"
]
}
}
]
}
}
}
But this doesn't return any post, and returning hits total 0. Anyone have any idea, or suggestions, what I'm doing wrong ?
match_phrase is very restrictive and in most of cases is recommended to use it inside a should clause to increase the score instead of a must, because it requires the user to type the value exactly as it is.
Example document
POST test_jakub/_doc
{
"query": "Something & Something",
"post_type": {
"raw": "post"
},
"post_status": "publish",
"post_date_gmt": "2021-01-01T12:10:30Z",
"post_date": "2021-01-01T12:10:30Z"
}
With this document searching for "anotherthing Something & Something" will return no results, that's why is a bad idea to use match_phrase here.
You can take 2 approaches
If you need this kind of tight queries take a look to the slop parameter that adds some flexibility to the match_phrase query allowing omit or transpose words in the phrase
Switch to a regular match query (recommended). In most cases this will work fine, but if you want to do extra score to the phrase matches you can add it as a should clause.
POST test_jakub/_search
{
"from": 0,
"size": 10,
"sort": {
"post_date": {
"order": "desc"
}
},
"query": {
"function_score": {
"query": {
"bool": {
"should": [
{
"match_phrase": {
"query": {
"query": "anotherthing something & something",
"slop": 2
}
}
}
],
"must": [
{
"match": {
"query": "anotherthing something & something"
}
}
]
}
},
"exp": {
"post_date_gmt": {
"scale": "270d",
"decay": 0.5,
"offset": "90d"
}
},
"score_mode": "avg",
"boost_mode": "sum"
}
},
"post_filter": {
"bool": {
"must": [
{
"terms": {
"post_type.raw": [
"post"
]
}
},
{
"terms": {
"post_status": [
"publish"
]
}
}
]
}
}
}
Last advice is to avoid using "query" as field name because leads to confusion and will break Kibana autocomplete on Dev Tools.
I'm trying to filter out a query based on a nested object (no array). I'm currently using AppSync and DynamoDB and the expression with expression values are executed correctly. But the filtering doesn't seem to work.
This is the sample data I'm trying to get (Filter by indicator.id):
Here's my query:
{
"version": "2017-02-28",
"operation": "Query",
"query": {
"expression": "pk = :pk and begins_with(sk, :sk)",
"expressionValues": {
":pk": { "S": "tenant:5fc30406-346c-42e2-8083-fda33ab6000a" },
":sk": {
"S": "school-year:2019-2020:grades:bVgA9abd:subject:m_kpc1Ae6:indicator:"
}
}
},
"filter": {
"expression": " contains(#indicatorId, :sk1) or contains(#indicatorId, :sk2) or contains(#indicatorId, :sk3)",
"expressionNames": { "#indicatorId": "indicator" },
"expressionValues": {
":sk1": {
"M": { "id": { "S": "07c658dd-999f-4e6f-95b8-c6bae422760a" } }
},
":sk2": {
"M": { "id": { "S": "0cf9f670-e284-4a93-b297-5e4a40c50228" } }
},
":sk3": { "M": { "id": { "S": "cd7902be-6512-4b47-b29d-40aff30c73e6" } } }
}
}
}
I've also tried:
{
"version": "2017-02-28",
"operation": "Query",
"query": {
"expression": "pk = :pk and begins_with(sk, :sk)",
"expressionValues": {
":pk": { "S": "tenant:5fc30406-346c-42e2-8083-fda33ab6000a" },
":sk": {
"S": "school-year:2019-2020:grades:bVgA9abd:subject:m_kpc1Ae6:indicator:"
}
}
},
"filter": {
"expression": " contains(#indicatorId, :sk1) or contains(#indicatorId, :sk2) or contains(#indicatorId, :sk3)",
"expressionNames": { "#indicatorId": "indicator.id" },
"expressionValues": {
":sk1": { "S": "07c658dd-999f-4e6f-95b8-c6bae422760a" },
":sk2": { "S": "0cf9f670-e284-4a93-b297-5e4a40c50228" },
":sk3": { "S": "cd7902be-6512-4b47-b29d-40aff30c73e6" }
}
}
}
I've also tried searching around StackOverflow, and Amazon forums and haven't found it directly to my problem:
How to filter by elements in an array (or nested object) in DynamoDB
Nested Query in DynamoDB returns nothing
Referring to this answer.enter link description here
According to DDB Nested Attributes doc, the filter expression should look like the following format:
"filter" : {
"expression" : "#path.#filter = :${fp}", ## filter path parent.target = :target
"expressionNames": {
"#path" : "${path}",
"#filter" : "${fp}"
},
"expressionValues" : {
":${fp}" : $util.dynamodb.toDynamoDBJson(${$target[$fp].eq}) ## :target : value to filter for
}
}
In Kibana I created 2 filters:
raw.browserJs.isWebDriver is true and raw.browserJs.isWebDriver isnot true. why the edit query DSL is the same for both:
{
"query": {
"match": {
"raw.browserJs.isWebDriver": {
"query": true,
"type": "phrase"
}
}
}
}
Also, how can i add condition in order to have one large DSL query with:
{
"query": {
"match": {
"appName": {
"query": "temp",
"type": "phrase"
}
}
}
}
The query DSL showing in Kibana is not the actual query which is send to elasticsearch. A range filter for the selected period is added and filters are inverted. You can see the actual query in the underlying request that is send in your browser.
You filter where raw.browserJs.isWebDriver is not true will end up in something like:
{
"query": {
"bool": {
"must_not": [
{
"match_phrase": {
"raw.browserJs.isWebDriver": true
}
}
]
}
}
}
You can combine multiple conditions in one DSL query with the bool query.(https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-bool-query.html)
The following query will work in your example:
{
"query": {
"bool": {
"must": [
{
"match_phrase": {
"raw.browserJs.isWebDriver": true
}
},
{
"match_phrase": {
"appName": "temp"
}
}
]
}
}
}
I have a problem with dynamic date tampletes
I'm using ElasticSearch 6.2.4
My steps:
1) Create index with next settings:
PUT /test1
{
"settings": {
"index":{
"number_of_shards" : 9,
"number_of_replicas" : 0,
"max_rescore_window" : 2000000000,
"max_result_window" : 2000000000
}
},
"mappings": {
"files": {
"properties": {
"Дата добавления в БД": {
"type": "date"
}
},
"numeric_detection": true,
"dynamic_templates": [
{
"integers": {
"match_mapping_type": "long",
"mapping": {
"type": "long"
}
}
},
{
"strings": {
"match_mapping_type": "string",
"mapping": {
"type": "keyword"
}
}
},
{
"dates": {
"match_mapping_type": "date",
"mapping": {
"format": "yyyy-MM-dd HH:mm:ss||yyyy/MM/dd HH:mm:ss||yyyyMMdd_HH:mm:ss",
"type": "date"
}
}
}
]
}
}
}
2) Try to put new records (I have only one)
POST /test1/files/_bulk
{"create":{"_index":"test1","_type":"files","_id":"0"}}
{"Дата добавления в БД":"2019/04/12 11:42:21"}
3) So, I have next output:
{
"took": 1,
"errors": true,
"items": [
{
"create": {
"_index": "test1",
"_type": "files",
"_id": "0",
"status": 400,
"error": {
"type": "mapper_parsing_exception",
"reason": "failed to parse [Дата добавления в БД]",
"caused_by": {
"type": "illegal_argument_exception",
"reason": "Invalid format: \"2019/04/12 11:42:21\" is malformed at \"/04/12 11:42:21\""
}
}
}
}
]
}
I can't understand where is my mistake??
I tried to find some information about this problem in Google, unfortunately, I have no solves of this problem. Maybe, this question is so stupid, but I've already broken my brain.
Please, help me...
I can't fully understand, but this option work:
{
"settings": {
"index":{
"number_of_shards" : 9,
"number_of_replicas" : 0,
"max_rescore_window" : 2000000000,
"max_result_window" : 2000000000
}
},
"mappings": {
"files": {
"dynamic_date_formats": ["yyyy-MM-dd HH:mm:ss","yyyy/MM/dd HH:mm:ss", "yyyyMMdd_HH:mm:ss"],
"numeric_detection": true,
"date_detection": true,
"dynamic_templates": [
{
"integers": {
"match_mapping_type": "long",
"mapping": {
"type": "long"
}
}
},
{
"strings": {
"match_mapping_type": "string",
"mapping": {
"type": "keyword"
}
}
}
]
}
}
}
Link to documentation:
https://www.elastic.co/guide/en/elasticsearch/reference/6.2/dynamic-field-mapping.html
Thanks for attention :)
From ES 5.0 Fielddata is disabled on text fields by default., How I could remove duplicates/ achieve the same result with existing settings i.e. when field data is disabled on query below?
{
"aggs": {
"query": {
"terms": {
"field": "name"
}
,
"aggs": {
"top": {
"top_hits": {
"size": 1
}
}
}
}
},
"size": 0,
"query": {
"multi_match": {
"query": "laura",
"operator": "OR",
"fields": [
"name"
]
}
}
}
You would have to enable fielddata on text fields for ES 5.x. Use it with caution as it consumes lot of heap space.
Update your mapping with
PUT your_index/_mapping/your_type
{
"properties": {
"name": {
"type": "text",
"fielddata": true
}
}
}
and then run the query.