Elasticsearch Date Range Aggregation with Sum - kibana

I've followed Elastic's docs and successfully queried my index to return data for two date ranges.
What I can't get right, is adding a sum of units to each range. I've managed to add a unit sum (see query below) but this only returns a sum for the entire range.
What I'm trying to achieve is a period A vs B comparison. i.e. This week you've sold X units vs Y units last week and therefore the % variance is Z.
{
"query":{
"filtered":{
"query":{
"query_string":{
"query":"*",
"analyze_wildcard":true
}
}
}
},
"size":0,
"aggs":{
"vendor_type":{
"terms":{
"field":"vendor_type",
"size":5
},
"aggs":{
"product_type":{
"terms":{
"field":"product_type",
"size":5,
"order":{
"unit_sum":"desc"
}
},
"aggs":{
"range":{
"date_range":{
"field":"date",
"format":"MM-yyy",
"ranges":[
{
"to":"now-1M/M"
},
{
"from":"now-1M/M"
}
]
}
},
"unit_sum":{
"sum":{
"field":"units"
}
}
}
}
}
}
}
}
Can anyone help? Thanks in advance.

You're almost there, you just need to move (or copy) your unit_sum inside the range aggregation, like this:
{
"query": {
"filtered": {
"query": {
"query_string": {
"query": "*",
"analyze_wildcard": true
}
}
}
},
"size": 0,
"aggs": {
"vendor_type": {
"terms": {
"field": "vendor_type",
"size": 5
},
"aggs": {
"product_type": {
"terms": {
"field": "product_type",
"size": 5
},
"aggs": {
"range": {
"date_range": {
"field": "date",
"format": "MM-yyy",
"ranges": [
{
"to": "now-1M/M"
},
{
"from": "now-1M/M"
}
]
},
"aggs": {
"unit_sum": {
"sum": {
"field": "units"
}
}
}
}
}
}
}
}
}
}

Related

Elasticsearch & Elasticpress search by math_phrase with & inside query

I have problem with my queries when I'm using " or ' - then I expect match_phrase, but I don't know how I can retrieve posts when I'm using match_phrase with &
For example I'm using Something & Something as phrase, and when I'm didn't using ' and " I can see posts with Something & Something but there I'm using multi_match.
Something what I've tried:
{
"from": 0,
"size": 10,
"sort": {
"post_date": {
"order": "desc"
}
},
"query": {
"function_score": {
"query": {
"bool": {
"must": [
{
"match_phrase": {
"query": "Something & Something"
}
}
]
}
},
"exp": {
"post_date_gmt": {
"scale": "270d",
"decay": 0.5,
"offset": "90d"
}
},
"score_mode": "avg",
"boost_mode": "sum"
}
},
"post_filter": {
"bool": {
"must": [
{
"terms": {
"post_type.raw": [
"post"
]
}
},
{
"terms": {
"post_status": [
"publish"
]
}
}
]
}
}
}
But this doesn't return any post, and returning hits total 0. Anyone have any idea, or suggestions, what I'm doing wrong ?
match_phrase is very restrictive and in most of cases is recommended to use it inside a should clause to increase the score instead of a must, because it requires the user to type the value exactly as it is.
Example document
POST test_jakub/_doc
{
"query": "Something & Something",
"post_type": {
"raw": "post"
},
"post_status": "publish",
"post_date_gmt": "2021-01-01T12:10:30Z",
"post_date": "2021-01-01T12:10:30Z"
}
With this document searching for "anotherthing Something & Something" will return no results, that's why is a bad idea to use match_phrase here.
You can take 2 approaches
If you need this kind of tight queries take a look to the slop parameter that adds some flexibility to the match_phrase query allowing omit or transpose words in the phrase
Switch to a regular match query (recommended). In most cases this will work fine, but if you want to do extra score to the phrase matches you can add it as a should clause.
POST test_jakub/_search
{
"from": 0,
"size": 10,
"sort": {
"post_date": {
"order": "desc"
}
},
"query": {
"function_score": {
"query": {
"bool": {
"should": [
{
"match_phrase": {
"query": {
"query": "anotherthing something & something",
"slop": 2
}
}
}
],
"must": [
{
"match": {
"query": "anotherthing something & something"
}
}
]
}
},
"exp": {
"post_date_gmt": {
"scale": "270d",
"decay": 0.5,
"offset": "90d"
}
},
"score_mode": "avg",
"boost_mode": "sum"
}
},
"post_filter": {
"bool": {
"must": [
{
"terms": {
"post_type.raw": [
"post"
]
}
},
{
"terms": {
"post_status": [
"publish"
]
}
}
]
}
}
}
Last advice is to avoid using "query" as field name because leads to confusion and will break Kibana autocomplete on Dev Tools.

Filter Expression based on a nested object DynamoDB AppSync

I'm trying to filter out a query based on a nested object (no array). I'm currently using AppSync and DynamoDB and the expression with expression values are executed correctly. But the filtering doesn't seem to work.
This is the sample data I'm trying to get (Filter by indicator.id):
Here's my query:
{
"version": "2017-02-28",
"operation": "Query",
"query": {
"expression": "pk = :pk and begins_with(sk, :sk)",
"expressionValues": {
":pk": { "S": "tenant:5fc30406-346c-42e2-8083-fda33ab6000a" },
":sk": {
"S": "school-year:2019-2020:grades:bVgA9abd:subject:m_kpc1Ae6:indicator:"
}
}
},
"filter": {
"expression": " contains(#indicatorId, :sk1) or contains(#indicatorId, :sk2) or contains(#indicatorId, :sk3)",
"expressionNames": { "#indicatorId": "indicator" },
"expressionValues": {
":sk1": {
"M": { "id": { "S": "07c658dd-999f-4e6f-95b8-c6bae422760a" } }
},
":sk2": {
"M": { "id": { "S": "0cf9f670-e284-4a93-b297-5e4a40c50228" } }
},
":sk3": { "M": { "id": { "S": "cd7902be-6512-4b47-b29d-40aff30c73e6" } } }
}
}
}
I've also tried:
{
"version": "2017-02-28",
"operation": "Query",
"query": {
"expression": "pk = :pk and begins_with(sk, :sk)",
"expressionValues": {
":pk": { "S": "tenant:5fc30406-346c-42e2-8083-fda33ab6000a" },
":sk": {
"S": "school-year:2019-2020:grades:bVgA9abd:subject:m_kpc1Ae6:indicator:"
}
}
},
"filter": {
"expression": " contains(#indicatorId, :sk1) or contains(#indicatorId, :sk2) or contains(#indicatorId, :sk3)",
"expressionNames": { "#indicatorId": "indicator.id" },
"expressionValues": {
":sk1": { "S": "07c658dd-999f-4e6f-95b8-c6bae422760a" },
":sk2": { "S": "0cf9f670-e284-4a93-b297-5e4a40c50228" },
":sk3": { "S": "cd7902be-6512-4b47-b29d-40aff30c73e6" }
}
}
}
I've also tried searching around StackOverflow, and Amazon forums and haven't found it directly to my problem:
How to filter by elements in an array (or nested object) in DynamoDB
Nested Query in DynamoDB returns nothing
Referring to this answer.enter link description here
According to DDB Nested Attributes doc, the filter expression should look like the following format:
"filter" : {
"expression" : "#path.#filter = :${fp}", ## filter path parent.target = :target
"expressionNames": {
"#path" : "${path}",
"#filter" : "${fp}"
},
"expressionValues" : {
":${fp}" : $util.dynamodb.toDynamoDBJson(${$target[$fp].eq}) ## :target : value to filter for
}
}

How to fix a problem with dynamic date templates?

I have a problem with dynamic date tampletes
I'm using ElasticSearch 6.2.4
My steps:
1) Create index with next settings:
PUT /test1
{
"settings": {
"index":{
"number_of_shards" : 9,
"number_of_replicas" : 0,
"max_rescore_window" : 2000000000,
"max_result_window" : 2000000000
}
},
"mappings": {
"files": {
"properties": {
"Дата добавления в БД": {
"type": "date"
}
},
"numeric_detection": true,
"dynamic_templates": [
{
"integers": {
"match_mapping_type": "long",
"mapping": {
"type": "long"
}
}
},
{
"strings": {
"match_mapping_type": "string",
"mapping": {
"type": "keyword"
}
}
},
{
"dates": {
"match_mapping_type": "date",
"mapping": {
"format": "yyyy-MM-dd HH:mm:ss||yyyy/MM/dd HH:mm:ss||yyyyMMdd_HH:mm:ss",
"type": "date"
}
}
}
]
}
}
}
2) Try to put new records (I have only one)
POST /test1/files/_bulk
{"create":{"_index":"test1","_type":"files","_id":"0"}}
{"Дата добавления в БД":"2019/04/12 11:42:21"}
3) So, I have next output:
{
"took": 1,
"errors": true,
"items": [
{
"create": {
"_index": "test1",
"_type": "files",
"_id": "0",
"status": 400,
"error": {
"type": "mapper_parsing_exception",
"reason": "failed to parse [Дата добавления в БД]",
"caused_by": {
"type": "illegal_argument_exception",
"reason": "Invalid format: \"2019/04/12 11:42:21\" is malformed at \"/04/12 11:42:21\""
}
}
}
}
]
}
I can't understand where is my mistake??
I tried to find some information about this problem in Google, unfortunately, I have no solves of this problem. Maybe, this question is so stupid, but I've already broken my brain.
Please, help me...
I can't fully understand, but this option work:
{
"settings": {
"index":{
"number_of_shards" : 9,
"number_of_replicas" : 0,
"max_rescore_window" : 2000000000,
"max_result_window" : 2000000000
}
},
"mappings": {
"files": {
"dynamic_date_formats": ["yyyy-MM-dd HH:mm:ss","yyyy/MM/dd HH:mm:ss", "yyyyMMdd_HH:mm:ss"],
"numeric_detection": true,
"date_detection": true,
"dynamic_templates": [
{
"integers": {
"match_mapping_type": "long",
"mapping": {
"type": "long"
}
}
},
{
"strings": {
"match_mapping_type": "string",
"mapping": {
"type": "keyword"
}
}
}
]
}
}
}
Link to documentation:
https://www.elastic.co/guide/en/elasticsearch/reference/6.2/dynamic-field-mapping.html
Thanks for attention :)

Elastic search 5.0 duplicate removal/optimisation

From ES 5.0 Fielddata is disabled on text fields by default., How I could remove duplicates/ achieve the same result with existing settings i.e. when field data is disabled on query below?
{
"aggs": {
"query": {
"terms": {
"field": "name"
}
,
"aggs": {
"top": {
"top_hits": {
"size": 1
}
}
}
}
},
"size": 0,
"query": {
"multi_match": {
"query": "laura",
"operator": "OR",
"fields": [
"name"
]
}
}
}
You would have to enable fielddata on text fields for ES 5.x. Use it with caution as it consumes lot of heap space.
Update your mapping with
PUT your_index/_mapping/your_type
{
"properties": {
"name": {
"type": "text",
"fielddata": true
}
}
}
and then run the query.

Running queries created by Kibana using Java API?

Is it feasible to run queries created by Kibana using Java API?
I mean get a ready queries from Kibana dashboards that are created by users dynamically, and pass it like a parameter in Java ?
this is an example of a query coming from Kibana dashboards :
{
"size": 0,
"query": {
"filtered": {
"query": {
"query_string": {
"analyze_wildcard": true,
"query": "*"
}
},
"filter": {
"bool": {
"must": [
{
"range": {
"#timestamp": {
"gte": 1274879129857,
"lte": 1432645529858
}
}
}
],
"must_not": []
}
}
}
},
"aggs": {
"3": {
"terms": {
"field": "ruleid",
"size": 20,
"order": {
"_count": "desc"
}
}
}
}
}

Resources