Drupal 8 Jsonapi Get request throws bad request error - drupal

Setting up drupal website API using json api, when accessing the link for get resource http://example.com/jsonapi/node/article - getting error
"title": "Bad Request",
"status": "400",
"detail": "The following query parameters violate the JSON:API spec: 'q'.",

Your web server seems to rewrite the request URL. In particular it adds a q query parameter. The full error message returned by Drupal includes the full URL that the application was receiving under links.via path of first errors object: http://207.148.125.64/jsonapi/node/article?q=%2Fjsonapi%2Fnode%2Farticle Please see above for full response.
This likely caused by a wrong configuration of web server used to serve Drupal. In particular the rewrite rule seems to be wrong. A similar issue has been reported in this bug.
I would recommend that you compare your web server configuration against the default .htaccess provided. If you are using nginx, you might want to have a look at an example configuration for nginx.
$ curl http://207.148.125.64/jsonapi/node/article | json_pp
{
"errors" : [
{
"title" : "Bad Request",
"status" : "400",
"meta" : {
"exception" : "Drupal\\Core\\Http\\Exception\\CacheableBadRequestHttpException: The following query parameters violate the JSON:API spec: 'q'. in /var/www/html/modules/contrib/jsonapi/src/EventSubscriber/JsonApiRequestValidator.php:78\nStack trace:\n#0 /var/www/html/modules/contrib/jsonapi/src/EventSubscriber/JsonApiRequestValidator.php(36): Drupal\\jsonapi\\EventSubscriber\\JsonApiRequestValidator->validateQueryParams(Object(Symfony\\Component\\HttpFoundation\\Request))\n#1 [internal function]: Drupal\\jsonapi\\EventSubscriber\\JsonApiRequestValidator->onRequest(Object(Symfony\\Component\\HttpKernel\\Event\\GetResponseEvent), 'kernel.request', Object(Drupal\\Component\\EventDispatcher\\ContainerAwareEventDispatcher))\n#2 /var/www/html/core/lib/Drupal/Component/EventDispatcher/ContainerAwareEventDispatcher.php(111): call_user_func(Array, Object(Symfony\\Component\\HttpKernel\\Event\\GetResponseEvent), 'kernel.request', Object(Drupal\\Component\\EventDispatcher\\ContainerAwareEventDispatcher))\n#3 /var/www/html/vendor/symfony/http-kernel/HttpKernel.php(127): Drupal\\Component\\EventDispatcher\\ContainerAwareEventDispatcher->dispatch('kernel.request', Object(Symfony\\Component\\HttpKernel\\Event\\GetResponseEvent))\n#4 /var/www/html/vendor/symfony/http-kernel/HttpKernel.php(68): Symfony\\Component\\HttpKernel\\HttpKernel->handleRaw(Object(Symfony\\Component\\HttpFoundation\\Request), 1)\n#5 /var/www/html/core/lib/Drupal/Core/StackMiddleware/Session.php(57): Symfony\\Component\\HttpKernel\\HttpKernel->handle(Object(Symfony\\Component\\HttpFoundation\\Request), 1, true)\n#6 /var/www/html/core/lib/Drupal/Core/StackMiddleware/KernelPreHandle.php(47): Drupal\\Core\\StackMiddleware\\Session->handle(Object(Symfony\\Component\\HttpFoundation\\Request), 1, true)\n#7 /var/www/html/core/modules/page_cache/src/StackMiddleware/PageCache.php(106): Drupal\\Core\\StackMiddleware\\KernelPreHandle->handle(Object(Symfony\\Component\\HttpFoundation\\Request), 1, true)\n#8 /var/www/html/core/modules/page_cache/src/StackMiddleware/PageCache.php(85): Drupal\\page_cache\\StackMiddleware\\PageCache->pass(Object(Symfony\\Component\\HttpFoundation\\Request), 1, true)\n#9 /var/www/html/modules/contrib/jsonapi/src/StackMiddleware/FormatSetter.php(45): Drupal\\page_cache\\StackMiddleware\\PageCache->handle(Object(Symfony\\Component\\HttpFoundation\\Request), 1, true)\n#10 /var/www/html/vendor/asm89/stack-cors/src/Asm89/Stack/Cors.php(49): Drupal\\jsonapi\\StackMiddleware\\FormatSetter->handle(Object(Symfony\\Component\\HttpFoundation\\Request), 1, true)\n#11 /var/www/html/core/lib/Drupal/Core/StackMiddleware/ReverseProxyMiddleware.php(47): Asm89\\Stack\\Cors->handle(Object(Symfony\\Component\\HttpFoundation\\Request), 1, true)\n#12 /var/www/html/core/lib/Drupal/Core/StackMiddleware/NegotiationMiddleware.php(52): Drupal\\Core\\StackMiddleware\\ReverseProxyMiddleware->handle(Object(Symfony\\Component\\HttpFoundation\\Request), 1, true)\n#13 /var/www/html/vendor/stack/builder/src/Stack/StackedHttpKernel.php(23): Drupal\\Core\\StackMiddleware\\NegotiationMiddleware->handle(Object(Symfony\\Component\\HttpFoundation\\Request), 1, true)\n#14 /var/www/html/core/lib/Drupal/Core/DrupalKernel.php(693): Stack\\StackedHttpKernel->handle(Object(Symfony\\Component\\HttpFoundation\\Request), 1, true)\n#15 /var/www/html/index.php(19): Drupal\\Core\\DrupalKernel->handle(Object(Symfony\\Component\\HttpFoundation\\Request))\n#16 {main}",
"trace" : [
{
"class" : "Drupal\\jsonapi\\EventSubscriber\\JsonApiRequestValidator",
"type" : "->",
"line" : 36,
"file" : "/var/www/html/modules/contrib/jsonapi/src/EventSubscriber/JsonApiRequestValidator.php",
"args" : [
{
"server" : {},
"headers" : {},
"cookies" : {},
"query" : {},
"request" : {},
"attributes" : {},
"files" : {}
}
],
"function" : "validateQueryParams"
},
{
"class" : "Drupal\\jsonapi\\EventSubscriber\\JsonApiRequestValidator",
"type" : "->",
"args" : [
{},
"kernel.request",
{
"_serviceId" : "event_dispatcher"
}
],
"function" : "onRequest"
},
{
"file" : "/var/www/html/core/lib/Drupal/Component/EventDispatcher/ContainerAwareEventDispatcher.php",
"args" : [
[
{
"_serviceId" : "jsonapi.custom_query_parameter_names_validator.subscriber"
},
"onRequest"
],
{},
"kernel.request",
{
"_serviceId" : "event_dispatcher"
}
],
"function" : "call_user_func",
"line" : 111
},
{
"class" : "Drupal\\Component\\EventDispatcher\\ContainerAwareEventDispatcher",
"type" : "->",
"line" : 127,
"args" : [
"kernel.request",
{}
],
"function" : "dispatch",
"file" : "/var/www/html/vendor/symfony/http-kernel/HttpKernel.php"
},
{
"file" : "/var/www/html/vendor/symfony/http-kernel/HttpKernel.php",
"args" : [
{
"attributes" : {},
"files" : {},
"request" : {},
"server" : {},
"headers" : {},
"query" : {},
"cookies" : {}
},
1
],
"function" : "handleRaw",
"class" : "Symfony\\Component\\HttpKernel\\HttpKernel",
"type" : "->",
"line" : 68
},
{
"args" : [
{
"request" : {},
"query" : {},
"headers" : {},
"cookies" : {},
"server" : {},
"files" : {},
"attributes" : {}
},
1,
true
],
"function" : "handle",
"file" : "/var/www/html/core/lib/Drupal/Core/StackMiddleware/Session.php",
"class" : "Symfony\\Component\\HttpKernel\\HttpKernel",
"line" : 57,
"type" : "->"
},
{
"line" : 47,
"type" : "->",
"class" : "Drupal\\Core\\StackMiddleware\\Session",
"function" : "handle",
"args" : [
{
"server" : {},
"query" : {},
"headers" : {},
"cookies" : {},
"request" : {},
"attributes" : {},
"files" : {}
},
1,
true
],
"file" : "/var/www/html/core/lib/Drupal/Core/StackMiddleware/KernelPreHandle.php"
},
{
"args" : [
{
"files" : {},
"attributes" : {},
"request" : {},
"headers" : {},
"query" : {},
"cookies" : {},
"server" : {}
},
1,
true
],
"function" : "handle",
"file" : "/var/www/html/core/modules/page_cache/src/StackMiddleware/PageCache.php",
"class" : "Drupal\\Core\\StackMiddleware\\KernelPreHandle",
"type" : "->",
"line" : 106
},
{
"class" : "Drupal\\page_cache\\StackMiddleware\\PageCache",
"type" : "->",
"line" : 85,
"file" : "/var/www/html/core/modules/page_cache/src/StackMiddleware/PageCache.php",
"args" : [
{
"files" : {},
"attributes" : {},
"query" : {},
"headers" : {},
"cookies" : {},
"server" : {},
"request" : {}
},
1,
true
],
"function" : "pass"
},
{
"file" : "/var/www/html/modules/contrib/jsonapi/src/StackMiddleware/FormatSetter.php",
"args" : [
{
"server" : {},
"headers" : {},
"cookies" : {},
"query" : {},
"request" : {},
"attributes" : {},
"files" : {}
},
1,
true
],
"function" : "handle",
"class" : "Drupal\\page_cache\\StackMiddleware\\PageCache",
"type" : "->",
"line" : 45
},
{
"file" : "/var/www/html/vendor/asm89/stack-cors/src/Asm89/Stack/Cors.php",
"function" : "handle",
"args" : [
{
"files" : {},
"attributes" : {},
"request" : {},
"query" : {},
"headers" : {},
"cookies" : {},
"server" : {}
},
1,
true
],
"type" : "->",
"line" : 49,
"class" : "Drupal\\jsonapi\\StackMiddleware\\FormatSetter"
},
{
"class" : "Asm89\\Stack\\Cors",
"line" : 47,
"type" : "->",
"file" : "/var/www/html/core/lib/Drupal/Core/StackMiddleware/ReverseProxyMiddleware.php",
"args" : [
{
"request" : {},
"headers" : {},
"cookies" : {},
"query" : {},
"server" : {},
"files" : {},
"attributes" : {}
},
1,
true
],
"function" : "handle"
},
{
"file" : "/var/www/html/core/lib/Drupal/Core/StackMiddleware/NegotiationMiddleware.php",
"function" : "handle",
"args" : [
{
"query" : {},
"headers" : {},
"cookies" : {},
"server" : {},
"request" : {},
"files" : {},
"attributes" : {}
},
1,
true
],
"line" : 52,
"type" : "->",
"class" : "Drupal\\Core\\StackMiddleware\\ReverseProxyMiddleware"
},
{
"file" : "/var/www/html/vendor/stack/builder/src/Stack/StackedHttpKernel.php",
"function" : "handle",
"args" : [
{
"request" : {},
"headers" : {},
"query" : {},
"cookies" : {},
"server" : {},
"files" : {},
"attributes" : {}
},
1,
true
],
"type" : "->",
"line" : 23,
"class" : "Drupal\\Core\\StackMiddleware\\NegotiationMiddleware"
},
{
"file" : "/var/www/html/core/lib/Drupal/Core/DrupalKernel.php",
"args" : [
{
"attributes" : {},
"files" : {},
"server" : {},
"headers" : {},
"query" : {},
"cookies" : {},
"request" : {}
},
1,
true
],
"function" : "handle",
"class" : "Stack\\StackedHttpKernel",
"line" : 693,
"type" : "->"
},
{
"line" : 19,
"type" : "->",
"class" : "Drupal\\Core\\DrupalKernel",
"function" : "handle",
"args" : [
{
"attributes" : {},
"files" : {},
"request" : {},
"server" : {},
"cookies" : {},
"headers" : {},
"query" : {}
}
],
"file" : "/var/www/html/index.php"
}
]
},
"detail" : "The following query parameters violate the JSON:API spec: 'q'.",
"source" : {
"file" : "/var/www/html/modules/contrib/jsonapi/src/EventSubscriber/JsonApiRequestValidator.php",
"line" : 78
},
"links" : {
"info" : {
"href" : "http://jsonapi.org/format/#query-parameters"
},
"via" : {
"href" : "http://207.148.125.64/jsonapi/node/article?q=%2Fjsonapi%2Fnode%2Farticle"
}
}
}
],
"jsonapi" : {
"version" : "1.0",
"meta" : {
"links" : {
"self" : {
"href" : "http://jsonapi.org/format/1.0/"
}
}
}
}
}

Related

for webserver group -Received 0 SUCCESS signal(s) out of 1. Unable to satisfy 100% MinSuccessfulInstancesPercent requirement cloudformation stack

enter image description here
Everything (ASG, EC2 instances) does appear to be created correctly in AWS. The issue is with web-server group only. I am using default templates available with cloudformation for wordpress multi-AZ RDS database instance for storage.
I'm using the following CloudFormation template--
{
"AWSTemplateFormatVersion" : "2010-09-09",
"Description" : "AWS CloudFormation Sample Template WordPress_Multi_AZ: WordPress is web software you can use to create a beautiful website or blog. This template installs a highly-available, scalable WordPress deployment using a multi-az Amazon RDS database instance for storage. It demonstrates using the AWS CloudFormation bootstrap scripts to deploy WordPress. **WARNING** This template creates an Amazon EC2 instance, an Application Load Balancer and an Amazon RDS database instance. You will be billed for the AWS resources used if you create a stack from this template."
"Parameters" : {
"VpcId" : {
"Type" : "AWS::EC2::VPC::Id",
"Description" : "VpcId of your existing Virtual Private Cloud (VPC)",
"ConstraintDescription" : "must be the VPC Id of an existing Virtual Private Cloud."
},
"Subnets" : {
"Type" : "List<AWS::EC2::Subnet::Id>",
"Description" : "The list of SubnetIds in your Virtual Private Cloud (VPC)",
"ConstraintDescription" : "must be a list of at least two existing subnets associated with at least two different availability zones. They should be residing in the selected Virtual Private Cloud."
},
"KeyName": {
"Description" : "Name of an existing EC2 KeyPair to enable SSH access to the instances",
"Type": "AWS::EC2::KeyPair::KeyName",
"ConstraintDescription" : "must be the name of an existing EC2 KeyPair."
},
"InstanceType" : {
"Description" : "WebServer EC2 instance type",
"Type" : "String",
"Default" : "t2.small",
"AllowedValues" : [ "t1.micro", "t2.nano", "t2.micro", "t2.small", "t2.medium", "t2.large", "m1.small", "m1.medium", "m1.large", "m1.xlarge", "m2.xlarge", "m2.2xlarge", "m2.4xlarge", "m3.medium", "m3.large", "m3.xlarge", "m3.2xlarge", "m4.large", "m4.xlarge", "m4.2xlarge", "m4.4xlarge", "m4.10xlarge", "c1.medium", "c1.xlarge", "c3.large", "c3.xlarge", "c3.2xlarge", "c3.4xlarge", "c3.8xlarge", "c4.large", "c4.xlarge", "c4.2xlarge", "c4.4xlarge", "c4.8xlarge", "g2.2xlarge", "g2.8xlarge", "r3.large", "r3.xlarge", "r3.2xlarge", "r3.4xlarge", "r3.8xlarge", "i2.xlarge", "i2.2xlarge", "i2.4xlarge", "i2.8xlarge", "d2.xlarge", "d2.2xlarge", "d2.4xlarge", "d2.8xlarge", "hi1.4xlarge", "hs1.8xlarge", "cr1.8xlarge", "cc2.8xlarge", "cg1.4xlarge"]
,
"ConstraintDescription" : "must be a valid EC2 instance type."
},
"SSHLocation": {
"Description": "The IP address range that can be used to SSH to the EC2 instances",
"Type": "String",
"MinLength": "9",
"MaxLength": "18",
"Default": "0.0.0.0/0",
"AllowedPattern": "(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})/(\\d{1,2})",
"ConstraintDescription": "must be a valid IP CIDR range of the form x.x.x.x/x."
},
"DBClass" : {
"Description" : "Database instance class",
"Type" : "String",
"Default" : "db.t2.small",
"AllowedValues" : [ "db.t1.micro", "db.m1.small", "db.m1.medium", "db.m1.large", "db.m1.xlarge", "db.m2.xlarge", "db.m2.2xlarge", "db.m2.4xlarge", "db.m3.medium", "db.m3.large", "db.m3.xlarge", "db.m3.2xlarge", "db.m4.large", "db.m4.xlarge", "db.m4.2xlarge", "db.m4.4xlarge", "db.m4.10xlarge", "db.r3.large", "db.r3.xlarge", "db.r3.2xlarge", "db.r3.4xlarge", "db.r3.8xlarge", "db.m2.xlarge", "db.m2.2xlarge", "db.m2.4xlarge", "db.cr1.8xlarge", "db.t2.micro", "db.t2.small", "db.t2.medium", "db.t2.large"]
,
"ConstraintDescription" : "must select a valid database instance type."
},
"DBName" : {
"Default": "wordpressdb",
"Description" : "The WordPress database name",
"Type": "String",
"MinLength": "1",
"MaxLength": "64",
"AllowedPattern" : "[a-zA-Z][a-zA-Z0-9]*",
"ConstraintDescription" : "must begin with a letter and contain only alphanumeric characters."
},
"DBUser" : {
"NoEcho": "true",
"Description" : "The WordPress database admin account username",
"Type": "String",
"MinLength": "1",
"MaxLength": "16",
"AllowedPattern" : "[a-zA-Z][a-zA-Z0-9]*",
"ConstraintDescription" : "must begin with a letter and contain only alphanumeric characters."
},
"DBPassword" : {
"NoEcho": "true",
"Description" : "The WordPress database admin account password",
"Type": "String",
"MinLength": "8",
"MaxLength": "41",
"AllowedPattern" : "[a-zA-Z0-9]*",
"ConstraintDescription" : "must contain only alphanumeric characters."
},
"MultiAZDatabase": {
"Default": "false",
"Description" : "Create a Multi-AZ MySQL Amazon RDS database instance",
"Type": "String",
"AllowedValues" : [ "true", "false" ],
"ConstraintDescription" : "must be either true or false."
},
"WebServerCapacity": {
"Default": "1",
"Description" : "The initial number of WebServer instances",
"Type": "Number",
"MinValue": "1",
"MaxValue": "5",
"ConstraintDescription" : "must be between 1 and 5 EC2 instances."
},
"DBAllocatedStorage" : {
"Default": "5",
"Description" : "The size of the database (Gb)",
"Type": "Number",
"MinValue": "5",
"MaxValue": "1024",
"ConstraintDescription" : "must be between 5 and 1024Gb."
}
},
"Mappings" : {
"AWSInstanceType2Arch" : {
"t1.micro" : { "Arch" : "HVM64" },
"t2.nano" : { "Arch" : "HVM64" },
"t2.micro" : { "Arch" : "HVM64" },
"t2.small" : { "Arch" : "HVM64" },
"t2.medium" : { "Arch" : "HVM64" },
"t2.large" : { "Arch" : "HVM64" },
"m1.small" : { "Arch" : "HVM64" },
"m1.medium" : { "Arch" : "HVM64" },
"m1.large" : { "Arch" : "HVM64" },
"m1.xlarge" : { "Arch" : "HVM64" },
"m2.xlarge" : { "Arch" : "HVM64" },
"m2.2xlarge" : { "Arch" : "HVM64" },
"m2.4xlarge" : { "Arch" : "HVM64" },
"m3.medium" : { "Arch" : "HVM64" },
"m3.large" : { "Arch" : "HVM64" },
"m3.xlarge" : { "Arch" : "HVM64" },
"m3.2xlarge" : { "Arch" : "HVM64" },
"m4.large" : { "Arch" : "HVM64" },
"m4.xlarge" : { "Arch" : "HVM64" },
"m4.2xlarge" : { "Arch" : "HVM64" },
"m4.4xlarge" : { "Arch" : "HVM64" },
"m4.10xlarge" : { "Arch" : "HVM64" },
"c1.medium" : { "Arch" : "HVM64" },
"c1.xlarge" : { "Arch" : "HVM64" },
"c3.large" : { "Arch" : "HVM64" },
"c3.xlarge" : { "Arch" : "HVM64" },
"c3.2xlarge" : { "Arch" : "HVM64" },
"c3.4xlarge" : { "Arch" : "HVM64" },
"c3.8xlarge" : { "Arch" : "HVM64" },
"c4.large" : { "Arch" : "HVM64" },
"c4.xlarge" : { "Arch" : "HVM64" },
"c4.2xlarge" : { "Arch" : "HVM64" },
"c4.4xlarge" : { "Arch" : "HVM64" },
"c4.8xlarge" : { "Arch" : "HVM64" },
"g2.2xlarge" : { "Arch" : "HVMG2" },
"g2.8xlarge" : { "Arch" : "HVMG2" },
"r3.large" : { "Arch" : "HVM64" },
"r3.xlarge" : { "Arch" : "HVM64" },
"r3.2xlarge" : { "Arch" : "HVM64" },
"r3.4xlarge" : { "Arch" : "HVM64" },
"r3.8xlarge" : { "Arch" : "HVM64" },
"i2.xlarge" : { "Arch" : "HVM64" },
"i2.2xlarge" : { "Arch" : "HVM64" },
"i2.4xlarge" : { "Arch" : "HVM64" },
"i2.8xlarge" : { "Arch" : "HVM64" },
"d2.xlarge" : { "Arch" : "HVM64" },
"d2.2xlarge" : { "Arch" : "HVM64" },
"d2.4xlarge" : { "Arch" : "HVM64" },
"d2.8xlarge" : { "Arch" : "HVM64" },
"hi1.4xlarge" : { "Arch" : "HVM64" },
"hs1.8xlarge" : { "Arch" : "HVM64" },
"cr1.8xlarge" : { "Arch" : "HVM64" },
"cc2.8xlarge" : { "Arch" : "HVM64" }
},
"AWSInstanceType2NATArch" : {
"t1.micro" : { "Arch" : "NATHVM64" },
"t2.nano" : { "Arch" : "NATHVM64" },
"t2.micro" : { "Arch" : "NATHVM64" },
"t2.small" : { "Arch" : "NATHVM64" },
"t2.medium" : { "Arch" : "NATHVM64" },
"t2.large" : { "Arch" : "NATHVM64" },
"m1.small" : { "Arch" : "NATHVM64" },
"m1.medium" : { "Arch" : "NATHVM64" },
"m1.large" : { "Arch" : "NATHVM64" },
"m1.xlarge" : { "Arch" : "NATHVM64" },
"m2.xlarge" : { "Arch" : "NATHVM64" },
"m2.2xlarge" : { "Arch" : "NATHVM64" },
"m2.4xlarge" : { "Arch" : "NATHVM64" },
"m3.medium" : { "Arch" : "NATHVM64" },
"m3.large" : { "Arch" : "NATHVM64" },
"m3.xlarge" : { "Arch" : "NATHVM64" },
"m3.2xlarge" : { "Arch" : "NATHVM64" },
"m4.large" : { "Arch" : "NATHVM64" },
"m4.xlarge" : { "Arch" : "NATHVM64" },
"m4.2xlarge" : { "Arch" : "NATHVM64" },
"m4.4xlarge" : { "Arch" : "NATHVM64" },
"m4.10xlarge" : { "Arch" : "NATHVM64" },
"c1.medium" : { "Arch" : "NATHVM64" },
"c1.xlarge" : { "Arch" : "NATHVM64" },
"c3.large" : { "Arch" : "NATHVM64" },
"c3.xlarge" : { "Arch" : "NATHVM64" },
"c3.2xlarge" : { "Arch" : "NATHVM64" },
"c3.4xlarge" : { "Arch" : "NATHVM64" },
"c3.8xlarge" : { "Arch" : "NATHVM64" },
"c4.large" : { "Arch" : "NATHVM64" },
"c4.xlarge" : { "Arch" : "NATHVM64" },
"c4.2xlarge" : { "Arch" : "NATHVM64" },
"c4.4xlarge" : { "Arch" : "NATHVM64" },
"c4.8xlarge" : { "Arch" : "NATHVM64" },
"g2.2xlarge" : { "Arch" : "NATHVMG2" },
"g2.8xlarge" : { "Arch" : "NATHVMG2" },
"r3.large" : { "Arch" : "NATHVM64" },
"r3.xlarge" : { "Arch" : "NATHVM64" },
"r3.2xlarge" : { "Arch" : "NATHVM64" },
"r3.4xlarge" : { "Arch" : "NATHVM64" },
"r3.8xlarge" : { "Arch" : "NATHVM64" },
"i2.xlarge" : { "Arch" : "NATHVM64" },
"i2.2xlarge" : { "Arch" : "NATHVM64" },
"i2.4xlarge" : { "Arch" : "NATHVM64" },
"i2.8xlarge" : { "Arch" : "NATHVM64" },
"d2.xlarge" : { "Arch" : "NATHVM64" },
"d2.2xlarge" : { "Arch" : "NATHVM64" },
"d2.4xlarge" : { "Arch" : "NATHVM64" },
"d2.8xlarge" : { "Arch" : "NATHVM64" },
"hi1.4xlarge" : { "Arch" : "NATHVM64" },
"hs1.8xlarge" : { "Arch" : "NATHVM64" },
"cr1.8xlarge" : { "Arch" : "NATHVM64" },
"cc2.8xlarge" : { "Arch" : "NATHVM64" }
}
,
"AWSRegionArch2AMI" : {
"af-south-1" : {"HVM64" : "ami-064cc455f8a1ef504", "HVMG2" : "NOT_SUPPORTED"},
"ap-east-1" : {"HVM64" : "ami-f85b1989", "HVMG2" : "NOT_SUPPORTED"},
"ap-northeast-1" : {"HVM64" : "ami-0b2c2a754d5b4da22", "HVMG2" : "ami-09d0e0e099ecabba2"},
"ap-northeast-2" : {"HVM64" : "ami-0493ab99920f410fc", "HVMG2" : "NOT_SUPPORTED"},
"ap-northeast-3" : {"HVM64" : "ami-01344f6f63a4decc1", "HVMG2" : "NOT_SUPPORTED"},
"ap-south-1" : {"HVM64" : "ami-03cfb5e1fb4fac428", "HVMG2" : "ami-0244c1d42815af84a"},
"ap-southeast-1" : {"HVM64" : "ami-0ba35dc9caf73d1c7", "HVMG2" : "ami-0e46ce0d6a87dc979"},
"ap-southeast-2" : {"HVM64" : "ami-0ae99b503e8694028", "HVMG2" : "ami-0c0ab057a101d8ff2"},
"ca-central-1" : {"HVM64" : "ami-0803e21a2ec22f953", "HVMG2" : "NOT_SUPPORTED"},
"cn-north-1" : {"HVM64" : "ami-07a3f215cc90c889c", "HVMG2" : "NOT_SUPPORTED"},
"cn-northwest-1" : {"HVM64" : "ami-0a3b3b10f714a0ff4", "HVMG2" : "NOT_SUPPORTED"},
"eu-central-1" : {"HVM64" : "ami-0474863011a7d1541", "HVMG2" : "ami-0aa1822e3eb913a11"},
"eu-north-1" : {"HVM64" : "ami-0de4b8910494dba0f", "HVMG2" : "ami-32d55b4c"},
"eu-south-1" : {"HVM64" : "ami-08427144fe9ebdef6", "HVMG2" : "NOT_SUPPORTED"},
"eu-west-1" : {"HVM64" : "ami-015232c01a82b847b", "HVMG2" : "ami-0d5299b1c6112c3c7"},
"eu-west-2" : {"HVM64" : "ami-0765d48d7e15beb93", "HVMG2" : "NOT_SUPPORTED"},
"eu-west-3" : {"HVM64" : "ami-0caf07637eda19d9c", "HVMG2" : "NOT_SUPPORTED"},
"me-south-1" : {"HVM64" : "ami-0744743d80915b497", "HVMG2" : "NOT_SUPPORTED"},
"sa-east-1" : {"HVM64" : "ami-0a52e8a6018e92bb0", "HVMG2" : "NOT_SUPPORTED"},
"us-east-1" : {"HVM64" : "ami-032930428bf1abbff", "HVMG2" : "ami-0aeb704d503081ea6"},
"us-east-2" : {"HVM64" : "ami-027cab9a7bf0155df", "HVMG2" : "NOT_SUPPORTED"},
"us-west-1" : {"HVM64" : "ami-088c153f74339f34c", "HVMG2" : "ami-0a7fc72dc0e51aa77"},
"us-west-2" : {"HVM64" : "ami-01fee56b22f308154", "HVMG2" : "ami-0fe84a5b4563d8f27"}
}
},
"Resources" : {
"ApplicationLoadBalancer" : {
"Type" : "AWS::ElasticLoadBalancingV2::LoadBalancer",
"Properties" : {
"Subnets" : { "Ref" : "Subnets"}
}
},
"ALBListener" : {
"Type" : "AWS::ElasticLoadBalancingV2::Listener",
"Properties" : {
"DefaultActions" : [{
"Type" : "forward",
"TargetGroupArn" : { "Ref" : "ALBTargetGroup" }
}],
"LoadBalancerArn" : { "Ref" : "ApplicationLoadBalancer" },
"Port" : "80",
"Protocol" : "HTTP"
}
},
"ALBTargetGroup" : {
"Type" : "AWS::ElasticLoadBalancingV2::TargetGroup",
"Properties" : {
"HealthCheckPath" : "/wordpress/wp-admin/install.php",
"HealthCheckIntervalSeconds" : 10,
"HealthCheckTimeoutSeconds" : 5,
"HealthyThresholdCount" : 2,
"Port" : 80,
"Protocol" : "HTTP",
"UnhealthyThresholdCount" : 5,
"VpcId" : {"Ref" : "VpcId"},
"TargetGroupAttributes" :
[ { "Key" : "stickiness.enabled", "Value" : "true" },
{ "Key" : "stickiness.type", "Value" : "lb_cookie" },
{ "Key" : "stickiness.lb_cookie.duration_seconds", "Value" : "30" }
]
}
},
"WebServerSecurityGroup" : {
"Type" : "AWS::EC2::SecurityGroup",
"Properties" : {
"GroupDescription" : "Enable HTTP access via port 80 locked down to the load balancer + SSH access",
"SecurityGroupIngress" : [
{"IpProtocol" : "tcp", "FromPort" : "80", "ToPort" : "80", "SourceSecurityGroupId" : {"Fn::Select" : [0, {"Fn::GetAtt" : ["ApplicationLoadBalancer", "SecurityGroups"]}]}},
{"IpProtocol" : "tcp", "FromPort" : "22", "ToPort" : "22", "CidrIp" : { "Ref" : "SSHLocation"}}
],
"VpcId" : { "Ref" : "VpcId" }
}
},
"WebServerGroup" : {
"Type" : "AWS::AutoScaling::AutoScalingGroup",
"Properties" : {
"VPCZoneIdentifier" : { "Ref" : "Subnets" },
"LaunchConfigurationName" : { "Ref" : "LaunchConfig" },
"MinSize" : "1",
"MaxSize" : "5",
"DesiredCapacity" : { "Ref" : "WebServerCapacity" },
"TargetGroupARNs" : [ { "Ref" : "ALBTargetGroup" } ]
},
"CreationPolicy" : {
"ResourceSignal" : {
"Timeout" : "PT15M"
}
},
"UpdatePolicy": {
"AutoScalingRollingUpdate": {
"MinInstancesInService": "1",
"MaxBatchSize": "1",
"PauseTime" : "PT15M",
"WaitOnResourceSignals": "true"
}
}
},
"LaunchConfig": {
"Type" : "AWS::AutoScaling::LaunchConfiguration",
"Metadata" : {
"AWS::CloudFormation::Init" : {
"configSets" : {
"wordpress_install" : ["install_cfn", "install_wordpress" ]
},
"install_cfn" : {
"files": {
"/etc/cfn/cfn-hup.conf": {
"content": { "Fn::Join": [ "", [
"[main]\n",
"stack=", { "Ref": "AWS::StackId" }, "\n",
"region=", { "Ref": "AWS::Region" }, "\n"
]]},
"mode" : "000400",
"owner" : "root",
"group" : "root"
},
"/etc/cfn/hooks.d/cfn-auto-reloader.conf": {
"content": { "Fn::Join": [ "", [
"[cfn-auto-reloader-hook]\n",
"triggers=post.update\n",
"path=Resources.LaunchConfig.Metadata.AWS::CloudFormation::Init\n",
"action=/opt/aws/bin/cfn-init -v ",
" --stack ", { "Ref" : "AWS::StackName" },
" --resource LaunchConfig ",
" --configsets wordpress_install ",
" --region ", { "Ref" : "AWS::Region" }, "\n"
]]},
"mode" : "000400",
"owner" : "root",
"group" : "root"
}
},
"services" : {
"sysvinit" : {
"cfn-hup" : { "enabled" : "true", "ensureRunning" : "true",
"files" : ["/etc/cfn/cfn-hup.conf", "/etc/cfn/hooks.d/cfn-auto-reloader.conf"]}
}
}
},
"install_wordpress" : {
"packages" : {
"yum" : {
"php73" : [],
"php73-mysqlnd" : [],
"mysql57" : [],
"httpd24" : []
}
},
"sources" : {
"/var/www/html" : "http://wordpress.org/latest.tar.gz"
},
"files" : {
"/tmp/create-wp-config" : {
"content" : { "Fn::Join" : [ "", [
"#!/bin/bash\n",
"cp /var/www/html/wordpress/wp-config-sample.php /var/www/html/wordpress/wp-config.php\n",
"sed -i \"s/'database_name_here'/'",{ "Ref" : "DBName" }, "'/g\" wp-config.php\n",
"sed -i \"s/'username_here'/'",{ "Ref" : "DBUser" }, "'/g\" wp-config.php\n",
"sed -i \"s/'password_here'/'",{ "Ref" : "DBPassword" }, "'/g\" wp-config.php\n",
"sed -i \"s/'localhost'/'",{ "Fn::GetAtt" : [ "DBInstance", "Endpoint.Address" ] }, "'/g\" wp-config.php\n"
]]},
"mode" : "000500",
"owner" : "root",
"group" : "root"
}
},
"commands" : {
"01_configure_wordpress" : {
"command" : "/tmp/create-wp-config",
"cwd" : "/var/www/html/wordpress"
}
},
"services" : {
"sysvinit" : {
"httpd" : { "enabled" : "true", "ensureRunning" : "true" }
}
}
}
}
},
"Properties": {
"ImageId" : { "Fn::FindInMap" : [ "AWSRegionArch2AMI", { "Ref" : "AWS::Region" },
{ "Fn::FindInMap" : [ "AWSInstanceType2Arch", { "Ref" : "InstanceType" }, "Arch" ] } ] },
"InstanceType" : { "Ref" : "InstanceType" },
"SecurityGroups" : [ {"Ref" : "WebServerSecurityGroup"} ],
"KeyName" : { "Ref" : "KeyName" },
"UserData" : { "Fn::Base64" : { "Fn::Join" : ["", [
"#!/bin/bash -xe\n",
"yum update -y aws-cfn-bootstrap\n",
"/opt/aws/bin/cfn-init -v ",
" --stack ", { "Ref" : "AWS::StackName" },
" --resource LaunchConfig ",
" --configsets wordpress_install ",
" --region ", { "Ref" : "AWS::Region" }, "\n",
"/opt/aws/bin/cfn-signal -e $? ",
" --stack ", { "Ref" : "AWS::StackName" },
" --resource WebServerGroup ",
" --region ", { "Ref" : "AWS::Region" }, "\n"
]]}}
}
},
"DBEC2SecurityGroup": {
"Type": "AWS::EC2::SecurityGroup",
"Properties" : {
"GroupDescription": "Open database for access",
"SecurityGroupIngress" : [{
"IpProtocol" : "tcp",
"FromPort" : "3306",
"ToPort" : "3306",
"SourceSecurityGroupId" : { "Ref" : "WebServerSecurityGroup" }
}],
"VpcId" : { "Ref" : "VpcId" }
}
},
"DBInstance" : {
"Type": "AWS::RDS::DBInstance",
"Properties": {
"DBName" : { "Ref" : "DBName" },
"Engine" : "MySQL",
"MultiAZ" : { "Ref": "MultiAZDatabase" },
"MasterUsername" : { "Ref" : "DBUser" },
"MasterUserPassword": { "Ref" : "DBPassword" },
"DBInstanceClass" : { "Ref" : "DBClass" },
"AllocatedStorage" : { "Ref" : "DBAllocatedStorage" },
"VPCSecurityGroups" : [{ "Fn::GetAtt": [ "DBEC2SecurityGroup", "GroupId" ]}]
}
}
},
"Outputs" : {
"WebsiteURL" : {
"Value" : { "Fn::Join" : ["", ["http://", { "Fn::GetAtt" : [ "ApplicationLoadBalancer", "DNSName" ]}, "/wordpress" ]]},
"Description" : "WordPress Website"
}
}
}

Converting a painless script into a visualisation on Kibana (Logs from AWS Connect)

I have logs being shipped from AWS Connect to Kibana through AWS OpenSearch. I have written the following script to return the latest status of an Agent like so:
GET agent-logs-*/_search
{
"script_fields": {
"data": {
"script": {
"lang": "painless",
"source": "params._source.CurrentAgentSnapshot.Configuration.Username + ', ' + params._source.CurrentAgentSnapshot.AgentStatus.Name + ', ' + params._source.EventTimestamp"
}
}
},
"collapse": {
"field": "CurrentAgentSnapshot.Configuration.Username.keyword"
},
"sort": [
{
"EventTimestamp": {
"order": "desc"
}
}
]
}
This returns a value of:
{
"took" : 29,
"timed_out" : false,
"_shards" : {
"total" : 65,
"successful" : 65,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 10000,
"relation" : "gte"
},
"max_score" : null,
"hits" : [
{
"_index" : "agent-logs-2022-06-28",
"_type" : "_doc",
"_id" : "",
"_score" : null,
"fields" : {
"data" : [
"al.pacino#email.com, Available, 2022-06-28T10:52:01.238Z"
],
"CurrentAgentSnapshot.Configuration.Username.keyword" : [
"al.pacino#email.com"
]
},
"sort" : [
1656413521238
]
},
{
"_index" : "agent-logs-2022-06-28",
"_type" : "_doc",
"_id" : "",
"_score" : null,
"fields" : {
"data" : [
"robert.deniro#email.com, Available, 2022-06-28T10:50:45.622Z"
],
"CurrentAgentSnapshot.Configuration.Username.keyword" : [
"robert.deniro#email.com"
]
},
"sort" : [
1656413445622
]
},
{
"_index" : "agent-logs-2022-06-26",
"_type" : "_doc",
"_id" : "",
"_score" : null,
"fields" : {
"data" : [
"marlon.brando#email.com, Offline, 2022-06-26T14:51:55.203Z"
],
"CurrentAgentSnapshot.Configuration.Username.keyword" : [
"marlon.brando#email.com"
]
},
"sort" : [
1656255115203
]
}
]
}
}
I wanted to take the data lines from the JSON - "al.pacino#email.com, Available, 2022-06-28T10:52:01.238Z" and represent this in a visualisation such as a Data Table to get a list of agents with their corresponding status.
By using the current agent-logs, there is a delay whereby the status change and heart beats overlap, causing an inaccurate count of the status, thus need to use this script.

Artifactory API - List all artifacts for a package version

What's the best way to list all assets for a given package via the Artifactory API?
I'm trying to write a script to get the assets for a package and I'd like for it to work with multiple repository types, like Maven and PyPI. I know I could use the Folder Info API to get what I need, but that relies on the repository layout, so it wouldn't work across repository types.
I'm currently using this AQL search:
curl -u user:password -X POST http://<artifactory_url>/artifactory/api/search/aql \
-H "Content-Type: text/plain" \
-d 'items.find({"repo": "libs-release-local"}, {"artifact.module.name": "com.foo.bar:fizz-buzz:1.2"})'
The response is almost what I want, but it seems to be including some assets from a different version of the package I'm searching for:
{
"results" : [ {
"repo" : "libs-release-local",
"path" : "com/foo/bar/fizz-buzz/1.0",
"name" : "fizz-buzz-1.0.properties",
"type" : "file",
"size" : 790,
"created" : "2020-09-29T15:35:59.233Z",
"created_by" : "user",
"modified" : "2020-09-29T15:35:59.181Z",
"modified_by" : "user",
"updated" : "2020-09-29T15:35:59.233Z"
},{
"repo" : "libs-release-local",
"path" : "com/foo/bar/fizz-buzz/1.1",
"name" : "fizz-buzz-1.1.properties",
"type" : "file",
"size" : 790,
"created" : "2020-09-29T15:42:34.982Z",
"created_by" : "user",
"modified" : "2020-09-29T15:42:34.931Z",
"modified_by" : "user",
"updated" : "2020-09-29T15:42:34.983Z"
},{
"repo" : "libs-release-local",
"path" : "com/foo/bar/fizz-buzz/1.2",
"name" : "fizz-buzz-1.2-javadoc.jar",
"type" : "file",
"size" : 391843,
"created" : "2020-09-30T18:54:41.599Z",
"created_by" : "user",
"modified" : "2020-09-30T18:54:40.650Z",
"modified_by" : "user",
"updated" : "2020-09-30T18:54:41.600Z"
},{
"repo" : "libs-release-local",
"path" : "com/foo/bar/fizz-buzz/1.2",
"name" : "fizz-buzz-1.2-sources.jar",
"type" : "file",
"size" : 1089,
"created" : "2020-09-30T18:54:41.764Z",
"created_by" : "user",
"modified" : "2020-09-30T18:54:41.710Z",
"modified_by" : "user",
"updated" : "2020-09-30T18:54:41.765Z"
},{
"repo" : "libs-release-local",
"path" : "com/foo/bar/fizz-buzz/1.2",
"name" : "fizz-buzz-1.2.jar",
"type" : "file",
"size" : 1410,
"created" : "2020-09-30T18:54:41.902Z",
"created_by" : "user",
"modified" : "2020-09-30T18:54:41.844Z",
"modified_by" : "user",
"updated" : "2020-09-30T18:54:41.903Z"
},{
"repo" : "libs-release-local",
"path" : "com/foo/bar/fizz-buzz/1.2",
"name" : "fizz-buzz-1.2.module",
"type" : "file",
"size" : 3481,
"created" : "2020-09-30T18:54:42.015Z",
"created_by" : "user",
"modified" : "2020-09-30T18:54:41.962Z",
"modified_by" : "user",
"updated" : "2020-09-30T18:54:42.015Z"
},{
"repo" : "libs-release-local",
"path" : "com/foo/bar/fizz-buzz/1.2",
"name" : "fizz-buzz-1.2.pom",
"type" : "file",
"size" : 781,
"created" : "2020-09-30T18:54:42.238Z",
"created_by" : "user",
"modified" : "2020-09-30T18:54:42.190Z",
"modified_by" : "user",
"updated" : "2020-09-30T18:54:42.238Z"
},{
"repo" : "libs-release-local",
"path" : "com/foo/bar/fizz-buzz/1.2",
"name" : "fizz-buzz-1.2.properties",
"type" : "file",
"size" : 790,
"created" : "2020-09-30T18:54:42.124Z",
"created_by" : "user",
"modified" : "2020-09-30T18:54:42.078Z",
"modified_by" : "user",
"updated" : "2020-09-30T18:54:42.125Z"
} ],
"range" : {
"start_pos" : 0,
"end_pos" : 8,
"total" : 8
}
}
Notice how it's including the properties file for fizz-buzz 1.0 and 1.1, even though I specified 1.2 in my search.
Is there a better way to get the information I'm looking for?
You can use the new GraphQL capability which was added in Artifactory 7.9.
This new capability allows you to query the rich metadata Artifactory holds about packages, version, artifacts and more using the GraphQL query language.
You can use the metadata REST API for queries. Please notice you need to use an admin access token for authentication. For example:
curl -H "Authorization: Bearer <Your Token>" -XPOST http://localhost:8082/metadata/api/v1/query -d '{"query":"..." }'
The following query, as an example, is fetching all the files which are part of versions 1.0* of a package named hello-world. This query will work for any type of package which can be managed in Artifactory.
query {
packages(
filter: {
name: "hello-world"
}
) {
edges {
node {
name
packageType
versions (filter: {name : "1.0*"}) {
name
repos {
name
leadFilePath
}
files {
name
}
}
}
}
}
}
The result would look something like
{
"data": {
"packages": {
"edges": [
{
"node": {
"name": "hello-world",
"packageType": "maven",
"versions": [
{
"name": "1.0-SNAPSHOT",
"repos": [
{
"name": "kotlin-local-snapshots",
"leadFilePath": "org/jetbrains/kotlin/hello-world/1.0-SNAPSHOT/hello-world-1.0-20171225.112927-1.pom"
}
],
"files": [
{
"name": "hello-world-1.0-20171225.112927-1.jar"
},
{
"name": "hello-world-1.0-20171225.112927-1.pom"
}
]
}
]
}
},
{
"node": {
"name": "hello-world",
"packageType": "maven",
"versions": [
{
"name": "1.0-SNAPSHOT",
"repos": [
{
"name": "kotlin-local-snapshots",
"leadFilePath": "org/jetbrains/kotlin/examples/hello-world/1.0-SNAPSHOT/hello-world-1.0-20171225.112138-1.pom"
}
],
"files": [
{
"name": "hello-world-1.0-20171225.112138-1.jar"
},
{
"name": "hello-world-1.0-20171225.112138-1.pom"
}
]
}
]
}
}
]
}
}
}
Try the below its using the path to only find artifacts that match com/foo/bar/fizz-buzz in the repo libs-release-local and then some jq at the end to make the output better. Noticed also the type : file which eliminates some noise in terms of metdata.
You will need to define or replace USER, API_KEY, and ARTIFACTORY_URL.
curl -su "${USER}:${API_KEY}" -X POST "${ARTIFACTORY_URL}/artifactory/api/search/aql" \
-H "content-type: text/plain" \
-d "items.find({\"type\" : \"file\",\"\$and\":[{\"path\" : {\"\$match\" : \"com/foo/bar/fizz-buzz*\"}, \"repo\" : {\"\$match\" : \"libs-release-local\"} }]}).include(\"name\",\"repo\",\"path\",\"size\").sort({\"\$desc\": [\"size\"]})" \
| jq -r "to_entries|map(\"\(.key)=\(.value|tostring)\")|.[]" | grep results | cut -f 2 -d = | jq .

Elasticsearch low indexing speed

I have a blog that contains 14k posts and tried to add these posts to the elastic search index.
I indexed some of the posts, but it's extremely slow, and it will take about 6 hours to estimate. All the performance optimization tips from the official site I made. In my opinion, I removed the redundant data such as post meta. Can I increase indexing speed? Add the index configuration below:
{
"test-post-1" : {
"aliases" : { },
"mappings" : {
"date_detection" : false,
"properties" : {
"ID" : {
"type" : "long"
},
"guid" : {
"type" : "keyword"
},
"menu_order" : {
"type" : "long"
},
"permalink" : {
"type" : "keyword"
},
"post_content" : {
"type" : "text"
},
"post_date" : {
"type" : "date",
"format" : "yyyy-MM-dd HH:mm:ss"
},
"post_excerpt" : {
"type" : "text"
},
"post_id" : {
"type" : "long"
},
"post_mime_type" : {
"type" : "keyword"
},
"post_modified" : {
"type" : "date",
"format" : "yyyy-MM-dd HH:mm:ss"
},
"post_name" : {
"type" : "text",
"fields" : {
"post_name" : {
"type" : "text"
},
"raw" : {
"type" : "keyword",
"ignore_above" : 10922
}
}
},
"post_parent" : {
"type" : "long"
},
"post_status" : {
"type" : "keyword"
},
"post_title" : {
"type" : "text",
"fields" : {
"post_title" : {
"type" : "text",
"analyzer" : "standard"
},
"raw" : {
"type" : "keyword",
"ignore_above" : 10922
},
"sortable" : {
"type" : "keyword",
"ignore_above" : 10922,
"normalizer" : "lowerasciinormalizer"
}
}
},
"post_type" : {
"type" : "text",
"fields" : {
"post_type" : {
"type" : "text"
},
"raw" : {
"type" : "keyword"
}
}
}
}
},
"settings" : {
"index" : {
"mapping" : {
"total_fields" : {
"limit" : "5000"
},
"ignore_malformed" : "true"
},
"number_of_shards" : "1",
"provided_name" : "test-post-1",
"max_shingle_diff" : "8",
"max_result_window" : "1000000",
"creation_date" : "1582745447768",
"analysis" : {
"filter" : {
"shingle_filter" : {
"max_shingle_size" : "5",
"min_shingle_size" : "2",
"type" : "shingle"
},
"edge_ngram" : {
"min_gram" : "3",
"side" : "front",
"type" : "edgeNGram",
"max_gram" : "10"
},
"ewp_word_delimiter" : {
"type" : "word_delimiter",
"preserve_original" : "true"
},
"ewp_snowball" : {
"type" : "snowball",
"language" : "russian"
}
},
"normalizer" : {
"lowerasciinormalizer" : {
"filter" : [
"lowercase",
"asciifolding"
],
"type" : "custom"
}
},
"analyzer" : {
"ewp_lowercase" : {
"filter" : [
"lowercase"
],
"type" : "custom",
"tokenizer" : "keyword"
},
"shingle_analyzer" : {
"filter" : [
"lowercase",
"shingle_filter"
],
"type" : "custom",
"tokenizer" : "standard"
},
"default" : {
"filter" : [
"ewp_word_delimiter",
"lowercase",
"stop",
"ewp_snowball"
],
"char_filter" : [
"html_strip"
],
"language" : "russian",
"tokenizer" : "standard"
}
}
},
"number_of_replicas" : "1",
"uuid" : "cWGjSF4FQ1Or0A_0oSlA2g",
"version" : {
"created" : "7050299"
}
}
}
}
}
Wordpress version: 5.3.2
Elasticsearch version: 7.5.2
Enabled plugins: ElasticPress

querying elasitcsearch parent child documents

We work with two types of documents on elastic search (ES): items and slots, where items are parents of slot documents.
We define the index with the following command:
curl -XPOST 'localhost:9200/items' -d #itemsdef.json
where itemsdef.json has the following definition
{
"mappings" : {
"item" : {
"properties" : {
"id" : {"type" : "long" },
"name" : {
"type" : "string",
"_analyzer" : "textIndexAnalyzer"
},
"location" : {"type" : "geo_point" },
}
}
},
"settings" : {
"analysis" : {
"analyzer" : {
"activityIndexAnalyzer" : {
"alias" : ["activityQueryAnalyzer"],
"type" : "custom",
"tokenizer" : "whitespace",
"filter" : ["trim", "lowercase", "asciifolding", "spanish_stop", "spanish_synonym"]
},
"textIndexAnalyzer" : {
"type" : "custom",
"tokenizer" : "whitespace",
"filter" : ["word_delimiter_impl", "trim", "lowercase", "asciifolding", "spanish_stop", "spanish_synonym"]
},
"textQueryAnalyzer" : {
"type" : "custom",
"tokenizer" : "whitespace",
"filter" : ["trim", "lowercase", "asciifolding", "spanish_stop"]
}
},
"filter" : {
"spanish_stop" : {
"type" : "stop",
"ignore_case" : true,
"enable_position_increments" : true,
"stopwords_path" : "analysis/spanish-stopwords.txt"
},
"spanish_synonym" : {
"type" : "synonym",
"synonyms_path" : "analysis/spanish-synonyms.txt"
},
"word_delimiter_impl" : {
"type" : "word_delimiter",
"generate_word_parts" : true,
"generate_number_parts" : true,
"catenate_words" : true,
"catenate_numbers" : true,
"split_on_case_change" : false
}
}
}
}
}
Then we add the child document definition using the following command:
curl -XPOST 'localhost:9200/items/slot/_mapping' -d #slotsdef.json
Where slotsdef.json has the following definition:
{
"slot" : {
"_parent" : {"type" : "item"},
"_routing" : {
"required" : true,
"path" : "parent_id"
},
"properties": {
"id" : { "type" : "long" },
"parent_id" : { "type" : "long" },
"activity" : {
"type" : "string",
"_analyzer" : "activityIndexAnalyzer"
},
"day" : { "type" : "integer" },
"start" : { "type" : "integer" },
"end" : { "type" : "integer" }
}
}
}
Finally we perform a bulk index with the following command:
curl -XPOST 'localhost:9200/items/_bulk' --data-binary #testbulk.json
Where testbulk.json holds the following data:
{"index":{"_type": "item", "_id":35}}
{"location":[40.4,-3.6],"id":35,"name":"A Name"}
{"index":{"_type":"slot","_id":126,"_parent":35}}
{"id":126,"start":1330,"day":1,"end":1730,"activity":"An Activity","parent_id":35}
I'm trying to make the following query: search for all items within a certain distance to a location that have children (slots) in the specified days and within certain start and end ranges.
An item with more slots fulfilling the condition should score higher.
I tried starting with existing samples but the docs are really scarce and its hard to move forward.
Clues?
I don't think there is a way to write an efficient query that would do something like this without moving location to slots. You can do something like this, but it can quite inefficient for some data:
{
"query": {
"top_children" : {
"type": "blog_tag",
"query" : {
"constant_score" : {
"query" : {
... your query for children goes here ...
}
}
},
"score" : "sum",
"factor" : 5,
"incremental_factor" : 2
}
},
"filter": {
"geo_distance" : {
"distance" : "200km",
"location" : {
"lat" : 40,
"lon" : -70
}
}
}
}
}
Basically, what this query is doing is this, it takes your range query or filter for children and whatever other conditions you need and wraps it into constant_score query to make sure that all children have score of 1.0. The top_children query collects all these children and accumulates their scores to the parents. And then filter filters out parents that are too far away.

Resources