I am working with AWS kinesis stream to create prefix for delivery output (S3), I chose JQ as parser and one of the prefix key is optional and I tried alternative option provided by jq but it is giving me an error. I have attached random AWS cloudtrail and the code snippet.
`
{
"eventVersion": "1.08",
"OptionalKey": "Optional" ---> I want to get the optional field and if it does not exist return "undefined"
"userIdentity": {
"type": "AssumedRole",
"principalId": "XXXXXXXXXXXXXXXXX:dkboss",
"arn": "XXXXXXXXXXXXXXXXXXXXXXXXX/dkboss",
"accountId": "XXXXXXXXXXXXXXXX",
"sessionContext": {
"sessionIssuer": {
"type": "Role",
"principalId": "XXXXXXXXXXXXXXXXXXXXXXXXXX",
"arn": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
"accountId": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
"userName": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
},
"webIdFederationData": {},
"attributes": {
"creationDate": "2022-11-27T15:24:28Z",
"mfaAuthenticated": "false" }
}
},
"eventTime": "2022-11-27T15:24:29Z",
"eventSource": "signin.amazonaws.com",
"eventName": "ConsoleLogin",
"awsRegion": "us-east-1",
"sourceIPAddress": "1.1.1.1",
"userAgent": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
"requestParameters": null,
"responseElements": {
"ConsoleLogin": "Success"
},
"additionalEventData": {
"MobileVersion": "No",
"MFAUsed": "No"
},
"eventID": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
"readOnly": false,
"eventType": "AwsConsoleSignIn",
"managementEvent": true,
"recipientAccountId": "XXXXXXXXXXXXXXXXXXXXXXX",
"eventCategory": "Management",
"tlsDetails": {
"tlsVersion": "TLSv1.2",
"cipherSuite": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
"clientProvidedHostHeader": "signin.aws.amazon.com"
}
}
`
I want to get build S3 prefix, but I couldn't figure out how optional values are supported.
`
CfnDeliveryStream.ProcessorParameterProperty(
parameter_name="MetadataExtractionQuery",
parameter_value="{"
"tlsDetais : .tlsDetails.tlsVersion,"
'OptionalKey : .OptionalKey // "undefined",'
"}"
`
Inside ProcessorParameterProperty, But it failed with the following error:
JQ Subprocess failed due to: jq: error: syntax error, unexpected //, expecting '}' (Unix shell quoting issues?) at <top-level>,
You have to group your expression with parentheses:
{
tlsDetais : .tlsDetails.tlsVersion,
OptionalKey : (.OptionalKey // "undefined")
}
Related
I have defined a state machine in AWS step functions and one of my states is storing an item to DynamoDB
...
"Store item": {
"End": true,
"Type": "Task",
"Resource": "arn:aws:states:::dynamodb:putItem",
"Parameters": {
"Item": {
"foo": {
"S.$": "$.data.foo"
},
"bar": {
"S.$": "$.data.bar"
},
"baz": {
"S.$": "$.data.baz"
},
},
"TableName": "nrp_items"
}
},
...
The problem starts from the fact that baz property is optional, ie not exist in some cases.
On those cases, the putItem task fails:
An error occurred while executing the state 'Store item' (entered at the event id #71). > The JSONPath '$.data.baz' specified for the field 'S.$' could not be found in the input
My backup plan is to use a lambda to perform that type of operation, but can I do it directly using the putItem task in steps function?
I was wondering if:
Is possible to somehow inject via JSONPath my whole $.data item to the "Item" property, something like:
...
"Store item": {
"End": true,
"Type": "Task",
"Resource": "arn:aws:states:::dynamodb:putItem",
"Parameters": {
"Item": "$.data",
"TableName": "nrp_items"
}
},
...
OR
2) Define that the baz property is optional
TL;DR We can deal with optional variables with a "Variable": "$.baz", "IsPresent": true Choice condition to handle no-baz cases.
The Amazon States Language spec does not have optional properties: Step Functions will throw an error if $.baz does not exist in the input. We can avoid undefined paths by inserting a two-branch Choice State, one branch of which handles baz-exists cases, the other no-baz cases. Each branch continues with a Pass State that reworks the data input into dynamo-format Item syntax, using Parameters. The put-item task's "Item.$": "$.data" (as in your #1) contains only foo-bar when baz is not defined, but all three otherwise.
{
"StartAt": "HasBazChoice",
"States": {
"HasBazChoice": {
"Type": "Choice",
"Choices": [
{
"Variable": "$.baz",
"IsPresent": true,
"Next": "MakeHasBazItem"
}
],
"Default": "MakeNoBazItem"
},
"MakeHasBazItem": {
"Type": "Pass",
"Parameters": {
"data": {
"foo": { "S.$": "$.foo"},
"bar": { "S.$": "$.bar"},
"baz": { "S.$": "$.baz"}
}
},
"Next": "PutItemTask"
},
"MakeNoBazItem": {
"Type": "Pass",
"Parameters": {
"data": {
"foo": {"S.$": "$.foo"},
"bar": {"S.$": "$.bar"}
}
},
"Next": "PutItemTask"
},
"PutItemTask": {
...
"Parameters": {
"TableName": "my-table",
"Item.$": "$.data"
}
},
}
}
If you have more than one optional field, your lambda backup plan is the better option - the above workaround would become unwieldy.
I've this resource of type Paramaters as follow:
{
"resourceType": "Parameters",
"parameter": [
{
"name": "operation",
"part": [
{
"name": "type",
"valueCode": "add"
},
{
"name": "path",
"valueString": "Organization"
},
{
"name": "name",
"valueString": "active"
},
{
"name": "value",
"valueString": "true"
}
]
}
]
}
Because I want to add active value on my Organization.
So I write, in according with HAPI FHIR documentation - 4.2.7 paragraph, this code to execute a patch:
fhirClientR4Configuration.clientFhihrR4().patch().withBody(body).withId("Organization/2381").execute();
But I have this error:
ca.uhn.fhir.rest.server.exceptions.InvalidRequestException: HTTP 400 :
Cannot deserialize instance of
java.util.ArrayList<com.github.fge.jsonpatch.operation.JsonPatchOperation>
out of FIELD_NAME token at [Source: UNKNOWN; line: -1, column: -1]
I've noticed I must put in the same array all informations about patch (in the tag part), but I don't know I can resolve it.
Tha HAPI FHIR version is 4.2.0
I'm in the process of creating a Logic App which should copy all documents from specific containers in the database to a different database and its corresponding collections.
For this, I've created a workflow which retrieves all documents, loops through them and tries to do a Create or Update document on the target location.
As you can see on the image, this looks fairly straightforward.
I've specified the PartitionKey in the header and Upsert is set to true, so if a document already exists it'll be updated, otherwise created.
Originally, the body was filled with #items('[blurred]_-_For_each_document').
However, I got an error stating there was a conflict on the id.
I've also tried to remove all 'system' properties like so:
#removeProperty(removeProperty(removeProperty(removeProperty(removeProperty(items('[blurred]_-_For_each_document'), 'id'), '_rid'), '_self'), '_ts'), '_etag')
but it appears not having an id in place isn't valid, so now I've got this as I'm only interested in having the contents of the 'actual' document:
#removeProperty(removeProperty(removeProperty(removeProperty(items('[blurred]_-_For_each_document'), '_rid'), '_self'), '_ts'), '_etag')
This still fails.
The raw input looks a bit like this:
{
"method": "post",
"headers": {
"x-ms-documentdb-raw-partitionkey": "1050",
"x-ms-documentdb-is-upsert": "True"
},
"path": "/dbs/[myDatabase]/colls/[myCollection]/docs",
"host": {
"connection": {
"name": "/subscriptions/[subscriptionGuid]/resourceGroups/[resourcegroup]/providers/Microsoft.Web/connections/[connectionName]"
}
},
"body": {
"id": "2faee185-4a51-4797-bff9-3ce23a603690",
"MyPartitionKeyNumber": 1050,
"SomeValue": false,
}
}
With the following response:
{
"code": "Conflict",
"message": "Entity with the specified id already exists in the system., ... ResourceType: Document, OperationType: Upsert ..."
}
I know for a fact, the id doesn't exist, but changed the step to do a PUT anyway.
{
"method": "put",
"headers": {
"x-ms-documentdb-raw-partitionkey": "1050",
"x-ms-documentdb-is-upsert": "True"
},
"path": "/dbs/[myDatabase]/colls/[myCollection]/docs",
"host": {
"connection": {
"name": "/subscriptions/[subscriptionGuid]/resourceGroups/[resourcegroup]/providers/Microsoft.Web/connections/[connectionName]"
}
},
"body": {
"id": "2faee185-4a51-4797-bff9-3ce23a603690",
"MyPartitionKeyNumber": 1050,
"SomeValue": false,
}
}
With a response I'm expecting to see:
{
"statusCode": 404,
"message": "Resource not found"
}
I've also tried to do a Delete document before running the Create or Update document step, but got the same error(s).
There's some post here on Stack Overflow stating the x-ms-documentdb-raw-partitionkey should be x-ms-documentdb-partitionkey (without the raw-part), but doing so results in the following error message:
PartitionKey extracted from document doesn't match the one specified in the header
For completeness sake, these are the relevant steps of my workflow:
"[blurred]_-_Get_all_documents": {
"type": "ApiConnection",
"inputs": {
"host": {
"connection": {
"name": "#parameters('$connections')['documentdb_3']['connectionId']"
}
},
"method": "get",
"path": "/v2/dbs/#{encodeURIComponent('myDatabase')}/colls/#{encodeURIComponent(items('[blurred]_-_For_each_collection'))}/docs"
},
"runAfter": {}
},
"[blurred]_-_For_each_document": {
"type": "Foreach",
"foreach": "#body('[blurred]_-_Get_all_documents')?['value']",
"actions": {
"[blurred]_-_Create_or_Update_document_on_blurred2": {
"type": "ApiConnection",
"inputs": {
"host": {
"connection": {
"name": "#parameters('$connections')['documentdb_4']['connectionId']"
}
},
"method": "post",
"body": "#removeProperty(removeProperty(removeProperty(removeProperty(items('[blurred]_-_For_each_document'), '_rid'), '_self'), '_ts'), '_etag')",
"headers": {
"x-ms-documentdb-raw-partitionkey": "#{items('[blurred]_-_For_each_document')['MyPartitionKeyNumber']}",
"x-ms-documentdb-is-upsert": true
},
"path": "/dbs/#{encodeURIComponent('myDatabase')}/colls/#{encodeURIComponent(items('[blurred]_-_For_each_collection'))}/docs"
},
"runAfter": {}
}
},
"runAfter": {
"[blurred]_-_Get_all_documents": [
"Succeeded"
]
}
}
I am trying to use a query with a filter:
query queryPitchesByApprovedIndex($approved: Boolean = true) {
queryPitchesByApprovedIndex(approved: $approved) {
items {
id
}
}
}
The mapping template, made by AppSync, looks like this:
{
"version": "2017-02-28",
"operation": "Query",
"query": {
"expression": "#approved = :approved",
"expressionNames": {
"#approved": "approved",
},
"expressionValues": {
":approved": {"B": $util.dynamodb.toBinary($ctx.args.approved)},
},
},
"index": "approved-index",
"limit": $util.defaultIfNull($ctx.args.first, 20),
"nextToken": $util.toJson($util.defaultIfNullOrEmpty($ctx.args.after, null)),
"scanIndexForward": true,
"select": "ALL_ATTRIBUTES",
}
The error I get is:
Unable to parse the JSON document: 'Unrecognized token '$util': was expecting ('true', 'false' or 'null')\n at [Source: (String)\"{\n \"version\": \"2017-02-28\",\n \"operation\": \"Query\",\n \"query\": {\n \"expression\": \"#approved = :approved\",\n \"expressionNames\": {\n \"#approved\": \"approved\",\n },\n \"expressionValues\": {\n \":approved\": {\"B\": $util.dynamodb.toBinary($ctx.args.approved)},\n },\n },\n \"index\": \"approved-index\",\n \"limit\": 20,\n \"nextToken\": null,\n \"scanIndexForward\": true,\n \"select\": \"ALL_ATTRIBUTES\",\n}\"; line: 10, column: 31]'
Any idea how I can fix that?
$util.dynamodb.toBinary(String data) takes a String as input, but you are passing a Boolean and this is why it fails evaluation.
This is good feedback, I will check with the team if it's possible to make the utility more lenient and take a Boolean as well $util.dynamodb.toBinary(Boolean data)
Here is a possible workaround in the meantime:
#if($ctx.args.approved)
#set($approved = $util.dynamodb.toBinaryJson("true"))
#else
#set($approved = $util.dynamodb.toBinaryJson("false"))
#end
{
"version": "2017-02-28",
"operation": "Query",
"query": {
"expression": "#approved = :approved",
"expressionNames": {
"#approved": "approved",
},
"expressionValues": {
":approved": $approved
},
},
"index": "approved-index",
"limit": $util.defaultIfNull($ctx.args.first, 20),
"nextToken": $util.toJson($util.defaultIfNullOrEmpty($ctx.args.after, null)),
"scanIndexForward": true,
"select": "ALL_ATTRIBUTES",
}
I am using https://www.npmjs.com/package/swagger and I am trying to create a config. I have such a part of my swagger.js
"dummy\/{id}\/related_dummies": {
"get": {
"tags": [
"RelatedDummy"
],
"operationId": "get_by_parentRelatedDummyCollection",
"produces": [
"application\/ld+json",
"application\/hal+json",
"application\/xml",
"text\/xml",
"application\/json",
"text\/html"
],
"parameters": [
{
"name": "id",
"in": "path",
"required": true,
"type": "integer"
}
],
"summary": "Retrieves the collection of RelatedDummy resources.",
"responses": {
"200": {
"description": "RelatedDummy collection response",
"schema": {
"type": "array",
"items": {
"$ref": "#\/definitions\/RelatedDummy"
}
}
}
}
}
But when I run swagger validate swagger.js I keep getting this error:
Project Errors
--------------
#/paths: Additional properties not allowed: dummy/{id}/related_dummies
Results: 1 errors, 0 warnings
What could be the reason of the error? Thanks
The problem is that the path dummy/{id}/related_dummies doesn't begin with a slash.
To be recognized as a valid path item object, it needs to be /dummy/{id}/related_dummies.
Using the escaped syntax from your example, the name should be "\/dummy\/{id}\/related_dummies"
The relevant part of the Swagger–OpenAPI 2.0 specification is in the Paths Object definition.