Storing optional attributes in DynamoDB's putItem via step functions - amazon-dynamodb

I have defined a state machine in AWS step functions and one of my states is storing an item to DynamoDB
...
"Store item": {
"End": true,
"Type": "Task",
"Resource": "arn:aws:states:::dynamodb:putItem",
"Parameters": {
"Item": {
"foo": {
"S.$": "$.data.foo"
},
"bar": {
"S.$": "$.data.bar"
},
"baz": {
"S.$": "$.data.baz"
},
},
"TableName": "nrp_items"
}
},
...
The problem starts from the fact that baz property is optional, ie not exist in some cases.
On those cases, the putItem task fails:
An error occurred while executing the state 'Store item' (entered at the event id #71). > The JSONPath '$.data.baz' specified for the field 'S.$' could not be found in the input
My backup plan is to use a lambda to perform that type of operation, but can I do it directly using the putItem task in steps function?
I was wondering if:
Is possible to somehow inject via JSONPath my whole $.data item to the "Item" property, something like:
...
"Store item": {
"End": true,
"Type": "Task",
"Resource": "arn:aws:states:::dynamodb:putItem",
"Parameters": {
"Item": "$.data",
"TableName": "nrp_items"
}
},
...
OR
2) Define that the baz property is optional

TL;DR We can deal with optional variables with a "Variable": "$.baz", "IsPresent": true Choice condition to handle no-baz cases.
The Amazon States Language spec does not have optional properties: Step Functions will throw an error if $.baz does not exist in the input. We can avoid undefined paths by inserting a two-branch Choice State, one branch of which handles baz-exists cases, the other no-baz cases. Each branch continues with a Pass State that reworks the data input into dynamo-format Item syntax, using Parameters. The put-item task's "Item.$": "$.data" (as in your #1) contains only foo-bar when baz is not defined, but all three otherwise.
{
"StartAt": "HasBazChoice",
"States": {
"HasBazChoice": {
"Type": "Choice",
"Choices": [
{
"Variable": "$.baz",
"IsPresent": true,
"Next": "MakeHasBazItem"
}
],
"Default": "MakeNoBazItem"
},
"MakeHasBazItem": {
"Type": "Pass",
"Parameters": {
"data": {
"foo": { "S.$": "$.foo"},
"bar": { "S.$": "$.bar"},
"baz": { "S.$": "$.baz"}
}
},
"Next": "PutItemTask"
},
"MakeNoBazItem": {
"Type": "Pass",
"Parameters": {
"data": {
"foo": {"S.$": "$.foo"},
"bar": {"S.$": "$.bar"}
}
},
"Next": "PutItemTask"
},
"PutItemTask": {
...
"Parameters": {
"TableName": "my-table",
"Item.$": "$.data"
}
},
}
}
If you have more than one optional field, your lambda backup plan is the better option - the above workaround would become unwieldy.

Related

Put Azure Key Vault value in parameter array

I am trying to deploy a App service webapp via ARM template and need to put a secret from a key vault into an app setting (env variable).
I have always simply used an array of values from a parameters file to populate these app settings, but now I am struggling to get a keyvault value into that array. Something like shown below in an ARM parameter file.
{
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"someStringParam": {
"value": "stringLiteralValueHere"
},
"envVars": {
"value": [
{
"name": "envVarKeyName",
"value": "stringLiteralValueHere"
},
{
"name": "KVsecret1",
"value": ##KEY VAULT SECRET HERE##
}
]
}
}
}
I have tried using a reference to the keyvault for the value but that errors on deployment.
{
"name": "KVsecret1",
"reference": {
"keyVault": {
"id": "/subscriptions/<subscription_id>/resourceGroups/<resource_group>/providers/Microsoft.KeyVault/vaults/<vault_name>"
},
"secretName": "secret1"
}
}
I have also tried using a parameter inside of the parameter file, but that just used the literal string for the value.
"parameters": {
"KVsecret1": {
"reference": {
"keyVault": {
"id": "/subscriptions/<subscription_id>/resourceGroups/<resource_group>/providers/Microsoft.KeyVault/vaults/<vault_name>"
},
"secretName": "KVsecret1"
}
},
"envVars": {
"value": [
{
"name": "envVarKeyName",
"value": "stringLiteralValueHere"
},
{
"name": "KVsecret1",
"value": "[parameters('KVsecret1')]"
}
]
}
}
Is this possible??
EDIT: Adding some detail here.
I am also trying to shoe horn a reference to another resource to get put the app insights instrumentation key into an app setting. Below is what I would like to do, but the copy function needs to use the name of the property and that is dynamic in this case as it changes with the each member of the array from the parameter file.
{
"type": "Microsoft.Web/sites/config",
"apiVersion": "2022-03-01",
"name": "[concat(parameters('backEndwebAppName'),'/appsettings')]",
"kind": "string",
"properties": {
"APPINSIGHTS_INSTRUMENTATIONKEY": "[reference(concat('microsoft.insights/components/',parameters('appInsightsName')),'2020-02-02').InstrumentationKey]",
"secret1FromKeyvault": "[parameters('secret1FromKeyvault')]",
"copy": [
{
"name": "envVarsFromParams",
"count": "[length(parameters('backEndEnvVariables'))]",
"input": {
"name": "[parameters('backEndEnvVariables')[copyIndex('envVarsFromParams').name]]",
"value": "[parameters('backEndEnvVariables')[copyIndex('envVarsFromParams').value]]"
}
}
]
},
"dependsOn": [
"[resourceId('Microsoft.Web/sites', parameters('backEndwebAppName'))]"
]
},
This isn't possible today within the param file, but in your scenario (if it's as simple as your OP example) you can just union the two in your template. So in your parameter file, you have 2 params kvSecret (the reference) and envVars (all your other env vars) and then in the template use:
"variables": {
"keySecretObj": {
"name": "kvSecret",
"value": "[parameters('kvSecret')]"
},
"envVarsFinal": "[union(parameters(variables('kvSecretObj`), parameters(`envVars`))]"
That help?

The language expression property '0' can't be evaluated, property name must be a string - ARM Template error while adding Key Vault access policy

I've been working on an issue and seem to be stuck, so asking on so in case anyone can help.
To describe the issue, I've got an existing Azure Key Vault setup, and wish to add a number of access policies to this resource group. It needs to be conditional as if the function name is "false" then that function should not be added to key vault access policy.
variable section:
"variables": {
"functionAccess": {
"value": [
{
"name": "[parameters('Function_1')]"
},
{
"name": "[parameters('Function_2')]"
},
{
"name": "[parameters('Function_3')]"
}
]
}
}
My Template :
{
"apiVersion": "2016-10-01",
"condition": "[not(equals(variables('functionAccess')[CopyIndex()].name, 'false'))]",
"copy": {
"batchSize": 1,
"count": "[length(variables('functionAccess'))]",
"mode": "Serial",
"name": "accessPolicies"
},
"name": "[concat(parameters('KeyVault_Name'), '/add')]",
"properties": {
"accessPolicies": [
{
"tenantId": "[subscription().tenantId]",
"objectId": "[if(not(equals(variables('functionAccess')[CopyIndex()].name, 'false')), reference(concat('Microsoft.Web/sites/', variables('functionAccess')[CopyIndex()].name), '2016-08-01', 'Full').identity.principalId, json('null'))]",
"permissions": {
"keys": [
"get",
"list"
],
"secrets": [
"get",
"list"
],
"certificates": [
"get",
"list"
]
}
}
]
},
"type": "Microsoft.KeyVault/vaults/accessPolicies"
}
When I deploy my ARM template for the azure key vault I got this error message:
The language expression property '0' can't be evaluated, property name must be a string.
also tried below, but same error:
{
"apiVersion": "2018-02-14",
"name": "[concat(parameters('KeyVault_Name'), '/add')]",
"properties": {
"copy": [
{
"batchSize": 1,
"count": "[length(variables('functionAccess'))]",
"mode": "serial",
"name": "accessPolicies",
"input": {
"condition": "[not(equals(variables('functionAccess')[copyIndex('accessPolicies')].name, 'false'))]",
"tenantId": "[subscription().tenantId]",
"objectId": "[if(not(equals(variables('functionAccess')[copyIndex('accessPolicies')].name, 'false')), reference(concat('Microsoft.Web/sites/', variables('functionAccess')[copyIndex('accessPolicies')].name), '2016-08-01', 'Full').identity.principalId, json('null'))]",
"permissions": {
"keys": [
"get",
"list"
],
"secrets": [
"get",
"list"
],
"certificates": [
"get",
"list"
]
}
}
}
]
},
"type": "Microsoft.KeyVault/vaults/accessPolicies"
}
There are a few options for dealing with filtering an array for copy operation. I deploy my ARM templates from PowerShell scripts and use PowerShell to setup parameter values. When I need special logic handle different inputs for different environments, I let PowerShell handle it.
If you must handle the filtering in ARM and you have the option to input a CSV list of functions, then perhaps the following will work. You can then use the functionAccessArray to iterate over in the copy operation.
{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
},
"variables": {
"functionAccessCsv": "Function-0,Function-1,false,Function-4,false,Function-6,Function-7",
"functionAccessFiltered": "[replace(replace(variables('functionAccessCsv'), 'false', ''), ',,', ',')]",
"functionAccessArray": "[split(variables('functionAccessFiltered'), ',')]"
},
"resources": [
],
"outputs": {
"functionAccessCsvFiltered": {
"type": "string",
"value": "[variables('functionAccessFiltered')]"
},
"functionAccessArray": {
"type": "array",
"value": "[variables('functionAccessArray')]"
}
}
}
The result:
I just had the same issue. By using an array parameter with a default value instead of a variable, I got it to work.
"parameters": {
"functionAccess": {
"type": "array",
"defaultValue": [
"value1",
"value2",
"value3"
]
}
}

Logic App workflow creating or updating documents in CosmosDB fails with conflict

I'm in the process of creating a Logic App which should copy all documents from specific containers in the database to a different database and its corresponding collections.
For this, I've created a workflow which retrieves all documents, loops through them and tries to do a Create or Update document on the target location.
As you can see on the image, this looks fairly straightforward.
I've specified the PartitionKey in the header and Upsert is set to true, so if a document already exists it'll be updated, otherwise created.
Originally, the body was filled with #items('[blurred]_-_For_each_document').
However, I got an error stating there was a conflict on the id.
I've also tried to remove all 'system' properties like so:
#removeProperty(removeProperty(removeProperty(removeProperty(removeProperty(items('[blurred]_-_For_each_document'), 'id'), '_rid'), '_self'), '_ts'), '_etag')
but it appears not having an id in place isn't valid, so now I've got this as I'm only interested in having the contents of the 'actual' document:
#removeProperty(removeProperty(removeProperty(removeProperty(items('[blurred]_-_For_each_document'), '_rid'), '_self'), '_ts'), '_etag')
This still fails.
The raw input looks a bit like this:
{
"method": "post",
"headers": {
"x-ms-documentdb-raw-partitionkey": "1050",
"x-ms-documentdb-is-upsert": "True"
},
"path": "/dbs/[myDatabase]/colls/[myCollection]/docs",
"host": {
"connection": {
"name": "/subscriptions/[subscriptionGuid]/resourceGroups/[resourcegroup]/providers/Microsoft.Web/connections/[connectionName]"
}
},
"body": {
"id": "2faee185-4a51-4797-bff9-3ce23a603690",
"MyPartitionKeyNumber": 1050,
"SomeValue": false,
}
}
With the following response:
{
"code": "Conflict",
"message": "Entity with the specified id already exists in the system., ... ResourceType: Document, OperationType: Upsert ..."
}
I know for a fact, the id doesn't exist, but changed the step to do a PUT anyway.
{
"method": "put",
"headers": {
"x-ms-documentdb-raw-partitionkey": "1050",
"x-ms-documentdb-is-upsert": "True"
},
"path": "/dbs/[myDatabase]/colls/[myCollection]/docs",
"host": {
"connection": {
"name": "/subscriptions/[subscriptionGuid]/resourceGroups/[resourcegroup]/providers/Microsoft.Web/connections/[connectionName]"
}
},
"body": {
"id": "2faee185-4a51-4797-bff9-3ce23a603690",
"MyPartitionKeyNumber": 1050,
"SomeValue": false,
}
}
With a response I'm expecting to see:
{
"statusCode": 404,
"message": "Resource not found"
}
I've also tried to do a Delete document before running the Create or Update document step, but got the same error(s).
There's some post here on Stack Overflow stating the x-ms-documentdb-raw-partitionkey should be x-ms-documentdb-partitionkey (without the raw-part), but doing so results in the following error message:
PartitionKey extracted from document doesn't match the one specified in the header
For completeness sake, these are the relevant steps of my workflow:
"[blurred]_-_Get_all_documents": {
"type": "ApiConnection",
"inputs": {
"host": {
"connection": {
"name": "#parameters('$connections')['documentdb_3']['connectionId']"
}
},
"method": "get",
"path": "/v2/dbs/#{encodeURIComponent('myDatabase')}/colls/#{encodeURIComponent(items('[blurred]_-_For_each_collection'))}/docs"
},
"runAfter": {}
},
"[blurred]_-_For_each_document": {
"type": "Foreach",
"foreach": "#body('[blurred]_-_Get_all_documents')?['value']",
"actions": {
"[blurred]_-_Create_or_Update_document_on_blurred2": {
"type": "ApiConnection",
"inputs": {
"host": {
"connection": {
"name": "#parameters('$connections')['documentdb_4']['connectionId']"
}
},
"method": "post",
"body": "#removeProperty(removeProperty(removeProperty(removeProperty(items('[blurred]_-_For_each_document'), '_rid'), '_self'), '_ts'), '_etag')",
"headers": {
"x-ms-documentdb-raw-partitionkey": "#{items('[blurred]_-_For_each_document')['MyPartitionKeyNumber']}",
"x-ms-documentdb-is-upsert": true
},
"path": "/dbs/#{encodeURIComponent('myDatabase')}/colls/#{encodeURIComponent(items('[blurred]_-_For_each_collection'))}/docs"
},
"runAfter": {}
}
},
"runAfter": {
"[blurred]_-_Get_all_documents": [
"Succeeded"
]
}
}

How to insert empty string in DynamoDB using the output of a Lambda in Step Functions?

I'm trying to save the output of a Lambda which calls Lex to DynamoDB using Step Functions.
The intentName in a Lex response is sometimes null (unknown). The problem is that in the state (task) that saves the response to DynamoDB, because of this empty string I get an error from DynamoDB.
Is there any workaround, maybe using JsonPath or the state machine diagram of the Step Function, in order to insert null or maybe no insert that specific property in DynamoDB?
Here is the JSON for the state machine:
{
"StartAt": "ProcessLex",
"States": {
"ProcessLex": {
"Type": "Task",
"Resource": "arn:aws:lambda:<Region>:<Account Id>:function:getIntent",
"ResultPath": "$.lexResult",
"Next": "ChoiceIfIntent"
},
"SaveToDynamo": {
"Type": "Task",
"Resource": "arn:aws:states:::dynamodb:putItem",
"Parameters": {
"TableName": "MyTable",
"Item": {
"dateTime": {
"S.$": "$.dateTime"
},
"intentName": {
"S.$": "$.lexResult.intentName"
},
"analysis": {
"M.$": "$.lexResult.sentimentResponse"
}
}
},
"End": true
},
"Comprehend": {
"Comment": "To be implemented later",
"Type": "Pass",
"End": true
},
"ChoiceIfIntent": {
"Type": "Choice",
"Choices": [
{
"Variable": "$.lexResult.intentName",
"StringGreaterThanEquals": "",
"Next": "SaveToDynamo"
}
],
"Default": "Comprehend"
}
}
}
The problem is not the null value, the problem is that in DynamoDB with the PutItem Api you cannot insert empty strings.
I know this is frustrating but the quickest solution is to replace "" with NULL.
The solution that I prefer is to set the convertEmptyValue to true in your DynamoDb client settings.
const dynamodb = new AWS.DynamoDB.DocumentClient({ convertEmptyValues: true })
UPDATE
Since yesterday, DynamoDB supports empty values for string!
Take a look here.

Send a list of objects whose information is obtained by asynchronous functions

I hope I can express myself a way you can all understand.
Here's the thing, I'm developing a back-end for an app in nodeJS. There's a table named 'chat' in dynamoDB with the ID of the chat's creator and a list of messages IDs, also there's a table named 'messages' with the author of each message and the text.
When the front-end asks the server for all the chats it has to send a list of chats objects with this sctructure:
[
{
"ID_CHAT": {
"S": "asd123asd"
},
"author": {
"name": "Name",
"lastname": "Lastname"
},
"date": {
"S": "19/11/2017 16:00"
},
"USER_DESTINATION": {
"S": "222asd222"
},
"messages": [
{
"Item": {
"author": {
"name": "Name",
"lastname": "Lastname"
},
"text": {
"S": "Hi!"
},
"MESSAGE_ID": {
"S": "456asd456"
},
"date": {
"S": "19/11/2017 16:05"
}
}
},
{
"Item": {
"author": {
"name": "Name",
"lastname": "Lastname"
},
"text": {
"S": "zup?"
},
"MESSAGE_ID": {
"S": "789asd789"
},
"date": {
"S": "19/11/2017 16:06"
}
}
}
]
}
{
"ID_CHAT": {
"S": "123asd123asd"
},
"author": {
"S": "123asd123"
}, and so on...
]
But in order to send this I need to get the chat's authors data from dynamodb (asynchronous function) and the list of messages data again from dynamodb.
I tried combining map() to get the chats authors and Promise.all() for the messages but it won't work cause the second iteration of the map() function runs after the Promise.then().
I'll appretiate your help!

Resources