Query global secondary index for items where field is undefined - amazon-dynamodb

I have this table test1 with a hash key s1 and range key s2 and the table has a global secondary index gsi1 with a hash key s3 and range key s4.
$ aws dynamodb describe-table \
--table-name test1
{
"Table": {
"AttributeDefinitions": [
{
"AttributeName": "s1",
"AttributeType": "S"
},
{
"AttributeName": "s2",
"AttributeType": "S"
},
{
"AttributeName": "s3",
"AttributeType": "S"
},
{
"AttributeName": "s4",
"AttributeType": "S"
}
],
"TableName": "test1",
"KeySchema": [
{
"AttributeName": "s1",
"KeyType": "HASH"
},
{
"AttributeName": "s2",
"KeyType": "RANGE"
}
],
"TableStatus": "ACTIVE",
"CreationDateTime": "2020-11-19T23:36:40.457000+08:00",
"ProvisionedThroughput": {
"LastIncreaseDateTime": "1970-01-01T08:00:00+08:00",
"LastDecreaseDateTime": "1970-01-01T08:00:00+08:00",
"NumberOfDecreasesToday": 0,
"ReadCapacityUnits": 3,
"WriteCapacityUnits": 3
},
"TableSizeBytes": 50,
"ItemCount": 3,
"TableArn": "arn:aws:dynamodb:ap-southeast-1:000000000000:table/test1",
"GlobalSecondaryIndexes": [
{
"IndexName": "gsi1",
"KeySchema": [
{
"AttributeName": "s3",
"KeyType": "HASH"
},
{
"AttributeName": "s4",
"KeyType": "RANGE"
}
],
"Projection": {
"ProjectionType": "ALL"
},
"IndexStatus": "ACTIVE",
"ProvisionedThroughput": {
"ReadCapacityUnits": 3,
"WriteCapacityUnits": 3
},
"IndexSizeBytes": 37,
"ItemCount": 1,
"IndexArn": "arn:aws:dynamodb:ddblocal:000000000000:table/test1/index/gsi1"
}
]
}
}
I have this item. Note that s4 is not there.
$ aws dynamodb get-item \
--table-name test1 \
--key '{"s1":{"S":"chen"}, "s2":{"S":"yan"}}'
{
"Item": {
"s3": {
"S": "fei"
},
"s1": {
"S": "chen"
},
"s2": {
"S": "yan"
}
}
}
I tried to query the index gsi1 for that item with this query and got the following error.
$ aws dynamodb query \
--table-name test1 \
--index-name gsi1 \
--expression-attribute-values '{
":s3": {"S": "fei"},
":s4": {"NULL": true}
}' \
--key-condition-expression 's3 = :s3 AND s4 = :s4'
An error occurred (ValidationException) when calling the Query operation: One or more parameter values were invalid: Condition parameter type does not match schema type
I also tried this query and got no items.
$ aws dynamodb query \
--table-name test1 \
--index-name gsi1 \
--expression-attribute-values '{
":s3": {"S": "fei"}
}' \
--key-condition-expression 's3 = :s3'
{
"Items": [],
"Count": 0,
"ScannedCount": 0,
"ConsumedCapacity": null
}
Any idea how I can query that index for that item?

You can't.
If the item doesn't have an attribute required for the GSI keys, it's not added to the GSI.
This is known as a "sparse index"

Related

Parsing json file with jq from HPE iLO

I have a json file pulled from an HPE iLO interface with the snmp configuration. It looks like:
[
{
"Comments": {
"BIOSDate": "01/23/2021",
"BIOSFamily": "U30",
"Manufacturer": "HPE",
"Model": "ProLiant DL380 Gen10",
"SerialNumber": "5UNESX378",
"iLOVersion": "iLO 5 v2.65"
}
},
{
"#HpeiLOSnmpService.v2_3_0.HpeiLOSnmpService": {
"/redfish/v1/Managers/1/SnmpService/": {
"#odata.context": "/redfish/v1/$metadata#HpeiLOSnmpService.HpeIloSnmpService",
"#odata.id": "/redfish/v1/Managers/1/SnmpService",
"Actions": {
"#HpeIloSnmpService.SendSNMPTestAlert": {
"target": "/redfish/v1/Managers/1/SnmpService/Actions/HpeILOSnmpService.SendSNMPTestAlert/"
}
},
"AlertDestinationAssociations": [
{
"SNMPAlertProtocol": "SNMPv3Trap",
"SecurityName": null
}
],
"AlertDestinations": [
"1.2.3.4",
"5.6.7.8",
null,
null
],
"AlertsEnabled": true,
"Name": "SnmpService"
},
"PeriodicHSATrapConfig": "Disabled",
"ReadCommunities": [
"",
"",
""
],
"Role": "",
"RoleDetail": "",
"SNMPAlertDestinations": {
"#odata.id": "/redfish/v1/Managers/1/SnmpService/SNMPAlertDestinations/"
},
"SNMPUsers": {
"#odata.id": "/redfish/v1/Managers/1/SnmpService/SNMPUsers/"
},
"SNMPv1Enabled": false,
"SNMPv3EngineID": "0x8920000000E3028329E002033",
"SNMPv3InformRetryAttempt": 2,
"SNMPv3InformRetryIntervalSeconds": 15,
"Status": {
"State": "Enabled"
},
"TrapCommunities": [
"",
"",
"",
"",
"",
"",
""
],
"TrapSourceHostname": "Manager",
"Users": [
{
"AuthProtocol": "MD5",
"PrivacyProtocol": "DES",
"SecurityName": "",
"UserEngineID": null
},
{
"AuthProtocol": "MD5",
"PrivacyProtocol": "DES",
"SecurityName": "",
"UserEngineID": null
},
{
"AuthProtocol": "SHA",
"PrivacyProtocol": "AES",
"SecurityName": "oneview_4849283d97929392",
"UserEngineID": null
},
{
"AuthProtocol": "MD5",
"PrivacyProtocol": "DES",
"SecurityName": "",
"UserEngineID": null
}
]
}
}
]
I want to select an element in the Users array that has SecurityName set to "" and change that element. I don't need the Comments portion. So, I try to select the section starting with #HpeiLOSnmpService.v2_3_0.HpeiLOSnmpService with:
jq -r '.[] | .#HpeiLOSnmpService.v2_3_0.HpeiLOSnmpService' snmp.json
but it gives me everything without the enclosing array. Anyone have a suggestion?
Thanks!
# starts a comment and your jq program degenerates to .[]|. which is identical to the program .[] (|. is a no-op/the identity filter). This program will simply select all values from the input.
You must quote certain characters, such as #, when they are part of propery names. The following will work with your JSON file:
jq -r '.[] | ."#HpeiLOSnmpService".v2_3_0.HpeiLOSnmpService'
Thanks. Using the quotes around the key with a '#' works. Ultimately, selecting the SecurityName that was unset was done with:
jq -r '.Users[] | select (.SecurityName == "") | {"AuthProtocol":.AuthProtocol,"PrivacyProtocol":.PrivacyProtocol,"SecurityName":.SecurityName,"UserEngineID":.UserEngineID}'

dynamodb Local, duplicate key

I'm using dynamodb Local. I'm trying to assign two addresses to one person but I'm getting a "Duplicate Key" error.
I want to add more countries and more addresses for one single person. but whenever I try to add another country and address for the person it gives a Duplicate Key error
Can someone pls help me out? Help is appreciated.
The image of my localhost code
{
"AttributeDefinitions": [
{
"AttributeName": "AddressID",
"AttributeType": "S"
},
{
"AttributeName": "CustomerID",
"AttributeType": "S"
},
{
"AttributeName": "Country",
"AttributeType": "S"
}
],
"TableName": "Addresses",
"KeySchema": [
{
"AttributeName": "AddressID",
"KeyType": "HASH"
},
{
"AttributeName": "CustomerID",
"KeyType": "RANGE"
}
],
"TableStatus": "ACTIVE",
"CreationDateTime": "2021-04-06T09:06:29.622Z",
"ProvisionedThroughput": {
"LastIncreaseDateTime": "1970-01-01T00:00:00.000Z",
"LastDecreaseDateTime": "1970-01-01T00:00:00.000Z",
"NumberOfDecreasesToday": 0,
"ReadCapacityUnits": 5,
"WriteCapacityUnits": 5
},
"TableSizeBytes": 732,
"ItemCount": 6,
"TableArn": "arn:aws:dynamodb:ddblocal:000000000000:table/Addresses",
"GlobalSecondaryIndexes": [
{
"IndexName": "UserIndex",
"KeySchema": [
{
"AttributeName": "AddressID",
"KeyType": "HASH"
},
{
"AttributeName": "Country",
"KeyType": "RANGE"
}
],
"Projection": {
"ProjectionType": "ALL"
},
"IndexStatus": "ACTIVE",
"ProvisionedThroughput": {
"ReadCapacityUnits": 5,
"WriteCapacityUnits": 5
},
"IndexSizeBytes": 732,
"ItemCount": 6,
"IndexArn": "arn:aws:dynamodb:ddblocal:000000000000:table/Addresses/index/UserIndex"
}
]
}

DynamoDB: list_append alternative for sets

I am trying to do an update operation on a dynamodb string set attribute. For lists, the operation would be
set #key = list_append(if_not_exists(#key, :empty_list), :newValue)
But this produces a list attribute. Is there an alternative for list_append but for sets?
Since DynamoDB can't store empty sets this is actually fairly easy, you can just use the ADD operator.
Here's an example I've built in Python:
import boto3
TABLE_NAME = "set-demo"
def create_table():
ddb = boto3.client("dynamodb")
ddb.create_table(
AttributeDefinitions=[
{"AttributeName": "PK", "AttributeType": "S"},
{"AttributeName": "SK", "AttributeType": "S"}
],
TableName=TABLE_NAME,
KeySchema=[
{"AttributeName": "PK", "KeyType": "HASH"},
{"AttributeName": "SK", "KeyType": "RANGE"}
],
BillingMode="PAY_PER_REQUEST"
)
def add_to_set(item_id: str, value: str):
table = boto3.resource("dynamodb").Table(TABLE_NAME)
table.update_item(
Key={
"PK": f"ITEM#{item_id}",
"SK": f"METADATA",
},
UpdateExpression="ADD #set_name :set_value",
ExpressionAttributeNames={
"#set_name": "values"
},
ExpressionAttributeValues={
":set_value": {value}, # needs to be a set type
}
)
if __name__ == "__main__":
# create_table()
add_to_set("a", "value_1")
add_to_set("a", "value_2")
add_to_set("a", "value_1")
In python it's sufficient to pass a value with the datatype set in the ExpressionAttributeValues for boto3 to know it needs to convert it into a set under the hood.
When I call add_to_set for the first time, it will create the set attribute and subsequent calls are just updates to the attribute.
This is what the item looks like in the end:
{
"PK": {
"S": "ITEM#a"
},
"SK": {
"S": "METADATA"
},
"values": {
"SS": [
"value_1",
"value_2"
]
}
}

What is the JsonPath expression for selecting an object based on sub-object values?

I need to be able to select elements within a JSON document based on the values in sub-elements which, unfortunately, reside in a list of key-value pairs (this is the structure I have to work with). I'm using Jayway 2.4.0.
Here is the JSON document:
{
"topLevelArray": [
{
"elementId": "Elem1",
"keyValuePairs": [
{
"key": "Length",
"value": "10"
},
{
"key": "Width",
"value": "3"
},
{
"key": "Producer",
"value": "alpha"
}
]
},
{
"elementId": "Elem2",
"keyValuePairs": [
{
"key": "Length",
"value": "20"
},
{
"key": "Width",
"value": "8"
},
{
"key": "Producer",
"value": "beta"
}
]
},
{
"elementId": "Elem3",
"keyValuePairs": [
{
"key": "Length",
"value": "15"
},
{
"key": "Width",
"value": "5"
},
{
"key": "Producer",
"value": "beta"
}
]
}
]
}
Here is the JsonPath I thought would do the trick:
$..topLevelArray[ ?( #.keyValuePairs[ ?(#.key=='Producer' && #.value=='beta') ] ) ]
and
$.topLevelArray[ ?( #.keyValuePairs[ ?(#.key=='Producer' && #.value=='beta') ] ) ]
Unfortunately, both are returning everything in the list, including the entry with Producer of 'alpha'. Thx in advance.

JQ Cross reference or how to replace one value with another part of the input

I want to parse terraform.tfstate (where openstack provider is used), to return instance name and it's internal + floating IP (if assigned).
First select what we are interested in:
jq -r '.modules?[]|.resources[]?|select(.type == "openstack_compute_floatingip_v2", .type == "openstack_compute_instance_v2")' < terraform.tfstate
For simplicity, pre-parsed example with the above part (one FIP and one instance):
{
"type": "openstack_compute_floatingip_v2",
"depends_on": [
"openstack_networking_router_interface_v2.management"
],
"primary": {
"id": "48b039fc-a9fa-4672-934a-32d6d267f280",
"attributes": {
"address": "209.66.89.143",
"fixed_ip": "10.10.10.5",
"id": "48b039fc-a9fa-4672-934a-32d6d267f280",
"instance_id": "597e75e8-834d-4f05-8408-e2e6e733577e",
"pool": "public",
"region": "RegionOne"
},
"meta": {},
"tainted": false
},
"deposed": [],
"provider": "provider.openstack"
}
{
"type": "openstack_compute_instance_v2",
"depends_on": [
"openstack_compute_floatingip_v2.management",
"openstack_compute_secgroup_v2.ssh_only",
"openstack_networking_network_v2.management"
],
"primary": {
"id": "597e75e8-834d-4f05-8408-e2e6e733577e",
"attributes": {
"access_ip_v4": "10.10.10.5",
"access_ip_v6": "",
"all_metadata.%": "1",
"all_metadata.habitat": "sup",
"availability_zone": "nova",
"flavor_id": "eb36e84e-17c1-42ab-b359-4380f6f524ae",
"flavor_name": "m1.large",
"force_delete": "false",
"id": "597e75e8-834d-4f05-8408-e2e6e733577e",
"image_id": "c574aeed-e47c-4fb7-9da0-75550b76ee56",
"image_name": "ubuntu-16.04",
"key_pair": "vault-etcd_test_tf",
"metadata.%": "1",
"metadata.habitat": "sup",
"name": "ctl01",
"network.#": "1",
"network.0.access_network": "false",
"network.0.fixed_ip_v4": "10.10.10.5",
"network.0.fixed_ip_v6": "",
"network.0.floating_ip": "",
"network.0.mac": "02:c6:61:f9:ee:7e",
"network.0.name": "management",
"network.0.port": "",
"network.0.uuid": "f2468669-e321-4eb4-9ede-003e362a8988",
"region": "RegionOne",
"security_groups.#": "1",
"security_groups.1845949017": "vault-etcd_test_ssh_only",
"stop_before_destroy": "false"
},
"meta": {
"e2bfb730-ecaa-11e6-8f88-34363bc7c4c0": {
"create": 1800000000000,
"delete": 1800000000000,
"update": 1800000000000
}
},
"tainted": false
},
"deposed": [],
"provider": "provider.openstack"
}
Required is to take from "type": "openstack_compute_floatingip_v2" replace .primary.attributes.address and .fixed_ip and from corresponding .instance_id the .name.
So, sth like:
{"address": "209.66.89.143",
"fixed_ip": "10.10.10.5",
"name": "ctl01"}
Well, I came with an idea while using walk, but miss how to actually assign the proper value from corresponding instance id:
jq -r "$(cat floating.jq)" terraform.tfstate
floating.jq:
def walk(f):
. as $in
| if type == "object" then
reduce keys[] as $key
( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f
elif type == "array" then map( walk(f) ) | f
else f
end;
.modules?[]|.resources[]?|select(.type ==
"openstack_compute_floatingip_v2", .type ==
"openstack_compute_instance_v2")|
.primary|walk( if type == "object" and .attributes.address then
.attributes.instance_id |= "REFERRED VALUE HERE") else . end)
Let's assume the two related objects are in a file named two.json. Then one way to merge the information from both objects is using the -s command-line option, e.g.
jq -s '
(.[0].primary.attributes | {address, fixed_ip})
+ {name: .[1].primary.attributes.name}' two.json
Output
With your example input, the output would be:
{
"address": "209.66.89.143",
"fixed_ip": "10.10.10.5",
"name": "ctl01"
}

Resources