Exclude Path in Azure Cosmos DB - azure-cosmosdb

What is the correct JSON to exclude certain keys from an input json to be not indexed by Azure CosmosDB. We are using the CosmosDB in mongodb mode. Was planning to change the index configuration on the Azure Portal after creating the collection.
Sample Input Json being
{
"name": "test",
"age": 1,
"location": "l1",
"height":5.7
}
If I were to include name and age in the index and remove location and height from the index, what does the includedPaths and excludedPaths look like.

Finally got it to work with the below spec:-
{
"indexingMode": "consistent",
"automatic": true,
"includedPaths": [{
"path": "/*",
"indexes": [{
"kind": "Range",
"dataType": "Number",
"precision": -1
},
{
"kind": "Hash",
"dataType": "String",
"precision": 3
}
]
}],
"excludedPaths": [{
"path": "/\"location\"/?"
},
{
"path": "/\"height\"/?"
}
]
}

It looks like underlying implementation has changed and at the time of writing the documentation does not cover changing indexPolicy in MongoDB flavoured CosmosBD. Because the documents are really stored in a wired way where all the keys start from root $v and all scalar fields are stored as documents, containing value and type information. So your document will be stored something like:
{
'_etag': '"2f00T0da-0000-0d00-0000-4cd987940000"',
'id': 'SDSDFASDFASFAFASDFASDF',
'_self': 'dbs/sMsxAA==/colls/dVsxAI8MBXI=/docs/sMsxAI8MBXIKAAAAAAAAAA==/',
'_rid': 'sMsxAI8MBXIKAAAAAAAAAA==',
'$t': 3,
'_attachments': 'attachments/',
'$v': {
'_id': {
'$t': 7,
'$v': "\\ร™\x87\x14\x01\x15['\x18m\x01รบ"
},
'name': {
'$t': 2,
'$v': 'test'
},
'age': {
'$t': 1,
'$v': 1
},
...
},
'_ts': 1557759892
}
Therefore the indexingPolicy paths need to include the root $v and use /* (objects) instead of /? (scalars).
{
"indexingMode": "consistent",
"automatic": true,
"includedPaths": [
{
"path": "/*",
"indexes": [
{
"kind": "Range",
"dataType": "Number"
},
{
"kind": "Hash",
"dataType": "String"
}
]
}
],
"excludedPaths": [
{"path": "/\"$v\"/\"location\"/*"},
{"path": "/\"$v\"/\"height\"/*"}
]
}
PS:
Also mongo api can be used to drop all the default indexes and create specific indexes as required
https://learn.microsoft.com/en-us/azure/cosmos-db/mongodb-indexing

Related

Not able to filter out required property in Azure TSI Gen1 Get Events API response

I am using the below request body to fetch only the required property values.
"searchSpan": {
"from": {
"dateTime": "2021-11-20T00:00:00.000Z"
},
"to": {
"dateTime": "2021-11-20T23:00:00.000Z"
}
},
"predicateString": "[Params.Name] = 'power'",
"take": 100
}
}
The URL is like below:
https://12345678a-bcde-3e91-blah-2292933292aa.env.timeseries.azure.com/events?api-version=2016-12-12
Despite specifying the required property the response returns all properties as if it has not seen the predicate string. What might I be doing wrong?
{
"warnings": [],
"events": [
{
"schema": {
"rid": 0,
"$esn": "my-event-hub",
"properties": [
{
"name": "mytimestamp",
"type": "DateTime"
},
{
"name": "Params.Name",
"type": "String"
},
{
"name": "Params.Value",
"type": "Double"
}
]
},
"$ts": "2021-11-20T10:01:50Z",
"values": [
"2021-11-20T10:01:50Z",
"energy",
60
]
},
{
"schemaRid": 0,
"$ts": "2021-11-20T10:01:50Z",
"values": [
"2021-11-20T10:01:50Z",
"power",
10
]
},
{
"schemaRid": 0,
"$ts": "2021-11-20T10:01:50Z",
"values": [
"2021-11-20T10:01:50Z",
"strength",
200
]
},
]
}
Edit
I'm getting "Properties count error" in the TSI overview page. This might quite be the root cause but I don't know for sure
"For Time Series Insights environment ABC: You have used all 641/600 properties in your environment".

The language expression property '0' can't be evaluated, property name must be a string - ARM Template error while adding Key Vault access policy

I've been working on an issue and seem to be stuck, so asking on so in case anyone can help.
To describe the issue, I've got an existing Azure Key Vault setup, and wish to add a number of access policies to this resource group. It needs to be conditional as if the function name is "false" then that function should not be added to key vault access policy.
variable section:
"variables": {
"functionAccess": {
"value": [
{
"name": "[parameters('Function_1')]"
},
{
"name": "[parameters('Function_2')]"
},
{
"name": "[parameters('Function_3')]"
}
]
}
}
My Template :
{
"apiVersion": "2016-10-01",
"condition": "[not(equals(variables('functionAccess')[CopyIndex()].name, 'false'))]",
"copy": {
"batchSize": 1,
"count": "[length(variables('functionAccess'))]",
"mode": "Serial",
"name": "accessPolicies"
},
"name": "[concat(parameters('KeyVault_Name'), '/add')]",
"properties": {
"accessPolicies": [
{
"tenantId": "[subscription().tenantId]",
"objectId": "[if(not(equals(variables('functionAccess')[CopyIndex()].name, 'false')), reference(concat('Microsoft.Web/sites/', variables('functionAccess')[CopyIndex()].name), '2016-08-01', 'Full').identity.principalId, json('null'))]",
"permissions": {
"keys": [
"get",
"list"
],
"secrets": [
"get",
"list"
],
"certificates": [
"get",
"list"
]
}
}
]
},
"type": "Microsoft.KeyVault/vaults/accessPolicies"
}
When I deploy my ARM template for the azure key vault I got this error message:
The language expression property '0' can't be evaluated, property name must be a string.
also tried below, but same error:
{
"apiVersion": "2018-02-14",
"name": "[concat(parameters('KeyVault_Name'), '/add')]",
"properties": {
"copy": [
{
"batchSize": 1,
"count": "[length(variables('functionAccess'))]",
"mode": "serial",
"name": "accessPolicies",
"input": {
"condition": "[not(equals(variables('functionAccess')[copyIndex('accessPolicies')].name, 'false'))]",
"tenantId": "[subscription().tenantId]",
"objectId": "[if(not(equals(variables('functionAccess')[copyIndex('accessPolicies')].name, 'false')), reference(concat('Microsoft.Web/sites/', variables('functionAccess')[copyIndex('accessPolicies')].name), '2016-08-01', 'Full').identity.principalId, json('null'))]",
"permissions": {
"keys": [
"get",
"list"
],
"secrets": [
"get",
"list"
],
"certificates": [
"get",
"list"
]
}
}
}
]
},
"type": "Microsoft.KeyVault/vaults/accessPolicies"
}
There are a few options for dealing with filtering an array for copy operation. I deploy my ARM templates from PowerShell scripts and use PowerShell to setup parameter values. When I need special logic handle different inputs for different environments, I let PowerShell handle it.
If you must handle the filtering in ARM and you have the option to input a CSV list of functions, then perhaps the following will work. You can then use the functionAccessArray to iterate over in the copy operation.
{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
},
"variables": {
"functionAccessCsv": "Function-0,Function-1,false,Function-4,false,Function-6,Function-7",
"functionAccessFiltered": "[replace(replace(variables('functionAccessCsv'), 'false', ''), ',,', ',')]",
"functionAccessArray": "[split(variables('functionAccessFiltered'), ',')]"
},
"resources": [
],
"outputs": {
"functionAccessCsvFiltered": {
"type": "string",
"value": "[variables('functionAccessFiltered')]"
},
"functionAccessArray": {
"type": "array",
"value": "[variables('functionAccessArray')]"
}
}
}
The result:
I just had the same issue. By using an array parameter with a default value instead of a variable, I got it to work.
"parameters": {
"functionAccess": {
"type": "array",
"defaultValue": [
"value1",
"value2",
"value3"
]
}
}

Azure ARM templates - empty values as a parameters, IF function

I am preparing ARM template for "Schedule update deployment" in Update Management service. I want to add parameters like: "excludedKbNumbers" and "includedKbNumbers". I am deploying my templates using powershell. When I am passing KB numbers using mentioned parameters templates completed successfully. In case when I am putting KB number using one of the mentioned parameters, second is empty, template completed successfully. Problem is when I dont want to pass Included/Exluded KB numbers, in my powershell deployment command I am not putting parameter names "excludedKbNumbers" and "includedKbNumbers", and then I am receiving below error: "message": "{\"Message\":\"The request is invalid.\",\"ModelState\":{\"softwareUpdateConfiguration.properties.updateConfiguration\":[\"Software update configuration has same KbNumbers in
includedKbNumbers and excludedKbNumbers.\"]}}"
I am using this structure in my template json('null') and this is a problematic area.
extract from my template:
"parameters": {
"excludedKbNumbers": {
"type": "string",
"defaultValue": "",
"metadata": {
"description": "Specify excluded KB numbers, required data structure: 123456"
}
},
"includedKbNumbers": {
"type": "string",
"defaultValue": "",
"metadata": {
"description": "Specify included KB numbers, required data structure: 123456"
}
},
"resources": [
{
"type": "Microsoft.Automation/automationAccounts/softwareUpdateConfigurations",
"apiVersion": "2017-05-15-preview",
"name": "[concat(parameters('automationAccountName'), '/', parameters('scheduleName'))]",
"properties": {
"updateConfiguration": {
"operatingSystem": "[parameters('operatingSystem')]",
"windows": {
"includedUpdateClassifications": "[parameters('Classification')]",
"excludedKbNumbers": [
"[if(empty(parameters('excludedKbNumbers')), json('null'), parameters('excludedKbNumbers'))]"
],
"includedKbNumbers": [
"[if(empty(parameters('includedKbNumbers')), json('null'), parameters('includedKbNumbers'))]"
],
"rebootSetting": "IfRequired"
},
"targets": {
"azureQueries": [
{
"scope": [
"[concat('/subscriptions', '/', parameters('subscriptionID'))]"
],
"tagSettings": {
"tags": {
"[parameters('tagKey')]": [
"[parameters('tagValue')]"
]
},
"filterOperator": "All"
},
"locations": []
}
]
},
"duration": "PT2H"
},
"tasks": {},
"scheduleInfo": {
"isEnabled": false,
"startTime": "2050-03-03T13:10:00+01:00",
"expiryTime": "2050-03-03T13:10:00+01:00",
"frequency": "OneTime",
"timeZone": "Europe/Warsaw"
}
}
}
],
try doing this:
"excludedKbNumbers": "[if(empty(parameters('excludedKbNumbers')), json('null'), array(parameters('excludedKbNumbers')))]",
"includedKbNumbers": "[if(empty(parameters('includedKbNumbers')), json('null'), array(parameters('includedKbNumbers')))]"

Updating CosmosDb indexing policy through ARM templates

I am trying to use ARM templates to update the indexing policy for cosmos container. I tried 2 methods, one to simply declare the indexing policy in while declaring the container in ARM.
{
"apiVersion": "[variables('cosmosDbApiVersion')]",
"type": "Microsoft.DocumentDB/databaseAccounts/apis/databases/containers",
"dependsOn": [ /* resourceId */ ],
"name": "/* containerName */",
"properties": {
"resource": {
"id": "/* id */",
"partitionKey": {
"paths": [
"/partitionKey"
],
"kind": "Hash"
},
"indexes": [
{
"indexingMode": "consistent",
"automatic": true,
"includedPaths": [
{
"path": "/*",
"indexes": [
{
"kind": "Range",
"dataType": "Number",
"precision": -1
},
{
"kind": "Hash",
"dataType": "String",
"precision": 3
}
]
}
]
}
],
"defaultTtl": "[variables('defaultTtlValueToEnableTtl')]"
}
}
},
The second was to use to use ARM to deploy container setting as such:
{
"apiVersion": "[variables('cosmosDbApiVersion')]",
"type": "Microsoft.DocumentDB/databaseAccounts/apis/databases/containers/settings",
"name": "[/* name */",
"dependsOn": [ " /* container name */" ],
"properties": {
"resource": {
"throughput": "/* some throughput */",
"indexes": [
{
"indexingMode": "consistent",
"automatic": true,
"includedPaths": [
{
"path": "/*",
"indexes": [
{
"kind": "Range",
"dataType": "Number",
"precision": -1
},
{
"kind": "Hash",
"dataType": "String",
"precision": 3
}
]
}
]
}
]
}
}
},
Both techniques do not fail deployment but the indexing policy does not change.
Would appreciate some help.
this is the example from the template reference (looks slightly different to what you are doing):
"resource": {
"id": "string",
"indexingPolicy": {
"automatic": "boolean",
"indexingMode": "string",
"includedPaths": [
{
"path": "string",
"indexes": [
{
"dataType": "string",
"precision": "integer",
"kind": "string"
}
]
}
],
"excludedPaths": [
{
"path": "string"
}
],
"spatialIndexes": [
{
"path": "string",
"types": [
"string"
]
}
]
},
xxx
}
https://learn.microsoft.com/en-us/azure/templates/microsoft.documentdb/2019-08-01/databaseaccounts/sqldatabases/containers
Range and hash index types are ignored by the Cosmos resource provider now for new containers or containers that were created within the past year or so. ARM does not validate the index policy which is why the template will deploy successfully.
Hash index was deprecated for these newer container because the performance of the range index in the new indexer is surpasses what hash index provided so was no longer necessary.
To create/modify index policy refer to this article below. There are multiple examples of index policies that implement everything from very simple to more complex policies that include composite indexes, spatial indexes and unique keys.
https://learn.microsoft.com/en-us/azure/cosmos-db/manage-sql-with-resource-manager#create-resource

GraphDb Indexing Policies

Consider the following json responses..
If you run the graph query g.V().hasLabel('customer'), the response is:
[
{
"id": "75b9bddc-4008-43d7-a24c-8b138735a36a",
"label": "customer",
"type": "vertex",
"properties": {
"partitionKey": [
{
"id": "75b9bddc-4008-43d7-a24c-8b138735a36a|partitionKey",
"value": 1
}
]
}
}
]
If you run the sql query select * from c where c.label = 'customer', the response is:
[
{
"label": "customer",
"partitionKey": 1,
"id": "75b9bddc-4008-43d7-a24c-8b138735a36a",
"_rid": "0osWAOso6VYBAAAAAAAAAA==",
"_self": "dbs/0osWAA==/colls/0osWAOso6VY=/docs/0osWAOso6VYBAAAAAAAAAA==/",
"_etag": "\"2400985f-0000-0c00-0000-5e2066190000\"",
"_attachments": "attachments/",
"_ts": 1579181593
}
]
Q: With this difference in structure around the partitionKey section, should this be referenced as /properties/partitionKey/*, or /partitionKey/? in the indexing policy?
Currently i have hedged by bets with...
{
"indexingMode": "consistent",
"automatic": true,
"includedPaths": [{
"path": "/properties/partitionKey/*"
},{
"path": "/partitionKey/?"
},{
"path": "/label/?"
}
],
"excludedPaths": [{
"path": "/*"
},{
"path": "/\"_etag\"/?"
}
]
}
TIA! ๐Ÿ˜
This should be stored as "/partitionKey" in your index policy, not "/properties/partitionKey"
btw, another thing to point out here is it's generally better to only exclude paths you will never query on rather than have to include those you will. This way, if you add properties to your graph you won't have to rebuild the index to query on the new properties.

Resources