I am trying to use ARM templates to update the indexing policy for cosmos container. I tried 2 methods, one to simply declare the indexing policy in while declaring the container in ARM.
{
"apiVersion": "[variables('cosmosDbApiVersion')]",
"type": "Microsoft.DocumentDB/databaseAccounts/apis/databases/containers",
"dependsOn": [ /* resourceId */ ],
"name": "/* containerName */",
"properties": {
"resource": {
"id": "/* id */",
"partitionKey": {
"paths": [
"/partitionKey"
],
"kind": "Hash"
},
"indexes": [
{
"indexingMode": "consistent",
"automatic": true,
"includedPaths": [
{
"path": "/*",
"indexes": [
{
"kind": "Range",
"dataType": "Number",
"precision": -1
},
{
"kind": "Hash",
"dataType": "String",
"precision": 3
}
]
}
]
}
],
"defaultTtl": "[variables('defaultTtlValueToEnableTtl')]"
}
}
},
The second was to use to use ARM to deploy container setting as such:
{
"apiVersion": "[variables('cosmosDbApiVersion')]",
"type": "Microsoft.DocumentDB/databaseAccounts/apis/databases/containers/settings",
"name": "[/* name */",
"dependsOn": [ " /* container name */" ],
"properties": {
"resource": {
"throughput": "/* some throughput */",
"indexes": [
{
"indexingMode": "consistent",
"automatic": true,
"includedPaths": [
{
"path": "/*",
"indexes": [
{
"kind": "Range",
"dataType": "Number",
"precision": -1
},
{
"kind": "Hash",
"dataType": "String",
"precision": 3
}
]
}
]
}
]
}
}
},
Both techniques do not fail deployment but the indexing policy does not change.
Would appreciate some help.
this is the example from the template reference (looks slightly different to what you are doing):
"resource": {
"id": "string",
"indexingPolicy": {
"automatic": "boolean",
"indexingMode": "string",
"includedPaths": [
{
"path": "string",
"indexes": [
{
"dataType": "string",
"precision": "integer",
"kind": "string"
}
]
}
],
"excludedPaths": [
{
"path": "string"
}
],
"spatialIndexes": [
{
"path": "string",
"types": [
"string"
]
}
]
},
xxx
}
https://learn.microsoft.com/en-us/azure/templates/microsoft.documentdb/2019-08-01/databaseaccounts/sqldatabases/containers
Range and hash index types are ignored by the Cosmos resource provider now for new containers or containers that were created within the past year or so. ARM does not validate the index policy which is why the template will deploy successfully.
Hash index was deprecated for these newer container because the performance of the range index in the new indexer is surpasses what hash index provided so was no longer necessary.
To create/modify index policy refer to this article below. There are multiple examples of index policies that implement everything from very simple to more complex policies that include composite indexes, spatial indexes and unique keys.
https://learn.microsoft.com/en-us/azure/cosmos-db/manage-sql-with-resource-manager#create-resource
Related
I've been working on an issue and seem to be stuck, so asking on so in case anyone can help.
To describe the issue, I've got an existing Azure Key Vault setup, and wish to add a number of access policies to this resource group. It needs to be conditional as if the function name is "false" then that function should not be added to key vault access policy.
variable section:
"variables": {
"functionAccess": {
"value": [
{
"name": "[parameters('Function_1')]"
},
{
"name": "[parameters('Function_2')]"
},
{
"name": "[parameters('Function_3')]"
}
]
}
}
My Template :
{
"apiVersion": "2016-10-01",
"condition": "[not(equals(variables('functionAccess')[CopyIndex()].name, 'false'))]",
"copy": {
"batchSize": 1,
"count": "[length(variables('functionAccess'))]",
"mode": "Serial",
"name": "accessPolicies"
},
"name": "[concat(parameters('KeyVault_Name'), '/add')]",
"properties": {
"accessPolicies": [
{
"tenantId": "[subscription().tenantId]",
"objectId": "[if(not(equals(variables('functionAccess')[CopyIndex()].name, 'false')), reference(concat('Microsoft.Web/sites/', variables('functionAccess')[CopyIndex()].name), '2016-08-01', 'Full').identity.principalId, json('null'))]",
"permissions": {
"keys": [
"get",
"list"
],
"secrets": [
"get",
"list"
],
"certificates": [
"get",
"list"
]
}
}
]
},
"type": "Microsoft.KeyVault/vaults/accessPolicies"
}
When I deploy my ARM template for the azure key vault I got this error message:
The language expression property '0' can't be evaluated, property name must be a string.
also tried below, but same error:
{
"apiVersion": "2018-02-14",
"name": "[concat(parameters('KeyVault_Name'), '/add')]",
"properties": {
"copy": [
{
"batchSize": 1,
"count": "[length(variables('functionAccess'))]",
"mode": "serial",
"name": "accessPolicies",
"input": {
"condition": "[not(equals(variables('functionAccess')[copyIndex('accessPolicies')].name, 'false'))]",
"tenantId": "[subscription().tenantId]",
"objectId": "[if(not(equals(variables('functionAccess')[copyIndex('accessPolicies')].name, 'false')), reference(concat('Microsoft.Web/sites/', variables('functionAccess')[copyIndex('accessPolicies')].name), '2016-08-01', 'Full').identity.principalId, json('null'))]",
"permissions": {
"keys": [
"get",
"list"
],
"secrets": [
"get",
"list"
],
"certificates": [
"get",
"list"
]
}
}
}
]
},
"type": "Microsoft.KeyVault/vaults/accessPolicies"
}
There are a few options for dealing with filtering an array for copy operation. I deploy my ARM templates from PowerShell scripts and use PowerShell to setup parameter values. When I need special logic handle different inputs for different environments, I let PowerShell handle it.
If you must handle the filtering in ARM and you have the option to input a CSV list of functions, then perhaps the following will work. You can then use the functionAccessArray to iterate over in the copy operation.
{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
},
"variables": {
"functionAccessCsv": "Function-0,Function-1,false,Function-4,false,Function-6,Function-7",
"functionAccessFiltered": "[replace(replace(variables('functionAccessCsv'), 'false', ''), ',,', ',')]",
"functionAccessArray": "[split(variables('functionAccessFiltered'), ',')]"
},
"resources": [
],
"outputs": {
"functionAccessCsvFiltered": {
"type": "string",
"value": "[variables('functionAccessFiltered')]"
},
"functionAccessArray": {
"type": "array",
"value": "[variables('functionAccessArray')]"
}
}
}
The result:
I just had the same issue. By using an array parameter with a default value instead of a variable, I got it to work.
"parameters": {
"functionAccess": {
"type": "array",
"defaultValue": [
"value1",
"value2",
"value3"
]
}
}
There are multiple ways how to create composite index. But I see no ARM templates there and latest scheme for containers also has no it. When we can use ARM templates to define composite indexes?
Can you try something like the following:
{
"name": "string",
"type": "Microsoft.DocumentDB/databaseAccounts/sqlDatabases/containers",
"apiVersion": "2020-03-01",
"location": "string",
"tags": {},
"properties": {
"resource": {
"id": "string",
"indexingPolicy": {
"automatic": "boolean",
"indexingMode": "string",
"includedPaths": [
{
"path": "string",
"indexes": [
{
"dataType": "string",
"precision": "integer",
"kind": "string"
}
]
}
],
"excludedPaths": [
{
"path": "string"
}
],
"spatialIndexes": [
{
"path": "string",
"types": [
"string"
]
}
],
"compositeIndexes":[
[
{
"path":"/name",
"order":"ascending"
},
{
"path":"/age",
"order":"descending"
}
]
]
},
"partitionKey": {
"paths": [
"string"
],
"kind": "string",
"version": "integer"
},
"defaultTtl": "integer",
"uniqueKeyPolicy": {
"uniqueKeys": [
{
"paths": [
"string"
]
}
]
},
"conflictResolutionPolicy": {
"mode": "string",
"conflictResolutionPath": "string",
"conflictResolutionProcedure": "string"
}
},
"options": {
"additionalProperties": {},
"throughput": "string"
}
},
"resources": []
}
I simply took the template definition from here and added definition for composite index from here under indexing policy section.
Consider the following json responses..
If you run the graph query g.V().hasLabel('customer'), the response is:
[
{
"id": "75b9bddc-4008-43d7-a24c-8b138735a36a",
"label": "customer",
"type": "vertex",
"properties": {
"partitionKey": [
{
"id": "75b9bddc-4008-43d7-a24c-8b138735a36a|partitionKey",
"value": 1
}
]
}
}
]
If you run the sql query select * from c where c.label = 'customer', the response is:
[
{
"label": "customer",
"partitionKey": 1,
"id": "75b9bddc-4008-43d7-a24c-8b138735a36a",
"_rid": "0osWAOso6VYBAAAAAAAAAA==",
"_self": "dbs/0osWAA==/colls/0osWAOso6VY=/docs/0osWAOso6VYBAAAAAAAAAA==/",
"_etag": "\"2400985f-0000-0c00-0000-5e2066190000\"",
"_attachments": "attachments/",
"_ts": 1579181593
}
]
Q: With this difference in structure around the partitionKey section, should this be referenced as /properties/partitionKey/*, or /partitionKey/? in the indexing policy?
Currently i have hedged by bets with...
{
"indexingMode": "consistent",
"automatic": true,
"includedPaths": [{
"path": "/properties/partitionKey/*"
},{
"path": "/partitionKey/?"
},{
"path": "/label/?"
}
],
"excludedPaths": [{
"path": "/*"
},{
"path": "/\"_etag\"/?"
}
]
}
TIA! ๐
This should be stored as "/partitionKey" in your index policy, not "/properties/partitionKey"
btw, another thing to point out here is it's generally better to only exclude paths you will never query on rather than have to include those you will. This way, if you add properties to your graph you won't have to rebuild the index to query on the new properties.
Using a TablesDB table in Cosmos DB I'm trying to index only the PartitionKey and RowKey.
My CosmosDB index below compiles correctly but when I run a query on the PartitionKey/RowKey I get the error "An invalid query has been specified with filters against path(s) excluded from indexing. Consider adding allow scan header in the request."
Does anyone know how to use CosmosDB TablesDB that indexes only the PartitionKey and RowKey and nothing else?
{
"indexingMode": "consistent",
"automatic": true,
"includedPaths": [
{
"path": "/PartitionKey/?",
"indexes": [
{
"kind": "Hash",
"dataType": "String",
"precision": -1
},
{
"kind": "Range",
"dataType": "Number",
"precision": -1
}
]
},
{
"path": "/RowKey/?",
"indexes": [
{
"kind": "Range",
"dataType": "String",
"precision": -1
},
{
"kind": "Range",
"dataType": "Number",
"precision": -1
}
]
}
],
"excludedPaths": [
{
"path": "/*"
}
]
}
We use the almost same index configuration, only changing the '?' character by '*'. Our configuration looks like this:
{
"indexingMode": "lazy",
"automatic": true,
"includedPaths": [
{
"path": "/PartitionKey/*",
"indexes": [
{
"kind": "Range",
"dataType": "String",
"precision": -1
},
{
"kind": "Range",
"dataType": "Number",
"precision": -1
}
]
},
{
"path": "/RowKey/*",
"indexes": [
{
"kind": "Range",
"dataType": "String",
"precision": -1
},
{
"kind": "Range",
"dataType": "Number",
"precision": -1
}
]
}
],
"excludedPaths": [
{
"path": "/*"
}
]
}
What is the correct JSON to exclude certain keys from an input json to be not indexed by Azure CosmosDB. We are using the CosmosDB in mongodb mode. Was planning to change the index configuration on the Azure Portal after creating the collection.
Sample Input Json being
{
"name": "test",
"age": 1,
"location": "l1",
"height":5.7
}
If I were to include name and age in the index and remove location and height from the index, what does the includedPaths and excludedPaths look like.
Finally got it to work with the below spec:-
{
"indexingMode": "consistent",
"automatic": true,
"includedPaths": [{
"path": "/*",
"indexes": [{
"kind": "Range",
"dataType": "Number",
"precision": -1
},
{
"kind": "Hash",
"dataType": "String",
"precision": 3
}
]
}],
"excludedPaths": [{
"path": "/\"location\"/?"
},
{
"path": "/\"height\"/?"
}
]
}
It looks like underlying implementation has changed and at the time of writing the documentation does not cover changing indexPolicy in MongoDB flavoured CosmosBD. Because the documents are really stored in a wired way where all the keys start from root $v and all scalar fields are stored as documents, containing value and type information. So your document will be stored something like:
{
'_etag': '"2f00T0da-0000-0d00-0000-4cd987940000"',
'id': 'SDSDFASDFASFAFASDFASDF',
'_self': 'dbs/sMsxAA==/colls/dVsxAI8MBXI=/docs/sMsxAI8MBXIKAAAAAAAAAA==/',
'_rid': 'sMsxAI8MBXIKAAAAAAAAAA==',
'$t': 3,
'_attachments': 'attachments/',
'$v': {
'_id': {
'$t': 7,
'$v': "\\ร\x87\x14\x01\x15['\x18m\x01รบ"
},
'name': {
'$t': 2,
'$v': 'test'
},
'age': {
'$t': 1,
'$v': 1
},
...
},
'_ts': 1557759892
}
Therefore the indexingPolicy paths need to include the root $v and use /* (objects) instead of /? (scalars).
{
"indexingMode": "consistent",
"automatic": true,
"includedPaths": [
{
"path": "/*",
"indexes": [
{
"kind": "Range",
"dataType": "Number"
},
{
"kind": "Hash",
"dataType": "String"
}
]
}
],
"excludedPaths": [
{"path": "/\"$v\"/\"location\"/*"},
{"path": "/\"$v\"/\"height\"/*"}
]
}
PS:
Also mongo api can be used to drop all the default indexes and create specific indexes as required
https://learn.microsoft.com/en-us/azure/cosmos-db/mongodb-indexing