Using a TablesDB table in Cosmos DB I'm trying to index only the PartitionKey and RowKey.
My CosmosDB index below compiles correctly but when I run a query on the PartitionKey/RowKey I get the error "An invalid query has been specified with filters against path(s) excluded from indexing. Consider adding allow scan header in the request."
Does anyone know how to use CosmosDB TablesDB that indexes only the PartitionKey and RowKey and nothing else?
{
"indexingMode": "consistent",
"automatic": true,
"includedPaths": [
{
"path": "/PartitionKey/?",
"indexes": [
{
"kind": "Hash",
"dataType": "String",
"precision": -1
},
{
"kind": "Range",
"dataType": "Number",
"precision": -1
}
]
},
{
"path": "/RowKey/?",
"indexes": [
{
"kind": "Range",
"dataType": "String",
"precision": -1
},
{
"kind": "Range",
"dataType": "Number",
"precision": -1
}
]
}
],
"excludedPaths": [
{
"path": "/*"
}
]
}
We use the almost same index configuration, only changing the '?' character by '*'. Our configuration looks like this:
{
"indexingMode": "lazy",
"automatic": true,
"includedPaths": [
{
"path": "/PartitionKey/*",
"indexes": [
{
"kind": "Range",
"dataType": "String",
"precision": -1
},
{
"kind": "Range",
"dataType": "Number",
"precision": -1
}
]
},
{
"path": "/RowKey/*",
"indexes": [
{
"kind": "Range",
"dataType": "String",
"precision": -1
},
{
"kind": "Range",
"dataType": "Number",
"precision": -1
}
]
}
],
"excludedPaths": [
{
"path": "/*"
}
]
}
Related
There are multiple ways how to create composite index. But I see no ARM templates there and latest scheme for containers also has no it. When we can use ARM templates to define composite indexes?
Can you try something like the following:
{
"name": "string",
"type": "Microsoft.DocumentDB/databaseAccounts/sqlDatabases/containers",
"apiVersion": "2020-03-01",
"location": "string",
"tags": {},
"properties": {
"resource": {
"id": "string",
"indexingPolicy": {
"automatic": "boolean",
"indexingMode": "string",
"includedPaths": [
{
"path": "string",
"indexes": [
{
"dataType": "string",
"precision": "integer",
"kind": "string"
}
]
}
],
"excludedPaths": [
{
"path": "string"
}
],
"spatialIndexes": [
{
"path": "string",
"types": [
"string"
]
}
],
"compositeIndexes":[
[
{
"path":"/name",
"order":"ascending"
},
{
"path":"/age",
"order":"descending"
}
]
]
},
"partitionKey": {
"paths": [
"string"
],
"kind": "string",
"version": "integer"
},
"defaultTtl": "integer",
"uniqueKeyPolicy": {
"uniqueKeys": [
{
"paths": [
"string"
]
}
]
},
"conflictResolutionPolicy": {
"mode": "string",
"conflictResolutionPath": "string",
"conflictResolutionProcedure": "string"
}
},
"options": {
"additionalProperties": {},
"throughput": "string"
}
},
"resources": []
}
I simply took the template definition from here and added definition for composite index from here under indexing policy section.
I am trying to use ARM templates to update the indexing policy for cosmos container. I tried 2 methods, one to simply declare the indexing policy in while declaring the container in ARM.
{
"apiVersion": "[variables('cosmosDbApiVersion')]",
"type": "Microsoft.DocumentDB/databaseAccounts/apis/databases/containers",
"dependsOn": [ /* resourceId */ ],
"name": "/* containerName */",
"properties": {
"resource": {
"id": "/* id */",
"partitionKey": {
"paths": [
"/partitionKey"
],
"kind": "Hash"
},
"indexes": [
{
"indexingMode": "consistent",
"automatic": true,
"includedPaths": [
{
"path": "/*",
"indexes": [
{
"kind": "Range",
"dataType": "Number",
"precision": -1
},
{
"kind": "Hash",
"dataType": "String",
"precision": 3
}
]
}
]
}
],
"defaultTtl": "[variables('defaultTtlValueToEnableTtl')]"
}
}
},
The second was to use to use ARM to deploy container setting as such:
{
"apiVersion": "[variables('cosmosDbApiVersion')]",
"type": "Microsoft.DocumentDB/databaseAccounts/apis/databases/containers/settings",
"name": "[/* name */",
"dependsOn": [ " /* container name */" ],
"properties": {
"resource": {
"throughput": "/* some throughput */",
"indexes": [
{
"indexingMode": "consistent",
"automatic": true,
"includedPaths": [
{
"path": "/*",
"indexes": [
{
"kind": "Range",
"dataType": "Number",
"precision": -1
},
{
"kind": "Hash",
"dataType": "String",
"precision": 3
}
]
}
]
}
]
}
}
},
Both techniques do not fail deployment but the indexing policy does not change.
Would appreciate some help.
this is the example from the template reference (looks slightly different to what you are doing):
"resource": {
"id": "string",
"indexingPolicy": {
"automatic": "boolean",
"indexingMode": "string",
"includedPaths": [
{
"path": "string",
"indexes": [
{
"dataType": "string",
"precision": "integer",
"kind": "string"
}
]
}
],
"excludedPaths": [
{
"path": "string"
}
],
"spatialIndexes": [
{
"path": "string",
"types": [
"string"
]
}
]
},
xxx
}
https://learn.microsoft.com/en-us/azure/templates/microsoft.documentdb/2019-08-01/databaseaccounts/sqldatabases/containers
Range and hash index types are ignored by the Cosmos resource provider now for new containers or containers that were created within the past year or so. ARM does not validate the index policy which is why the template will deploy successfully.
Hash index was deprecated for these newer container because the performance of the range index in the new indexer is surpasses what hash index provided so was no longer necessary.
To create/modify index policy refer to this article below. There are multiple examples of index policies that implement everything from very simple to more complex policies that include composite indexes, spatial indexes and unique keys.
https://learn.microsoft.com/en-us/azure/cosmos-db/manage-sql-with-resource-manager#create-resource
I'm trying to filter the child documents returned when querying a parent document using the sql api in cosmos db.
For example given this document:
{
"customerName": "Wallace",
"customerReference": 666777,
"orders": [
{
"date": "20181105T00:00:00",
"amount": 118.84,
"description": "Laptop Battery"
},
{
"date": "20181105T00:00:00",
"amount": 81.27,
"description": "Toner"
},
{
"date": "20181105T00:00:00",
"amount": 55.12,
"description": "Business Cards"
},
{
"date": "20181105T00:00:00",
"amount": 281.00,
"description": "Espresso Machine"
}]
}
I would like to query the customer to retrieve the name, reference and orders over 100.00 to produce a results like this
[{
"customerName": "Wallace",
"customerReference": 666777,
"orders": [
{
"date": "20181105T00:00:00",
"amount": 118.84,
"description": "Laptop Battery"
},
{
"date": "20181105T00:00:00",
"amount": 281.00,
"description": "Espresso Machine"
}]
}]
the query I have so far is as follows
SELECT c.customerName, c.customerReference, c.orders
from c
where c.customerReference = 666777
and c.orders.amount > 100
this returns an empty set
[]
and if you remove "and c.orders.amount > 100" it matches the document and returns all orders.
To reproduce this issue I simply set up a new database, added a new collection and copied the json example in as the only document. The index policy is left as the default which I've copied below.
{
"indexingMode": "consistent",
"automatic": true,
"includedPaths": [
{
"path": "/*",
"indexes": [
{
"kind": "Range",
"dataType": "Number",
"precision": -1
},
{
"kind": "Range",
"dataType": "String",
"precision": -1
},
{
"kind": "Spatial",
"dataType": "Point"
}
]
}
],
"excludedPaths": []
}
Cosmos DB doesn't support the deep filtering in the way I attempted in my original query.
To achieve the results described you need to use a subquery using a combination of ARRAY and VALUE as follows:
SELECT
c.customerName,
c.customerReference,
ARRAY(SELECT Value ord from ord in c.orders WHERE ord.amount > 100) orders
from c
where c.customerReference = 666777
note the use of 'ord' - 'order' is a reserved word.
The query then produces the correct result - eg
[{
"customerName": "Wallace",
"customerReference": 666777,
"orders": [
{
"date": "20181105T00:00:00",
"amount": 118.84,
"description": "Laptop Battery"
},
{
"date": "20181105T00:00:00",
"amount": 281.00,
"description": "Espresso Machine"
}
]
}]
What is the correct JSON to exclude certain keys from an input json to be not indexed by Azure CosmosDB. We are using the CosmosDB in mongodb mode. Was planning to change the index configuration on the Azure Portal after creating the collection.
Sample Input Json being
{
"name": "test",
"age": 1,
"location": "l1",
"height":5.7
}
If I were to include name and age in the index and remove location and height from the index, what does the includedPaths and excludedPaths look like.
Finally got it to work with the below spec:-
{
"indexingMode": "consistent",
"automatic": true,
"includedPaths": [{
"path": "/*",
"indexes": [{
"kind": "Range",
"dataType": "Number",
"precision": -1
},
{
"kind": "Hash",
"dataType": "String",
"precision": 3
}
]
}],
"excludedPaths": [{
"path": "/\"location\"/?"
},
{
"path": "/\"height\"/?"
}
]
}
It looks like underlying implementation has changed and at the time of writing the documentation does not cover changing indexPolicy in MongoDB flavoured CosmosBD. Because the documents are really stored in a wired way where all the keys start from root $v and all scalar fields are stored as documents, containing value and type information. So your document will be stored something like:
{
'_etag': '"2f00T0da-0000-0d00-0000-4cd987940000"',
'id': 'SDSDFASDFASFAFASDFASDF',
'_self': 'dbs/sMsxAA==/colls/dVsxAI8MBXI=/docs/sMsxAI8MBXIKAAAAAAAAAA==/',
'_rid': 'sMsxAI8MBXIKAAAAAAAAAA==',
'$t': 3,
'_attachments': 'attachments/',
'$v': {
'_id': {
'$t': 7,
'$v': "\\Ù\x87\x14\x01\x15['\x18m\x01ú"
},
'name': {
'$t': 2,
'$v': 'test'
},
'age': {
'$t': 1,
'$v': 1
},
...
},
'_ts': 1557759892
}
Therefore the indexingPolicy paths need to include the root $v and use /* (objects) instead of /? (scalars).
{
"indexingMode": "consistent",
"automatic": true,
"includedPaths": [
{
"path": "/*",
"indexes": [
{
"kind": "Range",
"dataType": "Number"
},
{
"kind": "Hash",
"dataType": "String"
}
]
}
],
"excludedPaths": [
{"path": "/\"$v\"/\"location\"/*"},
{"path": "/\"$v\"/\"height\"/*"}
]
}
PS:
Also mongo api can be used to drop all the default indexes and create specific indexes as required
https://learn.microsoft.com/en-us/azure/cosmos-db/mongodb-indexing
I have wriiten the query to get the all the list of Event Data entities. The result is Coming like this from the google Data Store.
[{
"key": {
"id": 5678669024460800,
"kind": "Event",
"path": [
"Event",
5678669024460800
]
},
"data": {
"createdAt": "2017-03-27T06:28:58.000Z",
"users":["test1#xxx.com","test2#xxx.com","test3#xxx.com"]
}
},
{
"key": {
"id": 5678669024460800,
"kind": "Event",
"path": [
"Event",
5678669024460800
]
},
"data": {
"createdAt": "2017-03-27T06:28:58.000Z",
"users":["test1#xxx.com"]
}
},
{
"key": {
"id": 5678669024460800,
"kind": "Event",
"path": [
"Event",
5678669024460800
]
},
"data": {
"createdAt": "2017-03-27T06:28:58.000Z",
"users":["test2#xxx.com","test3#xxx.com"]
}
}]
but i need to Write a Query to filter by Email'id. means i need to fetch the entities which are match with the Email id. For Eg if i pass the emailid as "test1#xxx.com" i should get final Result like this. Can anybody help me on this.
[{
"key": {
"id": 5678669024460800,
"kind": "Event",
"path": [
"Event",
5678669024460800
]
},
"data": {
"createdAt": "2017-03-27T06:28:58.000Z",
"users":["test1#xxx.com","test2#xxx.com","test3#xxx.com"]
}
},
{
"key": {
"id": 5678669024460800,
"kind": "Event",
"path": [
"Event",
5678669024460800
]
},
"data": {
"createdAt": "2017-03-27T06:28:58.000Z",
"users":["test1#xxx.com"]
}
}]
The GQL query would be something like -
SELECT * FROM Event WHERE users='test1#xxx.com'
You need to make sure the users property is indexed in order for the search to work, otherwise you may not get any results back.