Firestore Pagination: how to define **unique** 'startAt'-cursor for REST? - firebase

This is a follow up question to an already solved one.
For this previous question an answer was given, how to define a cursor for query-pagination with 'startAt' for REST, that relates to a range of documents. In the example below, the cursor relates to all documents with an 'instructionNumber.stringValue' equal to "instr. 101". According to my testing, this results in skipping of documents.
New question:
How has the cursor to be defined, to not only relate to the stringValue of a field, that the query is ordered by? But instead to a distinct document (usually defined by its document-id)?
"structuredQuery": {
"from": [{"collectionId": "instructions"}],
"where": {
"fieldFilter": {
"field": {
"fieldPath": "belongsToDepartementID"
},
"op": "EQUAL",
"value": {
"stringValue": "toplevel-document-id"
}
}
},
"orderBy": [
{
"field": {
"fieldPath": "instructionNumber"
},
"direction": "ASCENDING"
}
],
"startAt": {
"values": [{
"stringValue": "instr. 101"
}]
},
"limit": 5
}
}
For better understanding, here is the condensed schema of the documents.
{
"document": {
"name": "projects/PROJECT_NAME/databases/(default)/documents/organizations/testManyInstructions/instructions/i104",
"fields":
"belongsToDepartementID": {
"stringValue": "toplevel-document-id"
},
"instructionNumber": {
"stringValue": "instr. 104"
},
"instructionTitle": {
"stringValue": "dummy Title104"
},
"instructionCurrentRevision": {
"stringValue": "A"
}
},
"createTime": "2022-02-18T13:55:47.300271Z",
"updateTime": "2022-02-18T13:55:47.300271Z"
}
}

For a query with no ordering:
"orderBy": [{
"direction": "ASCENDING",
"field": {"fieldPath": "__name__"}
}],
"startAt": {
"before": false,
"values": [{"referenceValue": "last/doc/ref"}]
}
For a query with ordering:
"orderBy": [
{
"direction": "DESCENDING",
"field": {"fieldPath": "instructionNumber"}
},
{
"direction": "DESCENDING",
"field": {"fieldPath": "__name__"}
}
],
"startAt":
{
"before": false,
"values": [
{"stringValue": "instr. 101"},
{"referenceValue": "last/doc/ref"}
]
}
Be sure to use the same direction for __name__ as the previous "orderBy" or it will need a composite index.

To ensure you have identify unique document for starting at you'll always want to include the document ID in your call to startAt.
I'm not sure of the exact syntax for the REST API, but the Firebase SDKs automatically pass this document ID when you call startAt with a DocumentSnapshot.

Related

BatchWrite REST API JSON Structure for FireStore

I know how to compose JSON when you want to create a single document on FireStore.
# Query String
"updateMask.fieldPaths": [
"`price`",
"`status`"
]
# Body
"fields": {
"price": {
"stringValue": "165.00"
},
"status": {
"stringValue": "active"
}
}
But when it comes to BatchWrite I cannot understand what the doc implies
https://cloud.google.com/firestore/docs/reference/rest/v1/projects.databases.documents/batchWrite
Currently I'm doing this with Python library I deployed on cloud computing resource(pipedream) but I want to do this within Make.com and its "Make API Call" module so I need to know the curl style or REST JSON syntax.
I'm assuming fields go to "writes" but not sure what goes to "labels" object.
Exactly what's the write structure for this call?
{
"writes": [
{
"update": {
"name": "projects/project_id/databases/(default)/documents/service_order/352003090342435.5f357270173e8c70878.-9486573",
"fields": {
"managed_by": {
"booleanValue": "false"
},
"location2": {
"booleanValue": "false"
},
"made_by": {
"stringValue": "352003090342435.5eda4f3c17284c58460.90921842"
},
"phone2": {
"stringValue": "0665439307"
},
"state": {
"stringValue": "confirm"
},
"write_date": {
"stringValue": "1597664391000"
},
"made_by_name": {
"stringValue": "FaridBenabdallah"
},
"description": {
"stringValue": "e1"
},
"accepted_offer": {
"booleanValue": "false"
},
"id": {
"stringValue": "352003090342435.5f357270173e8c70878.-9486573"
},
"location1": {
"stringValue": "352003090342435.5f3572a0173e8c7c336.-8281409"
},
"date_order": {
"booleanValue": "false"
},
"order_type": {
"stringValue": "Minuiserie"
},
"phone1": {
"stringValue": "0657331995"
}
}
},
"updateMask": {
"fieldPaths": [
"managed_by",
"location2",
"made_by",
"phone2",
"state",
"write_date",
"made_by_name",
"description",
"accepted_offer",
"id",
"location1",
"date_order",
"order_type",
"phone1"
]
}
}
]
}
This another stackoverflow article helped me
If there is null you need to change the intended value type to nullValue.

Firestore Pagination: how to define 'startAt'-cursor for REST?

I am trying to use a cursor and 'startAt', to paginate REST requests to Firestore. According to the Paginate-documentation, the cursor should equal to the last document of the previous query. As the REST-documentation is without an example, I tried to run it by inserting an entire document as cursor in the startAt-object; like this:
POST https://firestore.googleapis.com/v1/PROJECT-NAME/databases/(default)/documents/organizations/testManyInstructions:runQuery
{
"structuredQuery": {
"from": [
{
"collectionId": "instructions"
}
],
"where": {
"fieldFilter": {
"field": {
"fieldPath": "belongsToDepartementID"
},
"op": "EQUAL",
"value": {
"stringValue": "toplevel-document-id"
}
}
},
"orderBy": [
{
"field": {
"fieldPath": "instructionNumber"
},
"direction": "ASCENDING"
}
],
"startAt": {
"values": [{
"document": {
"name": "projects/PROJECT-NAME/databases/(default)/documents/organizations/testManyInstructions/instructions/i0",
"fields": {
"checkbox": {
"booleanValue": false
},
"retrainTimespanDays": {
"integerValue": "365000"
},
"approvedByName": {
"stringValue": ""
},
"instructionNumber": {
"stringValue": "instr. 0"
},
"instructionCurrentRevision": {
"stringValue": "A"
},
"instructionCurrentRevisionPublishingDate": {
"timestampValue": "1999-01-01T00:00:00Z"
},
"instructionFileURL": {
"stringValue": ""
},
"instructionTitle": {
"stringValue": "dummy Title0"
},
"instructionFileUploadDate": {
"timestampValue": "1999-01-01T00:00:00Z"
},
"belongsToDepartementID": {
"stringValue": "toplevel-document-id"
},
"approvedByEmailAdress": {
"stringValue": ""
}
},
"createTime": "2022-02-18T13:55:42.807103Z",
"updateTime": "2022-02-18T13:55:42.807103Z"
}
}
]
},
"limit": 5
}
}
without the "startAt"-Object, the following code works fine and returns 5 documents.
with the "startAt"-Object, this error is returned:
{
"error": {
"code": 400,
"message": "Invalid JSON payload received. Unknown name \"document\" at 'structured_query.start_at.values[0]': Cannot find field.",
"status": "INVALID_ARGUMENT",
"details": [
{
"#type": "type.googleapis.com/google.rpc.BadRequest",
"fieldViolations": [
{
"field": "structured_query.start_at.values[0]",
"description": "Invalid JSON payload received. Unknown name \"document\" at 'structured_query.start_at.values[0]': Cannot find field."
}
]
}
]
}
}
]
Please advise, how to set the cursor in the startAt-object correctly.
I've run a similar query using offset instead of startAt, so I tried modifying and got it to work. This is the rest api documentation I used.
startAt requires a Cursor object which is an array of Values.
https://firebase.google.com/docs/firestore/reference/rest/v1/StructuredQuery
https://firebase.google.com/docs/firestore/reference/rest/v1/Cursor
https://firebase.google.com/docs/firestore/reference/rest/Shared.Types/ArrayValue#Value
I would have preferred an example as well!
"startAt": {
"values": [{
"stringValue": "Cr"
}]
},
"orderBy": [{
"field": {
"fieldPath": "Summary"
}
}],
Good luck!

Not able to filter out required property in Azure TSI Gen1 Get Events API response

I am using the below request body to fetch only the required property values.
"searchSpan": {
"from": {
"dateTime": "2021-11-20T00:00:00.000Z"
},
"to": {
"dateTime": "2021-11-20T23:00:00.000Z"
}
},
"predicateString": "[Params.Name] = 'power'",
"take": 100
}
}
The URL is like below:
https://12345678a-bcde-3e91-blah-2292933292aa.env.timeseries.azure.com/events?api-version=2016-12-12
Despite specifying the required property the response returns all properties as if it has not seen the predicate string. What might I be doing wrong?
{
"warnings": [],
"events": [
{
"schema": {
"rid": 0,
"$esn": "my-event-hub",
"properties": [
{
"name": "mytimestamp",
"type": "DateTime"
},
{
"name": "Params.Name",
"type": "String"
},
{
"name": "Params.Value",
"type": "Double"
}
]
},
"$ts": "2021-11-20T10:01:50Z",
"values": [
"2021-11-20T10:01:50Z",
"energy",
60
]
},
{
"schemaRid": 0,
"$ts": "2021-11-20T10:01:50Z",
"values": [
"2021-11-20T10:01:50Z",
"power",
10
]
},
{
"schemaRid": 0,
"$ts": "2021-11-20T10:01:50Z",
"values": [
"2021-11-20T10:01:50Z",
"strength",
200
]
},
]
}
Edit
I'm getting "Properties count error" in the TSI overview page. This might quite be the root cause but I don't know for sure
"For Time Series Insights environment ABC: You have used all 641/600 properties in your environment".

Azure Time Series Insights query: how to denote a variable such as 'cpu.thermal.average' in the query?

My series' got the value: cpu.thermal.average. I see the values in the TSI explorer for the given range of time.
By executing the following request, for "temperatures" I only get "null".
I am not sure about how to write $event.cpu.thermal.average equivalent.
Any idea?
{
"aggregateSeries": {
"timeSeriesId": [
"MySeries",
],
"searchSpan": {
"from": "2021-03-10T07:00:00Z",
"to": "2021-03-10T20:00:50Z"
},
"interval": "PT3600M",
"filter": null,
"inlineVariables": {
"latitudes": {
"kind": "numeric",
"value": {
"tsx": "$event.latitude"
},
"filter": null,
"aggregation": {
"tsx": "avg($value)"
}
},
"temperatures": {
"kind": "numeric",
"value": {
"tsx": "$event['cpu.thermal.average']"
},
"filter": null,
"aggregation": {
"tsx": "avg($value)"
}
}
},
"projectedVariables": [
"latitudes",
"temperatures"
]
}
}
From here: https://learn.microsoft.com/en-us/rest/api/time-series-insights/reference-time-series-expression-syntax#value-expressions
When accessing nested properties, the Type is required.
It should work if you use $event.cpu.thermal.average.Double

The language expression property '0' can't be evaluated, property name must be a string - ARM Template error while adding Key Vault access policy

I've been working on an issue and seem to be stuck, so asking on so in case anyone can help.
To describe the issue, I've got an existing Azure Key Vault setup, and wish to add a number of access policies to this resource group. It needs to be conditional as if the function name is "false" then that function should not be added to key vault access policy.
variable section:
"variables": {
"functionAccess": {
"value": [
{
"name": "[parameters('Function_1')]"
},
{
"name": "[parameters('Function_2')]"
},
{
"name": "[parameters('Function_3')]"
}
]
}
}
My Template :
{
"apiVersion": "2016-10-01",
"condition": "[not(equals(variables('functionAccess')[CopyIndex()].name, 'false'))]",
"copy": {
"batchSize": 1,
"count": "[length(variables('functionAccess'))]",
"mode": "Serial",
"name": "accessPolicies"
},
"name": "[concat(parameters('KeyVault_Name'), '/add')]",
"properties": {
"accessPolicies": [
{
"tenantId": "[subscription().tenantId]",
"objectId": "[if(not(equals(variables('functionAccess')[CopyIndex()].name, 'false')), reference(concat('Microsoft.Web/sites/', variables('functionAccess')[CopyIndex()].name), '2016-08-01', 'Full').identity.principalId, json('null'))]",
"permissions": {
"keys": [
"get",
"list"
],
"secrets": [
"get",
"list"
],
"certificates": [
"get",
"list"
]
}
}
]
},
"type": "Microsoft.KeyVault/vaults/accessPolicies"
}
When I deploy my ARM template for the azure key vault I got this error message:
The language expression property '0' can't be evaluated, property name must be a string.
also tried below, but same error:
{
"apiVersion": "2018-02-14",
"name": "[concat(parameters('KeyVault_Name'), '/add')]",
"properties": {
"copy": [
{
"batchSize": 1,
"count": "[length(variables('functionAccess'))]",
"mode": "serial",
"name": "accessPolicies",
"input": {
"condition": "[not(equals(variables('functionAccess')[copyIndex('accessPolicies')].name, 'false'))]",
"tenantId": "[subscription().tenantId]",
"objectId": "[if(not(equals(variables('functionAccess')[copyIndex('accessPolicies')].name, 'false')), reference(concat('Microsoft.Web/sites/', variables('functionAccess')[copyIndex('accessPolicies')].name), '2016-08-01', 'Full').identity.principalId, json('null'))]",
"permissions": {
"keys": [
"get",
"list"
],
"secrets": [
"get",
"list"
],
"certificates": [
"get",
"list"
]
}
}
}
]
},
"type": "Microsoft.KeyVault/vaults/accessPolicies"
}
There are a few options for dealing with filtering an array for copy operation. I deploy my ARM templates from PowerShell scripts and use PowerShell to setup parameter values. When I need special logic handle different inputs for different environments, I let PowerShell handle it.
If you must handle the filtering in ARM and you have the option to input a CSV list of functions, then perhaps the following will work. You can then use the functionAccessArray to iterate over in the copy operation.
{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
},
"variables": {
"functionAccessCsv": "Function-0,Function-1,false,Function-4,false,Function-6,Function-7",
"functionAccessFiltered": "[replace(replace(variables('functionAccessCsv'), 'false', ''), ',,', ',')]",
"functionAccessArray": "[split(variables('functionAccessFiltered'), ',')]"
},
"resources": [
],
"outputs": {
"functionAccessCsvFiltered": {
"type": "string",
"value": "[variables('functionAccessFiltered')]"
},
"functionAccessArray": {
"type": "array",
"value": "[variables('functionAccessArray')]"
}
}
}
The result:
I just had the same issue. By using an array parameter with a default value instead of a variable, I got it to work.
"parameters": {
"functionAccess": {
"type": "array",
"defaultValue": [
"value1",
"value2",
"value3"
]
}
}

Resources